Damage what the players actually care about - but don't make them sit out
What does a player fear when they scoff in the face of death? Regression. Wasted resources. Loss of power.
Don't kill the characters, no one enjoys sitting out for a session. Take their precious resources. I'm not talking about gold, I'm talking about taking away their power. You can easily get more gold, but you can't pick up feats, powers, stats, skills, or hit points at the village temple.
If a player puts their character in a scenario that demands a punishment, rip away something that the player really cares about, but don't keep them from playing the game!
Maybe Bob's character is pushed into a spike pit with one hp because he lost a dexterity check. Instead of the character dying, the party could retrieve his lifeless body, only to find he is miraculously still breathing. As a ramification of being pushed into the spike pit, his dexterity goes down by one, or he loses 10 ranks of balance, or he loses 5 hp, permanently.
In my experience, the key to introducing this sort of permanent penalty mechanic is to first introduce permanent bonus mechanics. Fountains of +constitution, tomes of ancient knowledge that provide feats, or blessings of hit points from deities are all great ways to make the players feel like they achieved something really special.
I theorize that damaging the player's progression in a game is far more interesting than making the player sit out or play a throwaway secondary character.
How do you handle a player that decides to make a new character instead of accepting the permanent penalties?
I think that you can avoid this scenario in a couple ways:
- Play the sort of game where the players are attached to/invested in their characters (like in roleplay-intensive story games)
- Demonstrate the possibility of future growth through the use of permanent bonus mechanics as listed above
If your have implemented the above techniques and a player is still interested in creating a new character, I would work with the player to create an exciting and rewarding death for the old character. If the player is willing to sacrifice their character, perhaps they are more invested in their next cool character idea, which is a win for the player because the player gets to enjoy creating a new character, and a win for the DM because the player will be more responsive to permanent penalties and bonuses.