I've been playing 4e for a few months now and I'm really interested in creating a dungeon crawl in which the PCs can choose to stealthily kill off certain monsters before they are able to alert other enemies or bosses. I've been throwing some ideas around, but haven't really had any luck in trying to think of a reasonable instant-kill system. Although I want to reward my players for good rolls and inventive thinking on how to stealthily bypass the guards, I don't want my players to think that instant kills are a norm for every monster in every dungeon. I'd prefer not to make it a skill challenge. How can I implement a stealth-based system for allowing instant kills on monsters by PCs?
Take a look at Break & Enter by Emerald Press. I have a copy and it provides an excellent framework for stealth encounters complete with stealth kills and ways the entire party can participate. (Link to driveThruPress)
It offers a way of calculating "unaware HP" that only enemies not aware of the players' presences have, and adjudicating attacks against them. It's certainly worth trying if you find your players wanting to do quite a lot of sneaking, as the default rules do not support that kind of play.
Without going into detail about the excellent stealth mechanics, guards going on their rounds and so on, we have selected excerpts:
Unaware HP is a mob's Constitution modifier (minimum 1) with more if they're large, and wisdom modifier if they're elite or solo. Standard creating a mob calculations for everything else. (page 24-25)
Looking at how this plays out, the emphasis is on quiet, silent, attacks... rather than big flashy attacks that make noise, have arrows passing through visual range, or have showy magic. Because an elite or solo can be dropped by any character in a round, a scene's tension is not in 5 rounds of combat, but a slow build over many rounds of a successful sneak to get to that point. When considered over average numbers of rolls, the amount of time a character spends getting in position, cleaning up after the body, and the number of stealth checks necessary more than makes up for the quite neat consequence of dropping the solo in one hit if everything goes according to plan.
You could house rule the scenario by saying that making an attack against an opponent who is totally unaware that there is danger in the offing scores a coup-de-grasDDI, and so essentially you let them auto-crit, and can kill outright with a bigger-than-bloody-them hit.
Obviously against minion guards this doesn't really change much, but against tougher opponents it means that they may kill in one shot, and should have a half-reasonable chance of doing so... but may not. Which gives you a chance for tense die rolling.
Have them use skills to get into position.
If there are multiple targets, then let the players pick one character to attack each target, and let them get the coup de gras.
If anyone fails to kill, have the surviving monsters and players who've not acted roll initiative... players that win, can try to finish off the opposition before they can react.
How about adding a stealth check as bonus damage? At low levels that could be an instant kill, but isn't guaranteed. If you have the whole party jumping a single guard, it's probably a one turn kill. I'd also consider letting players inflict status effects depending on the result of the roll. Like, on a stealth 30, you can line up your shot just right and daze the enemy. On a 50 you stun them. That lets really stealthy characters set up enemies for the rest of the party.
I know you said no skill challenges, but you should make sure the party can use their other skills. You don't want stealth to be the win button that all players end up taking. Make them use their athletics to get into the hard to reach location that's really well hidden. Or dungeoneering to know if the troglodyte guards use sight or smell. Then once the players have done that, they can use stealth for a bonus on their ambush.
(Incidentally, MERP had a separate skill for ambush. MERP also involved rolling critical hits against a table. Some table results were instant kills. IIRC if you succeeded at ambushing someone you could adjust your critical result by your ambush skill. I think I was aiming for something along those lines, but without a critical hit table, it doesn't translate very well.)