As I have played D&D we have come to many points on the road where things don't seem quite fair and as the DM I have to be the arbiter of sticky wickets. For example: If you have a situation where the player has the blade of his knife to the throat of the evil baddy and there is no realistic reason the evil baddy wouldn't die in the next minute, but the rules tell you that a dagger in D&D 3.5 at most (meaning with a crit) can deal 8 + STR (mod) x2 damage, and the baddy is a level 3 fighter and thus beyond that spectrum how do you handle it? And moreover how do you handle it fairly? Sure for story's sake you allow the player to kill the evil baddy, but what happens in the next encounter when the roles are reversed and there is no reason the enemy shouldn't be able to end the life of the player? How do you make these decisions and make them feel real and somewhere in the realm of fairness, taking a step away from dice rolls and getting into the moment? The goal here is to find a way to make high level players afraid of things that are still dangerous. Even a massively high level fighter as a CHARACTER should consider the danger of a fight that the game would say "isn't dangerous"
Dungeons & Dragons is not a realistic game
It’s not intended to be. It is designed with a large number of abstractions and it is designed for larger-than-life, superhuman heroes. A first level human barbarian can break Usain Bolt’s world-record sprint time – and then keep up the pace for several minutes, and do it while wielding a battleaxe, wearing a chain shirt, and carrying fifty or more pounds of assorted other gear.
It only gets worse from there. A character limited by the realistic limits of the human body cannot reasonably challenge much of anything above first level, and even that is pushing it. Someone limited by realistic laws of physics caps out around being a challenge for roughly eighth-level characters; after that, things like flight or invisibility or whatever become so common that creatures with no answer to them are non-threats, regardless of how strong they are (see the Tarrasque, which is actually a fairly trivial fight for an intelligent eighth-level party, and can be taken down as early as third level if you’re clever and can justify some enormous knowledge checks).
HP is the biggest abstraction in the system
HP is not really (necessarily) “health,” and loss of HP is definitely not (necessarily) “wounds.” HP is more like “plot armor,” and can represent morale, determination, luck, blessings of the gods, magic mojo, and any number of other things, or more likely multiple things in combination. Just looking at some of the effects that can restore HP makes this rapidly clear; there even exist Extraordinary (read: non-magical) forms of HP healing based on being inspiring, rallying your allies to push on and fight through the pain. That couldn’t be possible if HP were strictly wounds.
So ultimately, the way D&D works, your players shouldn’t have been able to get a knife through an enemy’s throat unless the die rolls had already shown it was a killing blow. That’s how the game works: you roll the dice to see how well you do, then describe what you did to achieve that. You don’t get to describe a killing blow unless you actually score one. If they came up short, that should have been described accordingly; the enemy writhed out of the way or they messed up and only nicked him or whatever. And if that seems implausible given the circumstances, then they probably should have been using a Coup de Grace (and surviving a Coup de Grace is fairly implausible; that is a very difficult Fortitude save).
You might consider another system
But I honestly think this probably is not a very satisfying answer for you. Unfortunately, it’s the answer that D&D has, and trying to shoehorn realism into a system that is so fundamentally designed from the ground up with huge abstractions and intended for larger-than-life fantasy heroes, is just trying to force a round peg through a square hole. I’ve seen it tried, and it just does not work well at all: it’s a lot of work, lots of rules end up being really wonky, and I generally haven’t seen people with your interests actually be satisfied by it. If this is what you and your group is interested in, there are games that are more lethal and aim at greater levels of realism that may suit you better.
For the knife thing, there's always Coup de Grace
A Coup de Grace can happen in a situation where one character has another - PC or NPC - helpless. They normally take a full-round action to happen, and automatically deal damage as a critical hit, followed by forcing a fortitude save vs. death. While this isn't a certified kill, I should note for you that people in real life have survived having their throats cut - and none of these people could wrestle bears with their bare hands.
3.5 is just straight bad at realism. 10th level characters can swim, unaided, through lava and come out the other side alive. An untrained shopkeeper can run for miles in full plate through the high heat of summer and the worst thing that happens is he gets tired. These catches, these areas that utterly lack resemblance to real life, are built into every facet of the system and it's too integrated to really remove. You can house rule it every time, of course, but you may be better served finding another system.
Fight Dirty D&D vs. High Adventure D&D
The real question is what kind of game do you (the group) want to play?
There is a style of D&D where the knife to the throat is just ruled a kill and done. And yes, the bad guys can do it too. So a lot of play becomes finding ways to fight dirty and eke out victory. This is where "throw lantern oil on the bad guys and throw a torch" came from.
There's also D&D that's aimed at being more like a lot of fantasy fiction - where the characters are heroic and not ruthless murderers - they may kill someone in the heat of battle, but not someone helpless. If you want to do this kind of play, then you have to agree as a group that this is what works for you.
If you want something that supports more realistic play on it's own, a different system is the best answer.
Keeping the tension
The second question - how do you make things scary for high level players? It's hard, because D&D magic effectively removes a lot of the fear for PCs quickly - heroes heal in 1-2 days maximum, raise dead eliminates fear of death, etc. Other games like Burning Wheel can leave a character laid up for weeks or months with injuries - you just don't want to get hurt at all in those games, so every fight has an element of danger.
With D&D, it's really hard. I think WOTC did a study of play and saw when most parties died, the players never realized they were losing until it was too late. So, the thing is that what is actually combat effective, may not put fear in the players' hearts. In one game, I gave a character a "Midas Touch" attack - on a critical, the target would turn to gold. Someone had a hawk animal companion they sent to attack the guy first, and he happened to roll a critical. Everyone freaked out and ran after that. Even though the odds were low, they didn't know that and the potential consequences were high.
You can try to do cheap tricks like that, but players will get tired of them if overused. Otherwise, it's a lot of stacking together several monster powers/spells in combos to hit hard and fast. Which is a lot of work, because the players have had many, many sessions to master their combinations of powers, while you will be putting together new monster sets regularly and having to figure out how these new sets work together. It makes high level play very swingy - sometimes you miss something crucial or the players play a little different and they steamroll your encounters, sometimes they miss something crucial or you actually put together something terribly mean and they don't run, and you steamroll them.
The only sorta-useful trick I have in that case is to try to make a combo that takes out one PC really hard. "Wait, the Barbarian is down to 3 hitpoints, already?!? Oh crap." It keeps the party up to win the fight, but that one PC who gets hit freaks out, and if everyone else sees someone go down quick, they realize they might be next.