Called shots are a great story device that can make combat seem terrifying and more dangerous, but should never be incorporated mechanically. It creates a situation that slows down combat, which slows down the game and makes it harder for the DM to move the game experience along. It also gives insane and unnecessary power to the players. This isn't to say you can't make the illusion of called shots. I personally like to use damage dealt as my gauge on how I want to describe a called shot.
You are using a long sword and fighting a rogue with 8 hit points. You deal max damage of 8 on the die. This kills the rogue. You were aiming for his eye?
"You drive the point of your blade through his eye socket and out the other end. A small streak of blood races down to the point of your trusted weapon. As you slide your weapon from the rogue's skull, his body goes limp and falls to the cobblestone street." -DM
Same scenario vs the rogue but you deal minimum damage.
"You drive the tip of your weapon at his eye, thrusting with precision..." (Because you beat AC, you DID hit.) "... but the rogue sees your attack coming at the last moment and narrowly escapes your fatal strike, leaving but a gash on his brow." -DM
To me, Hit points are a representation of how tough a character is to kill, rather than just how tough they are. The latter of the two is the idea that X weapons sticking out of target = dead target. I call that the pin cushion concept. If you want more realistic combat, you should stay away from this. The first concept is the idea that health points are also a representation of how well a target can defend themselves from blows even after Armor Class has been bypassed. When I do combat with my players, I try to remember that a slashed throat is a slashed throat. There isn't much in the way of coming back from that, so I do my best to rationalize when it is acceptable for a player to be able to cut the throat of a target:
Let's say you are trying to lop off some brigands head. You are going to deal 10 damage to the target with 10 HP. If he is effectively dead from a single strike, there is little to no reason not to push the idea that your called shot did indeed lop off the head of the brigand especially when, as a player, you don't know how much HP the Brigand had in the first place. For all you know, he had 30 hitpoints and your called shot ended him.
Let's then say the next brigand has 20 hp because he's a higher level than the last. You make the EXACT same strike and deal the same damage. Instead of lopping off his head, he puts up his dagger and it clashes with your blade then he grinds it away from his neck. Unfortunately for him, it still slides into his side and chops through his mid section.
That is the illusion you can create for called shots. No new mechanics. Just good story telling.
As a final note, here are some alternative house rules I've used for called shots and stuff:
- Using inspiration to target a specific section of body. "Golum wants the ring of power that is on Frodo's Finger. He attacks and deals damage. He uses his inspiration to bite Frodo's finger off and the ring with it." This is a great way to balance called shots as the DM is the one balancing called shots out via usage and players will think twice before they waste a precious inspiration point.
- Think of a cool story effect when a player deals max damage.
- Have specific ideas already in place for characters and enemies to live up to when making called shots. "Darth Vader cuts off Luke's hand and vice versa." This is a great way to add story elements to your game using called shots and leaves a lot of the power to the DM, but lets the player decide if they want to go that route. It could be as simple as a side note on an enemy. "Luke's hand was removed in a lightsaber duel with Darth Vader and is now replaced with a mechanical one. If he chooses to do a called shot on Darth Vader's hand, he can remove it, but will have a moment of shock when he realizes that Darth Vader too has a mechanical hand and he is becoming more like him."