That's usually more balanced, except with high AC allies
The real question is: how much less damage does the party as a whole take when one member has a certain fighting style? The only defense-oriented fighting styles useful to a shield user are Defense, Protection, and Interception (TCE), so I'll also be looking at those, plus your version of Protection and an alternative.
A potentially more balanced homebrew Protection works as follows:
- If the attack has neither advantage nor disadvantage, one d20 is rolled. After seeing the roll, Protection may now be used. If it is, an additional d20 is then rolled to represent disadvantage.
- If the attack has advantage, one d20 is designated the "primary" die. The primary d20 and another d20 are rolled. After seeing the roll, Protection may now be used. If it is, only the primary die is kept.
Without this modification, your version of Protection can turn a normal hit into a critical hit. It also may be too powerful when protecting high AC allies, as I'll discuss later.
Meeting the party
For the sake of comparison, all five party members are level 5. The main stat we care about is AC, so without further ado:
- AC 19. The character with the Fighting Style (designated "Main"). Maybe a fighter or paladin.
- AC 14. A backliner who doesn't want to get hit. Most warlocks and bards.
- AC 17. Monks, two-handed weapon users.
- AC 19 (again). Another mid/frontliner. Fighters, paladins, clerics, etc.
- AC 24. Slayer of bounded accuracy. Bladesingers, Eldritch Knights, dips into classes with shield.
Setting the stage
The party will face off against a (nearly) Deadly encounter - two CR 5 creatures. I'll use an air elemental's Multiattack to represent a typical CR 5 creature.
For each comparison, only Main and one other party member (designated "Ally") are targeted by attacks. What about the rest of the party, though? If you're in an encounter with two typical enemies who spread their attacks across more than two targets, you've probably already won. Focus fire is king; even dumb enemies (who don't all gang up on one target) tend to each pick a target and stick to it.
Since this is a "party with an interest in optimisation", I'll assume that they have good tactical positioning. The AC 14 warlock will (try to) stand in the back and won't be targeted by as many attacks as the AC 19 fighter. They won't always succeed, though, so here's how often each party member is in a position to get hit:
- AC 14: 25%. The other 75% of the time, they are 60 feet away from the enemy, invisible, etc. This also accounts for times when the enemy chooses not to provoke opportunity attacks to reach them, can't get through a chokepoint, is stopped by Sentinel, etc.
- AC 17: 40%. They are often hitting enemies in melee, but may be able to hit-and-run (monk's Step of the Wind, for example).
- AC 19: 50%. They are standing next to Main trading blows with enemies on a regular basis. They're just as likely to be targeted as Main.
- AC 24: 50%. They are also up front next to Main. At level 5, most characters can only reach AC 24 using shield, meaning their normal AC is 19. Some DMs will never target them ("smart enemies should switch targets"), some will always target them ("they invested in AC, they should get to use it"), while others are somewhere in between. I'll use 50% as a middle-ground.
Main will pick up the remaining slack, as they try to protect their allies. When we focus on their ability to protect the AC 14 squishy, Main will be targeted the other 75% of the time due to the squishy's good tactical positioning.
Kicking in the door
Let's simulate a million rounds of combat (each with four attacks) to see how much less damage Main and Ally take (combined), based on Ally's AC:
Your version of Protection is strictly better than RAW in all cases. That said, RAW Protection is also worse than Interception in all cases. Interception falls off when defending an AC 24 Ally, since it's quite likely they won't be hit at all.
Defense trails behind the rest, as Main's AC 19 isn't high enough to break bounded accuracy. However, Defense leaves your reaction free for opportunity attacks, shield, Sentinel, etc.
What if the enemies had advantage?
Interception and your version of Protection are barely affected by advantage for an AC 24 Ally, while everything else weakens noticeably. My proposed alternative drops off considerably at AC 24, bringing it more in line with Interception.
Without advantage, your version of Protection is strong, but probably not overpowered in general. Interception is much better at protecting low AC, while your Protection is much better at protecting high AC. If attacks at your table are only directed at AC 20+ targets, though, your version of Protection may end up overpowered.
With advantage, your version of Protection definitely protects high AC allies too well. Given how powerful super high AC already is, it becomes a must-pick for higher AC parties. If you still prefer Protection to Interception, I'd recommend adopting my proposed changes to both tone down high AC defense and to avoid "helpfully" turning a hit into a crit, while still improving on the RAW version.