There are several other answers here, but, IMO, none of them actually answer the real underlying question:
How can I easily determine when it is mathematically correct for my character to use -5 to hit for +10 damage?
$$Maximum\,AC=\left\lfloor\frac{\left(2\,\times\,Attack\,Bonus\right)\,-Average\,Damage\,+\,32}{2}\right\rfloor$$
(The \$\lfloor\,\,\rfloor\$ indicate the mathematical floor operation or, here, rounding down.)
So, if you have a +8 attack bonus and deal 1d12 (i.e., 6.5) damage, your result is:
$$\left\lfloor\frac{\left(2\times8\right)-6.5+32}{2}\right\rfloor=20$$
So, with a +8 to attack, deal 1d12 damage on a hit, and the target's AC is 20 or lower, then it is mathematically correct to use -5/+10.
Let's say you have a level 9 Fighter with 20 Str, Great Weapon Fighting, Great Weapon Master, and a +1 greatsword. Your attack bonus is \$4+5+1=10\$. Your damage on a hit is \$2d6\,(8.33)+6=14.33\$. (The average of 2d6 here is 8.33 and not 7 due to Great Weapon Fighting.)
$$\left\lfloor\frac{\left(2\times10\right)-14.33+32}{2}\right\rfloor=18$$
So, mathematically, you should use -5/+10 on any target with an AC of 18 or less.
The above is assuming that you do not have advantage or disadvantage. I will cover that below, but the reality is that using -5/+10 is essentially always correct when you have advantage, and conversely it is essentially always incorrect when you have disadvantage.
Yes, you do have an outliers such as when you require a natural 20 to hit the target, or when you've got Dex 10 and a Blowgun with the Sharpshooter feat, or sometimes when you have a very large number of sneak attack dice, but none of those cases very common.
Further, critical hits also have no bearing on the calculation at all because critical hits add the same amount of damage to both types of attacks and occur at the same rate on both attacks. A natural 20 always hits, and essentially every high level Champion is already always going to hit any target's AC on a natural 18 or better even with the -5 no matter how your DM interprets a critical hit.
Let's see how we arrive at that formula.
I think it's easy to see that answer will depend on three factors:
- Your attack bonus
- Your average or expected damage on a hit
- The target's AC
The first two you can know pretty easily before the game even begins. The target's AC, however, is a value that varies for every combatant. Therefore, it will be most useful to determine what AC is the most effective.
So, what we want to know is:
$$Expected\,damage\,from\,normal\,attack<Expected\,damage\,from\,-5/+10\,attack$$
Expected damage from a normal attack is, in most cases, best understood as the mean average damage on a hit multiplied by the chance to hit.
$$Expected\,damage\,from\,normal\,attack=Average\,damage\times\frac{21+Attack\,Bonus-Target\,AC}{20}$$
Expected damage from a -5/+10 attack is the same, but we need to write it using the same terms as above. So, we get:
$$Expected\,damage\,from\,-5/+10\,attack$$
$$=(Average\,damage + 10)\times\frac{21+Attack\,Bonus-5-Target\,AC}{20}$$
$$=(Average\,damage + 10)\times\frac{16+Attack\,Bonus-Target\,AC}{20}$$
So, that gives us this inequality:
$$Average\,damage\times\frac{21+Attack\,Bonus-Target\,AC}{20}$$
$$<$$
$$(Average\,damage + 10)\times\frac{16+Attack\,Bonus-Target\,AC}{20}$$
Now we just need to solve for Target AC. However, I'm lazy, so I made Wolfram Alpha do it. I used \$a\$ for the average damage, \$b\$ for the attack bonus, and \$x\$ for the target AC. I get the solution:
$$x<\frac{1}{2} (-a + 2 b + 32)$$
Which is the same as:
$$Target\,AC<\frac{\left(2\,\times\,Attack\,Bonus\right)\,-Average\,Damage\,+\,32}{2}$$
When the above inequality is true, it is mathematically correct to use -5/+10 on your attack.
You can repeat the above method to determine functions for advantage and disadvantage by substituting the different equations for calculating to-hit into the above inequality. However, you quickly run into pretty monster equation solutions for both advantage and disadvantage due to square roots.
However, here's the solution for advantage:
$$\frac{-\sqrt{a^2+10a+1600}-a+2b-8}{2}<Target AC<\frac{\sqrt{a^2+10a+1600}-a+2b-8}{2}$$
Here's the solution for disadvantage:
$$Target\,AC<\frac{-a-\sqrt{a^2+10a}+2b+32}{2}$$
$$Target\,AC>\frac{-a+\sqrt{a^2+10a}+2b+32}{2}$$
Again, where \$a\$ is the average damage and \$b\$ is the attack bonus.
Let's take the same level 9 Fighter as above with +10 attack bonus and 14.33 average damage on a hit.
Advantage:
$$-23.24 < Target\,AC < 20.91$$
Disadvantage:
$$Target\,AC < 9.50$$
$$Target\,AC > 28.17$$
And you can always test a given AC by calculating the damage per attack for that specific AC.
Note that with disadvantage you're getting results that are basically off the die, since you can't roll a 9 on d20+10.
Let's take an extreme example to test the rule of thumb. Let's take a level 19 Rogue/Level 1 Fighter with Sharpshooter, Crossbow Expert, Dex 20, a +3 Hand Crossbow, Bracers of Archery (+2 damage), and Archery Weapon Style (+2 to hit). We've got \$6 + 5 + 3 + 2 = 16\$ to hit. We do \$1d6 (3.5) + 5 + 3 + 2 + 10d6 (35) = 48.5\$ average damage.
Normal:
$$Target\,AC < 7.75$$
Unsurprisingly, the -5 from Sharpshooter it too much here. The +10 damage doesn't pay off when we're dealing with nearly 50 damage on average.
Advantage:
$$-45.56 < Target\,AC < 21.06$$
It's still preferred for essentially all targets here (the number of enemies with AC greater than 21 can be counted on one hand, I believe).
Disadvantage:
$$Target\,AC < -18.89$$
$$Target\,AC > 34$$
And, unsurprisingly, we're off the die here.
One final note, but we're pretty explicitly ignoring the idea of overkill here. If you're fighting particularly weak enemies, then +10 damage might be entirely overkill damage. That is, you'd have killed the target without the +10 damage anyways. That's technically within the scope of the discussion above, but I've only answered when it's correct to use the ability based on maximizing the damage value. IMO, overkill is just something that isn't much of a concern.
It's pretty trivial to know the AC of a target but fairly difficult to know it's HP. Further, many situations where you might overkill are obvious (e.g., kobold scouts).
However, if you want to consider overkill, then what you need to do is change how you calculate average damage. However, this really sucks because your average damage is different between both sides of the inequality.
Say we're looking at a fighter with duelist style dealing \$1d12 + 7\$ damage and attacking a creature with 13 hit points. Average damage caps at \$13\$ then. You can't ever deal more than 13 damage. That means instead of \$6.5 + 7\$, we calculate the average of \$8, 9, 10, 11, 12, 13, 13, 13, 13, 13, 13, 13\$ which is \$11.75\$. Then we calculate average damage with the +10, which is \$13\$ since rolling a 1 still deals more damage than the creature has. Doing this calculation with weapons dealing \$2d6\$ damage is much more annoying because there are \$36\$ outcomes. If you're looking at calculating with Sharpshooter and \$5d6\$ Sneak Attack damage with a \$1d6\$ hand crossbow, good luck because that's \$6^{6}\,=\,46,656\$ outcomes to average. Not difficult, but not trivial.
Now we use our inequality method, but we can't write it in the same terms anymore so it's a much bigger pain.
Again I solved for \$x\$ with Wolfram Alpha. I used \$a\$ for the average non-GWM damage, \$b\$ for the attack bonus, \$d\$ for the average GWM damage, and \$x\$ for the target AC. You want to use the solution where \$a\,\ge\,0\$ and \$a\,\gt\,d\$ since both of those should always be true. When I have time I'll come back and translate it and update the question again.