How can I calculate the optimal amount of attack bonus to give up when using Power Attack?

Question

The Power Attack feat lets you sacrifice attack bonus (AB) to increase your damage.

It's easy to see that you don't always want to give up the same amount of AB, and that the optimal amount (assuming you're trying to maximize your expected damage output) depends on multiple factors. Consider the following two degenerate cases where it's easy to see that the optimal choice differs:

1. If your opponent's AC is so much higher than your AB that you will only ever hit them by rolling a natural 20, then you clearly want to sacrifice the maximum possible amount of AB, since it won't affect your chance to hit at all (5% in any case), and maximizing your Power Attack damage will result in more damage if you do roll the natural 20.
2. If your foolish DM has granted you the legendary Munchkin's Sword of +Graham's Number Damage at level 1, then you clearly don't want to use Power Attack at all (unless you'll only miss on a natural 1), because your base damage is so high that even a 5% decrease in your chance to hit will utterly dwarf any piddly damage you get from sacrificing your 1 BAB.

In between these silly cases, though, I'm not sure how to determine the best amount of AB to sacrifice when using Power Attack. It's not even clear to me what information I need to do so, though I think it includes some or all of the following:

• The attacker's AB
• The defender's AC
• The attacker's BAB (because it's the maximum amount of AB they can subtract)
• The number of attacks the attacker is making (if they're making a full attack)
• The amount of damage the attack(s) will do on hit
• How much damage the attacker gains per point of sacrificed AB (e.g., 2 points when using a 2-handed weapon instead of 1 point for a 1-handed weapon)
• Whether the attack could crit, and its crit stats (range/multiplier) if so

Given this sort of info, how can I calculate the amount of AB to sacrifice to Power Attack that maximizes my expected damage output for the round?

A couple notes:

• I will happily upvote partial solutions (e.g., ones that only apply to a single attack without considering iteratives, or ones that ignore crits for simplicity's sake)
• Ignore considerations that require you to know how close to death the defender is. I'm happy with answers that naively maximize the expected value of my damage output against an idealized combat dummy with infinite HP. Accounting for the desire to maximize the probability of dealing lethal damage against low-HP opponents is, I think, too complicated, and beyond the scope of this question.
• Ignore the Shock Trooper feat for purposes of this question; obviously if you're giving up AC for damage instead of AB, it's a risk/reward judgment call, not a case where there's an objectively optimal value.

Answer 1

It turns out someone has built a calculator for that. After playing around with that calculator for a bit, it looks like the general answer, assuming you'd normally need to roll a 10 to hit, is "subtract a bit less than 1/2 of your BAB to maximize damage output" over a decent range of settings; a little more for a single attack. If you know the target AC, the calculator will tell you exactly where the optimum number is, but it seems to be between about 1/3 and 2/3 BAB over a decent range of non-trivial (eg., "need a nat-20 to hit to start with") inputs.

As the questioner pointed out: higher base damage tends to push the optimum penalty down (ie., never Power Attack with a "Munchkin's Sword of +Graham's Number Damage" and always max Power Attack with a 1d2-1 toothpick). A more extreme total attack bonus, relative to the target AC, tends to push the optimum penalty up (ie., always max Power Attack if you need a nat-20 anyway, or if you'll still only miss on a 1).

Interestingly: increasing the threat range or multiplier did increase the optimum penalty, but never by more than a point (all else being equal). Also, the number of extra attacks at max BAB didn't affect the optimal penalty (ie., having an extra attack from, say, Haste changes to the amount of damage you can theoretically do at any given Power Attack penalty, but it doesn't seem to change which penalty maximizes that damage).

Some basic theory that bears that out.

On the surface, crits and crit failures should cancel out, but that really only applies with a basic "20/x2" crit weapon. IME, creatures with Power Attack are likely to prefer high crit threat ranges and/or multipliers. One confirmed crit might be worth 3 or 4 hits even before Power Attack is brought into play. Power Attack magnifies this difference, since the additional damage from Power Attack is multiplied.

With an x3 weapon, confirming 1 crit is worth missing twice. With Power Attack and a two-handed weapon, that x3 crit is doing 6 points of extra damage per point of Power Attack penalty, which can pretty easily be worth another miss or two. However, the Power Attack penalty cuts both ways here: it increases the chance that a hit is a crit threat (that is: given that an attack hits, the odds of that attack also being a crit threat are higher), but it also decreases the chances of confirming that crit. And, iteratives are especially hurt, since they're less likely to land anyway.

Having extra full-bonus attacks doesn't change how the Power Attack penalty affects iterative attacks, it just increases the number of attacks that care the least about the Power Attack penalty.

Related: the version of Power Attack in Pathfinder (which I recognize isn't 3.5; bear with me a moment) removes the question (and increases the bonus damage):

You can choose to take a –1 penalty on all melee attack rolls ...

When your base attack bonus reaches +4, and every 4 points thereafter, the penalty increases by –1 ...

(note that it also increases the damage to 2 points base, 1 point for off-hand weapons, 3 points for two-handed weapons)

So, the penalty is:

BaB	Penalty	Percent BAB
0	1	X
1	1	100
2	1	50
3	1	33
4	2	50
5	2	40
6	2	33
7	2	28
8	3	38
9	3	33
10	3	30
11	3	27
12	4	33
13	4	30
14	4	29
15	4	27
16	5	31
17	5	29
18	5	28
19	5	26
20	6	30

... which mostly comes out to "a bit less than 1/3 BAB". Barring knowledge of your target's AC, that's probably a reasonable starting point.