The bonus to damage is more beneficial when the probability to hit is greater than the expected damage -1 divided by 20.
When the chance to hit is greater than the average damage diminished by 1 and divided by 20, then a +1 on the damage roll is better:
$$
p > \frac{\text{E}[d]-1}{20}
$$
It then depends on the character, the average damage and on the average probability to hit. Consider a 1st level fighter with a longsword, as an example: the expected value for their damage is 7.5 (1d8+3, assuming an optimal strength of 16), then a +1 bonus on damage is convenient if they hit with a probability greater than 32.5%. Since the average hitting probability is 65%, such bonus is more convenient rather than a bonus to hit.
There is always a chance to miss.
Suppose that a fighter (5th level, STR 18) has a bonus to hit equal to +7, they are under the effect of Bless and they are fighting a group of Ogre Zombie. The probability to hit is 95% and not 100% since a natural 1 is always a miss.
Critical hits.
With a straight roll, if the only way to hit an enemy is a 20 on the attack roll then the expected damage is
$$
\frac{1}{20} 2\,\text{E}[d] = \frac{\text{E}[d]}{10}.
$$
A bonus to damage does not change the probability to hit, it increases the final damage:
$$
\frac{1}{20} \left(2\,\text{E}[d]+1\right) = \frac{\text{E}[d]}{10}+\frac{1}{20}.
$$
If even with a +1 on the attack roll one still needs a 20 to hit, then the expected damage is the same of a straight roll:
$$
\frac{1}{20} 2\,\text{E}[d] = \frac{\text{E}[d]}{10}.
$$
Target's AC is 20.
When the AC of the target is 20, with a straight roll or with a flat +1 damage bonus one needs a critical hit: then the damage is
\$10\% \text{E}[d]\$. On the other hand, a +1 on attack rolls allow to hit both with a critical hit and with a 19 on the roll: then the expected damage is \$15\%\text{E}[d]\$.
The Mathematical explanation is reported below, for the interested readers.
The expected value \$\text{E}[D]\$ for the damage is given by the sum of two terms:
- the probability \$p_{nc}\$ to hit without a crit multiplied by the expected value \$\text{E}[d]\$ of the damage roll.
- the probability \$p_{c}\$ to hit with a crit multiplied by the 2 times the expected value \$2\text{E}[d]\$ of the damage roll.
The final formula reads
$$
\text{E}[D] = p_{nc}\,\text{E}[d] + p_c\,2\text{E}[d].
$$
In order to clarify this, let's take an example. Suppose that, considering all the bonus, one needs a 14 or more on the d20 to hit an enemy: then, the probability to hit is (21-14)/20=7/20=35%. But we have to discern non critical hits and critical hits for the expected value. Indeed, for rolls between 14 and 19 (extrema included) the expected damage is \$d\$, but with a natural 20 the expected damage is doubled, i.e \$2\,d\$. Then, the formula for the expected damage is
$$
\begin{split}
&P(14\text{-}19 \text{ on d20 })d + P(\text{critical hit})d=\\\\
&\frac{6}{20}d + \frac{1}{20}2\,d =\\\\
&\frac{3}{10}d + \frac{1}{10}d =\\ \\
&\frac25 d.
\end{split}
$$
If the target’s AC is such that the d20 roll needed to hit is \$x\$, then
$$
p_{nc} = 1-F(x)+\frac{1}{20} - \frac{1}{20} = 1-F(x)
$$
where \$F(x)\$ is the cumulative distribution function, \$1-F(x)\$ is the probability to roll a number greater than \$x\$, the first 1/20 term takes into account the probability to roll a \$x\$ and the last 1/20 excludes the prob to get a nat 20. Consider the above example for an explanation: if one needs a 14 on the roll to hit, then the probability to hit without crit is 6/20, which is exactly \$1-F(14)\$, where \$F(14)= 14/20\$.
The probability \$p_c\$ to get a crit instead is simply 1/20.
Gathering everything, one then has
$$
\text{E}[D] = \left(1.1-F(x) \right)\,\text{E}[d]
$$
A flat +1 bonus to hit consists in increasing the probability to hit of a 5%\$=1/20\$ more. Then, the two scenarios are given by
$$
\text{E}_{hit}[D] = (1.15-F(x) )\,\text{E}[d]
$$
for the bonus to hit and
$$
\text{E}_{damage}[D] = (1.1-F(x) )\,(\text{E}[d]+1)
$$
for the bonus to damage roll. The breakeven point, i.e. where the two bonuses are equal, is given by
\begin{eqnarray}
(1.1-F(x) )\,(\text{E}[d]+1) &=& (1.15-F(x) )\,\text{E}[d]\\\\
F(x) &=& 1.1 - \frac{\text{E}[d]}{20}
\end{eqnarray}
and then having a +1 on the damage is more convenient when \$F(x) < 1.1 - \displaystyle\frac{\text{E}[d]}{20}\$. Then, since the total probability \$p\$ to hit is given by \$1-F(x)+\frac{1}{20}\$, one obtains the final result
$$
p > \frac{\text{E}[d]-1}{20}
$$
