What are the odds of rolling high all the time?

Question

So, I have a suspicion that a player in an online game is... being creative in the way they call their dice rolls. I play with them in several games, and in 2 games I have convinced the DM to enforce online rolling, but in the 3rd game that is unlikely to happen.

Today I noted all their D20 dice rolls:

18, 13, 20, 8, 15, 11, 19, 7, 2, 15, 20, 8, 17, 18, 17, 11, 16

So of 17 rolls:

1. 10 were 15 or above

2. Only 4 were below 10 (to make it even worse all the low rolls just so happened to be when the player knew a low roll would do, such as hitting a zombie with a +11 bonus, and the higher rolls on checks they cared about where the DC was uncertain, a common pattern)

What are the odds of this happening?

Note: I am not interested in ways to attempt to stop what I think is cheating. At some point I will just talk to the DM in question, but this particular distribution seems even worse than I expected.

Answer 1

This may not be as unlikely as you think

Preamble: Probability is Hard

So distributions like this are tricky, because a lot of people assume, wrongly, that they only need to calculate the odds of getting a distribution this good (or better), and then presume, based on the improbability of those odds, that it's evidence of tampering/cheating. But that's not quite right, and to demonstrate, I'm going to borrow an example Matt Parker used when he was assessing the odds that a speedrunner cheated an RNG mechanic in a video game.

In this example, we consider an experiment where someone flips a [presumed to be fair] coin 100 times. Then, after the experiment is concluded, a third party looks at the results, and notes that at one point during the experiment, there was a run of 12 flips which resulted in 10 Tails results and 2 Heads results. They then note that the odds of getting at least this many Tails results in a run of 12 flips is only about 1.9%, and conclude that this is improbable with a fair coin; therefore, they conclude, the person flipping a coin must have cheated, either by using an unfair coin, or by using some kind of technique to bias the results.

However, as Matt goes on to point out, you can't simply consider the odds that a run of 12 flips results in 10 (or more) Tails; you have to also consider the odds that, over the course of the entire experiment, you could get a run of 12 flips with an outcome this extreme. And as it turns out, those odds are actually about 88%. In other words, it's actually very likely, given enough trials of flipping coins, to get an individual run that's relatively improbable on its own.

So, in your case, the question we need to solve is not "how unlikely is it to get a run of d20 rolls this lucky in the course of a single night?", but rather "over the course of several sessions of a game, how unlikely is it for someone to get a run of d20 rolls that was at least this lucky?"

Let's do some math

So in your case, you've tracked 17 rolls from this player over the course of a single night, where the results were unusually high. Below are the odds of the two facts you've chosen to note:

• At least 10 of 17 rolls were 15 or higher: This has a probability of about 1.27%
• At most 4 of 17 rolls were below 10: This has a probability of about 5.96%

One more fact I'm going to track:

• Of 17 d20 rolls, the average result is 13.824 or higher: This has a probability of about 0.88%

So a brief sanity check we can perform on these odds, before we go further, is to note that none of these outcomes are terribly unlikely. The second condition happens more frequently than the odds of someone happening to roll a natural 20 on a d20 on any given roll, and we don't generally assume that any person who happens to roll a natural 20 is cheating based on that one roll. And there are a lot of improbable events that happen all the time that we generally would not think of as being evidence of cheating despite their absurdly low probability. As an example, I'll submit this combat log from a session my group had about a month ago, where on a 4d8 roll, our cleric rolled four ones, and then promptly rolled a natural 1 on her subsequent attack roll:

And, just so it doesn't go unstated: I can personally verify, as the DM and maintainer of the VTT these results were obtained upon, that these were fair results, despite the fact that the odds of this happening were about 0.00122% (or, 1 in 81,920).

But, I should also not leave unstated: that was a cherrypicked result from a long series of rolls over the course of a campaign that has run almost every week over two years. You could dig into any campaign and find individual runs that were at least as unlikely, perhaps even moreso.

Now, in your case, we don't have every single d20 roll to analyze; only the 17 from the session you chose to record. So we do have to make some educated guesses about, for example, how many sessions you've been in this campaign/game with this player, and how many d20 rolls they made in those other sessions. I'm going to assume that each session you've participated in has had a similar number of rolls (so, 17). We then have to ask, given the probabilities for each of the facts we're considering, how likely they are to have occurred at least once over X sessions.

Sessions	Odds of >=10 greater than 15	Odds of <=4 less than 10	Odds of average >=13.824
1	1.27%	5.96%	0.88%
2	2.52%	11.56%	1.75%
3	3.76%	16.84%	2.62%
4	4.98%	21.79%	3.47%
5	6.19%	26.45%	4.32%
6	7.38%	30.84%	5.17%
7	8.56%	34.96%	6.00%
8	9.72%	38.84%	6.83%
9	10.87%	42.48%	7.65%
10	12.00%	45.91%	8.46%
11	13.12%	49.13%	9.27%
12	14.22%	52.16%	10.06%
13	15.31%	55.02%	10.86%
14	16.38%	57.70%	11.64%
15	17.45%	60.22%	12.42%
16	18.49%	62.59%	13.19%
17	19.53%	64.82%	13.95%
18	20.55%	66.92%	14.71%
19	21.56%	68.89%	15.46%
20	22.56%	70.74%	16.20%

Now, it's important to note that not all of these numbers are simultaneously relevant. It's much more likely that one column is the most relevant, depending on how exactly you think this player is cheating (i.e. are they fudging die rolls higher? Are they making up numbers and just happening to choose them to be high? Are they making up numbers, but only when failure would be really bad for them?).

What's relevant for our purposes is that if you've played only 6 sessions with this player, then regardless of which properties we think are relevant, the player has at least a 5% chance of achieving those results at least once off a fair set of dice. If you've played more sessions with them, those odds get a lot better. A 5% chance is low; but again, 5% chances happen all the time.

And this assumes that we only care about the average result, i.e. we assume their method of cheating is that the player has been systematically nudging their dice results upwards. If we think their method of cheating has been to (secretly) reroll low results, then the results they achieved actually have a 30% chance of happening legitimately.

Conclusion: The results you sampled do not prove cheating

To be clear, they don't prove innocence either. A 5% chance is pretty low, and if the player is cheating, these results would be consistent with what you'd expect given a player who either systematically fudges die results, or who just makes up numbers and biases them to be high enough to succeed.

But what I would say is that, if you plan to accuse a player of cheating in this game, I think this data would not be good evidence to support it. The odds that they're playing fairly are just too high. At best, it suggests a need to monitor their results and see if they continue to get lucky or if this was just a hot streak, and I think that to say the player is definitively cheating, they'd have to have odds lower than what's being shown here. If you collect more of their rolls, you'd be able to run a similar analysis with both sessions' data and narrow things down a bit, which might get closer to proving that they are or aren't cheating.

Answer 2

You can use systems like AnyDice for answering this kind of question, although I am author of a Ruby gem games_dice which can also calculate probabilities of outcomes from simple dice systems, so I used that as I am more familiar with it.

I focused on the claim "what is probability of getting 15+ in 10 or more rolls out of 17".*

The maths for this is based on permutations and combinations. I won't go into the maths in detail, when another site is more suitable - for examples of the calculation you can go to https://math.stackexchange.com/questions/151810/probability-of-3-heads-in-10-coin-flips

I make the chances of observing this result or better (when stated as a target in advance of the rolls) is roughly 1%, or a probability of 0.01269

You follow up the issue of the 10 high rolls by also noting that there only a few low rolls (only 4 lower than 10). This is trickier to consider. The two events - many high rolls, and few low rolls - are not independent observations. The conditional probability of seeing 4 or less results under 10 out of 17, given that you already know 10 of them were greater than 14, is completely unremarkable, actually the expected result. My code doesn't do that conditional calculation directly, but I estimate you would see that secondary result - 4 or less under 10, given that at least ten dice were 15+ - about 85% of the time (update: My maths was wrong, see HighDiceRoller's answer for the correct number here).

In theory you could combine those two results and claim that you would expect the player's results to be that good only 0.85% of the time. However, there are issues with that, not least of which is applying the boundaries you are using after the fact. It is simpler to focus on the high rolls only, as it is reasonable to claim in advance that getting very many high rolls is unexpectedly lucky. The ideal scientific approach would be to choose the boundary (e.g. 15+) and rules for taking the samples (e.g. "every attack and save roll made by player X in next session") in advance. Post-event statistical analysis suffers from re-use of the data - you use it both to search for patterns, then analyse it for the same patterns - which skews how you think about probabilities unless you are very careful.

---

As comments have suggested on the question, there is more to consider here than the odds of something happening in a person's favour. I would not personally consider a 1% chance event worth mentioning much. It is very hard to use statistical analysis like this as proof of much at all - both mechanically, but maybe more importantly, socially within the group.

* For the record, the Ruby code to do this is:

require 'games_dice'

dice = GamesDice.create('17d20m15')

# The easy 10+ higher than 15
puts dice.probabilities.p_ge(10)

# The harder conditional 4 or less worse than 10 (agrees with HighDiceRoller)
p_combo_low = [*10..17].inject(0.0) do |t, n|
  # Calculate probability of n out of 17 >= 15, and 4 or less < 10
  remaining = 17 - n
  if remaining <= 4
    p_less_than_four = 1.0
  else
    remaining_dice = GamesDice.create("#{17-n}d14m:<10,1.")
    p_less_than_four = remaining_dice.probabilities.p_le(4)
  end

  t + dice.probabilities.p_eql(n) * p_less_than_four
end

puts p_combo_low / dice.probabilities.p_ge(10)

Answer 3

What are the odds of this happening?

You are looking at the wrong numbers.

Tell me, if they had rolled 17x a "19" and for their two really important rolls they bombed it with a natural "1", would you accuse them of cheating? Even though those odds are way more extreme in their favor, the fact that they bombed their most important rolls weights way more than any dice probability.

If they had to roll 3x 16 to succeed and they rolled 3x 15, you would not think they were cheating. I mean that is an incredibly good result... but it still wasn't enough. They failed. Nobody cheats to fail.

So look at other numbers: how often did they succeed. How were their chances to succeed? Do they always miracolously succeed when it's important for their character? Maybe even against the odds? While all their low rolls are on unimportant things? Those are the telltale signs of cheating.

Those signs above are way more telling then any dice probabilities. Because they are just that. Probabilities. For those dice to be fair, somebody has to roll incredibly good (or bad) once in a while. A dice set that rolled average all the time would be just as broken as one that rolls "too good" or "too bad".

Summary: Dice probabilities are just that. They don't tell any story. Look at what happened with them in the game, to find out if someone is cheating.

Answer 4

Those numbers are unlikely.

They could be naturally occurring, especially given the small sample size. However if the pattern continues over a larger one, it becomes increasingly unlikely they are properly random. To increasingly infinitesimal likelihoods, as that's quite an aggressively off-centre set of numbers.

There is also stronger evidence.

If the player in question appears to value certain checks more than others or has information about the difficulty of tasks, and the numbers rolled seem to match the difficulty of the task, it becomes far more likely they are not being randomly generated.

However, this is not necessarily dice-cheating or fudging.

One thing to keep in mind with things like this is at actual tables, people do reliably roll high numbers on certain dice. Certain 'lucky', aka poorly weighted dice. Unlike casino dice made to within exacting standards, ttrpg dice are made by whoever, and can be poorly weighted.

I mean, odds are good this is. But that kind of distribution can be the result of poorly weighted dice. There are also people who consistently roll above the odds, using table dice that do not roll above the odds for anyone else - whether this is simply luck or them unconsciously being able to toss dice to come out with the number they want (an ability that can be learnt by (or is claimed to be learnt by) professional gamblers and used under much tougher conditions than a typical flat wooden surface) isn't something I can say, but I have seen very atypical distributions of numbers over very long periods - sometimes a period of years.

There is no way to know for certain.

If the game changes to another rolling method and the numbers normalize for this player, that may be that he's simply no longer rolling his 'lucky' dice, or controlling the throws. If those were not options, then it normalizing would be much more weight in the theory that he is intentionally cheating. But with those factors, and simply the nature of random chance, it is impossible to be certain.

Reading the player's attitude, or asking after the game is completed may give more information than attempting to analyze the distribution of the rolls. Human interaction is pretty informationally dense, and it's one of the skills humans are naturally the best at. Although it's likely a player cheating at dice would lie about it if they felt there was no way for them to be caught in the lie, they may also admit or talk about it if it's phrased in a non-threatening enough way.

Answer 5

That is definitely worse (or better) than you would usually expect.

You can also visualize it yourself pretty easily

You can quickly visualize the skew of a small (less than 20 or so) group of numbers.

Arrange them in order and just to make it look right, pad to an even number of digits, like so:

02 07 08 08 11 11 13 15 15 16 17 17 18 18 19 20 20

Now, "balance" it by finding the number in the middle, in this case with 8 numbers on the left and 8 on the right:

02 07 08 08 11 11 13 15 15 16 17 17 18 18 19 20 20
                        ^^

This middle number is the median. On average, the median of a set of 17 numbers, 1 to 20, should be around 10 1/2, the average expected roll.

Even more telling, if you see where 10 1/2 would fall, only 4 numbers were below average, and 13 were above average.

02 07 08 08 | 11 11 13 15 15 16 17 17 18 18 19 20 20

That's skewed, baby.

What does this tell you?

Not much, with one set of numbers. Even unlikely things happen, all the time.

But if the trend over time doesn't head toward average, then that tells you whatever is generating the numbers is not behaving as expected.

It's one thing if one supposedly random set of 17 numbers is skewed, but over time, as you add more and more sets of numbers, if it doesn't trend toward average, your number generator just ain't random.

Trust but verify

In our game, we play online, and for die rolls we use D&D Beyond or Avrae on Discord, and players' die rolls are always seen by at least the GM.

It's not that we don't trust each other, it's just that it takes the issue off the table.

Just a thought.

Answer 6

Only 4 were below 10: slightly less likely than might be expected

Neil Slater has an excellent answer, especially in pointing out the dangers of forming a hypothesis after the fact. I just wanted to expand on point #2:

Only 4 were below 10

A tempting approach

We may be tempted to figure: We know that 10 out of 17 dice rolled 15+. That leaves 7 dice. Each of those dice has a 9/20 chance of rolling below a 10. Do a little binomial theorem and we get an 84.7% chance of at most 4 of those dice rolling below a 10.

The problem

The problem with this approach is that, conditioned on at least 10 out of 17 dice rolling 15+, the remaining 7 dice can no longer be considered as uniform and independent. The most likely case is that exactly 10 out of 17 dice rolled 15+, in which case none of the remaining dice rolled 15+, effectively reducing the d20 to a d14 for those remaining dice. Suddenly rolling a 9- looks more likely.

All-in-all, if my computation is correct, the chance of 4 dice or fewer rolling 9-, conditioned on 10 or more dice rolling 15+, is 54.9%.

Is this of any practical difference compared to 85%? IMO 85% vs. 55%---or even 100% vs. 55%---doesn't make an appreciable difference to the strength of the accusation; I am in complete agreement with Neil Slater that these are not enough to sustain an accusation.

Code

Here's a JupyterLite notebook which uses my hdroller Python library.

import hdroller

# Dimension 0 is the number of dice that scored 15+.
# Dimension 1 is the number that scored 9-.
die = hdroller.d20.sub(lambda x: (x >= 15, x < 10))

# Roll 17 of them and sum.
seventeen_rolls = 17 @ die
print('Chance of rolling 15+ on at least 10 out of 17 dice:')
print(seventeen_rolls.dim[0] >= 10)

# Filter out everything that didn't fit the above condition.
seventeen_rolls_cond = seventeen_rolls.reroll(lambda a, b: a < 10, max_depth=None)
print('Chance of rolling 9- on at most 4 dice, conditional on the above:')
print(seventeen_rolls_cond.dim[1] <= 4)

print('For comparison: if the remaining 7 dice were uniform and independent d20s:')
print(7 @ (hdroller.d20 < 10) <= 4)
print('For comparison: if the remaining 7 dice were uniform and independent d14s:')
print(7 @ (hdroller.d14 < 10) <= 4)

Chance of rolling 15+ on at least 10 out of 17 dice:

Denominator: 13107200000000000000000

Outcome	Weight	Probability
False	12940833918520166973440	98.730728%
True	166366081479833026560	1.269272%

Chance of rolling 9- on at most 4 dice, conditional on the above:

Denominator: 166366081479833026560

Outcome	Weight	Probability
False	75088211144104304640	45.134327%
True	91277870335728721920	54.865673%

For comparison: if the remaining 7 dice were uniform and independent d20s:

Denominator: 1280000000

Outcome	Weight	Probability
False	195747435	15.292768%
True	1084252565	84.707232%

For comparison: if the remaining 7 dice were uniform and independent d14s:

Denominator: 105413504

Outcome	Weight	Probability
False	54384129	51.591235%
True	51029375	48.408765%

Answer 7

A simple chi^2 test for 95% confidence gives the comparison between the numbers 30.1435272 (which is the 0.95 for chi^2 with 19 d.o.f.) and 14.55 (which is the sum of some numbers-long story). Since the second is smaller we conclude that these data are insufficient to conclude with a confidence 95%, that the dice is NOT honest.