24
\$\begingroup\$

So, I have a suspicion that a player in an online game is... being creative in the way they call their dice rolls. I play with them in several games, and in 2 games I have convinced the DM to enforce online rolling, but in the 3rd game that is unlikely to happen.

Today I noted all their D20 dice rolls:

18, 13, 20, 8, 15, 11, 19, 7, 2, 15, 20, 8, 17, 18, 17, 11, 16

So of 17 rolls:

  1. 10 were 15 or above

  2. Only 4 were below 10 (to make it even worse all the low rolls just so happened to be when the player knew a low roll would do, such as hitting a zombie with a +11 bonus, and the higher rolls on checks they cared about where the DC was uncertain, a common pattern)

What are the odds of this happening?

Note: I am not interested in ways to attempt to stop what I think is cheating. At some point I will just talk to the DM in question, but this particular distribution seems even worse than I expected.

\$\endgroup\$
13
  • 5
    \$\begingroup\$ @NautArch Not really in this case. I am genuinely interested in knowing the odds here. I know how to handle the situation, this is just adding some clarity for me, and I like clarity. \$\endgroup\$
    – SeriousBri
    Feb 23 at 22:15
  • 1
    \$\begingroup\$ @NeilSlater if this was once then definitely not, but honestly this has been in 3 games for years. I don't remember him failing a single check when rolling in 'secret', this specific question was just to satisfy a bit of curiosity. That said - can you make an answer out of your comment? \$\endgroup\$
    – SeriousBri
    Feb 23 at 22:18
  • 4
    \$\begingroup\$ This is a really tricky problem from a mathematical point of view, with a lot of counterintuitive phenomena. Simple indicators such as average and median are NOT a good answer. I advise you to post on the Mathematics or Cross Validated stacks. \$\endgroup\$
    – Rad80
    Feb 24 at 8:37
  • 1
    \$\begingroup\$ Anecdotally, I know a player who fairly consistently rolls similarly well, particularly with physical dice, regardless of which set of dice they are using, verifiable by being in person or on camera while doing it (or having rolls checked by someone else in the household trustworthy to not allow cheating, depending on the specific circumstance). \$\endgroup\$ Feb 24 at 16:03
  • 2
    \$\begingroup\$ Not being anything close to a mathematician or statistician, I've always looked at odds as being the way to show how unlikely something was to happen after you've just witnessed it. "Million-to-one chances crop up nine times out of ten." --Terry Pratchet. \$\endgroup\$
    – CitizenRon
    Feb 25 at 14:38

7 Answers 7

52
\$\begingroup\$

This may not be as unlikely as you think

Preamble: Probability is Hard

So distributions like this are tricky, because a lot of people assume, wrongly, that they only need to calculate the odds of getting a distribution this good (or better), and then presume, based on the improbability of those odds, that it's evidence of tampering/cheating. But that's not quite right, and to demonstrate, I'm going to borrow an example Matt Parker used when he was assessing the odds that a speedrunner cheated an RNG mechanic in a video game.

In this example, we consider an experiment where someone flips a [presumed to be fair] coin 100 times. Then, after the experiment is concluded, a third party looks at the results, and notes that at one point during the experiment, there was a run of 12 flips which resulted in 10 Tails results and 2 Heads results. They then note that the odds of getting at least this many Tails results in a run of 12 flips is only about 1.9%, and conclude that this is improbable with a fair coin; therefore, they conclude, the person flipping a coin must have cheated, either by using an unfair coin, or by using some kind of technique to bias the results.

However, as Matt goes on to point out, you can't simply consider the odds that a run of 12 flips results in 10 (or more) Tails; you have to also consider the odds that, over the course of the entire experiment, you could get a run of 12 flips with an outcome this extreme. And as it turns out, those odds are actually about 88%. In other words, it's actually very likely, given enough trials of flipping coins, to get an individual run that's relatively improbable on its own.

So, in your case, the question we need to solve is not "how unlikely is it to get a run of d20 rolls this lucky in the course of a single night?", but rather "over the course of several sessions of a game, how unlikely is it for someone to get a run of d20 rolls that was at least this lucky?"

Let's do some math

So in your case, you've tracked 17 rolls from this player over the course of a single night, where the results were unusually high. Below are the odds of the two facts you've chosen to note:

  • At least 10 of 17 rolls were 15 or higher: This has a probability of about 1.27%
  • At most 4 of 17 rolls were below 10: This has a probability of about 5.96%

One more fact I'm going to track:

  • Of 17 d20 rolls, the average result is 13.824 or higher: This has a probability of about 0.88%

So a brief sanity check we can perform on these odds, before we go further, is to note that none of these outcomes are terribly unlikely. The second condition happens more frequently than the odds of someone happening to roll a natural 20 on a d20 on any given roll, and we don't generally assume that any person who happens to roll a natural 20 is cheating based on that one roll. And there are a lot of improbable events that happen all the time that we generally would not think of as being evidence of cheating despite their absurdly low probability. As an example, I'll submit this combat log from a session my group had about a month ago, where on a 4d8 roll, our cleric rolled four ones, and then promptly rolled a natural 1 on her subsequent attack roll:

Image of Combat Log showing 4 1's rolled on 4d8, and then a natural 1 on an attack roll

And, just so it doesn't go unstated: I can personally verify, as the DM and maintainer of the VTT these results were obtained upon, that these were fair results, despite the fact that the odds of this happening were about 0.00122% (or, 1 in 81,920).

But, I should also not leave unstated: that was a cherrypicked result from a long series of rolls over the course of a campaign that has run almost every week over two years. You could dig into any campaign and find individual runs that were at least as unlikely, perhaps even moreso.

Now, in your case, we don't have every single d20 roll to analyze; only the 17 from the session you chose to record. So we do have to make some educated guesses about, for example, how many sessions you've been in this campaign/game with this player, and how many d20 rolls they made in those other sessions. I'm going to assume that each session you've participated in has had a similar number of rolls (so, 17). We then have to ask, given the probabilities for each of the facts we're considering, how likely they are to have occurred at least once over X sessions.

Sessions Odds of >=10 greater than 15 Odds of <=4 less than 10 Odds of average >=13.824
1 1.27% 5.96% 0.88%
2 2.52% 11.56% 1.75%
3 3.76% 16.84% 2.62%
4 4.98% 21.79% 3.47%
5 6.19% 26.45% 4.32%
6 7.38% 30.84% 5.17%
7 8.56% 34.96% 6.00%
8 9.72% 38.84% 6.83%
9 10.87% 42.48% 7.65%
10 12.00% 45.91% 8.46%
11 13.12% 49.13% 9.27%
12 14.22% 52.16% 10.06%
13 15.31% 55.02% 10.86%
14 16.38% 57.70% 11.64%
15 17.45% 60.22% 12.42%
16 18.49% 62.59% 13.19%
17 19.53% 64.82% 13.95%
18 20.55% 66.92% 14.71%
19 21.56% 68.89% 15.46%
20 22.56% 70.74% 16.20%

Now, it's important to note that not all of these numbers are simultaneously relevant. It's much more likely that one column is the most relevant, depending on how exactly you think this player is cheating (i.e. are they fudging die rolls higher? Are they making up numbers and just happening to choose them to be high? Are they making up numbers, but only when failure would be really bad for them?).

What's relevant for our purposes is that if you've played only 6 sessions with this player, then regardless of which properties we think are relevant, the player has at least a 5% chance of achieving those results at least once off a fair set of dice. If you've played more sessions with them, those odds get a lot better. A 5% chance is low; but again, 5% chances happen all the time.

And this assumes that we only care about the average result, i.e. we assume their method of cheating is that the player has been systematically nudging their dice results upwards. If we think their method of cheating has been to (secretly) reroll low results, then the results they achieved actually have a 30% chance of happening legitimately.

Conclusion: The results you sampled do not prove cheating

To be clear, they don't prove innocence either. A 5% chance is pretty low, and if the player is cheating, these results would be consistent with what you'd expect given a player who either systematically fudges die results, or who just makes up numbers and biases them to be high enough to succeed.

But what I would say is that, if you plan to accuse a player of cheating in this game, I think this data would not be good evidence to support it. The odds that they're playing fairly are just too high. At best, it suggests a need to monitor their results and see if they continue to get lucky or if this was just a hot streak, and I think that to say the player is definitively cheating, they'd have to have odds lower than what's being shown here. If you collect more of their rolls, you'd be able to run a similar analysis with both sessions' data and narrow things down a bit, which might get closer to proving that they are or aren't cheating.

\$\endgroup\$
4
  • \$\begingroup\$ Comments are not for extended discussion; this conversation has been moved to chat. \$\endgroup\$
    – Someone_Evil
    Feb 26 at 4:26
  • 1
    \$\begingroup\$ @Someone_Evil: But comments are for pointing out when an answer is objectively incorrect, which is what my highly upvoted (and now deleted) comment was. \$\endgroup\$ Feb 27 at 5:02
  • \$\begingroup\$ @BlueRaja-DannyPflughoeft Not sure I'm really seeing objectively incorrect, nor that that really is the job of the comments. There are several responses to that comment already (so now they're nicely in chat together), and the correct thing now would be to vote and/or provide a (in your eyes) correct answer if needed \$\endgroup\$
    – Someone_Evil
    Feb 27 at 12:07
  • 2
    \$\begingroup\$ 'Moved to chat' is just a way for moderators to delete comments without having to actually delete them. Nobody looks at chat. \$\endgroup\$
    – Brady Gilg
    Feb 28 at 17:46
14
\$\begingroup\$

You can use systems like AnyDice for answering this kind of question, although I am author of a Ruby gem games_dice which can also calculate probabilities of outcomes from simple dice systems, so I used that as I am more familiar with it.

I focused on the claim "what is probability of getting 15+ in 10 or more rolls out of 17".*

The maths for this is based on permutations and combinations. I won't go into the maths in detail, when another site is more suitable - for examples of the calculation you can go to https://math.stackexchange.com/questions/151810/probability-of-3-heads-in-10-coin-flips

I make the chances of observing this result or better (when stated as a target in advance of the rolls) is roughly 1%, or a probability of 0.01269

You follow up the issue of the 10 high rolls by also noting that there only a few low rolls (only 4 lower than 10). This is trickier to consider. The two events - many high rolls, and few low rolls - are not independent observations. The conditional probability of seeing 4 or less results under 10 out of 17, given that you already know 10 of them were greater than 14, is completely unremarkable, actually the expected result. My code doesn't do that conditional calculation directly, but I estimate you would see that secondary result - 4 or less under 10, given that at least ten dice were 15+ - about 85% of the time (update: My maths was wrong, see HighDiceRoller's answer for the correct number here).

In theory you could combine those two results and claim that you would expect the player's results to be that good only 0.85% of the time. However, there are issues with that, not least of which is applying the boundaries you are using after the fact. It is simpler to focus on the high rolls only, as it is reasonable to claim in advance that getting very many high rolls is unexpectedly lucky. The ideal scientific approach would be to choose the boundary (e.g. 15+) and rules for taking the samples (e.g. "every attack and save roll made by player X in next session") in advance. Post-event statistical analysis suffers from re-use of the data - you use it both to search for patterns, then analyse it for the same patterns - which skews how you think about probabilities unless you are very careful.


As comments have suggested on the question, there is more to consider here than the odds of something happening in a person's favour. I would not personally consider a 1% chance event worth mentioning much. It is very hard to use statistical analysis like this as proof of much at all - both mechanically, but maybe more importantly, socially within the group.

* For the record, the Ruby code to do this is:

require 'games_dice'

dice = GamesDice.create('17d20m15')

# The easy 10+ higher than 15
puts dice.probabilities.p_ge(10)

# The harder conditional 4 or less worse than 10 (agrees with HighDiceRoller)
p_combo_low = [*10..17].inject(0.0) do |t, n|
  # Calculate probability of n out of 17 >= 15, and 4 or less < 10
  remaining = 17 - n
  if remaining <= 4
    p_less_than_four = 1.0
  else
    remaining_dice = GamesDice.create("#{17-n}d14m:<10,1.")
    p_less_than_four = remaining_dice.probabilities.p_le(4)
  end

  t + dice.probabilities.p_eql(n) * p_less_than_four
end

puts p_combo_low / dice.probabilities.p_ge(10)
\$\endgroup\$
1
  • 2
    \$\begingroup\$ Thank you, and yes, even as the OP here I wouldn't recommend anyone to make an accusation based on a 1% chance. But that wasn't really the purpose of the question, just the background as to why I was asking. \$\endgroup\$
    – SeriousBri
    Feb 23 at 22:41
9
\$\begingroup\$

What are the odds of this happening?

You are looking at the wrong numbers.

Tell me, if they had rolled 17x a "19" and for their two really important rolls they bombed it with a natural "1", would you accuse them of cheating? Even though those odds are way more extreme in their favor, the fact that they bombed their most important rolls weights way more than any dice probability.

If they had to roll 3x 16 to succeed and they rolled 3x 15, you would not think they were cheating. I mean that is an incredibly good result... but it still wasn't enough. They failed. Nobody cheats to fail.

So look at other numbers: how often did they succeed. How were their chances to succeed? Do they always miracolously succeed when it's important for their character? Maybe even against the odds? While all their low rolls are on unimportant things? Those are the telltale signs of cheating.

Those signs above are way more telling then any dice probabilities. Because they are just that. Probabilities. For those dice to be fair, somebody has to roll incredibly good (or bad) once in a while. A dice set that rolled average all the time would be just as broken as one that rolls "too good" or "too bad".

Summary: Dice probabilities are just that. They don't tell any story. Look at what happened with them in the game, to find out if someone is cheating.

\$\endgroup\$
8
\$\begingroup\$

Those numbers are unlikely.

They could be naturally occurring, especially given the small sample size. However if the pattern continues over a larger one, it becomes increasingly unlikely they are properly random. To increasingly infinitesimal likelihoods, as that's quite an aggressively off-centre set of numbers.

There is also stronger evidence.

If the player in question appears to value certain checks more than others or has information about the difficulty of tasks, and the numbers rolled seem to match the difficulty of the task, it becomes far more likely they are not being randomly generated.

However, this is not necessarily dice-cheating or fudging.

One thing to keep in mind with things like this is at actual tables, people do reliably roll high numbers on certain dice. Certain 'lucky', aka poorly weighted dice. Unlike casino dice made to within exacting standards, ttrpg dice are made by whoever, and can be poorly weighted.

I mean, odds are good this is. But that kind of distribution can be the result of poorly weighted dice. There are also people who consistently roll above the odds, using table dice that do not roll above the odds for anyone else - whether this is simply luck or them unconsciously being able to toss dice to come out with the number they want (an ability that can be learnt by (or is claimed to be learnt by) professional gamblers and used under much tougher conditions than a typical flat wooden surface) isn't something I can say, but I have seen very atypical distributions of numbers over very long periods - sometimes a period of years.

There is no way to know for certain.

If the game changes to another rolling method and the numbers normalize for this player, that may be that he's simply no longer rolling his 'lucky' dice, or controlling the throws. If those were not options, then it normalizing would be much more weight in the theory that he is intentionally cheating. But with those factors, and simply the nature of random chance, it is impossible to be certain.

Reading the player's attitude, or asking after the game is completed may give more information than attempting to analyze the distribution of the rolls. Human interaction is pretty informationally dense, and it's one of the skills humans are naturally the best at. Although it's likely a player cheating at dice would lie about it if they felt there was no way for them to be caught in the lie, they may also admit or talk about it if it's phrased in a non-threatening enough way.

\$\endgroup\$
4
  • 1
    \$\begingroup\$ Regarding dices: I, for one, happen to use differently colored dice for different tasks (red for attack with green if multiple, teal for skill checks, gold for saves). Why? Because I like it. As far as I know my dice are relatively well balanced, however should one be poorly weighted, I would have an odd "spike" in only certain types of rolls, which could look suspicious from the outside. \$\endgroup\$ Feb 24 at 10:04
  • 1
    \$\begingroup\$ +1 for the point about most polyhedral dice not being properly balanced. \$\endgroup\$ Feb 24 at 13:37
  • \$\begingroup\$ I once had a clear d20 with a small irregular shaped air bubble slightly underneath the '20'. We swore that it was more likely to roll 20s (or the other numbers next to or near the 20), to the point where the die was not allowed for actual usage in a real game. Our assumption was that the air bubble made 1 'side' of the die lighter than the others, thus would be more likely to be roughly facing up when rolled. Had the die not been clear, we would have never known about the defect. I really, really wish I still had that thing, it was pretty cool. \$\endgroup\$
    – Graham
    Feb 25 at 14:38
  • \$\begingroup\$ "but over time, as you add more and more sets of numbers, if it doesn't trend toward average" To be fair you should not include this known sample of 17 rolls when looking at future rolls because they would bias the entire set. If you see someone roll a 20, the odds of them rolling another 20 is 1/20, not 1/400. The odds of their total for the two dice being at least 21 is 100% \$\endgroup\$ Feb 25 at 16:44
8
\$\begingroup\$

That is definitely worse (or better) than you would usually expect.

You can also visualize it yourself pretty easily

You can quickly visualize the skew of a small (less than 20 or so) group of numbers.

Arrange them in order and just to make it look right, pad to an even number of digits, like so:

02 07 08 08 11 11 13 15 15 16 17 17 18 18 19 20 20

Now, "balance" it by finding the number in the middle, in this case with 8 numbers on the left and 8 on the right:

02 07 08 08 11 11 13 15 15 16 17 17 18 18 19 20 20
                        ^^

This middle number is the median. On average, the median of a set of 17 numbers, 1 to 20, should be around 10 1/2, the average expected roll.

Even more telling, if you see where 10 1/2 would fall, only 4 numbers were below average, and 13 were above average.

02 07 08 08 | 11 11 13 15 15 16 17 17 18 18 19 20 20

That's skewed, baby.

What does this tell you?

Not much, with one set of numbers. Even unlikely things happen, all the time.

But if the trend over time doesn't head toward average, then that tells you whatever is generating the numbers is not behaving as expected.

It's one thing if one supposedly random set of 17 numbers is skewed, but over time, as you add more and more sets of numbers, if it doesn't trend toward average, your number generator just ain't random.

Trust but verify

In our game, we play online, and for die rolls we use D&D Beyond or Avrae on Discord, and players' die rolls are always seen by at least the GM.

It's not that we don't trust each other, it's just that it takes the issue off the table.

Just a thought.

\$\endgroup\$
0
6
\$\begingroup\$

Only 4 were below 10: slightly less likely than might be expected

Neil Slater has an excellent answer, especially in pointing out the dangers of forming a hypothesis after the fact. I just wanted to expand on point #2:

Only 4 were below 10

A tempting approach

We may be tempted to figure: We know that 10 out of 17 dice rolled 15+. That leaves 7 dice. Each of those dice has a 9/20 chance of rolling below a 10. Do a little binomial theorem and we get an 84.7% chance of at most 4 of those dice rolling below a 10.

The problem

The problem with this approach is that, conditioned on at least 10 out of 17 dice rolling 15+, the remaining 7 dice can no longer be considered as uniform and independent. The most likely case is that exactly 10 out of 17 dice rolled 15+, in which case none of the remaining dice rolled 15+, effectively reducing the d20 to a d14 for those remaining dice. Suddenly rolling a 9- looks more likely.

All-in-all, if my computation is correct, the chance of 4 dice or fewer rolling 9-, conditioned on 10 or more dice rolling 15+, is 54.9%.

Is this of any practical difference compared to 85%? IMO 85% vs. 55%---or even 100% vs. 55%---doesn't make an appreciable difference to the strength of the accusation; I am in complete agreement with Neil Slater that these are not enough to sustain an accusation.

Code

Here's a JupyterLite notebook which uses my hdroller Python library.

import hdroller

# Dimension 0 is the number of dice that scored 15+.
# Dimension 1 is the number that scored 9-.
die = hdroller.d20.sub(lambda x: (x >= 15, x < 10))

# Roll 17 of them and sum.
seventeen_rolls = 17 @ die
print('Chance of rolling 15+ on at least 10 out of 17 dice:')
print(seventeen_rolls.dim[0] >= 10)

# Filter out everything that didn't fit the above condition.
seventeen_rolls_cond = seventeen_rolls.reroll(lambda a, b: a < 10, max_depth=None)
print('Chance of rolling 9- on at most 4 dice, conditional on the above:')
print(seventeen_rolls_cond.dim[1] <= 4)

print('For comparison: if the remaining 7 dice were uniform and independent d20s:')
print(7 @ (hdroller.d20 < 10) <= 4)
print('For comparison: if the remaining 7 dice were uniform and independent d14s:')
print(7 @ (hdroller.d14 < 10) <= 4)

Chance of rolling 15+ on at least 10 out of 17 dice:

Denominator: 13107200000000000000000

Outcome Weight Probability
False 12940833918520166973440 98.730728%
True 166366081479833026560 1.269272%

Chance of rolling 9- on at most 4 dice, conditional on the above:

Denominator: 166366081479833026560

Outcome Weight Probability
False 75088211144104304640 45.134327%
True 91277870335728721920 54.865673%

For comparison: if the remaining 7 dice were uniform and independent d20s:

Denominator: 1280000000

Outcome Weight Probability
False 195747435 15.292768%
True 1084252565 84.707232%

For comparison: if the remaining 7 dice were uniform and independent d14s:

Denominator: 105413504

Outcome Weight Probability
False 54384129 51.591235%
True 51029375 48.408765%
\$\endgroup\$
2
  • 1
    \$\begingroup\$ Thanks, I knew I fudged the remaining below 10 calculation, but could not figure out which was the fairest approximation. Oddly enough if you just assume the remaining dice are d14, you get close to your answer. \$\endgroup\$ Feb 24 at 9:11
  • \$\begingroup\$ @Neil Slater Great point---I've added the d14 as another comparison. \$\endgroup\$ Feb 25 at 4:36
0
\$\begingroup\$

A simple chi^2 test for 95% confidence gives the comparison between the numbers 30.1435272 (which is the 0.95 for chi^2 with 19 d.o.f.) and 14.55 (which is the sum of some numbers-long story). Since the second is smaller we conclude that these data are insufficient to conclude with a confidence 95%, that the dice is NOT honest.

\$\endgroup\$
9
  • 1
    \$\begingroup\$ Welcome to the stack Nestle, take the tour when you have a moment. I too thought a chi-square test would be a good way to approach the problem here, and I think your answer could be improved by including a little bit more detail about the calculation and interpretation of the result. \$\endgroup\$ Feb 24 at 14:41
  • \$\begingroup\$ Welcome to RPG.SE! I think too that this approach is the best one, but as @ThomasMarkov said this answer may benefit from a more detailed explanation. Moreover, I think the main issue is that the frequency of each result is too low. \$\endgroup\$
    – Eddymage
    Feb 24 at 14:44
  • 5
    \$\begingroup\$ -1. Unfortunately, the sample size here is too small to support a chi-square test. The requirement is an expected frequency of 5 or more for each outcome, i.e., you'd need 100 rolls minimum for a chi-squared test of a d20 to be legitimate. E.g., See Weiss, Introductory Statistics, Procedure 13.1. (And aside from that, I'd note chi-squared tests are of extremely low power, i.e., false negatives are very common.) \$\endgroup\$ Feb 24 at 14:59
  • 1
    \$\begingroup\$ ... see also OpenStax Introductory Statistics, Ch. 11: The Chi-Square Distribution, Sec. 11.2 Goodness-of-Fit Test, first boxed Note: "The expected value for each cell needs to be at least five in order for you to use this test." openstax.org/books/introductory-statistics/pages/… \$\endgroup\$ Feb 24 at 15:06
  • 2
    \$\begingroup\$ It also should be heavily emphasized that 95% confidence means that the test will get it wrong 1 time in 20. Which is normally way too high, but is somewhat appropriate for a d20 task: you'll falsely accuse someone of cheating if you roll a natural 1 on your statistics check. \$\endgroup\$
    – Ray
    Feb 24 at 16:25

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .