It’s time for a blog post on randomness and how our rolls and averages are calculated! This is a common question or concern, so it’s time to discuss how we can determine whether Adventure drops and success rolls are working correctly.
Too Long; Didn’t Read
Randomness can be streaky, meaning you can have stretches or very poor or very good results. Only when averaged out over large sample sizes and long periods of time can the averages show.
MATH AND STATS AHEAD
DISCLAIMER: I like numbers, but I am far from a statistician, mathematician, or a data scientist. It's entirely possible that I am wrong in how I apply statistics or mathematics.
(image courtesy of XKCD)
You Are A Pattern Recognition Machine
Game designers, and game developers in general, have long known that people are very good at recognizing patterns in nature. Pattern recognition is an inherent ability in humans, and the ability to see patterns in randomness is called Apophenia or Patternicity. Michael Shermer calls Patternicity, “the tendency to find meaningful patterns in meaningless noise.” Many scholarly studies have noted that humans find patterns in otherwise random data, and that humans are inherently poor at seeing randomness without ascribing patterns to it.
There are volumes written about the human ability to detect patterns in otherwise meaningless data. Perhaps the most well-known example is the Gambler’s Fallacy, a mistaken belief that because something happens more often now, it will happen less often in the future, or that the opposite will be true.
The goal of this blog is to show that Adventure drops, the Random Number Generator, and Adventure success rates are constants that do not vary unless we call out changes to them. Hopefully, this blog can be a reference for future concerns that the Random Number Generator is not working correctly.
Sample Size Is Important
In order for statistical analysis to be significant, and account for random distribution, we need to have a sufficiently large sample size to account for the randomness inherent in a Random Number Generator.
The sample size for this analysis is 220,000,000 Adventure completions. Individual Adventure samples varied from a minimum of 45,324 completions to a maximum of 9,991,973 completions.
Have Adventure Drop Rates Changed Over Time?
This question comes up frequently. After streaks of bad luck, players will complain that drop rates have changed, that items are dropping less often than normal, that they in particular are affected, etc. This is a good example of a single observer making an inference from a statistically insignificant sample size. For actions that have two possible outcomes, such as failure and success, we can use smaller sample sizes and still have significant results. For Adventure success rates, with the same attack value and same defense value, a sample of 1,500 adventures would probably be sufficient for 95-99% accuracy on whether Adventures are succeeding or failing correctly. For loot table drops, however, we have a minimum of five possible outcomes, with up to 10 (or more) outcomes for more complex Adventures. This means that even numbers that would approach statistical significance in two-variables tests are not sufficient for a representative sample.
For an example, a player completed quite a few runs of Flea Bottom (~400+), and saw some streakiness in their results:
When compared to Flea Bottom daily drops, we see the averages appear:
And when compared to Flea Bottom monthly drops, the averages smooth even more:
The same is seen in Summerhall (Volume 3, Level 102):
And in Oldtown (Volume 3, Level 87):
So the answer is no, Adventure drop rates do not change over time, unless we say so in build notes. Observed streakiness is due to the relatively small sample size. Please remember that relative is, well, relative: for multiple-output Adventures, you may need results from tens of thousands of attempts to reach a valid conclusion.
Are Rolls Truly Random?
Some players have complained that their rolls are non-random, or show favoritism towards failure or success. These players point to odds-based reasoning to show that what occurred is statistically impossible. Here is an example:
“I sent out 8 Sworn Swords, and each one rolled a 50. This is impossible, and shows that your Random Number Generator is broken.”
The statement is false in two key ways. First, that this result is impossible. It is improbable, but entirely possible. Given the sample size we are working with - 220,000,000 results for this blog - and given the high number of players engaged in sending out Sworn Swords, even very improbably results have a good chance of occurring for our players at some point.
Second, looking at binomial distribution, the probability of the same result coming up for seven attempts is very rare - but not impossible. The probability of the same result coming up for eight attempts is tiny, but with 220,000,000 adventures, and tens of thousands of players, unlikely events can (and do) occur.
Fortunately, we don’t need to rely on inference or estimation - we have all of the roll results for every Adventure completed in the last eight months. Let’s review them:
They continue on up to rolling a 100. Sample size varies from 85,000 per roll result to 630,000 per roll result (based on total Adventuring per month). When averaged out, the Random Number Generator for rolls settles at 1% for each possible roll.
When looking at individual adventures, we see the same results. Here are four representative adventures, showing the actual rolls for each Adventure:
NOTE: Two of the results up above are from the Braavosi adventures.
The grey bars above show the deviation between each adventure’s rolls as a percentage of the total. The red bars show the absolute number of rolls for each Adventure. While each individual adventure varies in popularity, the percentage rate for each roll stays within a very tight range.
The Gambler’s Fallacy and Random Shop Packs
Wikipedia (I know, I'm citing Wikipedia) defines the Gambler’s Fallacy as “the mistaken belief that if something happens more frequently than normal during some period, then it will happen less frequently in the future, or that if something happens less frequently than normal during some period, then it will happen more frequently in the future (presumably as a means of balancing nature).” This comes up often in Game of Thrones Ascent, whether through purchasing random packs or by acquiring Adventure Party rolls. When the common reward is a seal, and the rare reward is a peerless item, getting five seals in a row can make you feel like the world will balance itself out and give you a peerless item soon.
This is the fallacy. The chance of receiving an item remains the same regardless of how many attempts are made. Wikipedia goes into this better than I can, using the example of flipping a coin:
“Now suppose that we have just tossed four heads in a row, so that if the next coin toss were also to come up heads, it would complete a run of five successive heads. Since the probability of a run of five successive heads is only 1⁄32 (one in thirty-two), a person subject to the gambler's fallacy might believe that this next flip was less likely to be heads than to be tails. However, this is not correct, and is a manifestation of the gambler's fallacy; the event of 5 heads in a row and the event of "first 4 heads, then a tails" are equally likely, each having probability 1⁄32.”
Getting four seals in a row does not mean you are more (or less) likely to acquire a peerless item from a random pack; the chance of receiving an item remains the same, and it is equally likely that you will or will not receive an item versus a seal. Especially in small sample sizes, where customers purchase a handful of random packs, even relatively high odds like a 40% chance of receiving a peerless item can result in no peerless items after five, or even ten purchases.
No More “Nodsplaining,” Please!
I realise this is a long blog post with lots of numbers, so I’ll end it here. As always, if you have any questions or comments, let us know on the forums!