The original, and full article, is on Qubit's Medium profile.
In a Medium blog, Qubit Data Science Lead Matthew Tamsett discusses how the Base Rate Fallacy can affect whether customer experiences are considered successful, and how Qubit Bayesian testing avoids this.
What is Base Rate Fallacy?
No statistical test is perfect.
Tests can fail in different ways, and they come with guarantees about the rates at which they fail. But, when humans fail to understand these failure rates, what they think is a failure rate of 5% might actually be a failure rate of 30%. One such misunderstanding is known as the base rate fallacy.
In A/B testing, you separate the wheat from the chaff. But, sometimes random and unrelated events in the world conspire to make a winning experience look like an insignificant or losing experience (a false negative), or make an insignificant or losing experience look like a winner (a false positive).
How Qubit protects you from this fallacy
The Bayesian A/B tests performed by Qubit measure the probability distribution of an uplift. Therefore, we can make a stronger guarantee: a winning experience has a less than 5% probability of being a fluke.
We do this by defining a “winner” as an experience where the probability of an uplift is 95% or more. Exactly 5% of these winning experiences are illusory, no matter the base rate of winning experiences. In this way, the Qubit platform preemptively prevents its users from committing the base rate fallacy.
The distinction between false positive rate and false discovery rate is very important one, but classical statistical testing makes it far too easy to forget about it and commit the base rate fallacy. This can, in theory, lead to A/B tests that are impractically stringent.
But, in practice, modern websites are already optimised to some degree and winning experiences are rare, leading to tests that are too lax and declare illusory winners too often. The Bayesian messaging in the Qubit platform doesn't let you make this mistake.