Do you want to be able to stop your tests early when you see overwhelmingly strong results, without inflating your false positive rate? Great! Me too! Read on!
If you are a Data Scientist, or if you are well versed on any form of Sequential Testing, this guide is probably not for you!
In that case you are probably looking for an advanced comparison of different Sequential Testing methodologies, and a writeup of different ways they could be implemented in large scale enterprise orgs with a strong Data Science resources, you are probably looking for one of these instead:
This guide is for the rest of us, working in orgs where we aren't lucky enough to be surrounded by world class staticticians. This is for those of us who aren't looking to implement an industry leading vision of perfection, we are just looking this question in a way that we can implement tomorrow and explain to our stakeholders quickly and easily: How can we run more sensitive experiments, while remaining agile when we see strong results?
We will go through an approach that I lean towards, that balances a sensitive runtime (i.e. a longer runtime) against reasonable flexibility, with an emphasis on being able to slot it into your org with minimal effort.
This approach is the perfect complement to a careful MDEâdriven test duration strategy. First, pick a runtime long enough to detect your target effect size; then apply OBF peeking to stop early if results are overwhelmingly strong. Both of these together are what I call RLSE Framework where RLSE stands for "Run Long, Stop Early" (pronounced as release).
As I have written about, I believe most companies doing AB testing would benefit from running their tests longer than they probably currently are. I recommend 5 week minimum as a reasonable heuristic.
When A/B testing teams set longer runtimes, they then need want to check in on experiments before they finish - but unadjusted "peeking" inflates your Type I error (false positives rate) beyond what you are happy with.
So basically, the longer you run:
My preferred simple solution is to use Group Sequential Designs with an OâBrienâFleming (OBF) We will come back to that, but first...
Most A/B Testing tooling allows a user to check on an experiment to see the results at any point during the runtime. While there is value to this (especially checking for issues early in the run cycle), there is also risk.
If you are willing to make a call on your test results any time your main metric goes above the confidence threshold, and you check it every day, you radically increase the likelihood of a False Positive (Type I error), because most tests will eventually go green â at some point if you just check them often enough. This is called peeking.
Even If you set up and run A/A tests with no actual change to the user - if you check your AA Test every day, you won't have to wait too long before you see your main metric report as conclusively positive.
This is what we are trying to avoid. Just checking your experiments all the time and making a decision when they are statistically significant - BIG NO-NO! This protocol is nothing more than a type-1 error (false positive) generation machine!
Contrary to what a lot of people who know me might think, I actually really am not concerned about the risk of a high false positive rate, in terms of rolling out red herrings. I used to be very concerned about that, but I'm not these days!
The risk of the "peeking" behaviour described above, and the inflated Flase Positive Rate, is actually - counter intuitively nothing to do with rolling out a change that was not actually improving anything.
I mean, yea, this is a risk and it will definitely happen, you will definitely end up rolling out false positives. But it's quite unimportant in the grand scheme of things. In fact, if your change was detected as a conclusively positive change, but in fact is was a False Positive, still it isstatistically highly unlikely unlikely that your change is negatively affecting users in reality. Even moreso: the chance that it is negativly affecting anything big enough to make a meaningful dent in your overall business is extremely unlikely (you would have detected it if that were the case).
The much bigger risk is that you then have a team who think they have struck solid gold đž and will spend the next 2 quarters iterating on it. They are chasing a ghost đ» and wasting their time.
Development time is typically the most precious resource in modern tech and product orgs. Wasting your dev team's time chasing a statistical ghost is far more damaging than the individual impact of rolling out a false positive. What better thing could your dev team have ben focussing on in that time, instead?
It's all about Opportunity Cost.
Remember the problems we are trying to solve. We want to:
The solution to these two problems mentioned above, and the balance between them, is almost certainly some form of Sequential Testing.
Though there are some complicated statistics involved in Sequential Testing, the concept is simple: When you see overwhelmingly strong results, you want to be able to stop your test early. We do this by deciding in advance how many times we will "peek", taking the 5% alpha, and spreading it across those peeks.
All we are doing with various calculation methods that go around this, with various different names, is we are trying to take this notion of Overwhelmingly Strong out of being just one person's opinion, and instead set predefined boundaries that we are all aligned on, to remove the human bias. There are many ways of doing that, each with various pros and cons. We will touch on various methods lightly (this is not a DS-heavy post) but first I will summarise my preferred version.
The O'Brien-Fleming method allows us to define high confidence thresholds at which we would make a decision early-on. This means we get to say that the definition of seeing Overwhelmingly Strong results is no longer dictated by the bias of the experimenter, but is aligned publically beforehand.
You will need
Let's look at an example below. In this hypothetical test I calculated that my runtime would be 34 days. I rounded that up to 35 because I always, always recommend rounding up to a whole week. From that I then know that if it's 5 weeks, that's 4 interim analysis points, plus the end runtime, for a total of 5 potential decision tims. With that I can calculate these O'Brien-Fleming numbers:
| Week | OBF confidence | No GST confidence |
|---|---|---|
| 1 | 99.95% | 95.00% |
| 2 | 98.57% | 95.00% |
| 3 | 97.29% | 95.00% |
| 4 | 96.25% | 95.00% |
| 5 | 95.45% | 95.00% |
Why O'BrienâFleming over others?
You can use the formulas below to calculate your adjusted confidence thresholds with OBF framework.
Just kidding haha don't do that.
I mean, you can if you want, but for real you don't need to do that. You open your LLM of choice and type:
"I'm running an AB test, I want to run it for [n] weeks, and I want to use alpha value of 5% aka 95% confidence level. How do I adjust my confidence levels if I want to have an interim analysis on every full week cycle, using OâBrienâFleming methodology to maintain my target alpha rate?"
Depending on what LLM you used and what kind of mood it's in, you'll see something similar to the table above.
One of the golden rules of AB testing is that we must not move the goalposts! The success criteria for an experiment should be decided before the experiment starts, and as much as possible, should be practically set in stone.
So if you want to stop your experiment early when you see big results, that's fine, but we do so using pre-defined criteria, which we document! Put it in the JIRA ticket, or the trello card, or write it on the post-it note, or whatever-the-heck you and your org are using to track A/B Test details, make sure this criteria is well defined there.
For a 5âweek experiment targeting the Minimum Detectable Effect - planned per MDE guidelinesâMonte Carlo simulations (n=100âŻ000) show:
In total, OBF enables ~30% of tests to finish early, saving time and resources without compromising statistical rigor.
OâBrienâFleming, when paired with a robust runtime-first approach, gives teams both precision and agility: plan for the effect size you care about, then peek responsibly each week without inflating false positives.