Principled Peeking: O’Brien–Fleming for Weekly Looks

Do you want to be able to stop your tests early when you see overwhelmingly strong results, without inflating your false positive rate? Great! Me too! Read on!

Preface: What is this about and who is it for?

If you are a Data Scientist, or if you are well versed on any form of Sequential Testing, this guide is probably not for you!

In that case you are probably looking for an advanced comparison of different Sequential Testing methodologies, and a writeup of different ways they could be implemented in large scale enterprise orgs with a strong Data Science resources, you are probably looking for one of these instead:

  1. Principled Peeking in A/B Testing - Professor Gabriela Cohen Freue, 2003
  2. Choosing a Sequential Testing Framework — Comparisons and Discussions (Spotify) - MĂ„rten Schultzberg (Staff Data Scientist) and Sebastian Ankargren (Sr. Data Scientist), 2023
  3. Sequential Testing at Booking.com - Dr Nils Skotara (Principal Data Scientist), 2023

This guide is for the rest of us, working in orgs where we aren't lucky enough to be surrounded by world class staticticians. This is for those of us who aren't looking to implement an industry leading vision of perfection, we are just looking this question in a way that we can implement tomorrow and explain to our stakeholders quickly and easily: How can we run more sensitive experiments, while remaining agile when we see strong results?

We will go through an approach that I lean towards, that balances a sensitive runtime (i.e. a longer runtime) against reasonable flexibility, with an emphasis on being able to slot it into your org with minimal effort.

🔗

This approach is the perfect complement to a careful MDE‑driven test duration strategy. First, pick a runtime long enough to detect your target effect size; then apply OBF peeking to stop early if results are overwhelmingly strong. Both of these together are what I call RLSE Framework where RLSE stands for "Run Long, Stop Early" (pronounced as release).

Problem Statement: What Are We Trying To Solve

As I have written about, I believe most companies doing AB testing would benefit from running their tests longer than they probably currently are. I recommend 5 week minimum as a reasonable heuristic.

When A/B testing teams set longer runtimes, they then need want to check in on experiments before they finish - but unadjusted "peeking" inflates your Type I error (false positives rate) beyond what you are happy with.

So basically, the longer you run:

My preferred simple solution is to use Group Sequential Designs with an O’Brien–Fleming (OBF) We will come back to that, but first...

What NOT to do: "peeking"

Most A/B Testing tooling allows a user to check on an experiment to see the results at any point during the runtime. While there is value to this (especially checking for issues early in the run cycle), there is also risk.

If you are willing to make a call on your test results any time your main metric goes above the confidence threshold, and you check it every day, you radically increase the likelihood of a False Positive (Type I error), because most tests will eventually go green ✅ at some point if you just check them often enough. This is called peeking.

Even If you set up and run A/A tests with no actual change to the user - if you check your AA Test every day, you won't have to wait too long before you see your main metric report as conclusively positive.

This is what we are trying to avoid. Just checking your experiments all the time and making a decision when they are statistically significant - BIG NO-NO! This protocol is nothing more than a type-1 error (false positive) generation machine!

Why is a Type 1 error (false positive) so toxic?

Contrary to what a lot of people who know me might think, I actually really am not concerned about the risk of a high false positive rate, in terms of rolling out red herrings. I used to be very concerned about that, but I'm not these days!

The risk of the "peeking" behaviour described above, and the inflated Flase Positive Rate, is actually - counter intuitively nothing to do with rolling out a change that was not actually improving anything.

I mean, yea, this is a risk and it will definitely happen, you will definitely end up rolling out false positives. But it's quite unimportant in the grand scheme of things. In fact, if your change was detected as a conclusively positive change, but in fact is was a False Positive, still it isstatistically highly unlikely unlikely that your change is negatively affecting users in reality. Even moreso: the chance that it is negativly affecting anything big enough to make a meaningful dent in your overall business is extremely unlikely (you would have detected it if that were the case).

The much bigger risk is that you then have a team who think they have struck solid gold 💾 and will spend the next 2 quarters iterating on it. They are chasing a ghost đŸ‘» and wasting their time.

Development time is typically the most precious resource in modern tech and product orgs. Wasting your dev team's time chasing a statistical ghost is far more damaging than the individual impact of rolling out a false positive. What better thing could your dev team have ben focussing on in that time, instead?

It's all about Opportunity Cost.

My solution: Longer initial runtime plus Group Sequential Testing

Remember the problems we are trying to solve. We want to:

The solution to these two problems mentioned above, and the balance between them, is almost certainly some form of Sequential Testing.

Though there are some complicated statistics involved in Sequential Testing, the concept is simple: When you see overwhelmingly strong results, you want to be able to stop your test early. We do this by deciding in advance how many times we will "peek", taking the 5% alpha, and spreading it across those peeks.

All we are doing with various calculation methods that go around this, with various different names, is we are trying to take this notion of Overwhelmingly Strong out of being just one person's opinion, and instead set predefined boundaries that we are all aligned on, to remove the human bias. There are many ways of doing that, each with various pros and cons. We will touch on various methods lightly (this is not a DS-heavy post) but first I will summarise my preferred version.

O'Brien–Fleming method: Higher Confidence Thresholds At First

The O'Brien-Fleming method allows us to define high confidence thresholds at which we would make a decision early-on. This means we get to say that the definition of seeing Overwhelmingly Strong results is no longer dictated by the bias of the experimenter, but is aligned publically beforehand.

You will need

Let's look at an example below. In this hypothetical test I calculated that my runtime would be 34 days. I rounded that up to 35 because I always, always recommend rounding up to a whole week. From that I then know that if it's 5 weeks, that's 4 interim analysis points, plus the end runtime, for a total of 5 potential decision tims. With that I can calculate these O'Brien-Fleming numbers:

Week OBF confidence No GST confidence
1 99.95% 95.00%
2 98.57% 95.00%
3 97.29% 95.00%
4 96.25% 95.00%
5 95.45% 95.00%

Why O'Brien–Fleming over others?

How to calculate for your own experiments

You can use the formulas below to calculate your adjusted confidence thresholds with OBF framework.

Just kidding haha don't do that.

I mean, you can if you want, but for real you don't need to do that. You open your LLM of choice and type:

"I'm running an AB test, I want to run it for [n] weeks, and I want to use alpha value of 5% aka 95% confidence level. How do I adjust my confidence levels if I want to have an interim analysis on every full week cycle, using O’Brien–Fleming methodology to maintain my target alpha rate?"

Depending on what LLM you used and what kind of mood it's in, you'll see something similar to the table above.

Here's an important part: DOCUMENT THIS.

One of the golden rules of AB testing is that we must not move the goalposts! The success criteria for an experiment should be decided before the experiment starts, and as much as possible, should be practically set in stone.

So if you want to stop your experiment early when you see big results, that's fine, but we do so using pre-defined criteria, which we document! Put it in the JIRA ticket, or the trello card, or write it on the post-it note, or whatever-the-heck you and your org are using to track A/B Test details, make sure this criteria is well defined there.

Expected Early Stopping

For a 5‑week experiment targeting the Minimum Detectable Effect - planned per MDE guidelines—Monte Carlo simulations (n=100 000) show:

In total, OBF enables ~30% of tests to finish early, saving time and resources without compromising statistical rigor.

How to Implement

  1. Plan for 5 weekly looks; record cumulative z‑values at each analysis.
  2. Compare against OBF thresholds: 3.51 → 2.99 → 2.61 → 2.32 → 1.96.
  3. Stop when observed z ≄ threshold; otherwise continue until final look.
  4. For background on selecting your test duration, see How To Set MDE article.

Conclusion

O’Brien–Fleming, when paired with a robust runtime-first approach, gives teams both precision and agility: plan for the effect size you care about, then peek responsibly each week without inflating false positives.

Sources

  1. Sequential Testing at Booking.com by Dr. Nils Skotara
    https://booking.ai/sequential-testing-at-booking-com-650954a569c7
  2. Choosing a Sequential Testing Framework — Comparisons and Discussions by by MĂ„rten Schultzberg and Sebastian Ankargren
    https://engineering.atspotify.com/2023/03/choosing-sequential-testing-framework-comparisons-and-discussions
  3. Flexible Stopping Boundaries When Changing Primary Endpoints after Unblinded Interim Analyses by Liddy M Chen, Joseph G Ibrahim, Haitao Chu
    https://pmc.ncbi.nlm.nih.gov/articles/PMC4024106/
  4. Principled Peeking in A/B Testing
    https://ubc-stat.github.io/stat301book/ch02_abPeeking.html
  5. An automated platform trial framework for A/B testing, by Wenru Zhou, Miranda Kroehl, Maxene Meier, Alexander Kaizer
    https://www.sciencedirect.com/science/article/pii/S2451865424001352
  6. Error Spending in Sequential Testing Explained, by Georgi Georgiev
    https://blog.analytics-toolkit.com/2020/error-spending-in-sequential-testing-explained/