The RLSE Framework

The RLSE Framework stands for Run Long, Stop Early and is how I recommend most orgs doing AB Testing set up their tests for greater success.

🔗️

This article can be considered an overview and combined summary of two topics I talk about in much more detail here:

Most Experiments Are Underpowered

Unless you're working at one of the big global experimentation giants like Facebook, Bing, or booking.com, chances are you don't have nearly as much traffic as you wish you did, to be able to get a clear signal on most changes.

Most of us would happily celebrate a 1% uplift on our revenue, but not that many of us are privileged to work at a place that has enough traffic to detect that kind of change.

How long are you running your AB Tests?

Most experiments at most places tend to run around 1-3 weeks, sometimes peeking towards 4.

In this kind of runtime, especially in the 1-3 range, most websites just do not have enough power to detect most changes. When you CRO program outputs primarily inconclusive results, it is not because your team's ideas aren't good enough, it is because the impacts (positive or negative) are likely below the threshold you can detect,

The clearest advice I propose is that most people, most of the time, need to get used to the idea of running their experiments for longer. Quite simply: The longer you run, the more sensitive your test is. Running longer is better, in most ways. To a point...

How Long Is Long Enough?

Probably longer than you think!

📉

Want to see for yourself? Use the MDE Calculator to graph how MDE changes with runtime. Look up what your MDE would be for a 2-week test at your traffic level - you’ll probably be surprised how insensitive it is.

The question is not really How Long Should You Run, it's more like How Long Are You Willing To Run - because the former question has no answer other than "longer is better", but the latter involves considering a few things limiting factors, chiefly:

  1. Agility: How many times do you want to do the plan➡️test➡️learn➡️repeat cycle in a quarter? The faster we learn the faster we make decisions, so there is a tradeoff to consider here between running longer to have more sensitive experiments, vs learning slower. It's a question of speed vs accuracy.
  2. Momentum: Humans are not robots, we get excited about topics, and when a team are passionnate about a topic, we want to vcapitalise on that enthusiasm and not slow it down.
  3. Blocking: Is another Experiment waiting to run on the same area, either from you or from another team/org? If these cannot be run together and your test is holding up other developments, this is a considering.

In my experience it tends to be #3 that factors in more often than the others: You are likely to have other teams tapping you on the shoulder telling you they are waiting patiently for your experiment to finish so they can start theirs on the same component. That's my experience but this will vary a lot depending on where you work!

Stop Early

If you want to run tests that are more sensitive to detect smaller changes, but avoid (or minmise as much as possible) the downsides mentioned above, the solution is almost certainly some form of Sequential Testing.

So, what is it?

Many experimenters have seen extremely strong results early into a run of an experiment and been temted to stop, but hopefully we have all been trained this this is bad as introducing multiple decision points significantly increases Type-1 error rate.

The "proper" way to do this is to use an alpha spending function such as O'Brien-Fleming Group Sequential Testing, which is what I like to recommend. What this does is give you a higher confidence threshold to overcome in the early stages of your experiment, so that you can pre-define and align on what would constitute results strong enough to stop early.

By doing this in advance and transparently we are able to "peek" while keeping our Type-1 error rate at the desired 5% (or whatever you choose to use, but most stick to 5%)

How To Implement

My advice is based on the notion that you have decided to run for a set number of full weeks.

How long will you run?

Number of Weeks Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7 Combined Type-1 error
4 99.99% 99.45% 98.19% 97.36% 5%
5 100.00% 99.81% 99.05% 98.30% 97.84% 5%
6 100.00% 99.93% 99.51% 98.92% 98.46% 98.18% 5%
7 100.00% 99.98% 99.75% 99.32% 98.91% 98.61% 98.43% 5%

If you want to run shorter than 4 weeks or more than 7 - well, that's between you and whatever God you pray to - I would not usually recommend either as a general practice!

Summary

🔗️

Remember, this is a shortened overview and combined summary. Much more detail here:

Run Long, Stop Early

  1. Run longer experiments for more sensitivity, and you will have more successes
  2. Pre-define early stop criteria so you can still move fast when you see strong results