The RLSE Framework stands for Run Long, Stop Early and is how I recommend most orgs doing AB Testing set up their tests for greater success.
This article can be considered an overview and combined summary of two topics I talk about in much more detail here:
Unless you're working at one of the big global experimentation giants like Facebook, Bing, or booking.com, chances are you don't have nearly as much traffic as you wish you did, to be able to get a clear signal on most changes.
Most of us would happily celebrate a 1% uplift on our revenue, but not that many of us are privileged to work at a place that has enough traffic to detect that kind of change.
Most experiments at most places tend to run around 1-3 weeks, sometimes peeking towards 4.
In this kind of runtime, especially in the 1-3 range, most websites just do not have enough power to detect most changes. When you CRO program outputs primarily inconclusive results, it is not because your team's ideas aren't good enough, it is because the impacts (positive or negative) are likely below the threshold you can detect,
The clearest advice I propose is that most people, most of the time, need to get used to the idea of running their experiments for longer. Quite simply: The longer you run, the more sensitive your test is. Running longer is better, in most ways. To a point...
Probably longer than you think!
Want to see for yourself? Use the MDE Calculator to graph how MDE changes with runtime. Look up what your MDE would be for a 2-week test at your traffic level - you’ll probably be surprised how insensitive it is.
The question is not really How Long Should You Run, it's more like How Long Are You Willing To Run - because the former question has no answer other than "longer is better", but the latter involves considering a few things limiting factors, chiefly:
In my experience it tends to be #3 that factors in more often than the others: You are likely to have other teams tapping you on the shoulder telling you they are waiting patiently for your experiment to finish so they can start theirs on the same component. That's my experience but this will vary a lot depending on where you work!
If you want to run tests that are more sensitive to detect smaller changes, but avoid (or minmise as much as possible) the downsides mentioned above, the solution is almost certainly some form of Sequential Testing.
Many experimenters have seen extremely strong results early into a run of an experiment and been temted to stop, but hopefully we have all been trained this this is bad as introducing multiple decision points significantly increases Type-1 error rate.
The "proper" way to do this is to use an alpha spending function such as O'Brien-Fleming Group Sequential Testing, which is what I like to recommend. What this does is give you a higher confidence threshold to overcome in the early stages of your experiment, so that you can pre-define and align on what would constitute results strong enough to stop early.
By doing this in advance and transparently we are able to "peek" while keeping our Type-1 error rate at the desired 5% (or whatever you choose to use, but most stick to 5%)
My advice is based on the notion that you have decided to run for a set number of full weeks.
| Number of Weeks | Week 1 | Week 2 | Week 3 | Week 4 | Week 5 | Week 6 | Week 7 | Combined Type-1 error |
|---|---|---|---|---|---|---|---|---|
| 4 | 99.99% | 99.45% | 98.19% | 97.36% | 5% | |||
| 5 | 100.00% | 99.81% | 99.05% | 98.30% | 97.84% | 5% | ||
| 6 | 100.00% | 99.93% | 99.51% | 98.92% | 98.46% | 98.18% | 5% | |
| 7 | 100.00% | 99.98% | 99.75% | 99.32% | 98.91% | 98.61% | 98.43% | 5% |
If you want to run shorter than 4 weeks or more than 7 - well, that's between you and whatever God you pray to - I would not usually recommend either as a general practice!
Remember, this is a shortened overview and combined summary. Much more detail here:
Run Long, Stop Early