This concept pairs perfectly with any kind of Sequential Testing, such as my preferred method Principled Peeking with O’Brien-Fleming corrections to stop early when results are overwhelmingly strong.
One of the most common pitfalls in A/B testing is setting your Minimum Detectable Effect (MDE) too high - especially when your primary metric is conversion rate, which rarely moves by more than 1-2%. While a high MDE may let you run shorter tests, it also means your test will completely miss most real-world effects.
You can put any number you want into a runtime calculator. It will happily give you the answer you’re looking for. But that doesn’t mean your assumption is realistic. If you plug in an MDE of 5%, your test might only need a week - but almost no real conversion uplift is that large.
To better understand this trade-off, try the Inverse Runtime/MDE Calculator. Use it to see how long you'd need to run your test to detect a 1-2% conversion uplift. You'll quickly realise a 2-week test isn’t long enough to detect anything realistic for most businesses.
Conversion is typically the hardest metric to shift. Anything above 5% relative uplift on conversion is almost unheard of in routine tests - and even 3% is rare. Seeing a genuine 5% conversion uplift, on a properly powered experiment, is like the CRO version of a poker player seeing a Royal Flush: you might see it once or twice in your career if you're lucky. Many will never see it.
Consider that there's really only a short list of changes that can truly boost conversion by ≥2% in one go:
If your test doesn’t involve one of these, plan for an MDE of ≤2% for conversion, otherwise your experiment is doomed to not detect anything, before you even start it.
A lot of people just don't think a lower uplift, such as 0.75% uplift (just for example) is something they would care about being able to detect. It just doesn't seem big enough to be exciting.
The really easy way to change your mind about that is to work out what that extra 0.75% conversion means in hard revenue when extrapolate for a year. If we take some assumptions and hypothesise a moderately successful ecom site:
Base assumptions:
Annual revenue (baseline):
2,000 × €50 × 365 = €36,500,000
With 0.75% relative uplift:
€36,500,000 × 0.0075 = €273,750 additional revenue per year
Now imagine your CRO is standing in front of you and they're telling you this Experiment will earn you €273,750 over the next year, but only if you're willing to extend your runtime from 1 to 5 weeks. Seems pretty appealing now doesn't it? The heuristic is simple: If you would celebrate it, then you would want to detect it.
The answer of course is simple: You need to run your Experiments for longer than you probably are now[1].
Probably longer than you think! Rather than guessing how long to run based on whatever MDE gives you a nice, tidy runtime, I recommend flipping the script: fix the runtime first, then calculate the MDE that runtime supports.
Want to see for yourself? Use the MDE Calculator to graph how MDE changes with runtime. Look up what your MDE would be for a 2-week test at your traffic level - you’ll probably be surprised how insensitive it is.
The question is not really How Long Should You Run, it's more like How Long Are You Willing To Run - because the former question has no answer other than "longer is better", but the latter involves considering a few things limiting factors, chiefly:
In my experience it tends to be #3 that factors in more often than the others: You are likely to have other teams tapping you on the shoulder telling you they are waiting patiently for your experiment to finish so they can start theirs on the same component. That's my experience but this will vary a lot depending on where you work!
My default recommendation is a 5-week test for most experiments. Here’s why:
Plenty of teams feel like five weeks is a long time. The experimentation industry has normalised 1-2 week tests, but in reality very few businesses have the traffic to detect realistic MDEs in that time.
Most of the advice on this page is about using conversion as your primary metric, which is notoriously hard to move, but is also usually the metric we care about the most. Upper-funnel metrics - such as Banner Clicks, Product Page Views, or Add To Cart - are typically much easier to shift, but the trade-off is that they’re further removed from actual business value, and are weaker proxies for actual revenue impact.
That said, Add to Cart is often a great compromise. It’s high enough in the funnel to be more sensitive, but low enough to still carry meaningful intent.
The approach I recommend to choosing a metric is to calculate the MDE for Conversion and also a more elastic KPI such as Add To Cart, and when you se the potential MDEs for both, it will often be cleaer that Add To Cart is far more realistic.
When testing on a non-Conversion Primary KPI, as an extra safeguard, you could run a one-tailed non-inferiority test on Conversion concurrently, ensuring you don’t harm your bottom-line metric. This would be good practice and very by-the-book, in practice, however, it is not uually a concern (assuming that we've chosen a metric which is at least mostly a good proxy for Conversion).
The biggest concern with long tests is speed. But you can still be agile with a 5-week plan by checking results weekly using O’Brien-Fleming corrections. This gives you a principled way to stop early when effects are truly overwhelming - often saving 1-2 weeks of runtime while maintaining strong statistical guarantees.
Many would argue that for such a complex topic, there is no one size fits all - to be honest I argue it myself pretty often too! So is it really valuable to suggest a heuristic that can be applied globally?
Let's wrap up this article by acknowledging the obvious: If you are running an advanced Exprimentation program, then it is unlikely that any off-the-shelf mass-market advice is going to be perfect for you. Once your testing program is sufficiently advanced you'll want to figure out the setup that's right for you, in your business, solving the problems you are facing, workign in the or you find yourslf in, not just some average approach recommended by a stranger. This guide is not for you, it is for those at beginner or intermediate level looking for some guidance to get them running decent setups quickly.