David's Supafly Really Good But Also Quite Basic And Just Prototype AB Test MDE Calculator

How To Use

How to set MDE is one of the hottest topics in Online Experimentation. I have written about this a lot! MDE determines how long you need to run your experiment. We all want to be able to detect smaller changes but it's not always easy to know how long we would need to run for that.

My recommendation is always the same: don't start with what effect you want to detect! Do it the other way round, start instead with how long you are willing to run (longer is better) and calculate what effect you could detect in that timeframe.

Willing to run 4 weeks? Then put in your traffic and covnersion rate and it will tell you what you could detect in that time. Easy!

Almost always the impact of this tool is that people realise they need to run their experiments longer than they currently are. Have you considered running for 5 weeks, with some version of Group Sequential Testing to allow you to stop wildly successful tests earlier? I think you should!


Copied!