AB Testing in New Product Development

I've been doing AB Testing since 2009, including having built my own AB Testing Tool back before mass market WYSIWYG tools existed, having spent years implementing and managed global AB Testing programs in huge multinational corporations, and having run hundreds (thousands?) of AB Tests myself.

One of the things I hear a lot is about how you "can't AB test from 0 to 1" or essentially that "AB testing is for tweaks and optimising, not for brand new features/new product development"

By now I'm pretty sure this statement is absolutely false. Let's pick apart the most common arguments people have...

Common Myths in AB Testing

1. "You can't AB test a new feature because there's nothing to compare against"

People love to call this "Building from 0 to 1" and they love to say you can't test that.

Not only is it possible to AB test from 0 to 1, but that's what every AB test is. Base has 0 changes. Variant has 1 change.

The idea that a bigger change shouldn't be tested, because you somehow already know it will work, but a smaller change should be checked, is completely inverted.

The bigger the change is the more unknowns there are and the more risks.

2. "We don't need to AB Test this because user research proved it already"

False! User research is great and I am a huge believer in qualitative data but User Research only tells you what people think, it doesn't tell you how they will act.

User Research tells you there's a problem, AB Testing can tell you if your solution has solved that problem.

3. "We dont have enough power to experiment"

False! ... mostly!

If you don't have even 10k traffic per day, you shouldn't even think about AB testing. But for those of us above that range: Power is a function of audience size and effect size (and other things but let's KISS for now). There is no such thing as "not enough power" there is only "we didn't properly balance the audience size against the size of the change"

4. "We can't AB Test a new feature because we can't code the entire feature into JQuery in Optimizely/VWO/Maxymiser/etc"

False! you don't need to code your entire feature in one of these third party AB Testing tools. The solution here is quite simple actually, you code the feature into your normal codebase, and you just use the AB Testing Tool to show/hide the entry point.

Actually this is potentially a better way to build bigger AB Tests as then more of your product will be server-side rendered which is good for SEO, and good for client side performance.

5. "You can't AB test your way into a strategy"

Aha! This one is not a myth at all this one is TRUE.

AB Testing tell you how users respond to your product change, but no amount of that is going to shape your direction for the next decade. AB Testing is in line with strategy, not instead of it.

So - just a few thoughts - a bit different from anything I've posted on LinkedIn before. Hope this is relevant and maybe helps somebody.