Why Most A/B Tests Fail (and Why the Best Teams Want Them To)

Every growth team eventually confronts the same uncomfortable number: the failure rate. Run your first A/B test and the odds are roughly one in seven that it wins. Run a hundred and expect about fourteen winners, if you are running them blind. Most teams read that as a verdict — on their judgment, their skills, their product. It isn’t. It’s the baseline physics of the industry, and the teams that treat it that way are the ones that compound.

Three numbers explain why.

The first is the null rate. In a 2022 Management Science paper, Wharton researchers Berman and Van den Bulte analyzed nearly 5,000 tested effects and found that about 70 percent of them were true nulls — the idea genuinely did nothing, no matter how well it was executed. This is not a talent problem. It’s a property of the idea space: most untested intuitions are simply wrong, and no amount of cleverness changes that distribution.

The second is the base win rate. Bing, one of the most mature experimentation organizations anywhere, has seen its idea success rate sit around 10 to 20 percent for years, per Kohavi and colleagues’ KDD 2014 analysis. Booking.com told QCon London in 2026 that fewer than 25 percent of its roughly 150,000 experiments succeeded. Industry platforms VWO and Convert both land near one winner in seven. The remarkable thing is not that these numbers are low — it’s that they are stable, across companies, decades and product categories.

The third is the learning rate, and this is where the arithmetic flips. A sporadic team re-rolls the same dice forever: every test inherits the same roughly 14 percent prior, because nothing from the last test survives into the next. A cultural team changes the prior. Each logged hypothesis, each documented null, each validated mechanism becomes part of the knowledge the next test starts from. Move the hit rate from one-in-seven to one-in-five — a modest shift, achievable with a decent learning library — and the expected winners in a 100-experiment program go from ~14 to ~20. To ~25 at one-in-four. The gains are not linear; they compound, because the prior keeps rising.

The fourth layer is depreciation. Tactics decay — a discount gets copied, a format gets saturated, an algorithm absorbs the trick. Knowledge doesn’t. A learning library is the rare growth asset that doesn’t depreciate with time, doesn’t get copied by competitors, and doesn’t leave when the growth lead leaves. Expedia has run thousands of tests a year on a stated expectation that one-third fail, one-third do nothing, one-third win. That’s not pessimism. It’s the only honest budget for a system whose output is learning.

So the win rate was never the metric. The learning rate was. A team that treats a null result as tuition, logs it, and lets it raise the odds of the next test is running the only game in town where the losses are the fuel. That’s the math behind the moat — and it’s why the most dangerous thing a growth team can do is keep its failures quiet.