Growth Is a Learning Engine, Not a Bag of Tricks

Most teams I meet genuinely believe they are already doing this. They run an A/B on a headline here, a landing-page tweak there, maybe a weekly “what if we…” session that produces a pile of half-finished ideas. Then they call it growth experimentation. It isn’t. It’s random optimization — and the difference matters more than most people want to admit.

Think about what actually happens after a sporadic test wins. The team celebrates, ships the change, and moves on. The insight — why it won, what it said about the customer, which hypothesis it confirmed — rarely gets written down anywhere searchable. Three months later someone runs the same test with slightly different wording and gets the same result, again with no learning attached. That’s not a pipeline of insight; it’s a slot machine with extra steps.

The data on how rarely ideas work is humbling. In a peer-reviewed 2020 paper in the journal Trials, Ron Kohavi and colleagues — who built experimentation systems at Microsoft, Google and LinkedIn — reported that even the best-run platforms operate hundreds of concurrent experiments on millions of users each, and that “most ideas fail.” At Bing, which has been experimenting for well over a decade, only about 10 to 20 percent of ideas improve the metric they were designed to improve, and true breakthroughs happen roughly once in every 500 tests, per the 2014 KDD paper by Kohavi, Deng, Longbotham and Xu. Booking.com, one of the most mature testing organizations on earth, told QCon London in 2026 that across roughly 150,000 experiments, fewer than 25 percent succeeded (self-reported figures from the company). Wharton researchers Berman and Van den Bulte, analyzing nearly 5,000 tested effects in a 2022 Management Science paper, found about 70 percent were true nulls — the idea genuinely did nothing.

None of this means testing is broken. It means the win rate is the wrong KPI, and “test more” is the wrong strategy. The right strategy is to make the failures teach.

Why Sporadic Tests Don’t Compound

The One-Off Win That Gets Thrown Away

A sporadic test produces a one-off outcome, not an asset. The result is a step up or a step sideways, and then the curve flatlines, because nothing was retained that could raise the odds of the next test. HBS professor Stefan Thomke, in his book Experimentation Works and the interviews that followed it, puts the number bluntly: for every experiment that succeeds, nearly ten don’t. A team that treats each loss as a dead end is discarding roughly 90 percent of the information it just paid for.

The Growth-Hack Mirage

The growth-hack mindset chases the silver bullet — one trick that spikes a metric. It’s romantic, and it’s wrong in a specific way: hacks decay. A discount mechanic gets copied, a viral format gets saturated, an algorithm change gets absorbed by the platform. The learning doesn’t decay if it’s captured; the trick does. Booking.com’s own engineering blog, in a 2026 post on scaling experimentation quality, makes the same point from the inside: the company runs roughly 1,000 experiments in parallel at any given moment, and its early quality dashboards showed about 80 percent of experiments lacked a power calculation — volume without rigor was producing noise, not knowledge.

The Knowledge-Waste Problem

Here’s the uncomfortable part: most teams throw away the only thing that actually compounds. When insight isn’t logged, shared and consulted, the team re-tests what it already tested and re-makes the mistakes it already made. At a 2026 experimentation conference hosted by Booking.com, a data scientist from Delivery Hero described building a searchable experiment knowledge base and summed up the whole thesis in one sentence: “Institutional knowledge doesn’t preserve itself. Structuring your experiment records for retrieval, not just storage, is what lets past learnings compound.” That sentence is worth more than most growth roadmaps.

What a Testing Culture Actually Is

A testing culture isn’t “more testing.” It’s a system with five parts, and the parts have to be connected.

High-Velocity Throughput

First, velocity. A compounding culture runs a portfolio of many small, parallel tests on a steady cadence, not one-at-a-time hero tests. The numbers at the top end are staggering: Microsoft’s internal ExP platform runs over 12,000 experiments a year, and the Cambridge University Press book by Kohavi, Tang and Xu (2020) reports that Google, LinkedIn and Microsoft each run tens of thousands of controlled experiments annually. Velocity is the learning rate — the more cycles you complete, the faster your knowledge grows. At the 2026 Booking.com conference, 150 practitioners from 49 companies reported running 56,000 experiments a year between them.

A Shared Learning Library

Second, a learning library. Every experiment logs its hypothesis, its result and its insight into a searchable knowledge base — the Delivery Hero model. The library is what makes the next test smarter. Booking.com institutionalized this with “Experiment Bashes,” where teams review past experiments and answer three questions: what did we expect, what did we learn, and how valid are the results?

Hypothesis-First Rigor

Third, hypothesis-first discipline. Every test starts with a falsifiable claim — if we change X, then Y will happen, because Z. The hypothesis is the unit of learning, not the tweak. This is exactly the discipline Kohavi and Thomke wrote about in the Harvard Business Review (2017): a small headline change at Bing, initially dismissed and shelved for months, turned out to lift revenue by 12 percent — worth more than $100 million a year in the US alone, the best revenue idea in Bing’s history. Rigor is what separates that story from a lucky screenshot.

Honest Loser Reporting

Fourth, honesty. A culture that reports losers as learning, and never punishes them, learns at full speed. Expedia’s leadership has said it outright: roughly one-third of ideas fail, one-third are neutral and one-third win — and the company runs thousands of tests a year across product, engineering and marketing on exactly that expectation. When a null result is treated as information rather than failure, the confirmation bias that quietly corrupts most testing programs loses its grip.

The Learn → Apply → Re-Test Loop

Fifth, the loop itself. Insights change the next test, which produces new insights. That closed loop is the compounding engine — and it’s also the part most teams never close, because closing it requires the library, the rigor and the honesty to all exist at once.

Why Culture Compounds and Tactics Don’t

The Arithmetic of the Hit Rate

Here’s the mechanism, and it’s genuinely arithmetic. A sporadic team re-rolls the dice on every test: each experiment starts with the same roughly one-in-seven chance of winning, which is where the industry platforms’ own analyses (VWO and Convert) consistently land. A cultural team raises its hit rate over time because it knows what’s been tried, what worked, what didn’t and why. Even a modest improvement — from one-in-seven to one-in-five — changes the economics of a 100-test program dramatically. Culture compounds the odds.

The Counterfactual Is the Hidden Cost

Not experimenting has a measurable price, and it’s almost never on anyone’s P&L. In the KDD 2014 rules-of-thumb paper, Kohavi and colleagues shared Greg Linden’s famous Amazon finding: a 100-millisecond slowdown decreased sales by about 1 percent. And the Cambridge University Press book documents that at Bing in 2017, every tenth of a second of speed improvement was worth roughly $18 million a year in incremental revenue. Small changes, captured by testing, are worth real money. The team that doesn’t test is paying the counterfactual tax silently.

The Institutional Memory Moat

There’s a second, less obvious moat: retention. Knowledge that lives in people’s heads leaves when they do. A learning library is an asset that survives turnover. This is what separates a growth organization from a growth individual — the company keeps learning even when the growth team changes. McKinsey’s 2023 global survey (n=1,086) found organizations with strong innovation cultures were more than twice as likely to complete digital transformations without stalling (45 percent vs 21 percent). Culture, not talent or tools, was the differentiator.

The Strategic Difference: Make Tests Teach Each Other

So the reframe: growth isn’t “do more tests.” It’s “make the tests teach each other.” That’s the whole game. The team that compounds its learning will outgrow the team that just runs more tests, because learning compounds and testing alone doesn’t. This is the same logic Reforge’s growth-loops framework applied to user acquisition: loops feed their output back into their input and compound, while funnels run one-directional and decay. The same principle applies to insight.

Building the Compounding Engine: What Changes for Your Team

Instrument for Learning, Not Just Winning

Change the output you optimize for. An experiment’s real output is the insight — what did we learn about the customer? — not just the lift. Log both, celebrate both, and make “learning per test” an explicit part of how the team is evaluated.

Build the Learning Library as a Product

Treat the knowledge base like a product: searchable, updated, and actually consulted before every new test. If a new experiment doesn’t start with a five-minute search of the library, the library is decoration, not infrastructure. Delivery Hero’s pattern — chunking experiment documents into hypothesis, decision and rationale, then making them searchable — is the one worth copying.

Set a Velocity Goal, Not Just a Win Goal

Measure experiment throughput and insights-per-week, not just wins. The capacity to learn is the growth metric. Booking.com’s quality programs show what this looks like in practice: they track power calculations, runtime adherence and decision alignment — quality metrics about the learning, not about the winning.

Democratize the Testing, De-Risk the Failure

Anyone should be able to propose a hypothesis, and everyone must be safe reporting a loser. Google’s Project Aristotle, the internal study of 180+ teams, found psychological safety was the single strongest predictor of team effectiveness. A testing culture without psychological safety is a testing culture that lies to itself — winners get cherry-picked and losers get buried, and the compounding stops.

Feed the Winners and the Insights Back In

Every result changes the next test. Ship the winners, yes — but also change the roadmap based on the nulls. The closing of the loop is the compounding.

The Honest Caveats

Activity ≠ Learning

A team can run lots of tests and learn nothing. Tests without hypotheses, without a library, without reflection: velocity without learning is just fast busywork. In CleverTap’s 2024 vendor survey, 81.9 percent of product managers admitted they rarely experiment before making changes — the gap isn’t volume, it’s the system around the volume.

The Confirmation-Bias Risk

A testing culture can also fool itself. Cherry-picked wins, weak statistics, “we tested it because we believe it” — these compound their own bias. The Wharton paper found roughly one in five “significant” results at the 5 percent level is actually a false positive. Honest testing isn’t a nice-to-have; it’s the non-negotiable that keeps the whole engine honest.

Testing Refines Strategy; It Doesn’t Replace It

A testing culture optimizes within a strategy. It doesn’t substitute for positioning, brand and the big bet. Booking.com’s QCon talk made the same point in passing: after 20 years and 150,000 experiments, the company’s biggest wins still came from strategic bets, not button colors. Culture refines; strategy directs.

Compounding Is Slow — and That’s Why Most Teams Quit

The learning library compounds slowly, and impatient teams abandon it for the tactic exactly when it would start paying off. Mastercard’s 2024 State of Business Experimentation survey of 100+ enterprises using its Test & Learn platform (self-reported) found that in retail, up to 46 percent of ideas never break even or prove their initial hypothesis — a “failure rate” that would sink a quarterly-metric team but is just the cost of admission for a compounding one. If you can’t tolerate the losing streak, you can’t have the compounding.

The Takeaway

Growth isn’t a hack — it’s a testing culture. The team that runs sporadic tactics gets isolated wins that decay, and re-rolls the dice every time. The team that builds a systematic, learning-driven experimentation engine — high velocity, a shared library, hypothesis-first rigor, honest losers, a closed loop — gets compounding returns, because the insight, not the tweak, is the asset that appreciates. The brands that compound their learning, and retain it against turnover, will outgrow the brands that just test more. Learning compounds. Testing alone doesn’t.