Why Fake A/B Test Winners Keep Getting Promoted

Here is a question the statistics textbooks never ask: if peeking is so obviously destructive, why does every growth team still do it?

The answer isn’t ignorance. It’s incentives. The false winner and the honest null travel through your organization on two completely different reward tracks — and once you see the asymmetry, the persistence of bad testing stops being a mystery.

Follow the false winner’s journey. A test crosses 95 percent confidence on day four because someone checked it every morning. The dashboard goes green, the screenshot goes into Slack, and the change ships with a “data-driven” label attached. Three things happen instantly: the team looks decisive, the manager gets a slide for the quarterly review, and the person who called the win gets a story for their performance conversation. The reward arrives now, in public, denominated in status.

The cost arrives later, in private, denominated in nothing. When the lift evaporates in production — and at 20 percent power, significant results overstate the true effect by about 130 percent on average, per Kohavi and colleagues’ KDD 2026 replication study — no ceremony marks the failure. Nobody re-audits a celebrated decision. The quarterly review already happened. The promotion story already landed. The feedback loop that should punish the false win simply doesn’t close.

Now follow the honest null. The team pre-registers the design, runs to the full sample size, and reports the statistically correct answer: no detectable effect. What do they get? An awkward silence. No celebration, no slide, no story. Worse, the null invites suspicion — did you design it wrong, was the traffic too small, should we re-slice by segment? The honest answer is socially expensive in the exact moment it is delivered, while its value — a decision not made, a rollback avoided — is invisible by construction. You cannot screenshot a disaster that didn’t happen.

This is a textbook incentive asymmetry: immediate, visible, personal rewards for noise; delayed, invisible, diffuse rewards for rigor. And it explains the data. Booking.com’s 2018 simulation showed 531 of 1,000 identical A/A tests flashing a 95 percent “winner” under peeking — the statistical raw material is abundant. The KDD 2026 team found only 18 percent of publicly shared experiments in the GoodUI repository are powered to detect even a 2 percent effect — the organizational filter selects for exactly the tests most likely to produce celebratable noise.

So the fix was never just “teach people statistics.” It’s changing what gets rewarded. Pre-registration works because it moves the decision to a moment before anyone has a favorite arm. Crediting nulls as saved rollbacks — not failed ideas — makes the invisible value visible. And auditing last quarter’s shipped “winners” against production data closes the loop that the incentive system leaves open.

The teams that test well aren’t the ones that know the math best. They’re the ones that made honesty cheaper than a fake win.