The Incentive Loop: Why Nobody Is Paid to Find Out If Synthetic Research Works

Four percent of insights buyers use synthetic respondents as primary decision evidence. Twenty-three percent have piloted them. Seventy-seven percent of practitioners name bias as a concern. Forty percent name data quality as the top barrier to generative AI in research.

And the category keeps growing.

That combination usually gets attributed to hype, or to buyers not understanding the technology, or to vendors overselling. All three are partly true. None of them is the mechanism. If the industry were simply confused, better education would have fixed it by now.

Something more ordinary is happening. Demand for synthetic panels is not driven by belief in synthetic panels. It is driven by a purchasing system that rewards a different quality than accuracy.

The metric on the brief is not accuracy

Read a research brief as a procurement document rather than a scientific one, and the whole thing comes into focus.

The brief does not ask how accurate the answer will be. It asks how fast, and how much, and whether the study can be in the room before the planning cycle locks. Those are the columns with numbers in them. Accuracy appears nowhere, because it has no delivery date and no invoice line.

Under that constraint, a synthetic panel is not a compromise. It is the correct answer. It returns a thousand rows and a clean topline by Thursday, exactly what the brief requested. The person who commissioned it is not naive; they are responding to the only criteria they were given.

This is the first turn of the loop, and it contains no bad faith.

The chart does its job before anyone asks what it measured

The second turn is subtler.

A synthetic study rarely enters an organization to answer a question. It enters to settle an argument. There is a stalled decision, two camps, and a meeting on the calendar. The deck arrives, presents a clean directional read, and the decision moves.

To everyone in that room, the research worked. It did the job research is hired to do inside a company: convert a deadlock into a decision with a defensible-looking artifact attached.

Nobody asks what instrument produced the number, because the number did what it was needed for.

Replication is nobody’s job

Then the loop closes, and this is what makes it stable.

For synthetic research to be falsified, someone has to fund a comparison. Someone has to take a completed study, wait for the outcome, and check whether the panel called it correctly. That work needs budget, takes months, and most likely produces a finding that reads “our previous decision was shakier than we thought.”

There is no sponsor for that. The commissioner has no incentive to fund the audit. The vendor has no incentive to publish a divergence rate. The agency has no incentive to reopen a recommendation it already made.

So the audit never happens, the divergence is never measured, and the next brief arrives with the same deadline.

What would actually break the loop

Not better models. Not clearer vendor marketing. Not another article about mode collapse.

Two things would break it, and both are boring.

The first is a line on the brief. If commissioning a study required stating what it may not be used for, the instrument’s status would become explicit at the point of purchase, not in an appendix nobody reads.

The second is one funded backtest per category, per year. Not a methodology paper — a number, specific to your audience, your stimuli, your price points. Once a team knows its own divergence, the loop loses its hiding place.

Until one of those exists, the honest summary of this market is not that buyers believe synthetic panels work. It is that nobody is currently paid to find out whether they do.