You Can’t Survey a Model About Its Own Blind Spots

A focus group costs somewhere between seven and twenty thousand dollars and takes three to six weeks to organize. You recruit eight people, put them in a room, watch one of them dominate, watch two of them agree with everything, and read the transcript two weeks after the decision was already made.

A synthetic panel costs a few hundred dollars and returns a dataset before lunch. Same brief. Same questions. Ten thousand respondents instead of eight.

If you have ever signed off on a research budget, that comparison does something to your chest. It should. The economics are not a marketing fiction — they are the single most honest thing about this category.

The problem is that the two instruments do not measure the same thing, and the industry has spent two years arguing about the price while quietly hoping nobody asks what the cheaper instrument is actually measuring.

A real consumer tells you what she thinks. A synthetic respondent tells you what a language model predicts a consumer like her would think. That sentence reads like a technicality. It is the entire argument.

What a synthetic panel actually is

Worth being precise, because the term now covers four different products sold under one word.

At the safe end sits augmentation: filling gaps in a real sample, boosting an under-quota cell, imputing a missing answer. This is the mature corner of the market, and it performs reasonably well. Next comes fully synthetic data, built from scratch with no primary data collection at all. Then synthetic personas — digital stand-ins for a segment. And finally digital twins, built from a single real person’s interview.

The ICC/ESOMAR Code, revised in June 2025, drew this line explicitly and defined synthetic data for the first time: artificially generated data intended to replace what would normally be gathered from people. Notably, ESOMAR’s framework covers augmentation but does not yet endorse the fully synthetic or persona-based versions. That asymmetry is a signal, not a footnote.

The practical consequence: you can build a “panel” in an afternoon with an API key and a persona spreadsheet, or you can build one calibrated against a decade of your own transaction data. Both get called synthetic. They perform nothing alike. Most public arguments about this technology are two people comparing different rungs of the same ladder and never noticing.

Two studies define the boundary

Before you spend a dollar, know both ends of the evidence.

The optimistic end. Leo Yeykelis and co-authors from Stanford and Boston University generated 19,447 AI personas to replicate 133 experimental findings drawn from 45 studies published in the Journal of Marketing. Identical stimuli, identical measures, identical sample specifications, identical statistical tests. The personas reproduced 76% of the original main effects. Across all 133 findings, including interaction effects, the replication rate landed at 68%.

Those numbers are not trivial. When 100 psychology studies were re-run with real humans in 2015, 39% replicated.

The pessimistic end. James Bisbee and four co-authors, writing in Political Analysis, prompted ChatGPT to adopt personas matching actual respondents in the American National Election Study, and then compared the output. The averages lined up beautifully. Almost nothing else did. Response variance came out at roughly half the human level. Nearly half the regression coefficients diverged significantly, and about a third came out with the wrong sign. Then they ran the same prompt three months later and got a materially different distribution — not because of anything they changed, but because the model underneath had moved.

So the averages can be right while the structure is wrong. Hold both findings at once and you have an accurate map of where this technology actually stands.

Where the numbers hold up

Strip away the vendor decks and a fairly consistent pattern appears across independent studies.

Synthetic panels are good at direction. If a concept clearly beats another, the model usually sees it. In the Yeykelis work, agreement rose to 88% when the effect size was medium or large — precisely the cases where you did not need expensive research to tell you which option was stronger.

They are good at scale and at breadth. Twenty thousand personas ranked across forty concepts is something no human panel can do, and the constraint that normally forces you to test three concepts instead of thirty simply disappears. That is a genuine change in what research can attempt.

They are good at pre-testing stimuli: packaging, headlines, subject lines, ad frames. Harvard Business School’s Ayelet Israeli and co-authors found that LLM-generated synthetic customers produced realistic willingness-to-pay estimates for familiar product attributes, and that teams could use them as a first filter across dozens of ideas before fielding anything.

And they are good at the thing nobody puts in the case study: killing bad ideas cheaply. If a concept fails against a synthetic panel, you have lost an afternoon. If it fails in a real group, you have lost six weeks and a chunk of credibility.

The mirror problem

Here is the question that decides everything, and it is a question about epistemology, not software.

When a synthetic respondent answers, are you learning about consumers, or about the model’s prior beliefs about consumers?

The distinction is not academic. NielsenIQ ran experiments while building its own synthetic capability and found that its synthetic respondents appeared to care more about human health than the actual humans did. That is a systematic bias in a direction nobody would have predicted, and it emerged only because NIQ had real panel data to check against. A team without that benchmark would have shipped the finding.

The HBS work found the same failure mode in a different costume: LLMs rate novelty higher than humans do. Synthetic customers were initially enthusiastic about pancake-flavored toothpaste. Fine-tuning on real study data corrected the enthusiasm and brought willingness-to-pay back toward human sentiment.

This is what people mean when they say a model is sycophantic. It is not that the model lies. It is that the model has absorbed a smoothed, polite, socially desirable version of human opinion — the version people write down, not the version they act on.

Mode collapse: the average survives, the segment does not

The sharpest public test of this came from Nielsen Norman Group in late 2026, and the design was admirably simple.

They took one open-ended survey question that 604 tech and design professionals had already answered, posed it to 120 synthetic respondents, and compared the two datasets.

Both datasets reached the same headline conclusion. So far, so reassuring.

Then the detail. The human respondents used more than fifty distinct labels for the same concept. The synthetic respondents collapsed onto about a dozen. Every uncommon answer — human-factors engineering, business analysis, digital management — vanished. NN/g’s term for it is mode collapse: the model reproduces the most common data point and drops the long tail.

It got worse inside the segments. Among human product managers, 44% used UX terminology. Among synthetic product managers, 93% did.

The reproducibility trap

There is a second failure mode that gets far less attention, and it is the one that should worry anyone in a regulated category.

Bisbee’s team ran an identical prompt in April 2023 and again in July 2023. Same instructions, same personas, different model version underneath. The distribution of responses shifted.

This means a synthetic finding is not a fact. It is a snapshot of one model on one day, and it carries an expiry date you cannot see. Re-run the study in six months and you may get a different answer to the same question, with no methodological change to point at.

For an agency, that is an inconvenience. For a brand making an advertising claim, it is a liability. When a regulator or a court asks you to substantiate a claim, “the model said so, but we cannot reproduce it” is not a defense. It is an admission.

The cost case, stated honestly

The numbers are real, and they are large enough to matter.

Drive Research’s 2025 pricing survey puts in-person focus groups at roughly $7,000 to $20,000 per group, and an agency-run online concept test of 400 responses at $5,000 to $15,000 or more. Bain & Company, working with clients on synthetic customer programs, reports tests completed in half the time at one-third of the cost — and, in a later study using a client’s own first-party data to build digital twins, roughly 90% of the key outcomes from a prior large-scale conjoint study replicated correctly, including which features drove choice and which products to launch.

Notice what made the Bain result work: proprietary data, backtesting against a study that already existed, and segment-level construction. The cheap version — a demographic tag in a prompt — is not the version that produced 90%.

Where a real room still wins

Some questions simply require a body in a chair.

Purchase behavior is the clearest. Every research method on earth struggles with the say-do gap, but synthetic panels struggle more, because a model has no spending money. It can tell you what sounds like a purchase intention. It cannot tell you what happens when the price rises and the mortgage payment clears on the same day.

Emotional and irrational response is the second. Real people are inconsistent, petty, distracted and occasionally hostile in ways that make research valuable. Models are none of those things. NN/g’s earlier three-study test caught this precisely: synthetic participants reported finishing every online course. Real people said three out of seven, then explained the job change and the content that missed.

Claimable evidence is the third, and it is non-negotiable. If the output will appear in an advertising claim, a regulatory filing, or a courtroom, it needs real respondents and a documented method. The 2025 ICC/ESOMAR Code now requires human oversight and introduces a “Minimum Viable Data” threshold below which augmentation should not be used to extrapolate — a governance line, not a suggestion.

The four-gate hybrid stack

The useful answer is not synthetic versus real. It is a sequence with exit conditions.

Gate one, explore. Uncalibrated synthetic panel, no governance burden, used to generate hypotheses, vocabulary and objections, and to kill obvious losers. Exit condition: at least one hypothesis worth real money.

Gate two, calibrate. Before you trust the panel for anything that matters, back-test it against studies you have already run with humans. Compare on the tasks you care about, not the ones in the vendor demo. Exit: you know your divergence, quantified, by segment.

Gate three, screen. Now use the calibrated panel to rank dozens of concepts down to the handful worth fielding. Exit: a shortlist of three to five.

Gate four, validate. Real respondents only. Purchase intent, price sensitivity, emotional reaction, claim support. Exit: evidence you can defend in a boardroom.

The order matters more than the tooling. Most teams invert it — buy volume first, then discover there is no budget left for the one study that would have changed the decision.

What the industry is actually doing

Adoption is running well ahead of validation, which is worth knowing before you feel behind.

The 2025 GRIT Insights Practice Report, drawn from 992 respondents, found 67% of suppliers embedding generative AI directly into client deliverables, with data quality named as the top barrier by 40% of researchers. Harvard Business Review’s survey of more than 170 practitioners put adoption higher: 81% already use or plan to use generative AI to create synthetic data — while only 31% rated the value of AI-generated data as extremely high, and 77% flagged bias as a concern.

Then the number that reframes all of it. Per GRIT 2025, 23% of insights buyers piloted synthetic respondent tools, but only 4% used them as primary decision evidence.

The industry has already answered the question. Synthetic panels are a pre-testing instrument. The gap between 23% and 4% is not hesitation. It is the collective judgment of people who have run the backtest.

What to do on Monday

Pick one study you ran in the last eighteen months where you know the answer. Rebuild it synthetically. Compare. That single exercise will teach you more about your own category’s synthetic reliability than any vendor evaluation will, because the divergence pattern is category-specific and audience-specific.

Then write down, in one page, which decisions in your organization are allowed to be informed by synthetic research and which are not. Name the person who approves the exception. This is the entire governance program, and it costs nothing.

Then set the rule that makes it durable: synthetic findings are labeled exploratory in every deck they appear in. Not buried in an appendix. Labeled.

The takeaway

Synthetic panels are the most useful research instrument to arrive in twenty years, and the most over-sold.

They will cut your exploration budget by an order of magnitude and let you test ideas you previously could not afford to think about. They will also hand you confident, well-formatted, reproducible-looking answers that are wrong in ways the topline cannot reveal — flatter than reality, kinder than reality, and quietly missing every person who would have said the thing you needed to hear.

Use them where the decision is cheap. Validate where the decision is expensive. And keep one real room in the calendar for the questions that only a difficult human being can answer.

Frequently asked questions

Is a synthetic panel the same as a synthetic sample?
No, and the distinction matters commercially. A synthetic sample usually means augmentation of real data — boosting a thin quota cell or imputing missing responses, which is mature and defensible. A synthetic panel built entirely from model-generated personas is a different instrument with different failure modes. Vendors use the terms interchangeably; your procurement contract should not.

How accurate are synthetic respondents, in one number?
There is no single number, and anyone offering one is selling. Replication rates run from 68% to 88% depending on task type and effect size, and accuracy collapses specifically on segment-level differences and on anything involving actual spending. The honest answer is that accuracy is high where the decision is cheap and low where the decision is expensive.

Why do synthetic respondents seem more enthusiastic than real customers?
Because they are. Models absorb the polite, socially desirable version of human opinion — the version people write in reviews and say in interviews, not the version they act on. NielsenIQ observed synthetic respondents caring more about health than real humans did; Harvard researchers watched synthetic customers rave about pancake-flavored toothpaste until the model was fine-tuned on real data.

Can I use synthetic research to substantiate an advertising claim?
Not today, and not soon. The 2025 ICC/ESOMAR Code requires human oversight and sets a Minimum Viable Data threshold below which synthetic augmentation should not be used to extrapolate. More practically, a finding you cannot reproduce is not evidence — the same prompt produced different distributions three months apart in the Bisbee replication. Claims need real respondents and a documented method.

What is the single best first step?
Back-test. Take one completed human study where you know the answer, rebuild it synthetically with the same stimuli and measures, and measure the divergence yourself. Everything else — vendor selection, governance policy, budget allocation — gets easier once you have your own calibration curve rather than someone else’s case study.