Chat Reaches One Visitor in a Hundred. It Gets Credit for Ten Percent of Sales.

There is a new employee on your website. It works nights. It never asks for a raise. Last month it had a few thousand conversations with people who were about to buy something, and it is genuinely unclear whether it helped them or got in their way.

You gave it a budget. You gave it a brand voice. You gave it a slide in the launch deck. The one thing you never gave it was a control group.

That omission is not an oversight. It is the default state of on-site AI in 2026, and it costs you in both directions: you are probably paying for a conversion lift you cannot prove, while also carrying a conversion leak you cannot see.

The Deployment Is Real. The Measurement Isn’t.

Start with the split in the data. Gartner surveyed 3,566 business and consumer customers in February and March 2026 and found that people are roughly three times more likely to take a service question to ChatGPT, Gemini or Copilot than to the chatbot on the company’s own site. Company-provided chatbot usage has been statistically flat since 2022.

Meanwhile, a separate Gartner survey of 1,303 senior leaders found service functions putting a median 12% of their 2025 budget into AI, the highest share of any business function they track. Only 24% could point to a positive financial return.

Read that pairing again. Top of the spending table, near the bottom of the returns table. That is what a measurement problem looks like from the outside. Not a technology that does nothing, but an organisation that never agreed on what “doing something” would be counted as.

Your audience is not uniformly delighted to meet the bot either. Pew’s February 2026 survey of 5,119 US adults found 51% never use a chatbot at all. Among non-users, 76% doubt the answers are accurate and 79% worry about how their data gets used. So for a meaningful slice of your traffic, the assistant is not a helpful colleague. It is an obstacle standing between them and the thing they came for.

Chat Reaches About One Visitor in a Hundred

Here is the figure that should reframe every conversational-commerce claim you have heard this year. Smartsupp analysed 4.78 billion ecommerce visits across 175,438 accounts in 2024 and counted 40.1 million chat conversations. That is 0.84 conversations per 100 visits. Fewer than one.

Vendors will tell you chatters convert four times better, 12.3% against 3.1% in one widely circulated dataset. They will also say AI reaches around 1% of visitors while taking credit for roughly 10% of sales. Both claims can be accurate and still tell you almost nothing, because “credited” is attribution rather than causation, and “people who chat convert better” is a description of who chooses to chat.

Think about who clicks a help widget. Someone with a question. Usually someone with intent, comparing two sofas at eleven at night rather than idling on a blog post. Of course they convert. They arrived pre-qualified.

The 4× That Isn’t

There is exactly one piece of evidence in this category I trust, and it comes from random assignment instead of a vendor deck.

A research team ran large-scale randomised field experiments on a major cross-border retail platform through 2023 and 2024, bolting generative AI onto seven different workflows. The published finding is unusually candid: effects ran from no detectable impact to a 16.3% lift in sales, depending on how much each workflow added beyond what the firm was already doing. Where it worked, the gain came from conversion rather than bigger baskets, and the shoppers who benefited most were the least experienced ones. Return rates and product ratings did not deteriorate, which quietly kills the objection that AI sells people things they regret.

The pre-sale assistant in that study lifted sales 16.3% and conversion 21.7% against no pre-sale support. Set that beside the 4× vendor gap and the arithmetic turns interesting. A 21.7% relative lift on a 3.1% baseline lands near 3.8%, not 12.3%. The defensible effect is roughly a fifth of the advertised one. Still excellent. Still worth real money. And just about the only number in this article you could survive a finance review with.

Why Almost Nobody Tests It

If the effect is that real, why is a holdout so rare? Four reasons, none of them stupid.

A conversation is not a variant. A headline test has clean states: version A, version B, one variable. A conversation is unbounded, personalised, and different for every visitor. What do you hold constant? The greeting? The model? The retrieval index? The tone? There is no button to flip.

The treatment is the whole site. Chat does not live in one place, so its effect is smeared across every page and every other change you shipped that month. Isolate it and you are effectively experimenting on your entire domain.

Nobody owns it. Is the bot support, marketing or product? Support bought it, marketing writes its tone, product maintains the knowledge base, finance pays the per-conversation bill. When four departments share a tool, measurement becomes everyone’s interest and no one’s job description.

The vendor holds the logs. Most enterprise deployments hand you a dashboard rather than event-level data: resolution rate, deflection rate, a rolling CSAT average. None of those is a conversion metric, and none lets you build a counterfactual.

Put those together and the real reason surfaces. The metrics you are handed are help-desk metrics. Deflection, containment, resolution time, all of them measure whether the bot absorbed work. Not one of them measures whether anybody bought anything.

What the Black Box Hides

An unmeasured assistant is not neutral. It sits in one of four states, and three of them look identical on the dashboard you already have.

A weak bot with no holdout is a silent leak. It interrupts people who were going to buy anyway, answers a pricing question with four paragraphs, and pushes a hot lead into a queue. Conversion dips, you blame the redesign, and the bot keeps its budget.

A strong bot with no holdout is a missed lift, which is the more common and more tragic case. The thing works, nobody can prove it, so it stays a line item instead of becoming a strategy. Next budget cycle it is the easiest item to cut, precisely because it has no number attached to it.

A weak bot that you do measure is a confirmed failure, and that is the only version of this story that ends well. You find out on your own schedule rather than your CFO’s.

Presence Is the Treatment

The usability evidence says the widget’s mere existence changes behaviour, which is exactly why “we installed it and conversions rose” is a narrative rather than a finding.

Nielsen Norman Group’s 2026 studies of site chatbots are brutal in a useful way. Regular Home Depot customers did not know the retailer’s AI assistant existed. The icon was small, unlabelled and camouflaged in brand colours; one participant, once shown it, said he had assumed it was a decorative graphic. Users who did try chatbots often found them slower than the search box they already had: fewer options on screen, more typing, worse comparison. One shopper gave up on Amazon’s Rufus mid-task and returned to plain search so she could see the whole catalogue instead of five suggestions.

The pattern holds across those studies. The bot earns its keep when it answers a specific question about the product in front of you, or handles the odd multi-part question no filter set can express. It loses whenever it duplicates navigation, promises “anything” in the welcome message, or vanishes at checkout, which happens to be the exact moment a hesitant buyer is worth the most. Consider that the average documented cart abandonment rate sits around 70%, and you begin to see how expensive that disappearing act is.

Placement moves the numbers too. Brands that put the assistant inside search, an inline panel and a product FAQ report engagement above 6%, against roughly 1% for a corner widget on its own, according to vendor-reported data. That is a design decision you can test, if you are willing to treat the bot’s absence as a legitimate variant.

Answer Quality Is the Variable You Aren’t Testing

When teams do measure chat, they usually measure engagement. Engagement is the wrong dependent variable, because a bot that talks a lot and answers badly can be extremely engaging.

Real variance in answer quality is enormous. An independent benchmark published in late 2026 put seven commercial models through 31 scripted support conversations, five times each. Hard failures, meaning refunds issued that should not have been, data disclosed to the wrong person, actions claimed but never performed, ranged from 0.6% to 27.1% depending on the model. That is not a rounding error. That is a different product wearing the same interface.

The fix is not always a bigger model, either. A peer-reviewed study presented at ACL 2026 trained a 12-billion-parameter model to recognise the limits of its own evidence and say “I don’t know.” Over seven months at a large US financial institution, resolution rose 7.1 percentage points across 3,297 sampled queries, even though the model refused nearly three times as often as its base version. Smaller, cheaper, more honest, and better at closing cases.

Which hands you a sharper experiment: don’t just test chat on versus chat off. Test answer policies against each other, grounded versus free-generating, escalating versus always-answering, and let revenue pick the winner.

The Playbook

Instrument before you optimise. Log every conversational event to the same analytics property that holds your purchase events: greeting shown, question asked, answer served, source cited, escalation, handoff, add-to-cart from chat, checkout. If the events live only in the vendor’s dashboard, you cannot test anything. Put raw event export in the contract, at signature, not as a favour afterwards.

Define a real control. Randomise at the visitor level where you can, and at the page-type or time-slice level where you cannot. Page-type randomisation is cleaner than most people assume: hold 10% of product-page traffic out of the assistant, keep the rest on it, compare conversion on the same product mix.

Test the conversation, not the widget. The interesting variants are not on and off. They are the opening line, whether suggestions are buttons or prose, whether the bot can see the page it is on, whether it routes toward checkout or just answers and stops. Those are the levers with effect sizes big enough to detect at realistic traffic.

Optimise revenue events. Choose one primary outcome before launch, whether that is revenue per eligible visitor, qualified lead rate, or checkout completion, and refuse to declare victory on deflection. Track device while you are at it: most chat happens on phones, where conversion is already lowest, so a bot that works on desktop can still lose money.

Run it continuously. Models get swapped, prompts get edited, catalogues change. A bot that won last quarter can lose this one, and a single test is a snapshot of a moving object.

Running a Holdout Without Statistical Theater

Two things will wreck your read, and neither is exotic.

The first is seasonality and promotion. A four-week test straddling a campaign will attribute the campaign to the bot. Keep both arms live at the same time, compare equivalent periods, and write down everything else that changed.

The second is peeking. Roughly 19% of audited A/B tests end with a statistically significant winner; on one platform’s onsite experiments the figure was under 10%. Most tests are inconclusive, and treating an inconclusive result as permission to ship is how mediocre features become permanent. Half of businesses, meanwhile, QA their experiments through no formal process at all. Fix your sample size and duration before launch, then leave it alone.

If true randomisation is unavailable, a concurrent comparable segment beats a before-and-after chart every time. And say so in the readout. “Directional, confounded by the March promotion” is a respectable sentence. “Chat increased conversion by 32%” is not, unless you earned it.

What a Good On-Site Agent Actually Does

Four jobs, and only one of them is answering questions.

It answers, then routes. Someone asking about shipping cost on a product page is ninety seconds from checkout. The bot should resolve the objection or move them forward, not deliver a policy paragraph and stop.

It qualifies. Budget, timeline, use case, seat count, the questions a sales rep would ask, asked in chat and logged as a lead score. This is the part that turns a help desk into a pipeline.

It handles objections. Cost, fit, returns, compatibility. Objection handling is where the conversion gain in that randomised experiment actually came from: fewer people leaving to think about it.

It hands off at the right moment, visibly, on request. In Gartner’s August 2026 survey, 87% of customers said human access is essential whenever AI is involved in service, and customers who get trapped in a loop after a failed attempt are the least likely to come back.

Risks Worth Admitting

A hallucinated price is not a UX bug. It is a liability. A bot that confidently misstates your return window, your stock level or your financing terms has just made a promise your operations team has to honour. Ground the answers, cap the scope, keep a human one click away.

An aggressive proactive bot is also a treatment you are inflicting. The vendors selling proactivity will show you engagement curves; ask them for the complaint rate on the same cohort. Prompting someone after 45 seconds of hesitation can rescue a sale, or it can recreate a pop-up ad from 2011 with better manners.

And the quality gap between the bot you bought and the assistant your customer met on ChatGPT last month is the real competitive pressure here. That is not an argument for skipping the test. It is the reason the test matters more every quarter.

A Thirty-Day Start

Week one, get the event log out of the vendor and into your own analytics. If that is blocked, that is your finding.

Week two, define one primary metric and one guardrail: revenue per eligible visitor, with refund rate watching the door.

Week three, launch a 10% page-type holdout on your highest-traffic commercial template. Do not touch the variant. Let it run.

Week four, read the result at the sample size you promised yourself, publish it internally whether it flatters anyone or not, and put the number in the budget deck.

The Point

On-site AI is the only surface in your stack that gets deployed on faith and judged on uptime. That combination is how you end up with the largest AI budget share in the company and a 24% hit rate on positive returns.

Fix the measurement and the argument about whether to keep the bot mostly resolves itself. Sometimes in the bot’s favour, occasionally against it, always with a number attached.

Because the real risk was never that the assistant would fail. It is that it would succeed quietly, and nobody would know to double the budget.