Why Chat Conversion Lift Vanishes Under Random Assignment
The chatbot conversion statistic is one of the most repeated numbers in marketing software, and it is not a lie. It is a number nobody is paid to disprove.
Here is the version you have seen. Visitors who use on-site chat convert at 12.3%. Visitors who do not convert at 3.1%. Four times. The slide is accurate, the sample is real, and the conclusion drawn from it is almost entirely wrong.
Why the gap is so wide: chat meets intent, it does not create it
Fewer than one in a hundred ecommerce visits produces a chat conversation — 0.84 per 100, measured across 4.78 billion visits in 2024. That number is the whole key. Chat is not a channel that reaches people; it is a channel that people reach for. By the time someone opens a widget, they have already read the product page, formed a question, and decided the answer is worth interrupting their own browsing. The population is pre-qualified by definition, which is precisely why the group that stays off chat looks so much colder by comparison.
Why the number survives: credit follows the sale, not the bot
Analytics credits a conversation when it occurs inside a window before purchase. Nothing about that window rules out the possibility that the visitor would have bought anyway. So credit accumulates on whoever touched the buyer last, and last is usually the thing they opened to answer a single question. This is not fraud. It is simply the only measurement most dashboards are capable of producing.
Why nobody runs the test: the people who could, do not want the answer
A holdout means withholding the assistant from real visitors, including several who were about to buy. That request is genuinely hard to approve, because it feels like choosing to give worse service to a slice of customers for a statistical reason. Vendors face the same problem in reverse. At signature, the correlational figure is the strongest asset in the deck. A randomised result would be smaller, slower, and more expensive to produce.
Which is why the test rarely fails on methodology. It fails on incentives, and incentives are much harder to fix than methods.
The part worth keeping
Peel the 4× and you do not get zero. Of the 3.0× excess over baseline, roughly 2.8× is selection and 0.2× is the tool. In a large randomised field experiment on a cross-border marketplace, the surviving effect was a 16.3% lift in sales and a 21.7% lift in conversion against no pre-sale support — real money, honestly earned, and about a fifth of the claim.
That is the awkward discovery. The tool works. The story about the tool does not. And the distance between those two facts is not a measurement failure but a business model: the number that sells the software and the number that proves it were never going to be the same number.
So the question worth asking at the next renewal is not what chat converts at. It is who is paying to find out whether it converts at all.
