The Economics Behind the Block: Why AI Crawler Refusals Doubled in a Year

Nobody chose to become invisible. A ratio did it.

The number first, because it sets the size of the thing. Across one large content delivery network, 13.47% of AI bot requests were refused in the third quarter of 2026, against 5.98% in the same quarter a year earlier. These are preliminary figures, so treat them as a direction rather than a decimal — but a refusal rate that doubles inside twelve months is not a wobble. Something structural moved.

The barter that broke. For twenty years the deal was implicit and generous: crawl my site, send me visitors, and I will keep publishing. The exchange rate held because the crawler and the referral arrived from the same company. That is no longer true. One network’s own crawl-to-refer measurements put Google at roughly 14 pages fetched per visitor returned in mid-2025. OpenAI sat near 1,700 to 1. Anthropic sat near 70,000 to 1.

At seventy thousand to one, “allow” stops being hospitality and becomes an unbilled subsidy. The person signing it has a budget meeting in the morning and a bandwidth invoice on the desk. Closing the door is not ideology. It is arithmetic.

What the composition reveals. The refusals are not evenly spread, and that is the interesting part. In a late-September snapshot of 4,242 robots.txt files, OpenAI’s training crawler was blocked 1.60 times for every time it was allowed. The crawler that feeds ChatGPT’s answers was blocked 0.91 times — allowed slightly more often than refused. Publishers are not blocking bots in a fit of pique. They are blocking a line item: refuse the crawler that costs nothing to feed and earns nothing back, keep the one that answers questions. Roughly four in five news sites now block AI training bots; far fewer block the retrieval ones. That is a deliberate split, made by people who ran the numbers.

Why the reflex travelled

Then the posture migrated, and this is where the economics stop being borrowed honestly. Brands adopted default-deny because it sounds prudent, because a security vendor shipped it that way, or because a template arrived with the rule already inside. They inherited a publisher’s posture without inheriting a publisher’s business model.

A publisher’s page earns money when it is displayed. A brand’s page earns money when it is read by someone deciding what to buy. Those two interests diverge in exactly one place — the retrieval crawler — and the reflex landed on precisely that spot. The block that protects nothing is the one that removes you from a conversation already in progress.

Field notes from the audits

I have run this check often enough now to recognise the shape. The robots.txt is usually fine. The block sits four layers above it, in a firewall rule or a bot-management profile installed during an incident nobody wrote down. The fix takes an afternoon. The recovery takes a quarter, because a crawler has to notice, come back, and re-index before any of it shows. And the person who owns that rule has never heard of the report in which their rule is the single biggest variable.

The failure is silent by construction. A 403 lives in an edge log, not in a search console. Rankings do not move, so no alert fires, so the weekly report says the site is healthy while the assistant tells your customer about a competitor.

One more number makes the diagnosis obvious. Between the second and third quarters of 2026, search- and user-triggered requests grew from 6.51% to 11.92% of AI crawl activity. The crawlers that exist to answer questions are becoming a larger share of what AI sends at your site at the very moment the sites with the least to lose are blocking them.

The tell

So the doubling is not a technology trend. It is a bargaining position, adopted by one group that has a reason and copied by another group that has none.

The remedy is a policy split rather than a firewall fight: allow retrieval, decide on training deliberately, and write the decision down where the next security contractor can find it. The only genuinely indefensible position left is the accidental one — a rule installed by somebody solving a different problem, quietly declining the question your buyer just asked an assistant.