The Boring Half of AI Visibility: Crawl Access, Rendering and Extractable Structure

The plumbing did not become optional. It became the building.

Two years ago the confident take was that technical SEO had been quietly demoted. The AI answers the question on the results page, so why crawl the page at all? Why fight over canonicals when the model reads the web and writes you a paragraph?

I believed part of that story myself, briefly. It is a seductive story, because it flatters everyone doing the fun work. Content strategy is fun. Arguing about schema is fun. Staring at a 403 log at eleven on a Friday night is not, and nobody puts a firewall rule on a conference slide.

Then you look at what actually gates an AI answer, and the story collapses. Every one of these systems — Google’s AI Overviews, ChatGPT’s search, Perplexity, Claude — reaches the web through a crawler that must be allowed in, must receive a page it can parse, and must find a sentence it can lift. Fail any one of those and your content is, from the model’s point of view, simply not there.

So no. Technical SEO was not demoted. It was promoted and then ignored, which is the worse fate of the two.

The argument nobody needed to have

The framing that dominated 2025 was SEO versus GEO, as though a new discipline had arrived and rendered the old one quaint. Most of that argument was positioning, and Google ended it on 15 May 2026 by publishing its first official guide to its own generative features, filed under a new “Generative AI fundamentals” section of Search Central. The line everyone quoted: optimizing for generative AI search is optimizing for the search experience, and thus still SEO.

Read past the headline and something more useful is there. Google states that AI Overviews and AI Mode run on the same index and the same quality systems as the ordinary results page. There is no separate AI index, no parallel ranking algorithm, and no markup you can bolt on to be chosen. Retrieval-augmented generation does the answering; query fan-out does the sourcing — one question is exploded into several related searches, and each of those searches pulls pages the classic results page would never have surfaced.

That last mechanism deserves more attention than it gets. Fan-out is the reason a page sitting at position 40 can be quoted in an answer. The model asked a narrower question than the one in the search box, and your page happened to be the best answer to that one.

The eligibility floor is equally plain. A page must be indexed, and it must be eligible to appear in Search with a snippet. Google’s own words: there are no additional technical requirements. No AI schema, no llms.txt, no chunking ritual, no separate version of the site for the robots.

Whether that reads as a relief or an insult depends on what you were sold last year. Either way, it moves the argument back to a place most teams have not audited since 2019: can the machine get in, can it read, and can it take something out.

Gate one: can they get in?

Access is the gate teams believe they have already passed, which is exactly why it fails.

robots.txt is the file everyone checks and the file that has never once been the whole story. Above it sits the CDN, the web application firewall, the bot-management rule set, and whatever a security team configured during an incident nobody documented. A site can welcome OAI-SearchBot in plain text and refuse it four layers up with a 403, and the marketing department will never see the refusal because it does not appear in a dashboard anyone reads.

The scale is not marginal. Cloudflare’s network was turning away 13.47% of AI bot requests in the third quarter of 2026, against 5.98% in the same quarter a year earlier — preliminary figures, worth treating as a direction rather than a decimal. Read the composition and the intent becomes obvious: in a late-September sample of 4,242 robots.txt files, OpenAI’s training crawler was blocked 1.60 times for every time it was allowed, while the crawler that feeds ChatGPT’s answers was blocked 0.91 times. Publishers are saying no to training and yes to retrieval. That is a considered policy, not an accident.

And the policy has a reason. Cloudflare’s crawl-to-refer ratio — how many pages a platform takes for every visitor it sends back — sat at roughly 14 to 1 for Google in mid-2025, 1,700 to 1 for OpenAI, and about 70,000 to 1 for Anthropic. When you are handed that arithmetic, closing the door is a rational response.

But here is the uncomfortable part for brands rather than publishers. Blocking training makes sense. Blocking the retrieval crawlers is a slow commercial suicide that produces no error message. If you sell to buyers who now ask an assistant instead of a search box, the crawler you just blocked was your only route into that conversation.

The audit is therefore not a file review. It is a log review: pull thirty days of requests from OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot, and check the status codes. 200s all the way down means you are in the room. A wall of 403s means every pound you spent on an AI visibility programme went into a site that answers the door with a locked gate.

Gate two: can they read it?

This is the gate that quietly disqualifies a huge share of modern marketing sites, and almost nobody tests for it.

A page built with client-side rendering is, to a browser, complete: the script runs, the hero appears, the pricing table draws itself. To a crawler that does not execute JavaScript, the same page is an empty shell with a loading spinner and a script tag.

Vercel’s analysis of its own network traffic, run with MERJ and published in December 2024, is still the clearest statement of the split. None of the major AI crawlers executed JavaScript. GPTBot fetched JavaScript files on 11.5% of its requests and ran none of them; Claude downloaded scripts on 23.84% of requests and did the same. Googlebot and Applebot were the exceptions, and Gemini inherits Google’s rendering pipeline, which is why one engine can see your page while three others cannot.

Two years on, that baseline has not been formally overturned: no vendor has published documentation adding browser-grade rendering to its crawler. The practical consequence has not changed either. Your page can rank perfectly well in classic Search — Google renders it — and be functionally invisible to the assistant your customer is actually asking.

There is a second-order problem in the same dataset, and it is the one I would put on an engineering ticket first. ChatGPT’s crawler received a 404 on 34.82% of its fetches; Claude’s on 34.16%; Googlebot’s on 8.22%. A third of the AI web’s reading is spent on dead ends. Some of that is the crawler working from stale URL lists it did not build. A great deal of it is our fault: redirect chains, parameter sprawl, pages that were deleted without a decision, faceted navigation generating thousands of near-duplicates. Duplicate content is not an AI-era problem in the sense of being new. It is an AI-era tax, and it is being paid in crawl budget.

The test takes one command. Request the URL with a GPTBot user agent and read what comes back. If the answer paragraph is not in the response body, no amount of headings, entities or brand mentions will put it in the answer.

Gate three: can they take something out?

Suppose the crawler gets in and reads. Now the question becomes whether there is anything in your page worth lifting.

This is where a lot of confident-sounding GEO advice gets the mechanism slightly wrong. Answers are not assembled from entire pages. They are assembled from passages, and the passages that survive selection are the ones that can be read, checked and re-attributed without ambiguity. A study published on arXiv in April 2026, built on 602 controlled prompts across ChatGPT, Google and Perplexity and more than 21,000 citations, separated selection from what the authors call absorption: being cited is stage one, influencing the answer is stage two. The pages with the highest influence were longer, more structured, semantically closer to the query, and richer in extractable evidence — definitions, hard numbers, comparisons, procedural steps.

Nothing in that list requires a plugin. It requires writing that behaves like reference material: a direct answer immediately under the question, numbers with units and dates attached, a table where a table is the honest format, a short list where a list is the honest format.

And it requires the invisible half of the job: one real H1, a hierarchy that means something, structured data that agrees with the visible text rather than contradicting it. Google’s guidance on this is narrower than the industry’s, and it is worth repeating precisely. Structured data does not have to match a schema recipe invented for AI, because no such recipe exists. It has to match what a reader can see on the page. Markup that claims a price the page does not show is worse than no markup, because it hands the model a contradiction.

Which brings us to the money.

The schema shortcut that did not pay

For eighteen months the single most repeated piece of AI-search advice was: add structured data, get cited more. It felt like the kind of lever SEOs are trained to look for. Cheap, technical, invisible to the client, easy to invoice.

In May 2026 Ahrefs tested it properly, and the result deserves to be quoted in full because the internet has mostly quoted the half it likes. A first pass across six million URLs had found that pages cited by AI were roughly three times more likely to carry JSON-LD — exactly the kind of correlation that gets screenshotted into a pitch deck, and exactly the kind that proves nothing. So they took 1,885 pages that introduced JSON-LD between August 2025 and March 2026, matched each against three control pages from other domains that had never added schema, and ran a difference-in-differences analysis to strip out platform-wide trends. The measured effect: plus 2.4% on AI Mode, plus 2.2% on ChatGPT, minus 4.6% on AI Overviews. The first two are indistinguishable from zero. The third is small but statistically notable — around a dozen fewer citations a day on pages that had been earning hundreds — and Ahrefs says plainly that it cannot attribute that decline to schema, because both groups were already falling before the markup went on. The raw before-and-after on AI Mode looked like a 43% lift; the control group had risen almost as much, because AI Mode was exploding for everybody.

There is a prior detail that makes the null result even more instructive. Every page in that dataset already carried more than 100 AI Overview citations before any schema was added. These were not invisible pages hoping for a first citation. They were already in the game. Schema did not move them.

A separate experiment, run by searchVIU in October 2025 and cited by Ahrefs, offers one mechanical explanation: five systems were tested on pages where facts existed only inside JSON-LD, and none of them returned those facts when fetching live. They read visible HTML. It is a single narrow test on one page, so treat it as a lead rather than a law — but it points the same direction as the controlled result.

So what is schema still for? Rich results in classic Search, entity clarity at index time, product and local eligibility, and consistency that keeps you out of trouble. Bing has supported JSON-LD in its index for years, and answers built on that index inherit whatever it resolved months earlier. That is real value. It is just not the value that was being sold, and none of it is a shortcut past gates one through three.

llms.txt: the file almost nobody reads

If schema was the expensive disappointment, llms.txt was the cheap one — a Markdown file at the root of the site, proposed in September 2024, that would tell language models which pages mattered.

In June 2026 Ahrefs published a study of 137,210 domains. Of the llms.txt files that existed, 97% received zero AI requests in a single month. Where requests did happen, AI retrieval bots accounted for 1.1% of them — search crawlers barely register — while agentic tools like coding assistants were the largest single consumer at 10.5%, and training crawlers fetched the file almost five times as often as retrieval bots.

Google has been blunt: its Search products, including AI Overviews and AI Mode, do not use the file, and its May 2026 guide lists it among the things you do not need. Creating one will not help or hurt.

My own view, for what it is worth: ship it if your documentation is already clean Markdown and your customers’ engineers use coding agents to find your API details. That use case works and costs almost nothing. Do not ship it believing it is a citation strategy. A convention awaiting adoption is not the same thing as a standard, and no major provider has committed to reading it.

Two obsessions that quietly died

Keyword density went first, and nobody held a funeral. Exact-match counts, keyword-in-every-H2, the thesaurus-swap paragraph at the bottom of the page — all of it assumed the machine matched strings. These systems match meaning. That debate is over, and the only people still having it are the ones selling tools for it.

The second death is the one that hurts, because it was the whole job for twenty years. Position one was a proxy for attention: you ranked first, you got the clicks, you got the traffic. In an answer-first surface, position one is a proxy for almost nothing.

Ahrefs’ March 2026 measurement of 863,000 SERPs and roughly four million cited URLs found that 37.9% of URLs cited in AI Overviews ranked in the organic top ten, 31.2% sat between 11 and 100, and 31.0% ranked beyond position 100 entirely. In July 2025 the same company had measured the top-ten share at about 76%. BrightEdge, using a different method, puts the overlap near 17%. Two credible shops, two very different numbers, one shared conclusion: the old map no longer describes the territory.

That is not an argument for giving up on rankings. It is an argument for adding a second and third number to the report.

Core Web Vitals: still real, still not an AI signal

Here I want to be careful, because this is where technical SEO writing usually overreaches.

Core Web Vitals remain a confirmed ranking input, and the thresholds have not changed: LCP at 2.5 seconds or better, INP at 200 milliseconds or better, CLS at 0.1 or better, assessed at the 75th percentile of real users. INP replaced First Input Delay in March 2024 and is now the only interactivity metric. Chrome’s August 2026 dataset had 55.6% of origins passing all three, with good LCP at 68.1% and good INP at 85.3% — a large minority of the web still on the wrong side of a fixable line.

But Google’s own description of the weight is precise: page experience is one of many factors and matters most when pages are similar in relevance. It is a tiebreaker. And no study I can find shows Core Web Vitals moving AI citations in any direct way. The connection is indirect and worth stating honestly: Google’s AI guide does list page experience among its recommendations, and a page that times out or shifts under the thumb is a page a person abandons — which affects everything downstream.

Fix what is failing. Stop chasing a perfect score. The difference between 2.3 seconds and 1.8 seconds is an engineering sprint that would be better spent rendering your pricing page server-side.

The duplicate tax, and the one tag that still decides everything

Two smaller items that belong in the same audit, and get skipped because neither is glamorous.

The first is URL discipline: one canonical per page, redirects collapsed to a single hop, faceted navigation controlled, thin pages removed rather than orphaned. A third of AI crawler fetches ending in a 404 is not a mystery; it is a maintenance backlog with a bill attached.

The second is the snippet controls, the only place where a tag directly decides whether you are quotable. nosnippet, max-snippet and data-nosnippet all gate inclusion in AI features, and Google documents them as display controls separate from crawling. A legacy CMS setting, inherited from a template nobody remembers approving, can quietly remove your best pages from every answer. That is a five-minute check, and it outranks most of the strategy deck.

What to do first

Not a list of thirty tactics. Five steps, in this order, because the order is the point.

1. Prove access. Thirty days of server or CDN logs, filtered for the named AI crawlers, sorted by status code. Fix every 403 before anything else. This is the step that invalidates the rest if skipped.
2. Prove readability. Fetch your ten most commercially important URLs with a GPTBot user agent and a plain curl. If the substance of the page is missing from the response body, the fix is server-side rendering or prerendering — not a content rewrite.
3. Restructure for extraction. A direct answer in the first sentence under each heading, numbers with their units, tables where they earn it, and markup that matches what a reader sees. This is editorial work with a technical constraint, which is why it falls between two teams and gets done by neither.
4. Fix the plumbing nobody audits. Canonicals, redirect chains, orphan pages, snippet directives, sitemap accuracy. Unglamorous, finite, and it compounds.
5. Decide your crawler policy on purpose. Most brands should allow retrieval crawlers and can reasonably refuse training crawlers. What is indefensible is allowing a training crawler while a WAF rule blocks the one that answers questions.

How to measure it now

Rank tracking alone will mislead you, and not because rank tracking is broken — because it answers a question about a surface that is no longer the only one that matters.

Three numbers, measured honestly, beat thirty vanity metrics.

Access rate. The share of AI crawler requests that receive a 200. If this is not near 100%, nothing else you measure is real.

Selection rate. For a fixed panel of prompts, how often your domain appears as a cited source. Keep the engine, the surface, the language, the locale and the date next to every observation, because answers vary by all five. Sample the same panel monthly and accept that some movement is noise.

Absorption. Of the answers that cite you, how many actually use your language, your number, your definition. This requires reading the answers, which is why almost nobody reports it and why it is the number that predicts whether the citation was worth having.

And keep a fourth, unglamorous one: brand demand. If citations are rising while nobody types your name, you have bought mentions rather than memory.

Where this reading could be wrong

Four caveats, because the honest version of this argument needs them.

Crawlability is necessary, not sufficient. A technically flawless page with a commodity answer still loses to a worse-built page with an original one. Google says as much directly: unique, useful content will likely influence AI presence more than any other recommendation in its guide. The gates get you into the competition. They do not win it.

The schema null result has a boundary. Every page in that study was already visible to AI. For a page that has never been crawled or parsed properly, markup may still help at index time. Absence of evidence in one population is not evidence of absence everywhere.

Citation behaviour is partly non-deterministic. Two AI Overview runs sampled two months apart have been reported sharing only 18% of cited pages, against 45% for organic results. Some of your month-to-month movement is the machine, not you. Budget for that in how you read the trend.

And the mechanics keep moving. The figures in this piece are dated on purpose, because the ratios, the block rates and the rendering behaviour have all changed materially inside two years and will change again. Re-audit on a schedule; do not build a process around a single quarter’s numbers.

The plumbing is the product

Here is the reframe I would put in front of a client, in one sentence: technical SEO did not become less important in the AI era, it became the layer everything else is standing on — and layers that hold weight are boring by definition.

Content quality, brand authority, entity work, digital PR, the whole apparatus of generative engine optimisation, all of it runs through three gates that were built by engineers and are audited by nobody. A page that cannot be reached cannot be quoted. A page that does not exist before JavaScript runs cannot be read. A page with nothing extractable in it cannot be absorbed, no matter how many citations it collects.

The tactics that died were tactics. The principles that survived are principles, and they survived because a retrieval pipeline is less forgiving than a browser. That is the whole lesson, and it is the least fashionable one available: the work that wins now looks almost exactly like the work that won in 2019, done more carefully, measured differently, and defended in meetings where somebody would rather talk about AI.