Schema, llms.txt and AI search: what technical work actually matters

The short answer

Two things are sold hard as the technical route into AI search: more schema, and an llms.txt file. The evidence on both is now unusually clear. It does not support the pitch.

Key takeaways

• Schema markup is a floor, not a lever. Get it right once, so machines can state facts about you without guessing. More of it does not buy more citations.
• A controlled study says so. Ahrefs tracked 1,885 pages that added JSON-LD against roughly 4,000 matched controls. AI Mode moved +2.4 percent and ChatGPT +2.2 percent, both indistinguishable from noise. AI Overviews moved minus 4.6 percent.
• Google names llms.txt as unnecessary for its AI features, alongside content chunking, AI-specific rewrites and structured-data over-optimisation, in guidance published 15 May 2026.
• The technical work that does matter is duller. Can crawlers reach you? Can the page be parsed? Is the answer near the top? Is the page fast enough to fetch?
• Pepper treats this as a gate, not a strategy. Nothing above the floor counts until the floor is solid. We check it in week one, before anyone writes anything.

Where this comes from. We run organic for more than 250 enterprises at Pepper and track over 10 million prompts across every major engine. We audit this exact list on every new account. So the ordering below is what we actually find broken, rather than a theoretical checklist.

Does schema markup help AI search?

Short answer: it helps a machine understand what your page is. There is now decent evidence that piling on more of it does not increase how often you get cited.

The best evidence available is a controlled study rather than a vendor claim. Ahrefs tracked 1,885 pages that added JSON-LD schema between August 2025 and March 2026. Each was matched against three control pages from different domains with similar citation levels, giving roughly 4,000 controls. The analysis was a matched difference-in-differences test over a 30-day window either side of the change.

The results:

• Google AI Overviews: minus 4.6 percent. Small, but statistically significant against the matched controls.
• Google AI Mode: plus 2.4 percent. Statistically indistinguishable from zero.
• ChatGPT: plus 2.2 percent. Also indistinguishable from zero.

Read the limitations before you act on this. They matter, and the study states them plainly. Every page in the sample already had 100 or more AI Overview citations before schema was added. So this tests whether schema helps pages that are already cited. It does not test whether schema helps a page nothing has ever cited. The authors also note they cannot fully separate schema from other changes made at the same time, and did not test individual schema types, longer windows, or JavaScript-injected markup.

So the honest reading is narrow and useful: if you are already visible, more schema is not the lever. Whether it helps a page from a standing start is genuinely untested.

See where you show up. Pepper’s GEO platform tracks Brand Visibility, Domain Prompt Presence and Share of Voice across ChatGPT, Perplexity, Gemini, Claude and Google AI Overviews. Your team can log in, connect Search Console and GA4, manage the prompt set and run your own agents in the Agent Atlas. A growth team works the same account alongside you. Book a growth audit or see where you show up.

What Google actually says

On 15 May 2026 Google published its first official guidance on optimising for its AI features, and filed it under SEO fundamentals rather than as a new discipline. Two parts matter here.

There is no separate AI index and no separate AI ranking algorithm. Both AI Overviews and AI Mode run on the core Search ranking and quality systems. So there is no parallel technical stack to build. The work you already do carries across.

Several widely sold tactics are named as unnecessary. The list includes llms.txt files, content chunking, AI-specific rewrites, inauthentic mentions and structured-data over-optimisation.

That is awkward for the current top results on this keyword. Several recommend building both an llms.txt file and an expanded schema layer as the route into AI search. One of those is named unnecessary by Google. The other has a controlled study showing no uplift.

What is llms.txt, and should you build one?

llms.txt is a proposed file placed at the root of a site, in the spirit of robots.txt, meant to give language models a curated map of your important content.

The idea is reasonable. The reality is simpler: Google has named it unnecessary for its AI features, and no major engine has committed to consuming it in a way that changes what gets cited.

Our position, which we give clients directly:

• It costs almost nothing to publish, so if a stakeholder insists, the argument is not worth having.
• Do not let it displace real work. The hour spent debating it is better spent checking whether AI crawlers can reach your pages at all.
• Treat it as a red flag in a proposal. A partner leading with llms.txt in 2026 has not read the guidance. That raises a fair question about what else they missed.

We reached the same conclusion independently in our llms.txt explainer.

Effort against impact

Not all technical tasks return the same thing, and the ones sold hardest tend to sit in the bottom-left.

The two green points are the ones we find broken most often, and they cost nothing to check. The two orange points are the ones most proposals lead with.

The technical work that does matter

This is duller than schema, which is exactly why it gets skipped. It is also, in our experience auditing new accounts, where the actual breakages are.

• Crawler access, checked per crawler. Different bots do different jobs, and blocking each fails differently. OpenAI runs separate crawlers for search indexing, on-demand fetches and training. Perplexity runs a scheduled indexer and a separate on-demand fetcher. Block the wrong one and you vanish from answers while believing you are open. The detail is in our guide to the crawler rules.
• Indexing, verified rather than assumed. An unindexed page cannot be retrieved, so nothing else here matters. This is the most common breakage we find, and it is free to check.
• Rendering without JavaScript. If the answer only exists after a script runs, some retrievers will never see it. Fetch the page as raw HTML and read what comes back.
• Server response and speed. A retriever fetching on demand will give up. Slow or flaky responses cost citations silently, with no error for anyone to notice.
• Clean, parseable structure. Real headings, real HTML tables, real lists. A table rendered as an image is invisible to a machine. So is an answer trapped inside a graphic.
• Schema, done once and correctly. Organization details, author, article type, and sameAs links that actually resolve. Correct beats elaborate, and the study above suggests elaborate buys nothing extra.

We wrote the technical detail up separately in can AI bots crawl your site and crawlability for AI.

What this looks like on a real account

Technical work rarely gets a case study of its own, because it is the thing that makes the rest possible rather than the thing anyone photographs.

On SalesHood, the programme included resolving 10 technical issues including sitemap and schema fixes, alongside a content engine of 20 new pieces and 8 refreshed, and GEO and AEO tracking in Atlas. Visibility in AI Overviews went from 14 keywords to 97. Rich results grew 291 percent in five months, and clicks rose 20 percent against an industry-wide decline.

Note what that ordering implies. The schema fixes got the floor right. They were not the mechanism that produced the visibility. That was the content and the tracking, running on a site made readable first.

TVS Eurogrip is the clearest technical-recovery example we have published. A site migration cost roughly half their organic traffic. The work began with post-migration stabilisation and indexation fixes. Their Head of Brand Marketing, Kavitha Ganesan, described the starting point plainly: “We actually saw a 50% drop in organic traffic right after migration.”

Our methodology: how we weighted the technical tasks

Technical tasks at a glance

—

What does the technical work cost?

Three of the seven rows are free. In our audits they are also where the breakages usually are, which is the whole argument of this page.

How to evaluate a technical AI-search proposal

Most proposals we see lead with the visible items and skip the gate. Score them the other way round.

The scorecard

Score any proposal out of 100.

The live test, over 90 days

Fix 30 prompts your buyers would actually type, and run them across Google, ChatGPT and Perplexity. Then do one extra thing most teams skip. For every answer where a competitor appears and you do not, fetch your own equivalent page as raw HTML.

Real questions to start from, such as “does schema markup help AI search”, “how do I check if AI bots can crawl my site”, “what is llms.txt for” and “why is my page not showing in AI Overviews”.

Log three things per prompt. Whether you are named, whether a page on your domain is cited, and whether the page you would want cited is even retrievable. Rerun the identical set monthly for 90 days. Three readings is the shortest honest window: one is a snapshot, two could be noise, three shows direction.

That third column is the technical one, and it is the one that turns a vague visibility problem into a specific ticket.

Weak approach versus strong approach

The weaker approach starts with a structured-data audit. Implement every recommendation, publish an llms.txt file, add markup to templates that lacked it, then report schema coverage as a percentage.

The stronger approach: verify crawler access and indexing in an afternoon. Fetch your top twenty pages as raw HTML and read them. Fix what is genuinely broken, get schema correct once, then spend the rest of the budget on content and corroboration.

The weak version produces a report full of green ticks on a site that may still be unreachable. Nobody checked the thing that gates everything else.

Red flags

• llms.txt as a headline recommendation. Named as unnecessary in Google’s own May 2026 guidance.
• Schema coverage as an ongoing monthly metric. It is a fix to complete, not a service to subscribe to.
• “AI-specific” versions of your content. Also named as unnecessary. There is no separate AI index to feed.
• A structured-data audit with no crawl check. Auditing markup on pages nobody has confirmed are reachable is the wrong order.
• Claims that a particular schema type lifts citations. The one controlled study available did not test individual types, so nobody has that evidence yet.
• Guaranteed rich results or citations. Nobody controls generated output, and Google warns against providers guaranteeing rankings.

Five questions worth asking

1. “Have you confirmed our pages are indexed and reachable by each AI crawler?” If not, everything else is premature.
2. “What in Google’s May 2026 guidance changes your recommendations?” Anyone unfamiliar with it is working from last year’s playbook.
3. “What evidence supports each item on this list?” You want a study or the guidance, not category consensus.
4. “Which of these is a one-off fix and which is ongoing?” Schema is the former, and pricing it as the latter is a choice worth questioning.
5. “What would you tell us not to bother with?” A good partner cuts something.

Reduced to one principle: fix the gate, get schema right once, then stop doing technical work and go do content.

One closing note that costs us something. Technical work is satisfying to sell because it produces a tidy report full of resolved items, ours included. It is also the smallest lever once the floor is solid. A partner treating it as the whole programme is selling you the easy part.

What nobody should promise you

That more schema produces more AI citations. A controlled study of 1,885 pages against matched controls found effects indistinguishable from noise, and a small negative for AI Overviews.

That llms.txt gets you into AI answers. Google names it as unnecessary for its AI features.

A guaranteed rich result, citation or position. Nobody controls generated output.

A specific citation lift from a named schema type. The available study explicitly did not test individual types, so anyone quoting that number is estimating.

When technical work is not your problem

The answer that costs us the sale. If your pages are indexed, reachable by the major AI crawlers, render without JavaScript and already carry correct Organization and author schema, your technical work is done and you should stop buying more of it.

At that point the constraint is content depth and third-party corroboration, not markup. We would rather say so than sell another audit. A second technical engagement on a healthy site produces a longer report and no change in the numbers that matter.

Frequently asked questions

Does schema markup help AI search?
It helps machines state facts about your page without guessing, which is worth doing once. A controlled study of 1,885 pages against roughly 4,000 matched controls found no meaningful citation uplift in AI Mode or ChatGPT, and a small decline in AI Overviews.

Should I create an llms.txt file?
Google’s May 2026 guidance names llms.txt as unnecessary for its AI features. It costs little to publish if a stakeholder wants one. It should not displace crawler access checks or content work, and a partner leading with it is not current.

Is structured data still worth implementing?
Yes, once and correctly. Organization details, author information, article type and sameAs links that resolve give machines unambiguous facts. What the evidence does not support is treating expanded markup as an ongoing lever for more citations.

What technical work actually affects AI search visibility?
Crawler access checked per bot, verified indexing, rendering that works without JavaScript, reliable server response, and clean parseable structure. These gate everything else. In our audits, they are where the real breakages sit.

Is there a separate AI index I need to optimise for?
No. Google confirmed in May 2026 that AI Overviews and AI Mode run on core Search ranking and quality systems with no separate AI index and no separate AI ranking algorithm.

Why is my page not appearing in AI answers despite good schema?
Usually because it is not retrievable, not because the markup is wrong. Check the page is indexed, that AI crawlers are not blocked, and that the answer exists in the raw HTML rather than only after JavaScript runs.

Do different AI crawlers need different rules?
Yes, and this catches people out. OpenAI and Perplexity each run separate crawlers for scheduled indexing and on-demand fetching, so blocking one has a different consequence from blocking another.

How much should technical AI-search work cost?
The diagnostic parts are free. Crawl checks and index verification take an afternoon in tools you already have. Real cost appears only when rendering or platform changes are needed. Schema itself is a few hours of developer time, once.

Where to go next

Spend an afternoon on the three free checks before you commission anything: are the pages indexed, can each AI crawler reach them, and does the answer exist in the raw HTML.

Most teams reading an article about schema discover their problem is one of those three. That is cheaper, and more annoying, than the answer they expected.

Learn AI Search · Book a growth audit · Explore Pepper’s platform

Sources and further reading

• Ahrefs. “We tracked 1,885 pages adding schema. AI citations barely moved.” 1,885 pages adding JSON-LD between August 2025 and March 2026, matched against roughly 4,000 control pages, matched difference-in-differences over a 30-day window. AI Overviews minus 4.6 percent, AI Mode plus 2.4 percent, ChatGPT plus 2.2 percent. Limitations stated by the authors. Link
• Google Search Central. AI search optimisation guidance, 15 May 2026, filed under SEO fundamentals. Confirms no separate AI index or AI ranking algorithm, and names llms.txt, content chunking, AI-specific rewrites, inauthentic mentions and structured-data over-optimisation as unnecessary.
• Pepper. SalesHood case study and TVS Eurogrip case study. Figures and quotes as published on those pages.
• Pepper. llms.txt explained and robots.txt for AI crawlers.
• Deliberately excluded: claims that specific schema types lift AI citations by a stated percentage. The one controlled study available did not test individual types, and we found no other controlled work that does.