What is llms.txt, and should you use it?

The short answer

For AI search visibility, no. For developer documentation, possibly.

Ahrefs checked 137,210 domains with traffic in May 2026, then read the server logs. 28 percent had published an llms.txt file. Of those, 97 percent received zero requests that month. Nothing fetched them at all.

Of the small share that did get traffic, the largest category of requester was SEO audit tools, at 21.7 percent. The main thing reading your llms.txt is the tool that flagged you for not having one.

Key takeaways

• The file is not being read. 97 percent got zero requests in a month, across a sample that skews technical.
• AI bots do not look for it. The authors found no evidence of AI bots requesting llms.txt files that do not exist. They never go looking.
• Google has said no twice. Gary Illyes said Google does not support it and is not planning to. Google’s own generative AI guidance says files like this are not needed.
• The exception is real but narrow. Anthropic, Cursor and Mintlify support it for developer documentation, which is a different job from web search.
• Building one costs little. The problem is what it displaces. The hour goes to a file nothing reads, instead of a page engines actually retrieve.

Where this comes from. We run organic for more than 250 enterprises at Pepper and track over 10 million prompts across every major engine. This question comes up in nearly every technical review we run, usually because an audit tool flagged it. That work shapes the view here, along with the client reviews we sit in and the conversations we have at the events we run.

What is llms.txt?

An llms.txt file is a Markdown file at your domain root. It lists your most important pages, so a language model can find them without crawling your whole site.

Three things are worth knowing about its status.

• It is a proposal rather than a standard. It was published as an open idea and picked up by some tooling. No search engine or model provider ratified it.
• It is not robots.txt. Robots.txt is honoured by crawlers and controls access. This file requests attention, and nothing is obliged to read it. The crawler table covers what does get honoured.
• Its sibling is llms-full.txt. A larger variant containing full page content, which carries the same evidence problem. Neither appears in the tooling stacks we reviewed as a measurable lever.

The idea is sensible. Whether anything consumes it is an empirical question, and that question now has an answer. For the wider technical picture, see schema markup and AI search.

See where you show up. Pepper’s GEO platform tracks Brand Visibility, Domain Prompt Presence and Share of Voice across ChatGPT, Perplexity, Gemini, Claude and Google AI Overviews. Your team can log in, connect Search Console and GA4, manage the prompt set and run your own agents in the Agent Atlas. A growth team works the same account alongside you. Book a growth audit or see where you show up.

Should you use llms.txt?

The evidence is unusually direct, because somebody read the server logs rather than speculating.

Ahrefs took 137,210 domains using its Web Analytics product that received traffic in May 2026. It checked each domain root for an llms.txt returning HTTP 200. It validated that each file was genuine Markdown rather than HTML or a soft 404. Then it examined every request to those paths in Bot Analytics.

On the sample’s limits, which the authors state. Ahrefs Web Analytics customers skew more technical and SEO-aware than the web at large. So the authors ask you to treat 28 percent adoption as an upper bound. That caution cuts in an interesting direction. This population is more likely than average to have the file and to be crawled attentively, and the files still went unread.

The authors’ own conclusion is blunt. If your goal is showing up in ChatGPT, Perplexity or AI Overviews, an llms.txt file is largely decoration.

Who actually reads these files

This is the finding that settles the argument, and it is funnier than it should be.

Among the requests that did arrive, SEO audit tools were the largest single category. Add tech profiling tools at 11.6 percent and GEO tools at 5.8 percent. Roughly 39 percent of the traffic is software that inspects sites for a living.

Named AI tools accounted for 19.5 percent of fetches. That sounds better until you remember the denominator. It is 19.5 percent of the 3 percent of files that got any request at all.

The most decisive detail is not in the percentages. The authors found that no AI bots request llms.txt files that do not exist. They never go looking. A crawler that wanted this file would probe for it, find nothing, and retry later. That is not happening.

What Google has said

Twice, and in the same direction.

In July 2025, Gary Illyes of Google confirmed that Google does not support llms.txt and is not planning to. John Mueller separately compared it to the keywords meta tag. That signal was abandoned decades ago, because it described a page rather than proving anything about it.

Then in late May 2026, Google published guidance on optimising for its generative AI features. It tells site owners that machine-readable files of this kind are not needed to appear in generative AI search. It also reiterates that AI Overviews and AI Mode run on core Search systems, with no separate AI index.

So the largest engine has said it does not use the file, does not plan to, and that you do not need it. That is about as clear as this category gets. The same guidance underpins how a results page is now assembled.

When it is genuinely worth building

There is a real exception, and it is narrower than the guides suggest.

Developer documentation. Anthropic, Cursor and Mintlify support llms.txt for docs, and several documentation platforms generate one automatically. Say you publish an API reference or a developer portal. A coding assistant pulling those docs into a context window is a genuine use case, and the file helps it.

Note what that is not. It is a documentation delivery format for AI coding tools, rather than a visibility signal for web search. Those are different jobs, and most articles blur them.

Two more cases where we would not argue against it:

• Your documentation platform generates it automatically. Then it costs nothing and you can ignore it.
• An enterprise client requires it for a compliance checklist. A cheap box to tick, provided nobody believes it is doing something else.

What to do with the hour instead

The file is cheap. The opportunity cost is the real argument.

An hour spent on llms.txt is an hour not spent on the checks that decide retrieval. We set those out in the free AI visibility audit, where the ordering matters more than any single item.

• Check your crawlers individually. Blocking GPTBot does not hide you from ChatGPT search. Blocking Google-Extended does not remove you from AI Overviews. Both beliefs are common, and both are wrong.
• Fetch your pages as raw HTML. If the answer only appears after JavaScript runs, some retrievers will never see it.
• Put the answer near the top. Engines score passages. A buried answer loses to a thinner page that leads with one.
• Write something a model cannot compose. A price, a sample size, a named method. This is the part no file substitutes for, and the citation evidence shows why it decides so much.

Our methodology: how we weighted this

**What we could not verify and left out.** A widely shared figure claiming only 408 requests to llms.txt across more than 500 million AI bot visits over 90 days. The number is striking and points the same way as everything above, and we could not trace it to a named study with a published method, so it plays no part in the argument.

The options at a glance

How to choose whether to build one

One question decides it, and it is about your content type rather than your ambition.

The scorecard

Score the decision out of 100.

Below roughly 40, skip it. The honest score for most marketing sites is under 20, and the effort is better spent on earned third-party coverage.

The live test, over 90 days

You can settle this on your own domain, and it costs nothing.

Publish an llms.txt if you want, then watch your server logs for requests to that path. Separately, fix 30 prompts in your buyers’ words and run them across Google, ChatGPT and Perplexity. Real questions, such as “what is llms.txt”, “best data observability platform for a small team”, “how do I get cited by ChatGPT” and “does llms.txt help with AI search”.

Record whether you are named and whether a page of yours is cited, monthly for 90 days. The free method is in how to measure AI search visibility without expensive tools. Three readings is the shortest honest window. One is a snapshot, two could be noise, three shows direction. Keep the set fixed, because changing it resets the comparison.

Then compare the two. If your citations move while nothing ever requests the file, you have your answer. Better still, you generated it from your own logs rather than taking ours.

Weak approach versus strong approach

The weaker approach: run an audit tool, see llms.txt flagged, build one, then mark the issue resolved believing something improved.

The stronger approach: check the server logs for whether anything reads it, and spend the hour on crawler access and retrievability instead.

The weak version optimises for a green tick in a tool. The strong version optimises for being retrievable, which is what the tick was supposed to represent.

Red flags

• An audit tool scoring you down for a missing llms.txt. SEO audit tools are the largest category of requester at 21.7 percent. That tells you who the file is really for.
• A vendor promising AI visibility gains from llms.txt. Nothing in the server-log evidence supports that.
• Guides that conflate documentation delivery with search visibility. They are different jobs, and only one has a working use case.
• “Google might support it later.” They have said twice that they do not and will not, most recently in their own generative AI guidance.
• llms-full.txt as a content strategy. Same evidence problem, with ongoing maintenance attached.
• Guaranteed citations from any file. Nobody controls generated output, and Google warns against providers guaranteeing rankings.

Five questions worth asking

1. “Has anything actually requested our llms.txt?” Your own server logs answer this in minutes.
2. “Do we publish developer documentation?” If no, the real use case does not apply to you.
3. “What would we do with that hour instead?” Usually something with evidence behind it.
4. “Which crawlers can reach us, by name?” The check that matters, and the one most often wrong.
5. “What is this supposed to improve, and how would we see it?” If nobody can answer, it is a box being ticked.

Reduced to one principle: read your server logs before you believe a checklist.

One closing note that costs us something. Shipping a file is satisfying, easy to report and easy to invoice. Agencies prefer deliverables that look like progress, ours included. The unglamorous version is checking three crawlers by name and fixing a rendering problem. It is free, and it decides whether you are retrieved.

What does it cost?

The first row is free, takes minutes, and will probably end the discussion in your organisation.

What nobody should promise you

AI visibility gains from publishing llms.txt. Across 137,210 domains, 97 percent of these files were never requested.

That Google will start using it. Google has said it does not and will not, and its own generative AI guidance says the file is not needed.

That AI crawlers will discover it. The study found no AI bots requesting llms.txt files that do not exist.

A guaranteed citation from any technical file. Nobody controls generated output.

When to build one anyway

The answer that costs us the sale on the opposite side. If you publish developer documentation, you probably should build one, and we would say so even though it is not work we would charge for.

Coding assistants pulling your API reference into a context window is a real use case, with real support from Anthropic, Cursor and Mintlify. If your documentation platform generates the file automatically, that is free value and you should leave it on. Just do not expect it to change anything on your marketing pages. A different set of systems reads those.

Frequently asked questions

What is llms.txt?
A Markdown file at your domain root, listing your most important pages so a language model can find them without crawling everything. It is a proposal picked up by some tooling, rather than a standard ratified by search engines or model providers.

Should I use llms.txt?
For AI search visibility, no. Across 137,210 domains, 97 percent of these files received zero requests in May 2026. For developer documentation it is a different question, and the answer there is often yes.

Does Google use llms.txt?
No. Gary Illyes confirmed in July 2025 that Google does not support it and is not planning to. Google’s own generative AI guidance from May 2026 says machine-readable files of this kind are not needed.

Do AI crawlers look for llms.txt?
The evidence says no. The study found no AI bots requesting llms.txt files at domains without one. Nothing is probing for the file and finding it absent.

Who actually reads llms.txt files?
Mostly tooling. Of the small share of files receiving any request, SEO audit tools were the largest single category at 21.7 percent, with tech profiling tools at 11.6 percent and GEO tools at 5.8 percent.

Is llms.txt the same as robots.txt?
No. Robots.txt controls crawler access and every major crawler honours it. This file requests attention, and nothing is obliged to read it. The server-log evidence bears that out.

What about llms-full.txt?
A larger variant carrying full page content rather than just links. It faces the same evidence problem as the shorter file, since nothing was observed fetching either, and it carries meaningfully more maintenance. If your docs platform generates one automatically, leave it on and ignore it.

What should I do instead?
Check your crawlers individually by name. Fetch your key pages as raw HTML to confirm the answer exists before JavaScript runs. Then move answers near the top of each page. Those three decide retrieval.

Where to go next

Open your server logs and search for requests to /llms.txt. That takes minutes and it will probably end the conversation.

Then spend the hour you saved checking whether OAI-SearchBot, PerplexityBot and Googlebot can reach you. That is the check most often wrong.

Run the technical audit · Learn AI Search · Book a growth audit

Sources and further reading

• Ahrefs. “We analyzed 137K sites: 97% of llms.txt files never get read.” Study of 137,210 domains using Ahrefs Web Analytics that received traffic in May 2026, using Web Analytics and Bot Analytics server logs. Each domain root was checked for an llms.txt returning HTTP 200, validated as genuine Markdown rather than HTML or a soft 404, with every request to those paths classified by HTTP response and user agent. 28 percent published a file, roughly 38,360 domains. 97 percent of those files received zero traffic in May 2026. Of the 3 percent that received any, 96 percent of requests came from bots. Requester breakdown: SEO audit tools 21.7 percent, unknown 14.9 percent, general web crawlers 13.1 percent, tech profiling tools 11.6 percent, AI agents 10.5 percent, GEO and AEO tools 5.8 percent, AI training crawlers 5.3 percent, llms.txt discovery bots 3.6 percent, service and social bots 2.9 percent, research bots 2.7 percent, AI assistants 2.5 percent, AI retrieval bots 1.1 percent. Named AI tools accounted for 19.5 percent of fetches. No AI bots were observed requesting llms.txt files that do not exist. The authors state that Ahrefs Web Analytics customers skew more technical and SEO-aware than the web at large, so the 28 percent adoption figure should be treated as an upper bound. Their conclusion: for showing up in ChatGPT, Perplexity or AI Overviews, an llms.txt file is largely decoration. Link
• Google. Gary Illyes, July 2025, confirming Google does not support llms.txt and is not planning to. John Mueller separately compared it to the keywords meta tag.
• Google Search Central. Guidance on optimising for generative AI features, late May 2026. States that machine-readable files of this kind are not needed to appear in generative AI search, and confirms AI Overviews and AI Mode run on core Search ranking and quality systems with no separate AI index. Also warns against providers guaranteeing rankings.
• Anthropic, Cursor and Mintlify support llms.txt for developer documentation, which is a documentation delivery use case rather than a web search visibility one.
• Pepper. The free AI visibility audit for the checks that do decide retrieval, schema markup and technical AI search work for the wider technical picture, and how to write content that AI engines actually cite for the part no file substitutes for.
• Deliberately excluded: a widely shared figure claiming only 408 requests to llms.txt across more than 500 million AI bot visits over 90 days. It points the same way as the evidence above, and we could not trace it to a named study with a published method.