Can AI search bots crawl my website? A technical guide for marketers

The short answer

Probably, but the question hides a more useful one. Which bots, and for what?

Every major AI vendor now runs the same three-way split. A training crawler, a search crawler, and a user-triggered fetcher. They do different jobs and blocking them has completely different consequences.

Get that pattern and the whole subject becomes simple. Miss it and you can block the wrong one for years without noticing.

Key takeaways

• The pattern repeats across vendors. OpenAI, Anthropic, Perplexity and Google all separate training from answering.
• Blocking a training bot costs you nothing in search visibility. GPTBot and ClaudeBot feed model training, not the answer surfaces.
• Blocking a search bot makes you invisible on that platform. OAI-SearchBot, Claude-SearchBot and PerplexityBot are the ones that matter.
• Sophisticated publishers already do this. Across roughly 4,000 robots.txt files, GPTBot was disallowed 2.33 times for every allow, while OAI-SearchBot came out net allowed.
• Blanket rules are the common failure. A rule matching everything with “GPT” or “AI” in the name takes out the answering bots too.

Where this comes from. We run organic for more than 250 enterprises at Pepper and track over 10 million prompts across every major engine. We read robots.txt files for a living and find this wrong often. We read every crawler description below from the vendor’s own documentation on 30 September 2026, and the dates are stated.

What are AI crawlers?

AI crawlers are automated agents that fetch web pages on behalf of an AI company. The important thing is that a single vendor runs several, and they are not interchangeable.

Three jobs exist, and almost every vendor now has one bot for each.

• Training crawlers collect content that may train future models. Blocking one affects what those models learn from. It does not remove you from anything users search today.
• Search crawlers build and maintain the index answers draw from. Blocking one removes you from that platform’s answers. This is the consequential one.
• User-triggered fetchers visit a page because a person asked the assistant to look at it. Blocking one stops the assistant opening your page on request.

Most confusion here comes from treating all three as “AI bots” and writing one rule for the lot. What a results page now contains explains why the search crawler matters more than it used to.

See where you show up. Pepper’s GEO platform tracks Brand Visibility, Domain Prompt Presence and Share of Voice across ChatGPT, Perplexity, Gemini, Claude and Google AI Overviews. Your team can log in, connect Search Console and GA4, manage the prompt set and run your own agents in the Agent Atlas. A growth team works the same account alongside you. Book a growth audit or see where you show up.

The full crawler table, read from vendor documentation

Eleven crawlers across four vendors. We quoted or paraphrased every purpose below from the vendor’s own docs, read on 30 September 2026.

Read the table by column rather than by row. The vendor barely matters. The **job** decides the consequence, and the same three jobs repeat everywhere. Coverage differs sharply once you are through the door, as [the top AI search engines](https://www.pepper.inc/blog/top-ai-search-engines-in-2026-the-complete-overview/) sets out.

Two rows catch people out most often. GPTBot is training only, so blocking it for privacy leaves you fully visible in ChatGPT search. And Google-Extended does not affect AI Overviews, because Google states it does not impact inclusion in Google Search or act as a ranking signal. AI Overviews and AI Mode run on core Search systems with no separate AI index.

What publishers actually do

The market has already learned this distinction, which is a useful check on whether the framing above is real.

An analysis built on Cloudflare Radar data examined roughly 4,000 robots.txt files per snapshot, with a Q1 2026 baseline refreshed through 31 July 2026. The split is stark.

• GPTBot: 696 disallow rules against 299 allows. A block-to-allow ratio of 2.33 to 1.
• ClaudeBot: 619 against 259. A ratio of 2.39 to 1.
• OAI-SearchBot: 250 against 266. Net allowed.
• ChatGPT-User: 300 against 267. Close to parity.

The authors put it plainly. Publishers are not blocking AI. They are blocking training and allowing answering.

That is the sophisticated position, and it is available to you in ten minutes of editing. The unsophisticated position, a blanket rule against anything that looks like an AI bot, gives away the answer surfaces to protect training data that was mostly collected already.

How to check, rather than assume

Four steps, all free, and none needs a developer.

The order matters because each step rules out a different failure. Most teams stop after the first and miss the ones that actually bite.

1. Read your robots.txt against the table

Open yourdomain.com/robots.txt and search it for each token in the table above. You are looking for three specific things.

• A wildcard rule that catches search bots. A Disallow under a pattern matching “GPT” or “AI” will take out OAI-SearchBot along with GPTBot.
• An explicit block on a search crawler. OAI-SearchBot, Claude-SearchBot, PerplexityBot or Googlebot.
• A stale blanket block, added during the 2023 wave of AI blocking, before vendors separated the bots.

Record a pass or fail per crawler. Do not record one line reading “AI crawlers: allowed”, because that is the assumption this article exists to break. The same discipline applies to measuring visibility, where one blended score hides the diagnosis.

2. Check subdomains separately

Anthropic’s documentation states this explicitly, and it applies generally. Robots.txt works per subdomain.

A correct rule on www.example.com does nothing for docs.example.com or blog.example.com. If your content lives on a subdomain, check that subdomain’s file directly.

3. Look at the server logs

This is the step that turns assumption into evidence, and marketers can usually get it without writing code.

Ask whoever owns your hosting for a list of user agents hitting the site over the last 30 days. You want to see whether OAI-SearchBot, PerplexityBot and Googlebot actually turn up. A crawler that is allowed but never visits is a different problem from one that is blocked, and robots.txt cannot tell them apart. If nothing arrives at all, the constraint is usually ranking rather than access, which the writing evidence covers.

4. Check what a bot would actually receive

A page can be reachable and still unreadable.

Fetch your most important pages as raw HTML and read what comes back. If the answer only appears once JavaScript runs, some retrievers never see it. That check, plus the template for recording all of this, is in the free AI visibility audit.

Our methodology: how we weighted the checks

**What we could not verify and left out.** Any single figure for what share of the web blocks AI crawlers. The published numbers span roughly 9 percent to 54 percent, because the studies sample different populations and count different bots. We have used one study’s within-sample comparison between bots, which is internally consistent, rather than treating any headline rate as the truth.

The crawler jobs at a glance

—

How to evaluate a crawler access proposal

Somebody will eventually recommend blocking or unblocking something. This is how to judge it.

The scorecard

Score any proposal out of 100.

The live test, over 90 days

Fix 30 prompts in your buyers’ words and run them across Google, ChatGPT and Perplexity. Real questions, such as “can AI bots crawl my website”, “best data observability platform for a small team”, “how much does sales enablement software cost” and “which GEO agency works with fintech”.

Record whether you are named and whether a page of yours is cited, per engine.

Then do the thing this article is really about. Compare your results per engine against your robots.txt. If you are absent from exactly one platform and present on the others, look at that platform’s search crawler first. That pattern is diagnostic, and it is far faster than a general audit.

Rerun the identical set monthly for 90 days. Three readings is the shortest honest window. One is a snapshot, two could be noise, three shows direction. Keep the set fixed, because changing it resets the comparison.

Weak approach versus strong approach

The weaker approach: add a blanket disallow for anything that looks like an AI crawler, feel protected, and wonder later why you never appear in ChatGPT.

The stronger approach: allow every search and user-triggered crawler by name, decide on training crawlers as a policy question, and confirm in the logs that the search bots arrive.

The weak version treats a visibility decision as a privacy decision. Those are separate choices and the vendors have separated the bots so you can make them separately.

Red flags

• A proposal that says “AI crawlers” without naming any. There are eleven across four vendors and they do three different jobs.
• A wildcard rule matching “GPT” or “AI”. It will catch the search bots along with the training ones.
• Blocking by IP address. Anthropic’s documentation warns this stops the crawler reading robots.txt at all.
• Only the root domain checked. Robots.txt is per subdomain, and content often lives on one.
• “Block Google-Extended to stay out of AI Overviews.” It does not affect AI Overviews, and Google says so.
• Guaranteed visibility from unblocking. Access is eligibility, not citation, and nobody controls generated output.

Five questions worth asking

1. “Which crawlers did you check, by name?” If the answer is a category, nothing useful was checked.
2. “Which of those are search bots?” The person should answer without looking.
3. “Did you check every subdomain?” The most common silent gap.
4. “Have you seen these bots in our logs?” Allowed and arriving are different facts.
5. “What are we protecting by blocking this?” Often the honest answer is training data already collected.

Reduced to one principle: decide per job, not per vendor.

One closing note that costs us something. This is a ten-minute check on a text file and it does not need an agency, ours included in the general point that technical access work is easy to bill and cheap to do. If a partner proposes a paid engagement to read your robots.txt, ask them to send you the four steps above instead.

What does this cost?

The first four rows are free, and together they answer this article’s question properly.

What nobody should promise you

Visibility from unblocking a crawler. Access makes you eligible to be retrieved. It does not make you cited.

That blocking GPTBot protects you from ChatGPT. It is a training crawler. OAI-SearchBot is the one that surfaces sites in ChatGPT search.

That blocking Google-Extended removes you from AI Overviews. Google states it does not impact inclusion in Google Search.

A complete list of AI crawlers that stays current. Vendors add them. Claude-SearchBot is new in 2026, and this table will need revisiting.

When this is not your problem

The answer that costs us the sale. If your robots.txt has no AI-specific rules at all, you do not need to do anything here and we would tell you not to buy a technical engagement.

The default state is allowed. A file with no AI rules is already correct for visibility, and the only remaining question is whether the pages render without JavaScript. That is a five-minute check you can do yourself. Come back to this the day somebody proposes adding a blocking rule, because that is when the distinctions above start costing money.

Frequently asked questions

Can AI search bots crawl my website?

By default yes, unless your robots.txt blocks them. The more useful question is which ones, because every major vendor runs a training crawler, a search crawler and a user-triggered fetcher, and only the search crawler decides whether you appear in answers.

Does blocking GPTBot hide me from ChatGPT?

No. OpenAI describes GPTBot as crawling content that may train foundation models. The crawler that surfaces sites in ChatGPT’s search features is OAI-SearchBot, and that is the one to check.

Which AI crawlers should I allow?

The search and user-triggered ones, at minimum. That means OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot, plus ChatGPT-User, Claude-User and Perplexity-User for on-demand fetches. Training crawlers are a separate policy decision with no search visibility cost.

What crawlers does Anthropic run?

Three. ClaudeBot collects content that could contribute to training, Claude-SearchBot navigates the web to improve search result quality, and Claude-User accesses sites when someone asks Claude a question.

Does Google-Extended affect AI Overviews?

No. Google states that Google-Extended does not impact a site’s inclusion in Google Search or act as a ranking signal. It governs Gemini training and grounding, and AI Overviews run on core Search systems.

How many sites block AI crawlers?

There is no reliable single figure, and published estimates span roughly 9 to 54 percent because studies sample different populations and count different bots. The within-sample pattern is more useful: training bots are blocked far more than search bots.

Do I need to check subdomains separately?

Yes. Robots.txt applies per subdomain, so a correct file at your root domain does nothing for a docs or blog subdomain. This is the most common silent gap we find.

Is blocking AI crawlers by IP address a good idea?

No. Anthropic’s documentation warns against it, because blocking by IP prevents the crawler from reading your robots.txt directives at all, which removes your ability to give it instructions.

Where to go next

Open your robots.txt and search it for OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot. Four searches, two minutes.

If any of them is disallowed, you have found something worth more than most content projects. If none is, check your subdomains next, because that is where this usually hides. Then move up to earned coverage, which is where the largest remaining gains sit.

Run the full audit · Learn AI Search · Book a growth audit

Sources and further reading

Studies and primary sources

• OpenAI. Crawler documentation, read 30 September 2026. OAI-SearchBot surfaces websites in ChatGPT’s search features. GPTBot crawls content that may be used in training generative AI foundation models. ChatGPT-User handles user actions in ChatGPT and Custom GPTs. OAI-AdsBot validates the safety of pages submitted as ads. Link
• Anthropic. Crawler documentation, read 30 September 2026. ClaudeBot “helps enhance the utility and safety of our generative AI models by collecting web content that could potentially contribute to their training.” Claude-SearchBot “navigates the web to improve search result quality for users. It analyzes online content specifically to enhance the relevance and accuracy of search responses.” Claude-User “supports Claude AI users. When individuals ask questions to Claude, it may access websites using a Claude-User agent.” The documentation states that robots.txt must be modified per subdomain, and that blocking by IP address is discouraged because it prevents the crawler reading robots.txt directives. Link
• Perplexity. Crawler documentation, read 30 September 2026. PerplexityBot is “designed to surface and link websites in search results on Perplexity. It is not used to crawl content for AI foundation models.” Perplexity-User supports user actions. Link

Supporting sources

• Google Search Central. Crawler documentation, read 30 September 2026. Google-Extended “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search”, and manages use for training Gemini models and grounding in Gemini Apps and Vertex AI. Link
• Robots.txt analysis built on Cloudflare Radar data, roughly 4,000 to 4,200 robots.txt files per snapshot, Q1 2026 baseline refreshed through 31 July 2026, published 3 April 2026 and updated 3 September 2026. GPTBot 696 disallow against 299 allow, ClaudeBot 619 against 259, OAI-SearchBot 250 against 266, ChatGPT-User 300 against 267. The authors conclude that publishers are blocking training and allowing answering. Link
• Google Search Central. AI search optimisation guidance, 15 May 2026. Confirms AI Overviews and AI Mode run on core Search ranking and quality systems with no separate AI index, and warns against providers guaranteeing rankings.

Further reading

• Pepper. The free AI visibility audit for the full template, and how to optimise for AI search for where crawler access sits in the wider sequence.
• Deliberately excluded: any single headline figure for what share of the web blocks AI crawlers. Published estimates span roughly 9 to 54 percent because the studies sample different populations and count different bots.