The short answer
A free AI visibility audit is four checks and a spreadsheet. It takes an afternoon.
Before you run it, one correction. Almost every audit checklist on this topic publishes a flat list of AI crawlers and tells you that blocking any of them makes you invisible on that platform. That is wrong for at least two of the five most commonly listed, and we can show you the vendors’ own documentation.
Key takeaways
- Blocking GPTBot does not hide you from ChatGPT. OpenAI’s docs say GPTBot crawls content “that may be used in training our generative AI foundation models”. The crawler that controls ChatGPT search visibility is OAI-SearchBot.
- Blocking Google-Extended does not hide you from AI Overviews. Google states it “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search”.
- So two of the blocks people make for privacy cost them nothing, while the one that actually matters often goes unchecked.
- The audit itself is free. Search Console, Bing Webmaster Tools, a raw HTML fetch and thirty prompts in a spreadsheet.
- Three of the four checks are technical and come before any content work. If engines cannot reach or read the page, nothing downstream counts.
Where this comes from. We run organic for more than 250 enterprises at Pepper and track over 10 million prompts across every major engine. This is the audit we run in week one of an engagement, given away in full. The crawler table below was read from OpenAI’s, Perplexity’s and Google’s own documentation on 18 September 2026.
What is AI visibility?
AI visibility is whether AI systems mention your brand, cite your pages, and describe you correctly when someone asks a question you should own.
It is three separate things, and collapsing them into one score is the most common mistake in this category.
- Being named. An engine mentions your brand in its answer. This is the awareness half. If you are absent entirely, the problem is usually identity or reputation rather than any page you control.
- Being cited. The engine links a page on your domain as a source. This is the capture half. Being named without being cited means engines talk about you and send people elsewhere.
- Being described correctly. The engine puts you in the right category, with the right product, at the right price. A high score here can still be bad news, because confident and wrong is worse than absent.
The audit below measures all three. Most free checkers measure only the first.
See where you show up. Pepper’s GEO platform tracks Brand Visibility, Domain Prompt Presence and Share of Voice across ChatGPT, Perplexity, Gemini, Claude and Google AI Overviews. Your team can log in, connect Search Console and GA4, manage the prompt set and run your own agents in the Agent Atlas. A growth team works the same account alongside you. Book a growth audit or see where you show up.
The crawler table, read from the vendors’ own docs
This is the part most checklists get wrong. The bots are not interchangeable, and they fail differently.
| Crawler | Token | What it actually does | What blocking it costs you |
|---|---|---|---|
| OAI-SearchBot | OAI-SearchBot/1.4 | Surfaces websites in ChatGPT’s search features | You disappear from ChatGPT search answers |
| ChatGPT-User | ChatGPT-User/1.0 | Visits pages when a user asks ChatGPT to | ChatGPT cannot fetch your page on request |
| GPTBot | GPTBot/1.4 | Crawls content that may train foundation models | Nothing, as far as ChatGPT search visibility goes |
| OAI-AdsBot | OAI-AdsBot/1.0 | Validates pages submitted as ads on ChatGPT | Nothing for organic visibility |
| PerplexityBot | PerplexityBot/1.0 | Surfaces and links sites in Perplexity results, explicitly not for training | You disappear from Perplexity results |
| Perplexity-User | Perplexity-User/1.0 | Visits a page when a user asks a question | Perplexity cannot fetch on demand |
| Google-Extended | Google-Extended | Gemini model training and grounding in Gemini Apps and Vertex AI | Nothing in Google Search or AI Overviews |
| Googlebot | Googlebot | The core Search index | You disappear from Search and AI Overviews |
Read the GPTBot and Google-Extended rows together. A site that blocked both to keep its content out of AI training is still fully visible in ChatGPT search and in AI Overviews. That is usually the intended outcome, and almost nobody realises it.
Now read the OAI-SearchBot row. A site that blocked that one, perhaps by pattern-matching anything starting with “OAI”, is invisible in ChatGPT while believing it is open.
Google’s May 2026 guidance explains why the Google-Extended row reads as it does. AI Overviews and AI Mode run on the core Search ranking and quality systems, with no separate AI index. There is no separate thing to allow. We went through the rest of that guidance in schema, llms.txt and what actually matters.
The indexation stack
Each layer gates the one above it. Auditing out of order wastes the afternoon.
The free AI visibility audit: four checks
Everything here is free and uses tools you already have.
Check one: can they reach you?
Open yourdomain.com/robots.txt and read it against the table above. You are looking for three specific things:
- Is OAI-SearchBot allowed? If not, you are not in ChatGPT search, regardless of what any dashboard says.
- Is PerplexityBot allowed? Same logic for Perplexity.
- Is Googlebot allowed? This one gates both Search and AI Overviews. The per-bot detail is in robots.txt for AI crawlers.
Record a pass or fail per crawler. Do not record a single “AI crawlers: allowed” row, because that is the assumption this whole section exists to break.
Check two: can they read you?
Fetch your most important pages as raw HTML, exactly as a retriever would, and read what comes back.
- Does the answer exist before JavaScript runs? If the substance only appears after a script executes, some retrievers will never see it.
- Are your tables real HTML? A table rendered as an image is invisible to a machine.
- Is the page indexed? Confirm in Search Console rather than assuming. If pages are missing, start with crawlability for AI and can AI bots crawl your site.
Check three: can they lift you?
This is the retrievability check, and it is a reading exercise rather than a tool.
- Do headings read as questions someone would actually type?
- Is the answer in the first two sentences under each heading, before the context and caveats?
- Are the specifics present? Numbers, dates, named methods. A model can paraphrase a generality without you. It cannot invent your sample size. The structural side is covered in how to structure content for AI citation.
Check four: do they cite you?
Now, and only now, run thirty prompts.
Write thirty questions your buyers actually ask, phrased as they would type them. Run each across Google, ChatGPT and Perplexity. Record three columns, not one: named, cited, and described correctly.
The full method, including the spreadsheet columns, is in how to measure AI search visibility without expensive tools.
The template
Copy this. It is the whole audit on one sheet.
| Column | What goes in it | Pass looks like |
|---|---|---|
| Crawler | One row per bot from the table above | Allowed, for the three that matter |
| Robots directive | The exact line from your robots.txt | No disallow on OAI-SearchBot, PerplexityBot or Googlebot |
| Indexed | Yes or no, from Search Console | Yes, for every page you care about |
| Answer in raw HTML | Yes or no, from a raw fetch | Yes, visible before JavaScript |
| Prompt | One row per question, thirty rows | Phrased as a buyer would type it |
| Engine | Google, ChatGPT or Perplexity | All three per prompt |
| Named | Yes or no | Rising across three monthly readings |
| Cited | Yes or no, plus which URL | Rising, and the gap to Named closing |
| Described correctly | Correct, vague or wrong | Correct. Wrong outranks everything else as a priority |
| Sources cited instead | The URLs the engine used | Your earned-media target list |
The last column is the one people leave out, and it is the most useful thing on the sheet. Working that list is its own discipline, covered in [LLM seeding](https://www.pepper.inc/blog/llm-seeding-an-ai-search-strategy-to-get-mentioned-and-cited/). It tells you which third parties the engine already trusts in your category. That matters because **84 percent of AI citations come from earned media**, against 0.3 percent from paid or advertorial content.
The free tools worth using
Four, and two of them are yours already.
- Google Search Console. Free permanently, and the only source of your own first-party data on Google’s AI surfaces. It added Search Generative AI performance reports on 3 June 2026.
- Bing Webmaster Tools. Free permanently. Its AI Performance report opened to public preview on 10 February 2026, showing cited URLs and citation share. Sources disagree on whether it reflects ChatGPT traffic or only Copilot, so read it as directional.
- A raw HTML fetch. Free, using curl or your browser’s view-source. This is the single most revealing check in the audit and it requires no product at all.
- A spreadsheet. Free. The template above, run monthly.
What we deliberately did not include. Free AI visibility checkers that sample a prompt set you did not choose. They are useful for a first glance and weak as an audit, because the prompts are theirs rather than yours. We covered which ones are genuinely free, and which are trials, in the free tools breakdown.
Our methodology: how we weight the audit
| Check | Weight | Why it carries this much |
|---|---|---|
| Crawler access, per bot | 35% | A blocked retriever makes everything else theoretical, and this is the check most often got wrong |
| Indexing and raw-HTML rendering | 30% | An unindexed or script-dependent page cannot be retrieved at all |
| Retrievability of the answer | 20% | Being reachable is not being liftable, and this closes that gap |
| Citation and description accuracy | 15% | The outcome measure. It tells you whether the three above are working |
Free audit options at a glance
| Option | What it covers | Typical cost | Where it falls short |
|---|---|---|---|
| This template, run by hand | All four checks, with prompts you chose | Free, one afternoon plus two hours a month | No history before you start, and it does not scale past a few hundred prompts |
| Search Console and Bing | Your own first-party data on two surfaces | Free | Google and Microsoft only, nothing on ChatGPT, Perplexity or Claude |
| A free visibility checker | A quick sample across a few engines | Free | The prompt set is theirs, so it measures their sample rather than your buyers |
| A free platform tier | One real baseline across several engines | Free credits, then roughly $295/month | Credit grants run out, so it is a baseline rather than a programme |
| A paid tracking platform | Automated, multi-engine, with history | Roughly $99 to $400 a month | Diagnoses, never fixes, and none of them sees Copilot inside tenants |
—
How to evaluate what the audit found
The order matters more than the tooling. Run it top-down and stop when you find the first failure.
The scorecard
Score your site out of 100.
| Area | Weight | How to score it honestly |
|---|---|---|
| Per-crawler access verified | 30% | You checked OAI-SearchBot, PerplexityBot and Googlebot individually, not as one line |
| Indexed and readable without JavaScript | 25% | Confirmed in Search Console and by a raw fetch, rather than assumed |
| Answer-led structure | 20% | Headings are questions, answers come first, tables are text |
| Named, cited and described tracked separately | 15% | Three columns in the sheet, not one score |
| Run on a fixed cadence | 10% | Monthly, on the same prompt set, for at least three readings |
The live test, over 90 days
Fix 30 prompts in your buyers’ words and hold them steady. Real questions, such as “what is AI visibility”, “which data observability platform suits a small team”, “is sales enablement software worth it” and “how do I check if AI bots can crawl my site”.
Run them across Google, ChatGPT and Perplexity monthly for 90 days. Three readings is the shortest honest window: one is a snapshot, two could be noise, three shows direction.
Watch the described-correctly column first. A brand being confidently mis-described is a more urgent problem than a brand being absent, and it is invisible to any tool that only counts mentions.
Then read the two columns together. Named rising while cited stays flat means engines know you and quote somebody else, which is a retrievability or authority problem rather than an awareness one. Both columns flat after three readings usually points back to check one, and it is worth re-reading robots.txt before commissioning any content.
Weak approach versus strong approach
The weaker approach: run a free checker, get a visibility score out of 100, screenshot it for the deck, and change nothing because the score does not say what to fix.
The stronger approach: check the three crawlers individually, fetch your top pages as raw HTML, then run your own thirty prompts with three outcome columns.
The difference is that the weak version produces a number. The strong version produces a list of tickets, and one of them is usually a robots.txt line somebody added two years ago.
Red flags
- A checklist with a flat list of AI crawlers. Two of the five most commonly listed do not affect the visibility people think they do.
- “Block GPTBot to protect your content.” Reasonable if you mean training. It does nothing about ChatGPT search visibility.
- “Allow Google-Extended to appear in AI Overviews.” Google states explicitly that it does not affect inclusion in Search.
- A single AI visibility score. It hides the named-versus-cited gap and cannot show a wrong description at all.
- An audit that starts with content. If the crawlers are blocked, the content review is theoretical.
- Guaranteed citations after an audit. Nobody controls generated output, and Google warns against providers guaranteeing rankings.
Five questions worth asking
- “Which specific crawlers did you check, by name?” If the answer is “AI crawlers”, they checked nothing useful.
- “Did you fetch our pages as raw HTML?” This catches the script-dependent answers no dashboard reports.
- “Whose prompts did you run?” Theirs measures their sample. Yours measures your buyers.
- “How are you separating named from cited?” Two numbers, or the diagnosis is lost.
- “What did you find that we can fix this week?” A good audit produces tickets, not a score.
Reduced to one principle: check the crawlers by name before you check anything else.
One closing note that costs us something. Everything in this article is free, and most teams could run it without us, ours included in the general point that audits are the easiest thing in this category to sell and the cheapest to do yourself. Pay for the fixing, not the finding.
What does a paid audit cost, and when is it worth it?
| What you are buying | Typical 2026 cost | What it covers |
|---|---|---|
| This audit, run in-house | Free, an afternoon | All four checks with your own prompt set |
| Ongoing manual tracking | Free, about two hours a month | Thirty prompts, three engines, three outcome columns |
| A free platform baseline | Free credits | One proper multi-engine reading before you commit |
| Automated tracking | Roughly $99 to $400 a month | History and multi-engine coverage you cannot get by hand |
| Technical remediation | Project cost, varies widely | Fixing rendering or platform issues the audit finds |
| Platform plus a growth team | Custom | Pepper: tracking, plus the people and agents doing the work it points at |
The honest split is that finding the problems is free and fixing them is not. Budget accordingly.
What nobody should promise you
That blocking GPTBot or Google-Extended protects your visibility. Both are training controls, and both vendors say so.
A guaranteed citation after an audit. Nobody controls generated output.
A single visibility score that tells you what to fix. It cannot distinguish absent from mis-described, which are opposite problems.
Historical data from before you started measuring. There is no backfill in this category, which is the honest argument for running the audit now rather than buying more later.
When you do not need to run this
The answer that costs us the sale. If your site is not indexed, you do not need an AI visibility audit yet and we would tell you not to run one.
Fix indexing first. It is free, it takes an afternoon in Search Console, and it gates every other check on this page. An AI visibility audit on an unindexed site produces four failures and one insight you already had.
Frequently asked questions
What is AI visibility?
Whether AI systems mention your brand, cite your pages, and describe you correctly when someone asks a question you should own. Those are three separate outcomes, and collapsing them into a single score hides which one is broken.
How do I run a free AI visibility audit?
Four checks. Read robots.txt against the real crawler list, confirm your pages are indexed and readable without JavaScript, check that answers sit near the top of each page, then run thirty of your own prompts across three engines.
Does blocking GPTBot hide me from ChatGPT?
No. OpenAI’s documentation says GPTBot crawls content that may be used to train foundation models. The crawler that surfaces sites in ChatGPT’s search features is OAI-SearchBot, and that is the one to check.
Does blocking Google-Extended remove me from AI Overviews?
No. Google states that Google-Extended “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search”. It controls Gemini training and grounding, not AI Overviews eligibility.
Which crawlers actually matter for AI visibility?
OAI-SearchBot for ChatGPT search, PerplexityBot for Perplexity, and Googlebot for both Google Search and AI Overviews. The user-triggered fetchers, ChatGPT-User and Perplexity-User, matter when someone asks an engine to visit your page directly.
What free tools should I use for an AI audit?
Google Search Console, Bing Webmaster Tools, a raw HTML fetch and a spreadsheet. The first two are permanently free and report your own data rather than a third-party sample.
How often should I run an AI visibility audit?
Run the technical checks once, then after any site change. Run the thirty-prompt tracking monthly, because you need at least three readings on a fixed set before a trend means anything.
Why does describing me correctly matter more than mentions?
Because a confident wrong answer actively costs you deals, while absence merely fails to win them. A tool that counts mentions will report a wrong description as a success.
Where to go next
Open your robots.txt and search it for OAI-SearchBot, PerplexityBot and Googlebot. That takes two minutes and it is the check most likely to be wrong.
Then copy the template above and run the four checks in order. If check one fails, stop there, because nothing below it counts until it passes.
Learn AI Search · Book a growth audit · Explore Pepper’s platform
Sources and further reading
- OpenAI. Crawler documentation, read 18 September 2026. Lists OAI-SearchBot (“used to surface websites in search results in ChatGPT’s search features”), GPTBot (“used to crawl content that may be used in training our generative AI foundation models”), ChatGPT-User and OAI-AdsBot. Link
- Perplexity. Crawler documentation, read 18 September 2026. PerplexityBot is “designed to surface and link websites in search results on Perplexity. It is not used to crawl content for AI foundation models.” Perplexity-User handles user-triggered fetches. Link
- Google Search Central. Google crawlers documentation, read 18 September 2026. States Google-Extended “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search”, and that it manages use for training future Gemini models and grounding in Gemini Apps and Vertex AI. Link
- Google Search Central. AI search optimisation guidance, 15 May 2026. Confirms AI Overviews and AI Mode run on core Search ranking and quality systems with no separate AI index, and warns against guaranteed-ranking claims.
- Muck Rack. “What Is AI Reading?” May 2026. More than 25 million cited links across ChatGPT, Claude and Gemini. Source of the 84 percent earned media and 0.3 percent paid figures. Link
- Microsoft. Bing Webmaster Tools AI Performance report, public preview from 10 February 2026.
- Deliberately excluded: any figure for the share of AI traffic each crawler represents, or for how many sites block each bot. Neither is published by the engines, and the circulating numbers come from third-party samples with no stated method.
Latest Blogs
A fill-in one-page business case for GEO, the four objections you will actually get, and what to do when finance says prove it first. The maths lives in our ROI model; this is the artefact you take into the room.
A single AI visibility score is one number standing in for several engines, dozens of prompts and two different failure modes. Here is what it hides, when it is still useful, and what to ask before you trust one.
There is no single best AI SEO tool, and the reason is structural. The category is five different jobs bought separately, with different winners in each, and two of the five cost nothing at all. Here is the full stack and what actually integrates with what.
Get your hands on the latest news!
