smart_toy GEO & AI Search Diagnostic

AI Crawler Access Checker

Check whether your robots.txt allows or blocks each AI crawler — the retrieval agents that decide citations, and the ones that only train models.

What Is an AI Crawler?

An AI crawler is a bot that fetches web pages on behalf of an AI system. Two jobs are routinely conflated: collecting pages for a training corpus, and fetching a page while answering a live question. They are different agents with different consequences, and blocking one does not necessarily affect the other.

Most AI crawlers identify themselves in the User-Agent string and are expected to follow robots.txt. That protocol is voluntary: it is a declaration of preference, not an access control. A crawler can also be stopped before it ever reads your HTML — by a CDN or WAF, a rate limit, a login wall, or a page that only renders its content with JavaScript.

For citation, the retrieval agents are the ones that matter. If a retrieval agent cannot fetch the page, the model cannot quote it, regardless of how well the page ranks in Google.

Why AI Crawler Access Matters for GEO (Generative Engine Optimization)

As AI-powered engines like ChatGPT, Claude, Perplexity, and Google Gemini handle more user search queries, getting your content cited by AI models requires proper crawler access.

Retrieval agents and training agents are not the same thing

Every AI vendor runs more than one crawler, and they do different jobs. A training agent collects data to train a model; blocking it declines that use and nothing else. A retrieval agent fetches a page while answering a question, and it is the one that decides whether you can be cited.

Sites routinely get this backwards — allowing the training agent while blocking the retrieval ones, which gives up the citation without protecting the content.

What each crawler actually governs

OAI-SearchBot, ChatGPT-User (OpenAI): decide ChatGPT citations. The first serves ChatGPT's search; the second fires when a user asks ChatGPT to open a URL. Blocking either costs citations.
GPTBot (OpenAI): training only. Blocking it keeps your content out of future GPT training data and has no bearing on whether ChatGPT can cite you.
Claude-SearchBot, Claude-User (Anthropic): decide Claude citations, fetching pages while answering.
ClaudeBot (Anthropic): training only, like GPTBot. Blocking it does not affect citation.
PerplexityBot, Perplexity-User (Perplexity): decide Perplexity citations. Perplexity answers lean heavily on live retrieval, so blocking these removes you from its results.
Googlebot (Google): both the ordinary search crawler and the source for AI Overviews.
Google-Extended (Google): the exception. One token covers Gemini training and grounding — the two cannot be separated — so blocking it gives up Gemini citations as well. It is widely mistaken for a training-only switch.
Applebot-Extended, Bytespider, CCBot: training or dataset use; none of them decide citation.

One robots.txt rule that catches people out

If a crawler has a User-agent group of its own, the User-agent: * group does not apply to it at all — the specific group replaces the wildcard rather than adding to it. Most "but I disallowed it under *" confusion comes from this.

Access is the precondition, not the goal. How to become the source AI answers cite covers what decides citation once the crawlers can reach you.

How to Test AI Crawler Access

Testing access is not the same as opening the page in a browser. The check has to follow the crawler's path: the robots.txt rule for that specific agent, the server response, the CDN layer in front of it, and the HTML the crawler actually receives.

Read the rule for the exact agent. A group written for GPTBot is separate from the User-agent: * group. If the named group exists, the wildcard does not apply to that crawler at all.
Fetch the URL with the agent's user agent. A request with -A "GPTBot" shows what your server returns for that string, but it does not prove the vendor's infrastructure can reach you through your CDN. Use it as a server-side check, not as proof of access.
Check the CDN or WAF in front of the origin. Bot management, managed WAF rules, and "block AI bots" switches can return a challenge or a 403 before the request reaches your application. The firewall log is the authoritative record.
Verify the rendered response. If the content only exists after client-side JavaScript runs, a crawler that does not execute JavaScript sees an empty shell. Fetch the raw HTML or use the AI raw HTML viewer to see what a non-JS crawler receives.
Check for login walls, geo-blocks, and rate limits. A public marketing page behind a login or a country block is invisible to a crawler in another region.
Re-test after each change. A robots.txt edit, a CDN rule, and a deploy can each change the answer; test the same URL and the same agent after each one.

robots.txt Rules for AI Crawlers

The rule that causes the most confusion is precedence: a named User-agent group replaces the wildcard group for that crawler. The example below opts out of OpenAI's training crawler while keeping its retrieval agent, opts out of Anthropic's training crawler while keeping its retrieval agents, and allows Perplexity's live retrieval.

# Training crawler: opt out of model training
User-agent: GPTBot
Disallow: /

# Retrieval crawler: keep ChatGPT citations
User-agent: OAI-SearchBot
Allow: /

# Training crawler: opt out of model training
User-agent: ClaudeBot
Disallow: /

# Retrieval crawler: keep Claude citations
User-agent: Claude-SearchBot
Allow: /

# Perplexity retrieves live pages to answer
User-agent: PerplexityBot
Allow: /

Sitemap: https://example.com/sitemap.xml

Two cautions. First, a named User-agent group replaces the wildcard group for that crawler, and rules within a group are resolved by specificity; when two rules are equally specific, the least restrictive one wins. Keep each crawler in its own group, as above. Second, robots.txt is advisory; a determined crawler can ignore it, and a CDN rule can still block a crawler that robots.txt allows. Use robots.txt for policy and the CDN for enforcement.

If you are unsure whether a rule resolves the way you expect, the robots.txt tester validates the syntax and the group structure before you deploy it.

Why AI Crawlers Get Blocked

Legacy robots.txt rules. An SEO plugin or an old security template can leave Disallow: / under a broad group that happens to match an AI agent.
CDN bot management. "Block AI bots" toggles, Bot Fight Mode, and managed WAF rules are designed to stop automation; some of them stop retrieval agents along with scrapers.
JavaScript-only content. The server returns an empty shell and the crawler never executes the client-side code that would have rendered the text.
Login walls and paywalls. The page is public to a browser with a session but not to an anonymous crawler.
Rate limits and origin errors. A 429 or a 5xx during a crawl is indistinguishable from a block; the crawler simply does not get the page.
Geo-blocking. A country-level rule silently removes the site from crawlers in that region.

An X-Robots-Tag: noindex is a different problem: it does not stop the fetch, it stops the indexing. A crawler can read the page and still decline to use it as a citation source if the directive or the canonical points elsewhere.

AI Crawler Access by Platform

Platform	Retrieval agent(s)	Training agent / control token	What blocking retrieval costs
ChatGPT	OAI-SearchBot, ChatGPT-User	GPTBot	Live search results and user-triggered page reads
Claude	Claude-SearchBot, Claude-User	ClaudeBot	Answers that would otherwise cite the page
Perplexity	PerplexityBot, Perplexity-User	No separate training agent documented	Perplexity's live retrieval and source list
Gemini / AI Overviews	Googlebot	Google-Extended (control token for Gemini training and grounding)	Gemini grounding; AI Overviews depend on Googlebot

The exact user-agent strings and the jobs they perform are listed in the AI crawlers glossary entry. The broader GEO workflow is covered in the GEO and AI visibility guide.

How to Fix Blocked AI Crawlers

Allow the retrieval agents first. Check that OAI-SearchBot, Claude-SearchBot, PerplexityBot, and Googlebot are not caught by a named group or by a wildcard rule you did not intend.
Check the CDN and WAF logs. Look for 403s and challenge pages from the AI agents' IP ranges; allow verified bots rather than disabling bot management entirely.
Serve the content in HTML. If the text only appears after hydration, the crawler never sees it. Server-render the primary content or provide a non-JS path.
Remove unnecessary gates. Public marketing and documentation pages should not sit behind a login or a geo-block if you want them cited.
Watch for 429s and 5xx. A rate limit or an origin error looks like a block from the outside; fix the reliability problem before adding more crawlers.
Re-test after each change. Use the checker above, the raw HTML viewer, and the server logs together; no single one tells the whole story.

Frequently Asked Questions

What is the difference between GPTBot and ChatGPT-User?

GPTBot is OpenAI's automatic web crawler used to fetch data for training future AI models. ChatGPT-User is triggered in real-time when a user explicitly asks ChatGPT to visit and analyze a live web URL.

Should I block AI crawlers in my robots.txt file?

If you want your website content to be discovered, cited, and recommended in AI Search Engines like ChatGPT, Perplexity, and Google AI Overviews, you should allow AI crawlers. Blocking them excludes your site from Generative Engine Optimization (GEO).

How do I unblock GPTBot and ClaudeBot?

Remove any "Disallow: /" directives for User-agent: GPTBot or User-agent: ClaudeBot in your website's robots.txt file, or explicitly declare "Allow: /" for them.

What is an AI crawler?

An AI crawler is a bot that fetches web pages for an AI system. Some collect pages for model training; others fetch a page while answering a live question. The second group — retrieval agents such as OAI-SearchBot, Claude-SearchBot, and PerplexityBot — is the one that decides whether your page can be cited in an answer.

Do AI crawlers follow robots.txt?

Most identify themselves and are expected to follow robots.txt, but the protocol is voluntary: it is a statement of preference, not an access control. A crawler can also be stopped by a CDN, WAF, rate limit, login wall, or JavaScript-only rendering before robots.txt is even relevant. Use robots.txt for policy and your edge configuration for enforcement.

How do I test whether GPTBot can access my site?

Check the GPTBot group in robots.txt, request the URL with the GPTBot user agent from your server, inspect your CDN or WAF logs for 403s and challenges, and confirm the page returns its main content in the initial HTML. A request from your own machine proves your server responds to that user agent; it does not prove the vendor's infrastructure is allowed through your edge.

Does blocking GPTBot stop ChatGPT from citing me?

No. GPTBot is OpenAI's training crawler; blocking it declines training use and does not by itself remove you from ChatGPT citations. The agents that decide citations are the retrieval ones: OAI-SearchBot and ChatGPT-User. Blocking those is what costs you the citation.

Why does Cloudflare block AI crawlers?

Cloudflare offers bot-management features that challenge or block automated traffic, including a one-click "block AI bots" option and managed WAF rules. Those controls can stop retrieval agents as well as scrapers. If you want citations, allow verified AI crawlers in the firewall rules and check the firewall events for challenges before assuming robots.txt is the cause.

auto_stories Complete Guide