AI Crawler Access Checker
Check whether your robots.txt allows or blocks each AI crawler — the retrieval agents that decide citations, and the ones that only train models.
mysite.com
AI & Search Bot Status Breakdown
Detected robots.txt Excerpt
Want a complete Search & AI visibility audit?
Checking crawler access is only the first step. Run a full audit to analyse Core Web Vitals, sitemap coverage, schema markup and content indexing.
Start Full Website Audit arrow_forwardWhy AI Crawler Access Matters for GEO (Generative Engine Optimization)
As AI-powered engines like ChatGPT, Claude, Perplexity, and Google Gemini handle more user search queries, getting your content cited by AI models requires proper crawler access.
Retrieval agents and training agents are not the same thing
Every AI vendor runs more than one crawler, and they do different jobs. A training agent collects data to train a model; blocking it declines that use and nothing else. A retrieval agent fetches a page while answering a question, and it is the one that decides whether you can be cited.
Sites routinely get this backwards — allowing the training agent while blocking the retrieval ones, which gives up the citation without protecting the content.
What each crawler actually governs
- OAI-SearchBot, ChatGPT-User (OpenAI): decide ChatGPT citations. The first serves ChatGPT's search; the second fires when a user asks ChatGPT to open a URL. Blocking either costs citations.
- GPTBot (OpenAI): training only. Blocking it keeps your content out of future GPT training data and has no bearing on whether ChatGPT can cite you.
- Claude-SearchBot, Claude-User (Anthropic): decide Claude citations, fetching pages while answering.
- ClaudeBot (Anthropic): training only, like GPTBot. Blocking it does not affect citation.
- PerplexityBot, Perplexity-User (Perplexity): decide Perplexity citations. Perplexity answers lean heavily on live retrieval, so blocking these removes you from its results.
- Googlebot (Google): both the ordinary search crawler and the source for AI Overviews.
- Google-Extended (Google): the exception. One token covers Gemini training and grounding — the two cannot be separated — so blocking it gives up Gemini citations as well. It is widely mistaken for a training-only switch.
- Applebot-Extended, Bytespider, CCBot: training or dataset use; none of them decide citation.
One robots.txt rule that catches people out
If a crawler has a User-agent group of its own, the User-agent: * group does not apply to it at all — the specific group replaces the wildcard rather than adding to it. Most "but I disallowed it under *" confusion comes from this.
Access is the precondition, not the goal. How to become the source AI answers cite covers what decides citation once the crawlers can reach you.
Frequently Asked Questions
What is the difference between GPTBot and ChatGPT-User?
GPTBot is OpenAI's automatic web crawler used to fetch data for training future AI models. ChatGPT-User is triggered in real-time when a user explicitly asks ChatGPT to visit and analyze a live web URL.
Should I block AI crawlers in my robots.txt file?
If you want your website content to be discovered, cited, and recommended in AI Search Engines like ChatGPT, Perplexity, and Google AI Overviews, you should allow AI crawlers. Blocking them excludes your site from Generative Engine Optimization (GEO).
How do I unblock GPTBot and ClaudeBot?
Remove any "Disallow: /" directives for User-agent: GPTBot or User-agent: ClaudeBot in your website's robots.txt file, or explicitly declare "Allow: /" for them.
Crawling & Indexing — Full Guide
How Googlebot and AI crawlers discover your pages — robots.txt, llms.txt, crawl budget, and what blocks access.
Read the guide arrow_forwardRelated Glossary Terms
Related Tools
