Just as Googlebot crawls the web to build a search index, AI companies operate their own crawlers to power their language models and answer engines. The major AI crawlers are:
- GPTBot — OpenAI's crawler, used to train GPT models and power ChatGPT's browsing feature. User-agent:
GPTBot. - ClaudeBot — Anthropic's crawler for Claude model training. User-agent:
ClaudeBot. - PerplexityBot — Perplexity AI's crawler for real-time web retrieval in search answers. User-agent:
PerplexityBot. - Google-Extended — Google's separate crawler for Gemini model training (distinct from Googlebot). User-agent:
Google-Extended. - Applebot-Extended — Apple's crawler for Apple Intelligence training.
Controlling AI crawler access: AI crawlers respect robots.txt directives. Each can be allowed or blocked independently by specifying their user-agent name. Blocking them prevents your content from being used for training or live retrieval, but also removes your site from AI-generated answers.
User-agent: GPTBot\nAllow: /\n\nUser-agent: ClaudeBot\nAllow: /\n\nUser-agent: PerplexityBot\nAllow: /\n\nUser-agent: Google-Extended\nAllow: /Reading your own robots.txt to work out which of these are currently allowed is easy to get wrong, because each crawler follows only the group that names it. The free AI crawler access checker resolves those rules for you and reports each bot as allowed, blocked, or partially restricted.
What SearchVitals checks: The GEO / AI Visibility module parses your robots.txt and reports whether each major AI crawler is allowed, blocked, or subject to partial restrictions — flagging blanket blocks as a critical or warning finding.
Related: llms.txt, AI Overviews, robots.txt