Just as Googlebot crawls the web to build a search index, AI companies operate their own crawlers to power their language models and answer engines. The major AI crawlers are:
- GPTBot — OpenAI's crawler, used to train GPT models and power ChatGPT's browsing feature. User-agent:
GPTBot. - ClaudeBot — Anthropic's crawler for Claude model training. User-agent:
ClaudeBot. - PerplexityBot — Perplexity AI's crawler for real-time web retrieval in search answers. User-agent:
PerplexityBot. - Google-Extended — Google's separate crawler for Gemini model training (distinct from Googlebot). User-agent:
Google-Extended. - Applebot-Extended — Apple's crawler for Apple Intelligence training.
Controlling AI crawler access: AI crawlers respect robots.txt directives. Each can be allowed or blocked independently by specifying their user-agent name. Blocking them prevents your content from being used for training or live retrieval, but also removes your site from AI-generated answers.
User-agent: GPTBot\nAllow: /\n\nUser-agent: ClaudeBot\nAllow: /\n\nUser-agent: PerplexityBot\nAllow: /\n\nUser-agent: Google-Extended\nAllow: /What SearchVitals checks: The GEO / AI Visibility module parses your robots.txt and reports whether each major AI crawler is allowed, blocked, or subject to partial restrictions — flagging blanket blocks as a critical or warning finding.