The robots.txt file lives at https://example.com/robots.txt. Crawlers fetch it before crawling any other page and follow its directives to determine which paths they can access.
Basic syntax:
User-agent: *\nDisallow: /admin/\nAllow: /\n\nSitemap: https://example.com/sitemap.xmlUser-agent: * applies rules to all crawlers. Individual crawlers (Googlebot, GPTBot, ClaudeBot) can be targeted with their specific user-agent name.
Critical distinction: robots.txt controls crawl access, not indexing. A URL blocked by robots.txt cannot be crawled, but Google can still index it if another page links to it. To prevent indexing, use a noindex meta tag — which requires the page to be crawlable.
Common mistakes: Accidentally blocking CSS or JavaScript; blocking AI crawlers (GPTBot, ClaudeBot, PerplexityBot) unintentionally; leaving Disallow: / from a staging environment.
What SearchVitals checks: The Technical SEO audit fetches and parses robots.txt, identifying rules that block important pages, AI crawlers, or the sitemap. The GEO / AI Visibility module specifically checks whether AI crawlers are allowed or blocked.