The robots.txt file lives at https://example.com/robots.txt. Crawlers fetch it before crawling any other page and follow its directives to determine which paths they can access.
Basic syntax:
User-agent: *\nDisallow: /admin/\nAllow: /\n\nSitemap: https://example.com/sitemap.xmlUser-agent: * applies rules to all crawlers. Individual crawlers (Googlebot, GPTBot, ClaudeBot) can be targeted with their specific user-agent name.
Critical distinction: robots.txt controls crawl access, not indexing. A URL blocked by robots.txt cannot be crawled, but Google can still index it if another page links to it. To prevent indexing, use a noindex meta tag — which requires the page to be crawlable.
Common mistakes: Accidentally blocking CSS or JavaScript; blocking AI crawlers (GPTBot, ClaudeBot, PerplexityBot) unintentionally; leaving Disallow: / from a staging environment.
Blocking AI crawlers is the easiest of these to miss, since nothing in Search Console reports it. An AI crawler checker fetches the live file and shows which AI user-agents it currently permits.
What SearchVitals checks: The Technical SEO audit fetches and parses robots.txt, identifying rules that block important pages, AI crawlers, or the sitemap. The GEO / AI Visibility module specifically checks whether AI crawlers are allowed or blocked.
Related: llms.txt, AI crawlers, AI Overviews, XML sitemap, canonical tag, hreflang, redirects
To check whether a specific URL is blocked by the rules you are editing, use the robots.txt tester; it validates the syntax and the group structure before you deploy the file.