AI Crawler robots.txt Policies

Scope and method

The archived sample contains 1000 domains, spanning Tranco ranks 1–1726. The historical run probed 3,000 filtered candidates; 1,931 passed its heuristic selection, then the first 1000 by rank were retained. Passing that test is not an independent certification of a content website. Infrastructure and domain aliases remain in the sample.

The ranking date was recorded as 2026-11-08; its permanent Tranco ID was not retained. Initial collection finished at 2026-11-09T02:48:00Z, with later retries lacking per-attempt timestamps. The collector used a 10-second timeout, at most five redirects and a 200,000-byte limit. This is a historical snapshot, not a current live check.

The old classifier marked 891 domains as parsable; the remaining 109 are outside these rate calculations. “Parsable” is that classifier’s flag, not independent validation. Empty files may legitimately contain no restrictions, while error text and mixed retry metadata need review. We no longer describe all excluded HTML responses as missing files or claim there are only four unresolved cases.

The historical matcher prefers a named token group to a wildcard group and evaluates the longest matching rule, with Allow winning a tie. It checks only /. Reproducing that implementation is distinct from proving complete RFC compliance or real crawler behavior.

Tokens and purposes

The same 14 historical tokens are retained for comparison, but their purposes differ. The main metric covers 12 AI-related tokens; Googlebot and Bingbot are ordinary search baselines. User-triggered fetching and mixed-use control tokens are not treated as a universal “decides citations” category.

Official definitions differ: OpenAI says ChatGPT-User does not control Search inclusion; Perplexity-User generally ignores robots.txt; Google-Extended combines training/grounding usage controls. CCBot collects an open dataset; Bytespider’s exact downstream purposes were not independently verified here. See the linked operator sources and method notes.

Restrictions by token

Scroll horizontally for the full chart; exact values are in the adjacent table.

Within the 12-token AI-related scope, 235 domains have a named-group restriction and 26 have only wildcard restrictions. The original all-14 scope has 238 named restrictions; those counts must not be interchanged.

Within-provider comparisons

Compare two tokens from the same provider rather than infer outcomes for an entire platform. Each matrix includes all 891 domains in the historical parser subset. “No Disallow match” includes default/no-rule cases and is not a verified permission or citation.

GPTBot / OAI-SearchBot

106 of 190 domains restricting GPTBot have no root-path Disallow match for OAI-SearchBot (55.8%). This describes declarations only.

Scroll horizontally for the full chart; exact values are in the adjacent table.

ClaudeBot / Claude-SearchBot

89 of 185 domains restricting ClaudeBot have no root-path Disallow match for Claude-SearchBot (48.1%). This describes declarations only.

Scroll horizontally for the full chart; exact values are in the adjacent table.

Distribution of restrictions

The histogram counts restrictions across the 12 AI-related tokens only. A domain with a root restriction can still allow other paths. In the original all-14 scope, 12 domains restrict every token and 628 restrict none; no industry-category conclusion follows from those counts.

Scroll horizontally for the full chart; exact values are in the adjacent table.

llms.txt association

The root llms.txt heuristic flagged 172 of 1000 domains (17.2%). That is detection, not format-validated adoption: the bodies were not retained and subpath llms files were not tested. The chart asks a specific conditional question: how often does a root restriction occur within the detected and not-detected groups?

Scroll horizontally for the full chart; exact values are in the adjacent table.

The direction of the probability matters. This is not the percentage of restricted sites that publish llms.txt, and it establishes no causal effect. Removing the two search baselines also changes the legacy all-14 comparison; the old-scope values are retained in metrics.json.

Rank groups

These groups partition 891 domains. Differences describe this filtered snapshot, not a general law about website size. Small groups, aliases and reachability selection limit inference.

Scroll horizontally for the full chart; exact values are in the adjacent table.

Historical fetch metadata records 583/891 files with redirects (65.4%) and a median recorded size of 1,289 bytes. Some retry fields and a truncated file remain unverified; these are archived metadata, not newly validated operational measurements.

Reading the rules

These short archived snippets illustrate interpretation, not evidence of actual access. They also show why root-path and whole-site conclusions differ.

registrar-servers.com

User-agent: *
Disallow: /

A wildcard restriction applies at /. Its presence does not establish that this domain is a content publisher.

aka.ms

User-agent: *
Allow:/fwlink/p/
Disallow:/

The root is restricted, while the more specific /fwlink/p/ prefix is allowed. “All paths closed” would be inaccurate.

Limits and unresolved cases

• Rules only: no verified provider IPs, actual crawler logs, indexing, model training or answer citations were measured.
• Root path only: homepage rules do not describe every article, API or allowed exception.
• Unresolved fetch states: HTML and server-error cases, mixed retry metadata and error text in the parser subset require review. No historical verdict was silently relabeled.
• Sampling: content eligibility is heuristic, infrastructure remains, aliases may represent one service, and the permanent ranking-list ID is missing.
• File validity: llms bodies were not archived. The old robots read limit is below the RFC’s 500 KiB parsing minimum; truncated content may contain unseen rules.
• Time: a new llms fetch cannot be joined to the old robots snapshot and described as one contemporaneous observation. A method-validated collection will use a new version.

Sources and definitions

• RFC 9309
• Tranco
• llms.txt
• Method and data dictionary

Data and corrections

Download the frozen v1.1 correction bundle. The CSV retains 1000 domains and 64 columns; metrics, charts and this page use the same version. Data/derived metrics are CC BY 4.0; reproduction code is MIT. The public bundle reproduces CSV aggregation; it does not include third-party raw robots bodies.

• Versioned CSV
• Data + offline reproduction bundle
• Metrics with denominators
• Token registry
• Checksums
• Method and data dictionary
• Python reproduction script
• Correction log

After extracting the bundle, run with Python 3 (standard library only). The output must be outside the input directory. No network or API keys are used.

python3 build_ai_crawler_benchmark.py --input . --output ../benchmark-rebuilt-v1.1

Suggested citation

SearchVitals (2026). AI Crawler robots.txt Policies, November 2026, v1.1 editorial correction. Quote the data version, root-path scope and denominator. https://searchvitals.com/research/ai-crawler-access-benchmark-2026/v1.1

Methods maintained by the SearchVitals team. Report errors to [email protected]; include the dataset version and affected row or metric.

FAQ

Does blocking GPTBot remove my site from ChatGPT Search?

GPTBot and OAI-SearchBot have separate purposes. The former concerns training collection; the latter controls automatic Search crawling. This study observes root-path declarations, not actual indexing or citations. Allowing a token does not guarantee inclusion.

Are ChatGPT-User and Perplexity-User search crawlers?

They support user-triggered visits. OpenAI says robots.txt may not apply to ChatGPT-User and it does not determine Search inclusion; Perplexity says its user fetcher generally ignores robots.txt. A local Disallow match is not proof that either fetcher was actually blocked.

Does Google-Extended control Google Search?

Google-Extended is a usage-control token for Gemini/Vertex training and grounding, not a separate HTTP crawler. Google says it does not affect Search inclusion or ranking. Googlebot is a separate search baseline in this study.

What does the reported restriction rate measure?

261 of 891 domains (29.3%) in the historical parser subset have a root-path Disallow match for at least one of the 12 AI-related tokens. Googlebot and Bingbot are excluded from that metric. The dataset is a filtered domain sample, not a random estimate of the whole web.

Does a detected llms.txt mean a valid file or more AI citations?

No. The historical heuristic checked a successful root URL, minimum body length and an HTML-like prefix. Bodies were not archived for format validation. The association in this report is neither a causal claim nor a measurement of AI citations.

Can I reproduce this report?

The versioned bundle contains the unchanged CSV, token metadata, metrics, method notes and a Python standard-library script. It reproduces the aggregation and charts offline; it does not independently revalidate the historical robots classifier.