# AI crawler robots.txt policies — v1.1 method and data dictionary

## Scope and limitations

This is an editorial correction of a historical 1,000-domain snapshot, not a new crawl. The CSV and all per-token verdicts are unchanged. Fourteen tokens were evaluated at path `/`; two ordinary search crawlers are separate baselines. The main AI-related metric uses the remaining twelve tokens. The denominator is the subset marked `robots_parse_ok=1` by the archived collector, not 1,000 independently verified content websites.

A match to Disallow is a declared root-path restriction. A row without that match does not establish actual network access, indexing, model training or citation. User-triggered fetchers have provider-specific robots behavior; control tokens are not independent HTTP user agents. Common Crawl collects an open dataset. Bytespider's exact downstream purposes are not independently established here.

## Sampling and collection provenance

The original run log reports 3,000 filtered candidates, 1,931 passing its heuristic eligibility test, then the first 1,000 by Tranco rank. The claimed ranking date is 2026-11-08, but the permanent Tranco ID and full input archive were not retained. Initial collection finished at 2026-11-09T02:48:00Z. Later retry timestamps are missing. Do not treat the ranking date as the exact fetch time of every row.

The historical collector used a 10-second timeout, up to five redirects and a 200,000-byte read limit. It requested robots.txt using SearchVitalsBot, with browser-UA retries, then later fallback attempts. This was not a probe from any provider's verified crawler IP. Readable robots responses could pass eligibility without a content-homepage check. Infrastructure, aliases and selection bias remain; the sample is not random and is not the unfiltered Tranco top 1,000.

## Historical columns

- `domain`, `rank`: domain sampling unit and rank recorded by the collector. Aliases are separate domains.
- `eligible`: the historical heuristic selection flag, not an independent content-site certification.
- `robots_status`, `robots_final_url`, `robots_redirects`, `robots_bytes`, `robots_content_type`, `robots_ua_used`: recorded fetch metadata; some fallback rows mix fields from different attempts.
- `robots_parse_ok`, `robots_state`, `robots_missing`, `robots_error`, `robots_groups`: legacy classifier fields. In particular, `absent` is not a validated no-file result, and the four rows labeled unknown are not proven to be all unresolved cases.
- `<token>`: `blocked`, `allowed`, `no_rule` or `unknown` under the archived root-path matcher. `allowed` is a classifier output, not verified access. `<token>_rule` and `<token>_group` record its matching evidence.
- `llms_status`, `llms_bytes`, `llms_content_type`, `has_llms_txt`: root `/llms.txt` heuristic (HTTP 200, body over 20 bytes and not HTML-like at the start). llms bodies/final URLs were not retained. This is not format-validated adoption; subpath llms files were not checked.

## Metrics and probability

`metrics.json` provides numerator, denominator and one-decimal percent for every rate. Main restriction rates use 12 AI-related tokens; `legacy_all14_*` retains the former scope. Ordinary search baselines remain in the per-token table. Rank groups partition the historical parser subset. Pair matrices compare two specified tokens only. llms groups compare P(root restriction | heuristic llms detected) with P(root restriction | not detected), within the same historical parser subset, without implying causation. Rounding is decimal half-up to one place.

## Known unresolved cases

HTML/5xx classification, mixed fallback metadata, error text marked parsable, truncated robots files and llms false positives require separate review or new collection. Empty or comment-only robots can legitimately contain no restrictions; not every zero-group record is an error. No historical verdict was silently changed. See `review-decisions.csv` and `snapshot.json`. New collection must store every attempt, preserve bodies and use a new time window; do not combine fresh llms values with old robots values as a contemporaneous study.

## Offline reproduction (Python 3, standard library only)

Unzip dataset.zip, then run from that directory:

```bash
python3 build_ai_crawler_benchmark.py --input . --output ../benchmark-rebuilt-v1.1
```

The output must be outside the input directory. Input CSV hash is checked. This script only aggregates the supplied CSV and renders charts; it does not crawl, re-parse robots bodies or independently validate the historical classifier. It makes no network requests and does not require API keys. The original robots archive fingerprint is recorded, but third-party raw files are not included in this public reproduction bundle.

## Sources and citation

- OpenAI: https://developers.openai.com/api/docs/bots
- Anthropic: https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
- Perplexity: https://docs.perplexity.ai/docs/resources/perplexity-crawlers
- Google: https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers#google-extended
- Common Crawl: https://commoncrawl.org/ccbot
- RFC 9309: https://www.rfc-editor.org/rfc/rfc9309
- Tranco: https://tranco-list.eu/
- llms.txt proposal: https://llmstxt.org/

Suggested citation: SearchVitals (2026). AI Crawler robots.txt Policies, November 2026, v1.1 (editorial correction). https://searchvitals.com/research/ai-crawler-access-benchmark-2026. Include the data version and metric denominator when quoting. Dataset/derived metrics: CC BY 4.0. Reproduction code: MIT. Report errors to service@searchvitals.com.