# AI crawler benchmark v2.0 — independent collection window

The v1.1 article and release stay frozen. This directory implements the second
batch (T08–T11), not an overwrite of the November 9 measurements.

## Preregistered method (2026-11-10)

* Keep all 1,000 domains and ranks from the original filtered Tranco cohort. No
  domain replacement or exclusion based on its robots policy. This is a filtered
  domain cohort, not 1,000 independently verified content websites or companies.
* First run a 30-domain pilot (25 previously identified anomalies/review cases,
  five deterministically selected by SHA256). Exclude the pilot from final rates.
* Fetch HTTPS `/robots.txt`, HTTPS `/llms.txt`, and the homepage in the same new
  full-cohort window. One declared research UA, 10 workers, one request per final
  host at a time, 10-second socket timeout, at most five redirects and one retry
  for network/transient failures. Respect Retry-After; defer waits over 60 seconds.
  Do not retry 403 with a browser UA, bypass challenges, or substitute HTTP.
* Cap both wire bytes and decoded bytes at 1 MiB. A truncated/invalid body is
  unknown in the primary complete-file analysis. Store wire and decoded bodies,
  timestamps, status, redirect events, content type, hashes and selection number.
  Recompute all classifications offline from those bodies and attempt records.
* Primary denominator: successful, completely archived, UTF-8 robots responses
  with supported rules, extension-only records, or empty/comment-only bodies.
  Show successful HTML, error text and other unrecognized bodies separately.
  Rule files with some invalid lines retain parseable supported rules and expose
  the invalid line numbers. 404/410 and other 4xx are not declared allow policies;
  network/5xx are not declared blocking policies. RFC defaults are separate.
* Evaluate only `/` for the 14 frozen registry tokens; primary AI grouping is 12
  excluding Googlebot and Bingbot. This is a declaration analysis, not a live
  test using provider IPs, a whole-site block test or a citation-outcome study.
* Exact case-insensitive product-token match, merged named groups, wildcard
  fallback, longest normalized-octet path match and Allow on equal specificity;
  percent escapes, UTF-8, `*`, `$`, BOM and extension lines have regression tests.
* llms.txt: publish endpoint status, readable text and a **minimum H1 candidate**
  flag. An H1 is required by the proposal checked November 10, but alone does not
  prove semantic validity, complete format conformance, provider adoption or
  citation gains. HTML and known error text are not candidates. A missing root
  file does not exclude an llms.txt at a subpath. Cross-tabs use only the same new
  window and distinguish unknown responses from observed non-candidates.
* Homepage labels are conservative review evidence, never eligibility gates.
  Archive every final cohort homepage including four previously missing audits.
  Separate manual category decisions from automated content candidates; do not
  mechanically discard walled, thin or off-domain redirects. The adult label
  heuristic flags complete labels only; it is not an automatic exclusion.
* Alias sensitivity: group exact final robots URLs within the analyzable subset;
  disclose conflicting policies, if any, and exclude conflicts from URL-level
  rates. This is endpoint grouping, not verified ownership/company deduplication.
* Record string presence and valid user-agent/Content-Signal extension records
  separately. No inference that `__usage_ai-training`, Content-Signal or legacy
  Claude tokens are effective for all providers.
* No random-sample inference, causal estimates, paid AI calls or automatic publish.

## Sources

* RFC 9309: https://www.rfc-editor.org/rfc/rfc9309
* Google reference parser: https://github.com/google/robotstxt
* llms.txt proposal (modified 2026-08-10): https://llmstxt.org/
* Tranco list API and permanent IDs: https://tranco-list.eu/
* Provider definitions are preserved in the versioned `bots.json`; null source
  verification dates remain null rather than implying a new verification.

## Run

```
python3 -m unittest discover -s scripts/research/tests -p 'test_crawler_v2.py'
python3 scripts/research/crawler_benchmark_v2/collect.py \
  --cohort /path/cohort.csv --bots /path/bots.json \
  --output /path/new-window --home
python3 scripts/research/crawler_benchmark_v2/analyze.py \
  --run /path/new-window --output /path/new-analysis
```

The collector refuses an existing output directory. Its code is copied into
each run. Network failures stay recorded rather than silently dropping domains.
The analyzer verifies hashes, complete cohort and selected attempt provenance
before producing any rates, then reclassifies every archived body using its
packaged parser. Any difference from collection-time provisional classifications
is listed in `collector_to_analysis_reclassifications`; source records never change.
The analysis parser conservatively requires known extension keys for an
extension-only robots response, so JSON error objects cannot become valid files
merely because they contain a colon. Raw bodies may contain incidental website content;
public exports should include only the evidence needed to reproduce the study.


## Final window and scope

2026-11-10 01:04:31.278–01:18:54.938 UTC. 1,000 domains, 3,045
resource attempts, 5,443 HTTP requests including redirects. Paid AI calls: zero.
The exact cohort source is Tranco 94P72 (list date 2026-11-07); `provenance.json`
records recovered ranking fingerprints and the unchanged published v1.1 CSV hash.
All 1,000 domain/rank pairs match that ranking. No new sampling list was substituted.
Routing used the client's configured proxies; egress geography was not verified.

## Conservative grammar and reference comparison

Unsupported User-agent values still establish group boundaries: rules for a token
such as `MJ12bot` or `panscient.com` cannot spill into a preceding wildcard group.
Only groups with a supported product token count as a supported rule group. Empty
files and known extension-only files remain analyzable. A file with only malformed
agent records, JSON errors or unidentified text stays outside the denominator.
Rules must begin with `/`; a bare `Disallow: *` is flagged, not silently repaired.
Two Google-Extended root verdicts (twitter.com, x.com) differ for this exact reason
from Google's permissive implementation. Both implementations produce the same
primary AI-any result (249/854). See `reference-comparison.json` for pinned commit,
binary hash and token-by-token counts. This is not a provider-behavior test.

The collection-time parser was provisional. The final parser is applied to every
archived body; `metrics.json.collector_to_analysis_reclassifications` records all
changed classifications, group counts and policies. The raw run and code copied
at collection start remain untouched. No historical response field is overwritten.

## Data dictionary

`sites.csv` has one row per frozen domain, using UTF-8 and LF newlines:

* `domain`, `rank`: original sample key and Tranco rank; never live re-ranked.
* `{robots,llms}_status`: final selected attempt status, zero for request failure.
* `{robots,llms}_fetch_state`: success, not_found (404/410), other_4xx,
  server_error, network_error, redirect_limit or other_status.
* `{robots,llms}_body_kind`: complete rules, rules_with_invalid_lines,
  empty_or_comments, no_supported_rules (known extensions only), html_or_challenge,
  error_text, truncated, decode_error, encoding_review, unrecognized_text,
  h1_candidate or nonconforming_text. A body classification never overrides a
  non-success status to turn it into a policy.
* `{robots,llms}_selected_attempt`: one-based last-attempt index in this window;
  `{robots,llms}_final_url`, `_body_sha256`, `_content_type` identify that same response.
* `robots_analyzable`: one only for successful, complete supported rule/empty/known
  extension bodies; 146 outside this denominator are retained in the CSV.
* `protocol_default`: RFC retrieval behavior, independent from declarations.
* `{token}_policy`: allow, disallow, no_applicable_rule, no_file or unknown. Only
  `disallow` enters a restriction numerator. `no_applicable_rule` is NOT observed
  provider access. `{token}_source`: named, wildcard or none.
* `llms_observation`: minimum_h1_candidate, observed_non_candidate, root_not_found,
  or unknown. This certifies neither full proposal compliance nor provider use.
* `home_observation`: conservative automated review label, not a verified business
  category. `home_final_url`, `home_title` are response metadata. All 1,000 were
  observed; inaccessible/thin pages are retained, not eligibility-excluded.
* `adult_label_review_flag`: complete-label heuristic only. It does not classify
  Essex/Sussex/Middlesex as adult and does not exclude any frozen cohort member.

`sample-review.csv` documents 16 targeted cases (12 historical all-token-restricted
and four replacement domains). Categories are based on observed homepage evidence;
walled/blank/script-gated cases remain unresolved. It is not a representative
category distribution. Original homepage bodies are retained in the local archive,
not redistributed in the public ZIP. Public annotations can be counted again but
the underlying homepage HTML cannot be independently inspected from this ZIP.

`dataset.zip` includes analysis code, frozen context and `evidence/` with all
robots/llms attempts, wire/decoded bodies, response timestamps, redirect events,
retry reasons and body hashes. Identical wire/body files are deduplicated through
`body-aliases.json`; the analyzer resolves aliases and verifies both fingerprints.
The original collection config is preserved as `collection-run-original.json`;
`run.json` explicitly marks public homepage metadata-only coverage. Every core
compressed wire body is checked against its archived decoded representation.

## Replication

Extract the ZIP and run:

```
python3 analyze.py --run evidence --output ../rebuilt-v2.0
```

`metrics.json` and `sites.csv` must match the downloaded files byte-for-byte.
Core evidence validation is offline and makes no public or paid API requests.
The archive's `inputs-manifest.json` covers raw inputs; the public manifest includes
all release-file checksums including the ZIP and excludes itself.

## Comparing versions

v1.1 is an immutable editorial correction of the November 9 dataset (261/891,
29.3%). v2.0 measures a new November 10 window (249/854, 29.2%), with a different
research UA and stricter body/group validation. A common-analyzable-domain table
is included but cannot isolate policy changes from time, UA or method changes.
Alias sensitivity groups exact final robots URLs, not verified companies. None
of these associations establish AI citation, ranking, training use or causation.