# Supplemental analysis v2.1 — methods and dictionary

Analysis date: 2026-11-10. Base observations: frozen v2.0 collection,
2026-11-10T01:04:31.278Z to 2026-11-10T01:18:54.938Z. This is post-hoc descriptive
analysis, not a new measurement window or a new primary restriction estimate.
Original v2.0 files are unchanged. No cohort domains, synthetic probe paths or
linked resources were fetched, and no paid AI API was called for this analysis.

## Frozen inputs

`inputs.json` fixes the v2.0 ZIP, its CSV/metrics, the exact v2.0 policy parser,
the CommonMark AST inventory and targeted reviewer annotations by SHA256. The
analyzer rejects mismatches, incomplete candidate coverage, duplicate domains
and review/body-hash mismatches. Every analyzed body is read from the base ZIP
and checked against the original selected-response fingerprint.

## 1. Policy patterns

Analyze the original 854 complete analyzable robots responses and 12 AI-related
tokens. Googlebot and Bingbot are ordinary-search baselines, excluded from AI-any.
The training pair contains GPTBot/ClaudeBot; the search trio contains
OAI-SearchBot/Claude-SearchBot/PerplexityBot. Other roles are not silently treated
as training crawlers; the six-token signature is reported literally.

Profiles are mutually exclusive, in order: no AI root Disallow; all 12 studied AI
tokens restricted; some training-pair AND search-trio token restricted; some
training-pair token but NO search-trio token restricted; some search-trio token
but NO training-pair token restricted; only other studied AI tokens restricted.
Their counts sum to 854. The profile CSV also retains all 146 outside-denominator
domains with an explicit outside label. “No Disallow” includes an Allow match or
no applicable rule. Neither means verified live access or whole-site permission.
Exact signatures count all 12 bits; omitted tokens have no root Disallow match.
The baseline comparison is a separate 2x2 table, not inferred owner intent.

## 2. Configuration review

Compare claude-web/anthropic-ai against the current ClaudeBot, Claude-SearchBot
and Claude-User tokens on the same archived robots body. Review candidates need
at least one NAMED legacy-token root Disallow and no ClaudeBot root Disallow.
The shortlist is not a list of proven misconfigurations. Provider documentation
is consulted for current definitions; legacy usage and historic owner intent are
not inferred from absence in that documentation.

Record the matched rule, line number, named/wildcard source and each modern
root verdict. Evaluate `/__benchmark_scope_probe__` OFFLINE only. A root exception
means Allow /$ matches / while the synthetic non-root path matches Disallow.
A named path-specific case additionally has an actual declared literal disallow
path that still evaluates as restricted. Other shortlist cases retain their
wildcard root outcomes. No synthetic path is fetched, no rules are auto-fixed,
and domains sharing the same body are not called independent configurations.

## 3. llms.txt structural inventory and targeted review

Keep all 156 original minimum-H1 candidates. Candidate status is not revised or
upgraded to a compliance certificate. Extract AST features using PHP and
league/commonmark 2.8.2, CommonMarkCoreExtension, max_nesting_level=100. Source is
provided as `.php.txt` so no executable PHP endpoint is deployed under public/.

AST links include inline/reference links and angle-bracket autolinks. Links in
code fences and plain bare URLs are not AST hyperlinks. Bare HTTP URL strings
are counted separately with `https?://[^\s<>"\]]+`; strings can include examples,
code, URL templates or trailing punctuation. This lexical count is not a valid
or reachable-URL count. Resolve parsed web links against the FINAL llms URL.
Host relationship is exact host/subdomain matching, not registrable-domain or
company identity. A .md suffix is a path-shape observation, not a MIME guarantee.

Complementary shape bins: H2+web hyperlinks; web hyperlinks without H2; hyperlinks
but only fragments/non-web targets; no AST links but literal HTTP URLs; neither.
Record multiple H1s and separator-only first H1s rather than assigning pass/fail.
Record byte median, nearest-rank p90 and the count over 100 KiB descriptively;
these are not model-context limits, token costs or performance measurements.
Exact body hashes and final URLs support deduplication sensitivity only.

All 156 candidates receive structural extraction. Targeted qualitative review
covers all 12 with zero AST hyperlinks plus the largest-body example (13 domain
cases). Reviewers inspect headings/intros, URL forms and relevant archived
passages; no claim of full semantic validation, factual verification of publisher
statements or representative content-category coverage is made. The embedded
scheme pattern `^https?://[^/?#]*https?://` flags a static concatenation lead,
not a verified dead link. No links are fetched and no quality/citation score is
assigned. The original 156-domain candidate count remains unchanged.

## Data dictionary

* `policy-profiles.csv`: all 1,000 domains; original rank; disjoint profile or
  outside label; baseline-root flag; exact restricted AI-token list; body hash.
* `configuration-cases.csv`: candidate domains, source body hash, classification,
  named legacy root restrictions, ClaudeBot root policy/source, synthetic-path
  result and literal declared-path example. Full matched rules and line numbers
  for all old/current tokens are in `insights.json.configuration.cases`.
* `llms-quality.csv`: all 156 candidates, final URL/hash, decoded bytes, H1/H2
  counts and title, blockquote flag, parsed/unique web link counts, host relation,
  relative links, .md suffixes, fragments/non-web targets, lexical URL-string
  count, complementary shape and concatenation candidates. URLs were not tested.
* `markdown-features.json`: frozen AST measurements bound to body hashes; parser
  name/version/extensions, headings and link targets, short intro excerpts.
* `review.json`: targeted notes, scope, date and body fingerprints. These are
  reviewed annotations, not automatic proof of provider or owner behavior.
* `insights.json`: all tables, exact signatures, deduplication and review data.
* `sources.json`: primary definition/format sources and their November 10 checks.

## Offline replication

Download the unchanged v2.0 dataset.zip and this v2.1 analysis.zip separately.
Extract only analysis.zip into a new directory. With Python 3.10+:

```
python3 deepen.py --base-archive ../v2.0/dataset.zip --features markdown-features.json --reviews review.json --output ../rebuilt-v2.1
```

Four outputs must match byte-for-byte: insights.json, policy-profiles.csv,
configuration-cases.csv and llms-quality.csv. This uses the supplied frozen AST
measurements and requires no PHP or network. For independent AST re-extraction,
provide a Composer autoloader with league/commonmark exactly 2.8.2 installed:

```
python3 extract_features.py --base-archive ../v2.0/dataset.zip --autoload /path/to/vendor/autoload.php --output ../markdown-features-rebuilt.json
```

The helper invokes local PHP only and requires byte-identical extraction. The
actual analysis cannot run until the base/input hashes match. Supplemental ZIP
does not duplicate the 27 MB base ZIP. Manifest covers all supplemental files,
including ZIP, and excludes itself. The code is MIT; derived annotations and
metrics are CC BY 4.0; archived third-party content retains its original rights.