# v2.2: Site purposes and AI crawler policies

SearchVitals · November 10, 2026 · CC BY 4.0 for the analysis and labels.

This is a supplemental analysis of the frozen v2.0 observations, not another robots/llms collection. Its internal label version is `site-types-full-v3`; its public release is `v2.2`. The existing v2.0 and v2.1 releases are separate.

## Cohort and labels

The exact 1,000-domain v2.0 cohort is retained. The robots/llms collection window was November 10, 2026, 01:04:31.278–01:18:54.938 UTC. There are 854 analyzable robots responses, 249 domains with a root Disallow for at least one studied AI token, and 156 minimum-H1 llms candidates. No cohort member was removed or substituted because of classification or policy outcomes.

The earlier knowledge-assisted 100-domain pilot labels were inherited unchanged. For the remaining 900, an initial manual knowledge pass assigned 790 and sent 110 to archive review. Targeted review made 19 revisions to initial knowledge labels. A reproducible 40-domain spot check uses the lowest SHA256 values of `full-type-qc-2026-11-10\nDOMAIN` among the initial 790: 31 had readable archived static text, nine could not be independently verified from that text, and three labels were revised. This was one reviewer, not independent inter-rater validation or an accuracy estimate.

The first label set was frozen before joining policy outcomes. The reviewer had prior exposure to the benchmark; review cards for the new 900 omitted policy fields, but this is not a strictly blinded study. A later metadata-only correction removed hashes of local blocked responses from official page-tool evidence fields; categories changed by zero. `classification-freeze.json` and `freeze-revision.json` retain that distinction.

There are 991 assigned domains and nine unknown. Primary evidence bases are:

| Basis | Domains |
|---|---:|
| Archived page | 172 |
| Background knowledge | 809 |
| User-supplied knowledge | 1 |
| Follow-up homepage | 4 |
| Official page-tool view | 5 |
| Insufficient evidence | 9 |

Thus 810 assignments rely on knowledge and 181 on page evidence. Assigned-label coverage is not current-homepage verification coverage or accuracy. Three follow-up-homepage assignments and two official-tool assignments were inherited from the pilot.

The extension requested 14 unresolved homepages once each, generating 23 local HTTP requests including redirects, at 08:06:40.593–08:06:58.104 UTC. These later materials did not change the earlier robots/llms measurements. Tool-view request counts are unavailable and not included in those 23. Sources, timestamps, original response statuses and body hashes are retained; a successful tool view does not turn a local blocked response into a successful body.

## Category definitions and boundaries

The unit is a domain, with one primary site/service-purpose label, not a company, ownership claim or formal industry code.

- **editorial_media**: editorial news, professional publishing or compiled reference content, including official editorial blogs.
- **community_ugc**: user/author publishing, sharing, collaborative editing or reviews; content production mode matters more than subject matter.
- **commerce_marketplace**: product, accommodation, asset and classified-listing markets; clearly shop-oriented manufacturer homepages are included.
- **software_service**: software, developer docs, account tools, real-time data tools, games and streaming services; pure hosting endpoints are infrastructure.
- **corporate_brand**: company, product catalogue and brand/service information; secondary purchase links do not automatically define a store.
- **institution_public**: educational, research, public-service or public-interest institutions; `.org` and self-described “official” status alone are insufficient.
- **infrastructure_adtech**: DNS, cloud hosting, CDNs, public RPC and ad delivery/attribution infrastructure, distinguished from ordinary business software.
- **other_known**: heterogeneous content/application catalogues, wagering services, parked domains, example domains and other evidenced uses. Not one industry.
- **unknown**: knowledge and available page material do not support a sufficiently clear primary purpose. Unknowns remain in the cohort.

Redirect destinations can inform observed purpose but do not prove original-domain ownership or all subdomain functions. For netcraze.io, a thin CrazeDNS-branded homepage was combined with related official service documentation; the documentation does not explicitly name `.io`, so this remains a flagged inference. Mixed-purpose labels can be debatable, and inherited pilot boundaries may not be fully uniform. `borderline_review_flag` identifies explicitly recorded cases, not every uncertain label. Per-domain notes use the `note_zh` Chinese field, with English machine category keys.

## Counts, denominators and limits

The primary policy measure evaluates `/` for the original 12 AI crawler/fetcher/control product tokens. It includes inherited wildcard matches and excludes Googlebot and Bingbot search baselines. A declared Disallow does not establish AI-specific targeting, real crawler identity, successful network blocking, indexing, citations or owner intent.

Category policy percentages divide AI-root-Disallow domains by that category's analyzable robots responses. Minimum-H1 llms candidates use **all domains in that category**; unknown llms observations are separate. Candidates are not full proposal conformance, provider adoption or evidence of citation effects.

Defined categories need at least 20 analyzable robots responses to display a policy percentage. Other and unknown categories have counts only. This display threshold is not a significance test. The frozen popularity cohort, its original eligibility filters, mixed knowledge/page evidence, multilingual/regional destinations and related domains preclude web-wide or industry-wide prevalence claims. No causal claim, representative-sample confidence interval or formal industry ranking is made.

## Sensitivity checks

854 analyzable domain records map to 837 exact final robots URLs. Three groups have conflicting purpose labels: vk.com/userapi.com, magnific.com/freepik.com, and pinterest.com/pinimg.com. Excluding them leaves 834 URL units and 243 AI-root-Disallow units. Groups with differing full AI policy vectors would also be excluded; none occurred here. This is endpoint sensitivity, not company deduplication. robots URL groups are not used to merge llms observations.

Media/community/software rates are respectively 59.9%/44.3%/16.3% at domain level and 59.3%/44.2%/16.2% after URL grouping. Excluding recorded boundary cases gives 61.1%/43.4%/16.3%. These checks preserve the observed ordering, not its causal interpretation or validity for all websites.

Archive-only evidence gives media 8/20 and software 5/41; community has only 15 analyzable cases. Other source-specific cells are often small. Page readability is selective, so source-basis checks cannot validate knowledge-based labels. Full source-basis summaries, per-token counts and rank-band composition are supplied for inspecting confounding.

## Reproduce without network access

Extract the v2.2 ZIP into `v2.2/` and the v2.0 dataset ZIP into `evidence/`. v2.0 places `sites.csv` and `metrics.json` directly in its extraction root. From their parent directory:

```bash
python3 v2.2/analyze.py --frozen v2.2 --base evidence --work v2.2 --output rebuilt-site-types
```

Python standard library only. The output directory must be new. The script checks the frozen label hash, available decision-input hashes and pinned v2.0 site/metric hashes before joining; it rejects cohort/rank mismatches and requires the original totals 854/249/156. Generated review HTML is a companion review artifact, not a copy of the published article.

`classifications.csv/json` omit policy outcomes; `classified-sites.csv` is the joined file; `summary.json` contains all sensitivity checks; other CSVs provide category/token/rank/alias counts. `checksums.json` verifies ZIP payload files; the separate release `manifest.json` also hashes the ZIP. `freeze.py` is included for auditing the freezing rules, but re-freezing original evidence requires the local raw homepage archive. Raw homepage HTML is not redistributed in this package. The labels and published aggregate analysis are reproducible from the provided files, not an independent revalidation of each knowledge assignment.

Sources are domain-specific archived evidence references and official page URLs in `archive-followup-decisions.json` and `source-register.json`. The original v2.0 release documents robots matching and collection. No new robots/llms requests or paid AI classification calls were used for this supplement.