AI Crawler robots.txt Policies

Sample and method

The 1000 domains are the exact published v1.1 cohort, originally selected from Tranco list 94P72. All ranks match that list, whose CSV exactly matches the original archived ranking. No domains were replaced because of their new policies. The original eligibility filters remain a selection limitation.

We requested HTTPS /robots.txt, /llms.txt and the homepage in one window, with one research user agent, 10 workers and at most one concurrent request per host. We followed up to five redirects, permitted one transient-error retry, respected Retry-After and capped both wire and decoded bodies at 1 MiB. No browser fallback or independent HTTP substitution was used.

Root-path matching uses case-insensitive exact product tokens, merged matching named groups, wildcard fallback, normalized percent-encoded paths, longest-pattern precedence and Allow on a tie. Empty/comment-only and known extension-only files can be analyzable. HTML, known error messages, truncated bodies and unknown encodings are excluded from the main rule denominator. Every analysis starts from archived bodies, with reclassifications disclosed.

5443 HTTP requests across 3045 resource attempts. 638 selected robots attempts followed redirects; 2 ended on HTTP through formal redirects. Full request chains and the original HTTPS authority are preserved.

Google reference-parser comparison: 11956 root verdicts checked; 2 differences. This is an independent implementation check for this corpus, not a guarantee of identical provider behavior or full RFC conformance.

The two differences concern Google-Extended at twitter.com and x.com: their bare Disallow: * has no leading slash. Strict matching flags the invalid path; Google accepts it. The primary AI-any result is unchanged under the reference parser: 249 / 854 (29.2%).

Coverage and unknowns

404/410 are observed missing endpoints. Other 4xx, 5xx and network failures remain distinct. RFC 9309 permits access after an unavailable 4xx response and requires assuming disallow for an unreachable file; neither default is counted as a declared rule in this report.

Rules by token

223 restricted domains have a named-token Disallow match; 26 are restricted only through wildcard groups.

Official definitions differ: OpenAI says ChatGPT-User does not control Search inclusion; Perplexity-User generally ignores robots.txt; Google-Extended combines training/grounding usage controls. CCBot collects an open dataset; Bytespider’s exact downstream purposes were not independently verified here. See the linked operator sources and method notes.

Restriction breadth

Rank bands

Each denominator includes only analyzable robots responses in that original rank band. The filtered sample and unknowns differ between bands; this is descriptive, not a population trend estimate.

Site purposes and AI crawler policies

We assigned a primary site or service purpose to 991 of 1000 domains and retained 9 as unknown. The original 100-domain pilot labels were inherited; the remaining 900 were classified using knowledge first, then archived pages and targeted homepage follow-up. No domain was replaced or removed.

810 assignments rely on background or user-supplied knowledge; 181 on page evidence. Classification coverage is neither an accuracy estimate nor current-homepage verification coverage. Labels describe site or service purposes, not company industries or ownership.

Within this fixed cohort, media/publishing domains had 88 / 147 (59.9%) AI root-path Disallow evaluations; community/UGC had 47 / 106 (44.3%); software/online services had 42 / 257 (16.3%). These are descriptive differences, not evidence that category causes a policy choice.

Swipe or scroll horizontally to see all table columns.

Purpose categories: declared root policies and minimum llms.txt candidates

Primary purpose	All domains	Analyzable robots	AI root Disallow	Share of analyzable robots	llms candidates / all in category
Media and publishing	165	147	88	59.9%	16 / 165
Community and user content	112	106	47	44.3%	5 / 112
Commerce and marketplaces	128	116	35	30.2%	20 / 128
Software, online services and docs	303	257	42	16.3%	80 / 303
Corporate and brand information	34	26	3	11.5%	8 / 34
Education, research and public institutions	75	61	5	8.2%	4 / 75
Infrastructure and ad delivery	96	63	15	23.8%	19 / 96
Other evidenced purposes	78	70	13	Counts only	4 / 78
Unknown or insufficient evidence	9	8	1	Counts only	0 / 9

Policy percentages use analyzable robots in each category. llms candidate counts use every domain in that category; the denominators differ. Disallow includes inherited wildcard rules across the 12 studied AI tokens, excluding Googlebot and Bingbot. It is not proof of AI-specific targeting, actual blocking, indexing or citations.

We show policy percentages only for defined categories with at least 20 analyzable robots responses. “Other” combines heterogeneous purposes and “Unknown” lacks a usable purpose label; both have counts only. The 20-response display threshold does not establish statistical significance or web-wide representativeness.

Software/online services contributed 80 of the 156 minimum-H1 candidates, from 303 domains in that category. A candidate is not complete format validation or evidence of model adoption; absolute counts also reflect category size.

Do aliases and classification boundaries change the observation?

The 854 analyzable domains map to 837 exact final robots URLs. We excluded 3 groups with conflicting purpose labels, leaving 834 URL units. These are not unique companies. This check and exclusion of recorded boundary cases preserve the observed ordering of the three groups below; neither establishes causality or removes classification bias. robots URL grouping is not used to collapse llms candidates.

Swipe or scroll horizontally to see all table columns.

Primary purpose	Domain units	Same final robots URL	Recorded boundary cases excluded
Media and publishing	88 / 147 (59.9%)	86 / 145 (59.3%)	88 / 144 (61.1%)
Community and user content	47 / 106 (44.3%)	46 / 104 (44.2%)	43 / 99 (43.4%)
Software, online services and docs	42 / 257 (16.3%)	40 / 247 (16.2%)	41 / 252 (16.3%)

Source-basis sensitivity is more limited: the archive-only media group has 8 / 20 (40%), versus 5 / 41 (12.2%) for software/online services; the archive-only community group has only 15 analyzable cases. Readable-page selection is itself biased, so this does not validate all knowledge labels.

Classification method, evidence sources and boundaries

Initial labels were frozen before joining policy outcomes. One reviewer had prior exposure to the benchmark; the new 900-domain review cards omitted policy fields, but this was not an independent or strictly blinded study. A fixed-hash 40-domain spot check found readable static text for 31 entries; nine remained knowledge-based. It provides no overall accuracy estimate. Boundary flags record explicit cases and are not an exhaustive uncertainty score.

This classification stage fetched 14 unresolved homepages with 23 local HTTP requests, including redirects. Official page-tool views are registered separately because their underlying request count is unavailable. These later materials do not change the frozen robots/llms observation window; original HTTP errors and script gates remain recorded.

Primary classification basis	All domains
Archived page	172
Background knowledge	809
Follow-up homepage	4
Insufficient evidence	9
Official page-tool view	5
User-supplied knowledge	1

Media and publishing

Editorial news, professional publishing or compiled reference content; an official brand blog is not necessarily an independent news organization.

Community and user content

User/author publishing, sharing, collaborative editing or reviews. Content-production mode, rather than subject matter, guides the label.

Commerce and marketplaces

Product, booking, asset or classified-listing markets, including manufacturer homepages that primarily function as shops.

Software, online services and docs

Software, development docs, account tools, real-time data tools, games and streaming services; pure hosting endpoints belong to infrastructure.

Corporate and brand information

Company information, product catalogues and brand/service presentation. A secondary shopping link does not by itself make a store.

Education, research and public institutions

Educational, research, public-service or public-interest institutions; an .org suffix or the word “official” is not proof of status.

Infrastructure and ad delivery

DNS, cloud hosting, CDNs, public RPC and ad delivery/attribution infrastructure, distinguished from ordinary business software.

Other evidenced purposes

Heterogeneous content/app catalogues, wagering services, parked or example domains and other evidenced uses; not one industry.

Unknown or insufficient evidence

Neither knowledge nor available page material supports a sufficiently clear primary purpose; the domain remains in the cohort.

The nine unresolved domains

Insufficient functional evidence remained after archive review and homepage follow-up. Access restrictions were not bypassed, and domain wording alone was not used to force a category.

• skybridge.click
• userapi.com
• expireddomains.com
• certcenter.com
• ukdevilz.com
• theblowers.com
• luxuretv.com
• arionplay.com
• ojogodobicho.com

Site-purpose data and reproduction · v2.2

Training and search token pairs

The conditional rate is restricted first-token domains without a second-token Disallow match, divided by all first-token-restricted domains. It does not measure retained citations.

GPTBot / OAI-SearchBot

100 / 178 (56.2%) first-token-restricted domains have no second-token Disallow match.

ClaudeBot / Claude-SearchBot

83 / 174 (47.7%) first-token-restricted domains have no second-token Disallow match.

Policy patterns

Among the 249 AI-token-restricted domains, 233 (93.6%) have no root Disallow match for either ordinary-search baseline. This describes a difference in declared rules; it does not prove owner intent or successful crawler access.

The profiles below are mutually exclusive across all 854 analyzable domains. The training pair is GPTBot/ClaudeBot; the search trio is OAI-SearchBot/Claude-SearchBot/PerplexityBot. “Any” means at least one token in that set. All 12 studied AI tokens form a separate first-priority profile. These are operational groupings, not inferred business strategies.

Root-policy profile	Exact rule	Domains
No AI-token Disallow match	None of the 12 studied AI tokens is restricted at /.	605
All 12 studied AI tokens restricted	Every studied AI token matches Disallow at /.	45
Training-pair and search-trio restrictions	At least one token from each set is restricted; not all 12 AI tokens.	73
Training-pair restriction without search-trio restriction	At least one training-pair token is restricted, and none of the three search tokens is restricted.	84
Search-trio restriction without training-pair restriction	At least one search token is restricted; neither training-pair token is restricted.	8
Other studied AI tokens only	Some of the other seven AI tokens are restricted; neither the training pair nor the search trio is restricted.	39

AI-related tokens and ordinary-search baselines

Any studied AI token restricted	Googlebot or Bingbot restricted	Domains
Yes	Yes	16
Yes	No	233
No	Yes	2
No	No	603

Most frequent exact restriction combinations

Each row is an exact 12-token signature: every omitted AI token has no root Disallow match. The six-token signature includes usage-control tokens and a crawler whose purpose is unverified in the registry; it is not labeled “six training crawlers.” Repeated signatures do not prove shared ownership or a shared template.

Restricted tokens	Domains
OAI-SearchBot, ChatGPT-User, GPTBot, Google-Extended, Claude-SearchBot, Claude-User, ClaudeBot, PerplexityBot, Perplexity-User, Applebot-Extended, Bytespider, CCBot	45
GPTBot, Google-Extended, ClaudeBot, Applebot-Extended, Bytespider, CCBot	31
Bytespider	18
OAI-SearchBot, ChatGPT-User, GPTBot, Google-Extended, Claude-SearchBot, Claude-User, ClaudeBot, PerplexityBot, Perplexity-User, Bytespider, CCBot	13
CCBot	7

Token configuration review

Of 122 domains with a named legacy-Claude-token root Disallow match, 9 have no ClaudeBot root Disallow match. These 9 domains correspond to 9 distinct archived robots bodies. This is a maintenance-review shortlist, not a count of broken configurations.

We compared claude-web/anthropic-ai with ClaudeBot, Claude-SearchBot and Claude-User on the same archived body. Current provider documentation lists the three latter tokens separately. Historical owner intent and whether legacy strings are still used were not established; adding a modern token is not automatically the correct fix.

All candidate domains and their evidence

Domain	ClaudeBot at /	Matched group	Review result
chatgpt.com	Allow match	Wildcard	Root exception; offline probe restricted
espn.com	No applicable root rule	Wildcard	Wildcard root allowance or no matching prohibition
trustpilot.com	No applicable root rule	Named	Named current group restricts specific paths
repubblica.it	Allow match	Wildcard	Wildcard root allowance or no matching prohibition
skroutz.gr	Allow match	Named	Root exception; offline probe restricted
e621.net	No applicable root rule	Wildcard	Wildcard root allowance or no matching prohibition
nikkansports.com	No applicable root rule	Wildcard	Wildcard root allowance or no matching prohibition
ovh.com	No applicable root rule	Wildcard	Wildcard root allowance or no matching prohibition
espn.com.ve	No applicable root rule	Wildcard	Wildcard root allowance or no matching prohibition

Two cases, chatgpt.com and skroutz.gr, allow /$ but disallow the synthetic path /__benchmark_scope_probe__ when evaluated offline. Trustpilot has a named ClaudeBot group with path-specific prohibitions. These are concrete reasons a root-only difference cannot establish site-wide access or a stale-token mistake. No request was sent to the probe path.

A maintainer should first identify the intended provider use, inspect the applicable group and representative real paths, then decide whether a rule needs changing. Preserve deliberate root exceptions and path-specific rules. The downloadable case data includes matched rule lines, group sources and offline probe results.

Root llms.txt observations

An H1 candidate is only a minimum-structure observation under the llms.txt proposal checked November 10. It is not full format validation, verified provider use or an adoption estimate for the whole web. We checked the root endpoint only; files at subpaths can be missed. MIME alone does not decide the result.

The cross-tab uses only this new window and analyzable robots files. Unknown llms responses stay in their own row. Association does not show that publishing llms.txt changes robots policy or causes more citations.

What the llms.txt files contain

All 156 minimum-H1 candidate domains were inventoried with league/commonmark 2.8.2. They map to 152 final URLs and 151 distinct bodies. These are endpoint/body counts, not independent publishers or confirmed llms.txt adoption.

Structural observation	Result
Contains parsed CommonMark hyperlinks	144 / 156 (92.3%)
Contains an H2 heading	151 / 156 (96.8%)
Contains a link whose URL path ends in .md	27 / 156 (17.3%)
More than one H1	13
First H1 consists only of separator characters	1
Body exceeds 100 KiB	10
Median body bytes	9030.5
90th-percentile bytes (nearest rank)	48384

Structural form	Domains
H2 plus parsed web hyperlinks	143
Literal HTTP URLs, no parsed hyperlinks	12
Parsed web hyperlinks without H2	1

All 12 files without parsed hyperlinks still contain literal HTTP URL strings. Cursor is a bare-URL documentation index; Porkbun contains substantial API guidance; some other files describe entities, identity or usage policies. Zero CommonMark links does not mean no useful content. The proposal’s minimal heading requirement is not a navigation-quality score.

One static repair lead appears in Cursor’s archived index: https://cursor.comhttps://cursor.com/changelog.md. The destination was not requested. Unity’s 950,573-byte index illustrates size variation, not proven excess token cost or poor model performance. A separator-only first H1 is also reported separately; candidate status does not verify the project name.

Targeted archive review: all 12 no-hyperlink cases and the largest body

Domain	Observation and scope
unity3d.com	A 950,573-byte archived index with 2,478 parsed links. It illustrates body-size variation; token cost, usefulness and model handling were not measured.
name.com	Introduces API-discovery, documentation and agent-skill resources using literal URLs; no CommonMark hyperlinks were parsed. This is substantive discovery information, not an empty file.
dailymotion.com	Contains platform background, topical sections and contact/resource URLs in plain text. Statements about the business were not independently verified.
cursor.sh	A documentation index expressed as bare URLs. One archived target concatenates two schemes (cursor.comhttps://cursor.com/changelog.md); this is a static repair lead, not a live broken-link test.
porkbun.com	Contains a substantial API/agent reference with endpoint examples, code blocks and literal URLs. Zero parsed hyperlinks does not imply absent documentation.
flightradar24.com	Provides background and primary-resource URLs as plain text. Link destinations and factual claims were not fetched or verified.
kwai.com	Lists localized paths, help-center patterns and resource URLs in plain text. Placeholder patterns are not confirmed page URLs.
xing.com	Uses entity definitions, URL patterns and search parameters rather than a CommonMark link list. Pattern placeholders are not verified destinations.
moe.video	Provides company-description/FAQ prose and literal resource URLs. Publication is not evidence that a provider follows these instructions.
marktplaats.nl	Contains multilingual platform guidance, category resources and plain-text URLs; multiple H1 headings are a structural observation, not an automatic failure.
jusbrasil.com.br	Uses YAML-like content/attribution guidance and URL patterns under an H1. This review does not certify legal effect or provider compliance.
eltiempo.es	Most lines start with # and express usage guidance; CommonMark interprets many as headings. This is policy-oriented text rather than a conventional navigation index; enforcement was not tested.
nustargame.com	Contains self-described identity and anti-impersonation guidance with literal URLs. Ownership, licensing and security assertions were not verified.

Link extraction supports CommonMark inline/reference links and angle-bracket autolinks, excludes links inside code fences, and resolves relative targets against the final llms URL. Bare HTTP strings are a separate lexical count and can include examples or templates. A .md suffix does not establish the target MIME or quality. No linked pages were fetched, and no semantic-compliance or citation-effect score is assigned.

Aliases and sample review

Grouping identical final robots URLs yields 837 endpoints. After excluding 0 endpoints with conflicting observed policies, 244 / 837 (29.2%) restrict an AI token. The domain-level rate is 29.2%. This is endpoint sensitivity, not company deduplication.

Homepage evidence was collected for all domains, including the four previously missing audits. Automated labels are review candidates, not verified business categories or eligibility decisions. The review table documents the 12 historical all-token-restricted cases and four replacement domains; inaccessible evidence stays unresolved. No domain is removed from the main cohort.

Sample review (CSV)

Domain	Review category	Evidence / limitation
reddit.com	Community content	The archived homepage presents the Reddit community platform; retained as a domain, not a separate-company claim.
registrar-servers.com	Moved-site landing page	The homepage is a moved-site notice linking to Namecheap. This response does not establish a content publication or a DNS-service policy.
skybridge.click	Script gate; unresolved	The title names SMP-74, but visible content asks for JavaScript. Business category remains unresolved.
netcraze.io	DNS-branded thin landing; unverified	A thin landing page uses the CrazeDNS title. Service purpose and company identity are not verified from this response.
sharethrough.com	Advertising business	Homepage title and readable navigation describe programmatic advertising solutions. Retained; no exclusion based on its robots policy.
hihonorcloud.com	Empty/script-only; unresolved	Homepage provides no readable title or substantive visible content in this response; category remains unresolved.
adform.net	Access denied; unresolved	Homepage returns 403. Its name alone is not used to assign a business category.
mailchi.mp	Cross-domain marketing landing	Homepage redirects to the Mailchimp marketing landing page, so this is not an independent content-publication observation.
btloader.com	Access denied; unresolved	Homepage returns 403; business category remains unresolved.
stackexchange.com	Community content	Homepage presents the Stack Exchange questions network. Homepage content and robots retrieval outcome are separate evidence.
aka.ms	Link-management landing	Homepage formally redirects to an Azure-hosted link-management landing page. It is not company deduplication evidence.
keirin.jp	Racing information	Homepage presents racing information and results under KEIRIN.JP. It stays in the frozen cohort.
xsmn.mobi	Lottery results	Homepage title and readable result sections identify a lottery-results site. This completes a previously missing audit.
minhngoc.net.vn	Lottery results	Homepage identifies lottery results. A missing robots endpoint does not invalidate the homepage observation.
kinnser.net	Login portal	Homepage redirects to login.cfm and displays the WellSky HHH login title. It is a login portal, retained in the domain cohort.
espn.com.ve	Sports publication	Homepage identifies the ESPN Venezuela sports publication, completing the fourth previously missing audit.

Additional signals

String presence and syntactically recognized records are separate counts. Legacy Claude tokens, Content-Signal and __usage_ai-training are not assumed to control every provider. Their actual effect was not tested.

What changed since v1.1

v1.1 corrects the historical aggregation; v2.0 changes both time window and validation method. Differences cannot be attributed to website policy changes alone. The common-subset comparison holds the domain list constant but still has both limitations.

Common analyzable domains: 853. Historical AI-token restriction rate: 29.1%; new rate: 29.1%.

Limits and practical use

• This filtered Tranco cohort is not random, not necessarily 1,000 content websites and not 1,000 independent companies. Rank patterns are descriptive; no population confidence interval or causal conclusion is claimed.
• Only the root path was evaluated. Providers differ in how they apply robots declarations; control tokens and user-triggered fetchers are not interchangeable with autonomous crawlers. No provider IP, WAF bypass, citation or training-use test was run.
• This is one client run with locally configured network/proxy routing. Egress geography and direct-origin behavior were not independently verified. CDN, geo and UA variation, transient failures, body limits and finite retries can affect observations. Unknown states are disclosed rather than converted into policy decisions.
• llms.txt H1 candidates require further semantic review. Absence at the root does not rule out subpath files. Neither llms.txt nor an allowed crawler token guarantees indexing, ranking or citation.

For your own site, review the tokens for the provider uses you intend to control, check important paths in addition to /, and verify real requests in server logs. Keep search access, training preferences, user fetches and llms documentation as distinct decisions.

Open the checker · What llms.txt does · How AI citations work

Data and reproduction

• Domain measurements (CSV)
• Evidence and offline reproduction bundle (ZIP)
• Computed metrics (JSON)
• Provider and token registry
• Checksums
• Method and data dictionary
• Analysis source
• Change log
• Sample review (CSV)
• Reference-parser results
• Frozen sample provenance

Extract dataset.zip, then run the command below with Python 3.10 or later. It makes no network requests. Core robots/llms response bodies and retry metadata are included; homepage HTML remains in the local research archive, with homepage classifications and response metadata provided in the bundle. Derived measurements are CC BY 4.0; third-party response bodies retain their original rights.

python3 analyze.py --run evidence --output ../rebuilt-v2.0

Cite: SearchVitals. AI Crawler robots.txt Policies, v2.0, November 10, 2026. Include the version and collection window when reusing a number.

Supplemental v2.1 evidence and reproduction

This analysis reuses the frozen v2.0 window; the original measurements and downloads are unchanged. Download both the v2.0 dataset ZIP above and the supplemental ZIP below. The supplemental bundle includes frozen AST measurements, targeted review notes, checksums and offline analysis code. Independent AST re-extraction also needs PHP and league/commonmark 2.8.2; see the method.

• Supplemental findings (JSON)
• All domain profiles (CSV)
• Configuration-review cases (CSV)
• All candidate structural measurements (CSV)
• Targeted review notes (JSON)
• Method and data dictionary
• Supplemental analysis bundle (ZIP)
• Supplemental checksums
• Offline analysis source

python3 deepen.py --base-archive ../v2.0/dataset.zip --features markdown-features.json --reviews review.json --output ../rebuilt-v2.1

Site-purpose data and reproduction · v2.2

The supplement keeps the frozen v2.0 measurements and v2.1 analysis separate. The ZIP includes labels, recorded decisions and evidence references, source/alias sensitivities, methods and Python standard-library scripts. Raw homepage HTML is not redistributed. Per-domain review notes are in Chinese, with English machine category keys.

• 1,000-domain classifications (CSV)
• Classifications joined to v2.0 policies (CSV)
• Category counts and denominators (CSV)
• Full summary and sensitivity checks (JSON)
• Shared final robots URL groups (CSV)
• Policy counts by AI token and category (CSV)
• Category composition by rank band (CSV)
• Classification method and limits
• Complete site-purpose analysis package (ZIP)
• File checksums (JSON)
• Analysis reproduction script (Python)

Extract the v2.2 ZIP into v2.2/ and the v2.0 ZIP into evidence/. From their parent directory, run the command below. It reads the frozen v2.0 sites.csv and metrics.json in evidence/, makes no network requests, and requires a new output directory.

python3 v2.2/analyze.py --frozen v2.2 --base evidence --work v2.2 --output rebuilt-site-types

Definition and format sources

• OpenAI
• Anthropic
• llms.txt
• league/commonmark

Method sources

• RFC 9309
• Google robots.txt parser
• Tranco 94P72
• llms.txt

Questions

What does the restriction rate mean?

249 of 854 analyzable files (29.2%) match Disallow at / for at least one of 12 AI-related tokens. It excludes the Googlebot and Bingbot baselines, unknown responses and unobserved paths.

Are failed requests counted as AI blocks?

No. HTTP outcome, body classification, declared rules and RFC defaults are separate. A 403 or 500 response is not a declared Disallow match.

Does blocking GPTBot remove a site from ChatGPT Search?

OpenAI documents GPTBot and OAI-SearchBot separately for training collection and automatic Search crawling. This report measures declarations, not Search inclusion or citations.

Does a detected llms.txt prove adoption or more citations?

No. This version reports a minimum H1 candidate from the root URL, with the response body available for further review. It does not certify full format compliance or measure provider use or citation gains.

Can the new data be compared with v1.1?

The domain cohort is fixed, but both collection time and content validation changed. The version comparison describes the observed difference; it cannot isolate policy changes from method and retrieval effects.