Research
Original data studies from SearchVitals. The crawler-policy report includes versioned data, an offline reproduction script, sources and a correction log.
robots.txt policies
2026-11-09
AI Crawler robots.txt Policies — November 2026
A new collection of 1000 frozen domains: 249 of 854 analyzable robots files (29.2%) restrict an AI-related token at /. Request logs and root llms.txt evidence included.
Read the report arrow_forwardFuture studies will investigate verified network access and web performance separately from declared crawler policies.