Measurements we ran ourselves and industry claims we traced to their sources: site migrations, redesigns, AI crawlers, and everything else that quietly changes the web. Written for SEO professionals who are tired of statistics nobody can footnote.
How we measure: the full method behind the AI-crawler dataset — sample, parser, denominators, limitations and a dated correction log — is at Methodology. It is open for comment, including the objections that make our numbers worse.
Drop the before and after crawls and get the report a client can act on: risky changes first, every row with its before and after value, pages that lost every crawlable path (and which upstream page took them down), redirects that turned temporary or grew a hop, mapped redirects that end at the wrong URL. Runs entirely in your browser.
Paste your robots.txt and see all 18 crawlers from our scan: allowed or blocked, whether you named the crawler or inherited the rule, and which ones are blocked only because a rule you wrote for a different crawler also governs them. Runs entirely in your browser.
A measured change rate for robots.txt, which nobody had published. The more useful finding is underneath it: of the files whose rules genuinely changed, 88% changed nothing an AI crawler would notice.
A robots.txt line declaring whether you allow AI training is about a year old. We found it on 1,104 of the web's most-visited domains, and found that 78.8% of them carry one byte-identical string a CDN wrote for them. Where a person wrote the line, 42.7% allow training.
We read robots.txt on the web's most-visited domains and found that sites block model-training crawlers several times more often than the AI-search crawlers that send readers back. Then we found that most of that gap was never a decision.
The most-quoted statistic in migration SEO, traced back through every citation we could find. What the evidence actually supports, and the honest numbers to use instead.