Research
Research from Growth Radar
So far we've published two studies. The first follows what AI crawlers and AI assistants did on nine small sites we run over four weeks. The second checks a sample of 4,000 sites for a robots.txt that lets an AI crawler in while the site turns away a request using that crawler's name. Each study explains how we measured it and what it can't tell you, and you can download the data behind it.
When robots.txt lets an AI crawler in but the firewall refuses its name
We sampled 4,000 sites from the Tranco list and checked how often robots.txt lets an AI crawler in while a request carrying that crawler's name gets refused.
Published Data as CSV
What AI crawlers did on the nine sites we run
Four weeks of AI crawler requests and AI assistant visits on nine small sites we run, from 6 October to 2 November 2026.
Published Updated Data as CSV
New studies, guides and product changes also go out in our Atom feed.