Founding$29/mo for life · ends December 31

Research

Research from Growth Radar

So far we've published two studies. The first follows what AI crawlers and AI assistants did on nine small sites we run over four weeks. The second checks a sample of 4,000 sites for a robots.txt that lets an AI crawler in while the site turns away a request using that crawler's name. Each study explains how we measured it and what it can't tell you, and you can download the data behind it.

  1. When robots.txt lets an AI crawler in but the firewall refuses its name

    We sampled 4,000 sites from the Tranco list and checked how often robots.txt lets an AI crawler in while a request carrying that crawler's name gets refused.

    Published Data as CSV

  2. What AI crawlers did on the nine sites we run

    Four weeks of AI crawler requests and AI assistant visits on nine small sites we run, from 6 October to 2 November 2026.

    Published Updated Data as CSV

New studies, guides and product changes also go out in our Atom feed.