Study · 6 October to 2 November 2026
What AI crawlers did on the nine sites we run
We run nine small websites. In the 28 days from 6 October to 2 November 2026, OpenAI's crawlers made 157 confirmed requests to the one site where we record crawler requests on the server. Perplexity made 3. Over the same four weeks AI assistants sent 333 visits across all nine sites, and 322 of them came from ChatGPT. None of the nine block AI crawlers, and all nine publish an llms.txt file.
This is nine sites we run, not a benchmark of the web. They're small and they're ours, and we set them up so AI crawlers can read them easily. Read the numbers as what happened here.
By Growth Radar. Published 4 November 2026. Download the data as CSV.
OpenAI made almost every confirmed AI crawler request
OpenAI made 157 of the 160 confirmed AI crawler requests: GPTBot 154, OAI-SearchBot 2 and ChatGPT-User 1. Perplexity made the other 3. AI crawlers were 52% of all 307 confirmed crawler requests, ahead of Bingbot (84) and Googlebot (63).
GPTBot came in bursts. Of its 154 requests, 101 landed on 20 October, and it showed up on only 4 days. Drop that one day and OpenAI still leads Perplexity 56 to 3.
Requests claiming to be ClaudeBot, Amazonbot and Meta's crawler reached the site too, but we couldn't confirm who sent them, so they aren't counted anywhere on this page.
Crawling didn't bring visits to the site that got crawled
The site we log crawlers on got 160 confirmed AI crawler requests and 0 visits from an AI assistant. Visits did happen elsewhere. Assistants sent 333 visits to 4 of the 9 sites. Two sites got 331 of them, 2 more got one each, and 5 got none. ChatGPT sent 322 (97%) and Gemini 11. We saw none from Perplexity, Claude or Copilot.
About half, 167 of 333, arrived without a referrer, carrying only the tag ChatGPT adds to its links. If you only read referrers you'd miss them.
None of our sites block AI crawlers
All nine robots.txt files let the 12 AI user agents we checked fetch the homepage. We also requested each homepage as GPTBot, ClaudeBot and PerplexityBot, and got a normal 200 response 27 times out of 27.
We chose to leave the door open, so this isn't typical. Plenty of sites block these bots on purpose, and a test from our own connection can't tell you what a firewall does to the real crawler.
GPTBot reads llms.txt, rarely
All nine sites publish an llms.txt. On the server-logged site, GPTBot asked for /llms.txt twice and /llms-full.txt twice in four weeks, 4 of its 154 requests. No other confirmed crawler asked for either file. We can't tell from this whether it changed anything ChatGPT says.
How we counted
Crawler counts come from server-side request data on the only one of the nine sites where we keep it. A crawler is counted only when its claimed identity checks out, and our own traffic is excluded. Assistant visits come from first-party analytics on all nine sites. We couldn't separate our own visits from other visitors in our country, so every visit from that country is left out of the visit counts. Days are UTC. The robots.txt and homepage checks ran on 3 November 2026.
We'll refresh this every quarter. Next review: 4 February 2027. For numbers that move every day, see the rolling AI crawlers report.
Cite this
Growth Radar, "What AI crawlers did on the nine sites we run, 6 October to 2 November 2026", published 4 November 2026, https://growthradar.app/research/ai-crawlers
Want the same numbers for your own site? Run the free audit.