When robots.txt lets an AI crawler in but the firewall refuses its name

We took 4,000 sites from the Tranco list and could test 2,250 of them. The share that refused at least one crawler name stays close to one in ten from the top 1,000 down to the millionth site.

What changes with rank is the kind of refusal. To tell them apart we sent two more names to each site: a plain one with no bot word in it, and Googlebot. The last two columns count the sites where both of those got through. That share goes from 4.2% in the top 1,000 to 8.4% in the last band, mostly because fewer sites further down the list refuse the Googlebot name too.

The bands are slices of very different size, so we don't quote a share for the sample as a whole. The weighted row counts each band for its share of the list's testable sites, so the last band counts for most of it. It only speaks for sites like the ones we could test.

Of the 244 sites, 157 let both the plain name and the Googlebot name through. A plain name passing doesn't rule out a rule against anything that looks like a bot, so we went back to those sites with one more request under a made-up name with "bot" in it. 155 of them still served the browser-like request, and none refused the made-up bot, while 154 let it through. So on these sites the refusal came from the particular names we sent rather than anything bot-like in general.

79 sites refused the Googlebot name as well, and that's what you'd expect from a firewall that checks whether a bot really is who it says it is, because our server isn't Google. Those sites may well welcome the real crawlers, so we wouldn't read them as blocking AI at all. They lean toward the top of the list: 30 of the 50 in the top 1,000.

Only 2 sites refused the plain name too, and on 6 a control ended without a clear answer. That kind of site is rare here partly because we seldom reached it: anything that refused the browser-like request was left out from the start.

Requests using the GPTBot and ClaudeBot names were each refused on 8.1% of the sites whose robots.txt allowed those crawlers, and the CCBot name on 7.0%. Those three crawlers gather data for training. The names of crawlers that fetch pages for search or for someone asking a question came in lower, at 4.0% for OAI-SearchBot, 4.5% for ChatGPT-User and 4.7% for PerplexityBot. On 116 of the 244 sites, the training names were the only ones refused, and 48 sites refused all six.

Each name has its own denominator, because we only sent a name robots.txt allowed. The gap holds on the 1,981 tested sites that allowed all six: 8.0% for GPTBot, 8.0% for ClaudeBot and 6.7% for CCBot, against 3.7% for OAI-SearchBot and 4.1% each for ChatGPT-User and PerplexityBot. CCBot's published user agent also has a different shape from the others, so the 26 sites that refused that name alone may be reacting to the format.

It would be easy to read this as sites contradicting themselves, and that isn't what we found. Of the 759 refusals, only 49 came from a file with rules written for that crawler by name. 646 fell under the general rules for every bot, and 64 had no rule that applied, usually because there was no robots.txt at all. A site can block a crawler at the firewall on purpose and never touch robots.txt, and from outside we can't tell that from an accident.

15 of the 244 files had a content signal line saying no to AI training but still let the crawler fetch the page under the usual rules. We counted those as allowed, because that line isn't a fetch rule. And where we could see who answered, 81 of the 157 sites that passed both controls were served through Cloudflare, which served 34.0% of all the sites we tested. Only 25 of those 81 came back with one of Cloudflare's recognisable block pages, so we can't say whose rule it was.

Roughly four sites in ten couldn't answer the question, and leaving them out changes what the number means, so here they all are. A site that refuses a browser-like request from a cloud server can't tell us anything about crawler names, and that group alone is 558 sites. We don't know how many of them would have refused a crawler name too, so nothing on this page describes them.

Plenty of crawler requests also ended in something other than a clean refusal, and none of these are in the headline. Across all six names, on tested sites, we counted 63 rate limit responses, 31 timeouts, 41 dropped connections or similar failures, 18 not-found pages, 34 other client errors, 16 server errors and 1 reply that was nearly empty or titled like a denial. Some of those are probably quiet refusals, so the headline leans low on that count.

The biggest limit is that we aren't the crawler. A request that says GPTBot from our server is a request pretending to be GPTBot, and a careful firewall is right to turn it away. So a refusal here doesn't mean OpenAI's real crawler is refused on that site, and the Googlebot control only catches firewalls that check Google's identity too. One that verifies AI crawlers alone would still look like a rule against them, and our follow-up with the made-up bot can't separate that case either.

The probe can also miss blocks. A firewall that blocks the published address ranges of real crawlers would never fire for us, so that kind of block is invisible here. We also add a short note to every user agent to identify ourselves, so a rule that matches the crawler's exact string would miss us.

We can't see intent. Many of these sites may have chosen this deliberately and be perfectly happy with it. We tested home pages only, from one cloud server in the United States, rechecked from one in Germany, on one day, and a firewall can treat other pages or other visitors differently. The list also includes infrastructure domains and several domains run by the same company, so sites aren't fully independent of each other, though removing Google's country domains left the shares where they were.

The browser-like request was refused more often late in the run than early, 120 of the first 1,000 sites against 162 of the last 1,000, which may mean our server's standing slipped as it went. The refusal share among tested sites stayed flat over the same period, and it didn't rise with a name's position in the shuffled order on each site.

The sample comes from the Tranco list JZPGY, created 4 November 2026 from rankings over 6 October to 4 November 2026. We tested the whole top 1,000 and drew 1,000 sites at random from each of ranks 1,001 to 10,000, 10,001 to 100,000 and 100,001 to 1,000,000, with seed 20261105. We expected a bit more than half to be testable, and about 500 tested sites per band puts the interval within roughly three points either way at a rate near 10%. Tranco's authors ask to be cited: Le Pochat et al., "Tranco: A Research-Oriented Top Sites Ranking Hardened Against Manipulation", NDSS 2019.

On 5 November 2026 one cloud server in the eastern United States asked each home page for itself with a browser-like user agent, then read robots.txt, then sent the same request under six AI crawler names plus the two controls in a random order, then repeated the browser-like request. Only the user agent changed. We only sent a name our reading of robots.txt allowed. A later correction to that reading found 12 sites whose files had said no to a name we sent, 33 requests in all, and those are left out of every count.

A refusal means a 401, 403, 406 or 451 response, or a recognised challenge page, with the browser-like request working both before and after. Robots.txt is read by the standard's rules at the root and at the page the home page lands on, and a file that names a crawler with a full disallow doesn't count as allowing it, even if another line also allows it. Intervals are 95% Wilson intervals, and the weighted row uses a normal approximation.

We kept the load low. We read robots.txt and the home page and nothing deeper, sent no cookies and submitted nothing. A typical site got 11 requests spaced two to three seconds apart, more only when a request redirected (we followed up to three) or failed (we retried once). The main pass made 29,611 requests, counting redirects and retries, at no more than five a second, each with a 10 second timeout. Every request, including the browser-like one, carries "GrowthRadarStudy/1.0" and a link to our contact page, so anyone can see who sent it. A 50-site trial run came first, and the recheck and follow-up below came after, so some sites got a second visit.

We drew 30 refusals at random and read the raw responses and robots.txt by hand. On 29 of them the site really had refused the name after serving the browser-like request both times, and robots.txt really did allow that crawler. One was wrong: its file named the crawler with a full disallow and also a line allowing it, which the standard's tie rule reads as allowed even though the owner plainly wrote no. That's a 3.3% error rate, and with only 30 looked at the true rate could be as high as 16.7%. We changed the rule so those files no longer count as allowing the crawler, which took 3 sites off the count of refusing sites. This check only looked at refusals, so it says nothing about sites we counted as letting a name through, and we didn't draw a fresh set after the change.

We also sent the same requests again from a server in Germany, to the 244 sites that had refused a name and to 304 that hadn't. Of the refusing sites, 232 could be tested again and 231 of them refused again. Counting each site and name separately, 695 of 725 refusals repeated. Of the 289 quiet sites we could test again, 2 refused a name this time. So the result holds across two servers in two countries, and that still says nothing about the real crawlers, since both servers are ordinary cloud machines and neither belongs to them.

Before publishing we gave the raw data and a draft of this page to a separate AI reviewer told only to try to break them. It recomputed the counts and they held, and it found no sign that a few operators were behind them. Its objections changed the page in four ways. We dropped a share pooled across the bands in favor of the weighted row, and added the made-up bot request after it pointed out that our plain control has no bot word in it. We reworded the crawler figures so they talk about names, and we stopped counting an empty page or a denial page as the browser getting through, which moved the tested total from 2,266 to 2,250.

The CSV has one row for each of the 4,000 sites: rank band, what robots.txt said to each crawler, and how each request ended. Site names are left out, because the point is the pattern and a refusal from our probe isn't evidence against any one site. We plan to run this again each quarter and publish it as a new snapshot. Next review: 5 February 2027.

If you want to see what your own site does, the AI crawler checker runs a similar comparison for one domain, and your firewall logs will show what the real crawlers got. For what crawlers did on sites we run ourselves, see our AI crawler study.