Field report from server logs
What AI crawlers actually do on small sites: OAI-SearchBot vs GPTBot vs ClaudeBot
This report counts every hit from named AI crawlers on Growth Radar's own small sites, read from first-party server logs rather than estimated. Each hit is checked against the IP ranges the crawler's operator publishes, so impersonators are counted apart. Every table states its n, its window and how many sites contributed, and the rows download as CSV.
Window: 2026-10-04 to 2026-10-31, the last 28 whole days in UTC. 7 sites had a named crawler visit. Download every table as CSV.
OAI-SearchBot, GPTBot and ClaudeBot side by side
OAI-SearchBot indexes pages for ChatGPT search, GPTBot collects pages for OpenAI's models, and ClaudeBot collects pages for Anthropic's. A verified hit came from an address inside the operator's published ranges.
- OAI-SearchBot
- 2 IP-verified of 2 claimed · impersonators 0.0% · 1 page
- GPTBot
- 133 IP-verified of 137 claimed · impersonators 1.5% · 46 pages
- ClaudeBot
- 32 claimed · not checkable, no published IP list · 31 pages
Every AI crawler that visited
Hits claimed is every request whose user agent named the crawler. Failed IP check means the request came from outside the operator's published ranges: someone borrowing the name. Not checkable means the operator doesn't publish a range list we can read, so the hit is neither confirmed nor called fake.
| Crawler | Hits claimed | IP-verified | Failed IP check | Not checkable | Impersonator share | Pages fetched | Sites |
|---|---|---|---|---|---|---|---|
| GPTBot | 137 | 133 | 2 | 2 | 1.5% | 46 | 1 |
| Meta-WebIndexer | 112 | 0 | 0 | 112 | not measured | 31 | 2 |
| Amazonbot | 43 | 0 | 0 | 43 | not measured | 43 | 1 |
| ClaudeBot | 32 | 0 | 0 | 32 | not measured | 31 | 1 |
| DuckAssistBot | 5 | 0 | 0 | 5 | not measured | 1 | 1 |
| Meta-ExternalAgent | 3 | 0 | 0 | 3 | not measured | 1 | 1 |
| OAI-SearchBot | 2 | 2 | 0 | 0 | 0.0% | 1 | 1 |
| ChatGPT-User | 1 | 1 | 0 | 0 | 0.0% | 1 | 1 |
n = 335 hits claiming an AI crawler · 2026-10-04 to 2026-10-31 (28 days, UTC) · 4 sites contributing
How much crawler traffic is AI?
AI share is named AI crawler hits divided by named AI plus named search-engine crawler hits, with impersonators taken out of both. Requests flagged as bots that aren't a named crawler, such as headless browsers and scripts, are left out; dividing by them too can only give the same or a smaller share of the same traffic. In this window it reads 0.7% against 16.2% here. Both left-out counts are listed so either version can be recomputed.
| Measure | Value | In the share |
|---|---|---|
| Named AI crawler hits | 333 | numerator and denominator |
| Named search-engine crawler hits | 1,724 | denominator |
| AI share | 16.2% | result |
| Hits that failed the IP check | 3 | left out |
| Other bot-flagged hits | 42,751 | left out |
| Impersonator share of IP-checked AI crawler hits | 1.4% | separate |
n = 2,057 named crawler hits after impersonators are removed · 2026-10-04 to 2026-10-31 (28 days, UTC) · 7 sites contributing
Do AI crawlers fetch llms.txt?
Requests for /llms.txt and /llms-full.txt by named crawlers, impersonators excluded. A fetch shows the file was read, not that it changed what any assistant answers.
| Crawler | Kind | /llms.txt | /llms-full.txt |
|---|---|---|---|
| Baidu Spider | search | 3 | 0 |
| Yandex Bot | search | 1 | 0 |
| GPTBot | AI | 2 | 2 |
| Googlebot | search | 0 | 1 |
| Amazonbot | AI | 1 | 0 |
n = 10 llms.txt and llms-full.txt fetches by named crawlers · 2026-10-04 to 2026-10-31 (28 days, UTC) · 1 site contributing
How to verify GPTBot, ClaudeBot and other AI crawlers
- Log the client IP address of every request whose user agent names a crawler, not only the user agent.
- Fetch the IP range list the crawler's operator publishes, the URLs below, and refresh it daily.
- A request from inside the ranges is the real crawler; from outside, it's an impersonator.
- When an operator publishes no list, record the hit as not checkable instead of guessing.
| Crawler | Published IP list | Prefixes loaded | List fetched | Hits checked |
|---|---|---|---|---|
| Baidu Spider | none published in the lists read | none | never | 0 |
| Yandex Bot | none published in the lists read | none | never | 0 |
| GPTBot | https://openai.com/gptbot.json | 18 | 2026-11-01 | 135 |
| Meta-WebIndexer | none published in the lists read | none | never | 0 |
| Googlebot | https://developers.google.com/static/search/apis/ipranges/googlebot.json | 317 | 2026-11-01 | 23 |
| Bingbot | https://www.bing.com/toolbox/bingbot.json | 28 | 2026-11-01 | 41 |
| Amazonbot | none published in the lists read | none | never | 0 |
| DuckDuckGo | https://duckduckgo.com/duckduckbot.json | 486 | 2026-11-01 | 0 |
| ClaudeBot | none published in the lists read | none | never | 0 |
| DuckAssistBot | none published in the lists read | none | never | 0 |
| Meta-ExternalAgent | none published in the lists read | none | never | 0 |
| OAI-SearchBot | https://openai.com/searchbot.json | 39 | 2026-11-01 | 2 |
| ChatGPT-User | https://openai.com/chatgpt-user.json | 235 | 2026-11-01 | 1 |
n = 202 hits checked against a published IP list · 2026-10-04 to 2026-10-31 (28 days, UTC) · 1 site contributing
Is there a wider benchmark?
Not published yet. We only cite the cross-account benchmark once at least 5 separate accounts contribute; 1 does today.
Method and limits
Each site forwards crawler requests from its server to Growth Radar with the path, the status code and the client address. Crawlers are named from the user agent, then checked against published IP ranges. The sample is a few small sites run by one operator, so a count of a handful is a handful: read it as what happened here, not as a rate for the web. The figures are read live on every visit and the window moves daily. Any site can check its own crawler access with the free AI visibility checker.
Questions about this report
Whose sites are these numbers from?
Only sites run by the person who builds Growth Radar. Customer sites are never counted here. The sample is small and says nothing about large sites. We publish it because it's measured, not because it's representative.
How is a crawler hit verified?
The site forwards each crawler request with its client address. That address is compared with the IP ranges the crawler's operator publishes. A hit from inside the ranges is verified, and one from outside is an impersonator. If the operator publishes no list, the hit counts as not checkable rather than either.
Why does AI share differ from other figures for the same sites?
AI share here is named AI crawler hits divided by named AI plus named search-engine crawler hits, with impersonators removed from both. A figure that divides by every bot-flagged request, headless browsers and scripts included, only adds hits to the denominator, so it reads the same or lower for the same traffic. In this window it reads 0.7% against 16.2% here. Both counts are in the table so either can be recomputed.
Do AI crawlers fetch llms.txt?
The llms.txt table shows which named crawlers requested /llms.txt or /llms-full.txt in the window. When none did, the section says not measured. A site that doesn't forward those requests looks the same as one nobody asked for, so we don't report the absence as a zero.
Is there a benchmark across many sites?
There is a cross-account benchmark, and we only cite it here once at least 5 separate accounts contribute to it. Below that, nothing from it is published, because a handful of accounts could be identified from their deciles.