Free tool, no signup

AI crawler access checker

robots.txt says which AI crawlers may read a site, but a firewall can still turn them away. This checker reads robots.txt for 15 AI crawlers, then requests the homepage with each crawler's published user agent and compares the answer with what a normal browser gets.

How the check works

  1. Read robots.txt

    We fetch /robots.txt and apply RFC 9309 to each crawler: its own group if it has one, otherwise the * group, longest rule wins.

  2. Ask as each crawler

    We request the homepage with each crawler's published user agent, then with a normal browser's, and compare the two answers.

  3. Name what blocks it

    A 403, a Cloudflare challenge or a different status for the crawler than for the browser is a firewall block, whatever robots.txt says.

The 15 crawlers checked

  • OAI-SearchBot · OpenAI
  • ChatGPT-User · OpenAI
  • GPTBot · OpenAI
  • Claude-SearchBot · Anthropic
  • Claude-User · Anthropic
  • ClaudeBot · Anthropic
  • PerplexityBot · Perplexity
  • Perplexity-User · Perplexity
  • Google-Extended · Google
  • Applebot-Extended · Apple
  • CCBot · Common Crawl
  • Bytespider · ByteDance
  • meta-externalagent · Meta
  • Amazonbot · Amazon
  • DuckAssistBot · DuckDuckGo

Questions

Which AI crawlers should I allow?

Search and user-triggered crawlers (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, Claude-User) fetch pages to answer questions, so blocking them keeps you out of those answers. Training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) only collect pages for model training; blocking them is a business choice that doesn't stop citations.

robots.txt allows a bot but it's still blocked. Why?

CDNs like Cloudflare can block or challenge AI crawlers by user agent no matter what robots.txt says. Cloudflare's 'Block AI bots' setting is a common cause. The probe column shows what the homepage answered to each crawler's user agent.

Is the probe the same as the real crawler?

No. We send the crawler's published user agent from our own servers, not from OpenAI's or Anthropic's addresses. A firewall that checks IP addresses may treat the real crawler differently, so a pass here shows there's no user-agent block, not that every request will get through.

What does Google-Extended control?

Google-Extended is a robots.txt switch only. It decides whether Gemini models may train on pages Googlebot already crawls. It doesn't affect Google Search or AI Overviews, which use Googlebot.