Founding$29/mo for life · ends December 31

Free tool, no signup

Robots.txt tester

Google retired the robots.txt Tester in Search Console in late 2023. This one fetches a site's live robots.txt, tells you whether Googlebot, Bingbot and seven more crawlers, AI ones included, may crawl the URL you give it, names the rule that decided, and lets you edit the file and test again before you publish it.

How the test decides

  1. Read the live file

    We fetch /robots.txt and note what its HTTP answer means: a 404 allows everything, a 5xx makes Google stop crawling the whole site.

  2. Pick each crawler's group

    A crawler obeys the group that names it, or User-agent: * when none does. Groups naming the same crawler are merged; * is never added on top.

  3. Match the URL

    Rules are matched against the path and query, with * and $ wildcards. The longest match decides, and Allow wins a tie, per RFC 9309.

Four robots.txt mistakes that cost search traffic

  • Disallow: / left over from staging

    Site builders and staging setups ship with the whole site blocked. If Googlebot shows Blocked on your homepage, this is usually why.

  • Blocking a page to remove it from Google

    robots.txt hides the page from Google's crawler, so Google never sees a noindex on it, and can still list the bare URL. Allow it and use noindex instead.

  • Blocking CSS and JavaScript

    Google renders pages like a browser. Rules like Disallow: /assets/ can stop it seeing your layout and content, which hurts how the page is judged.

  • A Googlebot group that forgets the basics

    Once a group names Googlebot, Google ignores everything under User-agent: *. Copy the rules you still want into the Googlebot group.

Want to know how often robots.txt says yes while a firewall says no? We probed 4,000 sites for the robots.txt vs firewall study, and the AI crawler checker runs the same live probe on your site.

The 9 crawlers tested

  • Googlebot · Google Search and AI Overviews
  • Bingbot · Bing and Copilot search
  • OAI-SearchBot · ChatGPT search results
  • PerplexityBot · Perplexity answers
  • Claude-SearchBot · Claude search results
  • Applebot · Siri and Spotlight
  • GPTBot · OpenAI model training
  • ClaudeBot · Anthropic model training
  • Google-Extended · Gemini training (not Search)

Questions

How do I test whether a URL is blocked by robots.txt?

Enter the URL above. We read the site's /robots.txt, pick the group each crawler obeys and apply its rules to the URL's path and query. The longest matching rule wins, and Allow wins a tie with Disallow of the same length, which is how Google and RFC 9309 decide.

Does blocking a page in robots.txt keep it out of Google?

No. robots.txt stops crawling, not indexing. Google can still list a blocked URL, without a description, if other pages link to it. To keep a page out of Google, let Googlebot crawl it and add a noindex meta tag or X-Robots-Tag header.

Which group of rules does Googlebot follow?

The group whose User-agent line names it. If several groups name Googlebot, Google merges them. Only when no group names it does it fall back to User-agent: *, and it never combines its own group with the * group.

What happens when robots.txt returns an error?

A 404 or other 4xx means no rules: everything may be crawled. A 5xx or 429 is worse: Google treats the whole site as blocked and stops crawling until the file answers again, falling back to its last copy if the errors go on.

Should I block GPTBot?

GPTBot collects pages to train OpenAI's models. Blocking it doesn't remove you from ChatGPT's search answers; that's OAI-SearchBot. The same split holds for Anthropic (ClaudeBot trains, Claude-SearchBot searches) and Google (Google-Extended only controls Gemini training, not Search).