Robots.txt Tester
Enter a page URL. We read the site's live robots.txt and tell you which bots may crawl it, and why.
What this robots.txt tester does
Enter the full address of a page and pick a bot. The tester downloads the live robots.txt from that site's root, works out which group of rules applies to the bot, and tells you whether it may crawl that exact path (including any ?query) — and which line decides it. You can test as Googlebot, Bingbot, Googlebot-Image, GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot or the * group.
Every common bot at once
Below the answer, a table tests the same path for 19 crawlers: the search bots Googlebot, Bingbot, Googlebot-Image, DuckDuckBot, YandexBot, Baiduspider and Applebot, and 12 AI bots from OpenAI, Anthropic, Perplexity, Google, Apple, Common Crawl, Meta and ByteDance. Each row shows Allowed or Blocked and the rule that matched, so one test shows whether a page is open to search engines but closed to AI training bots, or the other way round.
How matching works
We follow Google's rules: a bot uses the group with its own name (or * if none), the longest matching rule wins, and Allow wins a tie. * matches anything and $ marks the end of the URL.
The file and its warnings
The tester also shows the robots.txt it read, with warnings for lines it couldn't understand, Allow or Disallow rules that come before any User-agent line (crawlers ignore them), and unknown field names — often a typo such as Dissallow. If the site has no robots.txt, or the address returns an HTML page instead of a text file, every bot may crawl everything; if it returns a server error, you'll see a note on how Google handles that.
Blocked doesn't mean hidden
A blocked page can still show in Google without a description if other sites link to it. To keep a page out of results, allow crawling and add noindex.
Common mistakes it catches
Disallow: / under User-agent: * blocks the whole site — often left over from a staging site. A group for Googlebot replaces the * group for Google, so rules under * no longer apply to it. Paths are case-sensitive: /Blog/ and /blog/ are different. Blocking CSS or JavaScript folders can stop Google rendering your pages properly. And a blocked page with a noindex tag can stay indexed, because Google can't crawl it to see the tag.
Questions
What if there's no robots.txt?
Then every bot may crawl every page. That's fine for most small sites.
What if robots.txt returns an error?
Google treats a 5xx server error as 'crawl nothing' until it's fixed. A 404 means 'crawl everything'.
Does it test my live robots.txt?
Yes. It fetches the current file from the site's root on every test, so you can edit the file and test again straight away.
Can I write a new robots.txt here?
Use the free robots.txt generator to build one, upload it to your site's root, then test your important pages here.
Is the robots.txt tester free?
Yes. It's free and needs no signup.