Free tool · No signup
Robots.txt Tester
Enter a page URL. We read the site's live robots.txt and tell you which bots may crawl it, and why.
Do this for your whole site, every week.Skyfliq tracks rankings, audits your site, finds backlinks and writes SEO content — next to your social, email and CRM.
Try Skyfliq freeHow matching works
We follow Google's rules: a bot uses the group with its own name (or * if none), the longest matching rule wins, and Allow wins a tie. * matches anything and $ marks the end of the URL.
Blocked doesn't mean hidden
A blocked page can still show in Google without a description if other sites link to it. To keep a page out of results, allow crawling and add noindex.
Questions
What if there's no robots.txt?
Then every bot may crawl every page. That's fine for most small sites.
What if robots.txt returns an error?
Google treats a 5xx server error as 'crawl nothing' until it's fixed. A 404 means 'crawl everything'.