Free tool · No signup
Robots.txt Generator
Pick what to allow or block, then copy or download your robots.txt.
All bots
Block AI bots
Your robots.txt
User-agent: * Allow: /wp-admin/admin-ajax.php Disallow: /wp-admin/ Disallow: /cart/ Disallow: /checkout/ Sitemap: https://example.com/sitemap.xml
Try a path
Googlebot: blockedBingbot: blockedGPTBot: blockedClaudeBot: blockedPerplexityBot: blocked
Do this for your whole site, every week. Skyfliq tracks rankings, audits your site, finds backlinks and writes SEO content, alongside your social, email and CRM.
Try Skyfliq freeWhere it goes
Upload the file to the root of your site so it loads at https://yourdomain.com/robots.txt. One file per host (www and non-www count separately).
What it can't do
robots.txt asks crawlers not to fetch pages. It doesn't hide them: a blocked page can still be indexed if other sites link to it. Use noindex for that.
Questions
Should I block AI bots?
It's your choice. Blocking search-style bots (OAI-SearchBot, PerplexityBot) can keep you out of AI answers. Blocking training bots (GPTBot, ClaudeBot, Google-Extended) doesn't affect Google Search.
Does Google respect Crawl-delay?
No. Google ignores Crawl-delay. Bing and some others respect it.