Free tool · No signup
Robots.txt Generator
Pick what to allow or block, then copy or download your robots.txt.
All bots
Block AI bots
Your robots.txt
User-agent: * Allow: /wp-admin/admin-ajax.php Disallow: /wp-admin/ Disallow: /cart/ Disallow: /checkout/ Sitemap: https://example.com/sitemap.xml
Try a path
Googlebot: blockedBingbot: blockedGPTBot: blockedClaudeBot: blockedPerplexityBot: blocked
Do this for your whole site, every week.Skyfliq tracks rankings, audits your site, finds backlinks and writes SEO content — next to your social, email and CRM.
Try Skyfliq freeWhere it goes
Upload the file to the root of your site so it loads at https://yourdomain.com/robots.txt. One file per host (www and non-www count separately).
What it can't do
robots.txt asks crawlers not to fetch pages. It doesn't hide them: a blocked page can still be indexed if other sites link to it. Use noindex for that.
Questions
Should I block AI bots?
It's your choice. Blocking search-style bots (OAI-SearchBot, PerplexityBot) can keep you out of AI answers. Blocking training bots (GPTBot, ClaudeBot, Google-Extended) doesn't affect Google Search.
Does Google respect Crawl-delay?
No. Google ignores Crawl-delay. Bing and some others respect it.