Skyfliq
Free tool · No signup

Robots.txt Generator

Pick what to allow or block, then copy or download your robots.txt.

All bots
Block AI bots

Your robots.txt

User-agent: *
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-admin/
Disallow: /cart/
Disallow: /checkout/

Sitemap: https://example.com/sitemap.xml

Try a path

Googlebot: blockedBingbot: blockedGPTBot: blockedClaudeBot: blockedPerplexityBot: blocked
Do this for your whole site, every week. Skyfliq tracks rankings, audits your site, finds backlinks and writes SEO content, alongside your social, email and CRM.
Try Skyfliq free

Where it goes

Upload the file to the root of your site so it loads at https://yourdomain.com/robots.txt. One file per host (www and non-www count separately).

What it can't do

robots.txt asks crawlers not to fetch pages. It doesn't hide them: a blocked page can still be indexed if other sites link to it. Use noindex for that.

Questions

Should I block AI bots?

It's your choice. Blocking search-style bots (OAI-SearchBot, PerplexityBot) can keep you out of AI answers. Blocking training bots (GPTBot, ClaudeBot, Google-Extended) doesn't affect Google Search.

Does Google respect Crawl-delay?

No. Google ignores Crawl-delay. Bing and some others respect it.