About this tool
Build a reviewable robots.txt snippet that separates AI training controls from AI search discovery, applies bounded path rules, and leaves classic search groups untouched.
AI Crawler Policy Generator creates separate robots.txt groups for documented AI training controls, AI search crawlers, and optional user-requested fetchers. It deliberately avoids a User-agent: * group, so the generated snippet does not redefine classic search crawler policy.
- Covers nine documented tokens across training controls, AI search crawlers, and optional user-requested fetchers.
- Validates exact origins, absolute discovery URLs, path syntax, duplicates, conflicts, and exception coverage before enabling export.
- Emits a standard Sitemap directive and treats llms.txt only as an ignored explanatory comment, not a robots.txt directive.
- Exports the generated snippet and versioned JSON evidence while stating merge, deployment, identity, indexing, and compliance limits.
How to use AI Crawler Policy
Choose training and AI search access separately, decide whether to include user-requested fetchers, enter the exact site origin, and list protected path prefixes with any more-specific exceptions. Resolve every validation error, review warnings and token coverage, then merge the downloaded snippet into the existing root robots.txt after checking group interactions.
When this tool is useful
- Before publishing a new robots.txt policy after an AI search or AdSense quality reset.
- When deciding whether to allow AI search crawlers while blocking model-training crawlers.
- After adding or changing a sitemap, protected path prefixes, or an optional llms.txt reference.
Practical tips
- Do not block all AI crawlers unless you accept reduced visibility in assistant search experiences.
- Keep private paths protected by authentication; robots.txt is not access control.
- Treat Sitemap as a discovery directive and llms.txt only as a non-standard comment in this generated snippet.
- Do not assume user-requested fetchers obey robots.txt; review each provider's current documentation.
Examples you can test
Load an example, compare the result with the expected output, then replace it with your own input.
Block model training while keeping AI search discovery
Example input
Training: Block AI search: Allow Protected paths: /api/, /admin/, /private/ Sitemap: https://example.com/sitemap-index.xml
Expected output
GPTBot, Google-Extended, and ClaudeBot receive Disallow: /. OAI-SearchBot, Claude-SearchBot, and PerplexityBot receive the protected path rules.
This is often the most balanced policy for a site that wants search visibility without broadly allowing training crawlers.
Avoid accidental AI search opt-out
Example input
Training: Block AI search: Block Goal: improve AI search visibility
Expected output
Warning: broad blocking conflicts with the stated visibility goal. Use the training-only blocking mode instead.
Crawler names are increasingly purpose-specific, so a single broad block can create unintended discovery loss.
Validation checklist
- Confirm every protected path is also protected server-side if the content is sensitive.
- Verify that referenced sitemap and llms.txt URLs return the intended response before publishing the policy.
- Merge the snippet with the current root robots.txt and check duplicate user-agent group behavior.
- Check official crawler documentation for current user-agent tokens before deployment.
- Use Search Console and server logs after deployment to confirm important pages remain crawlable.