About this tool
Compose a standards-focused robots.txt draft and review validation evidence before deploying it at the site root.
Robots.txt Generator creates a bounded RFC 9309 policy draft from explicit user-agent groups, root-relative Allow and Disallow patterns, and optional absolute Sitemap records. It validates product tokens, paths, URLs, duplicates, and UTF-8 output size before exposing the file.
- Validates RFC 9309 product tokens and root-relative path patterns.
- Deduplicates rules and reports groups, agents, rules, sitemaps, and UTF-8 bytes.
- Copies or downloads only an error-free robots.txt draft.
How to use Robots.txt Generator
Choose an example or enter one value per line in each group. Review the group inheritance warning, longest-match implications, public-path security warning, byte evidence, and absolute sitemap URLs, then copy or download the draft and deploy it as UTF-8 text/plain at the exact root path `/robots.txt`.
When this tool is useful
- Validates RFC 9309 product tokens and root-relative path patterns.
- Deduplicates rules and reports groups, agents, rules, sitemaps, and UTF-8 bytes.
- Copies or downloads only an error-free robots.txt draft.
Examples you can test
Load an example, compare the result with the expected output, then replace it with your own input.
Create a basic robots file
Example input
User-agent: *, disallow: /admin, sitemap: https://example.com/sitemap.xml
Expected output
robots.txt rules with disallow and sitemap directives
A basic robots file can guide crawlers away from private utility paths while exposing the sitemap.
Allow public pages while blocking staging paths
Example input
Disallow paths: /staging, /internal
Expected output
Rules that block selected paths for all user agents
Robots rules are crawl hints and should not be used to protect sensitive content.
Allow a longer exception
Example input
Disallow: /media/ and Allow: /media/public/
Expected output
One wildcard group with both path rules
RFC 9309 uses the most specific path match, so review overlapping rules with representative deployed URLs.
Validation checklist
- Do not use robots.txt as access control for private or sensitive URLs.
- Test generated rules before deploying them at the domain root.
- Keep sitemap URLs absolute and aligned with the canonical host.
- Repeat shared rules inside specific crawler groups because they do not inherit the wildcard group.
- Serve UTF-8 text/plain at the top-level /robots.txt path for the exact protocol, host, and port.