About this tool
Review how one crawler product token and request target are treated by an authored file without confusing crawl permission with indexing, rendering, or access control.
Robots.txt Tester evaluates one pasted RFC 9309 rule set against one crawler product token and one request target. It preserves group boundaries, merges exact matching groups, uses wildcard groups only as a fallback, normalizes comparable URI octets, and explains the longest matching Allow or Disallow rule.
- Parses up to 500 KiB of UTF-8 text, preserves RFC group boundaries, merges matching product-token groups, and falls back to wildcard groups only when no exact group exists.
- Compares path plus query as normalized octets with case-sensitive *, terminal $, empty-rule, equivalent-rule, and longest-match behavior.
- Reports the selected groups, ordered matches, winning line, ignored records, and validated sitemap URLs while keeping deployment and crawler-specific behavior outside the verdict.
How to use Robots.txt Tester
Paste the authored file, enter an exact product token such as Googlebot, and provide one root-relative path plus query or absolute HTTP(S) URL. Review the selected group type, ordered matching rules, winning source line, ignored invalid records, and sitemap inventory before checking the same file at the live origin.
When this tool is useful
- Test whether important templates or folders are blocked before deploying robots.txt changes.
- Check specific user-agents like Googlebot against directories or parameter patterns.
- Audit client robots files quickly without opening a separate crawler or desktop SEO suite.
Practical tips
- Test both wildcard and exact product tokens because an exact group is selected without adding wildcard rules.
- Include query strings in representative targets and test case differences, percent escapes, wildcards, and terminal anchors explicitly.
- Use authentication or authorization for private content, and verify the deployed file, status, redirects, and crawler documentation separately.
Examples you can test
Load an example, compare the result with the expected output, then replace it with your own input.
Allow a public child path
Example input
Disallow: /private and Allow: /private/public for /private/public/guide
Expected output
Allowed by the longer /private/public rule
The evidence includes the selected group, both matches, source line, and rule specificity.
Test an anchored file pattern
Example input
Disallow: /*.php$ against /download.php?preview=1
Expected output
Allowed because the query prevents the terminal anchor from matching
The request target includes the query component; fragments are never part of an HTTP request target.
Validation checklist
- Enter the crawler product token rather than a full HTTP User-Agent string.
- Review ignored lines and confirm empty Allow or Disallow records create no restriction.
- Check the deployed root-level /robots.txt response and crawler-specific documentation after editing rules.