About this tool
Inspect X-Robots-Tag headers on live responses before publishing or migrating assets so PDFs, media files, and documents do not send conflicting indexing directives.
The X-Robots-Tag Checker inspects HTTP response headers to surface indexing directives that operate outside of HTML markup, covering PDFs, images, feeds, and other non-HTML resources that cannot carry meta tags. It follows redirect chains and reports directives at each hop so you can see whether an intermediate proxy or CDN layer is injecting unexpected noindex or nofollow signals. This is the only reliable way to audit crawl directives on assets that lack a document head.
- Fetches URL responses with redirect-hop visibility and records X-Robots-Tag header values per hop.
- Parses final response directives, flags conflicts like index with noindex, and checks preview directive values.
- Helps QA robots controls on non-HTML assets where meta robots tags are unavailable.
How to use X-Robots Checker
Provide a URL for any resource type and the tool issues a request, captures every X-Robots-Tag header in the response (including across redirects), and parses the directives for each user-agent scope present. It highlights conflicts between hops, overly broad noindex rules that may block assets you intend to index, and missing directives on resources you want to exclude. Adjust your server, CDN, or proxy configuration based on the findings and re-check to confirm the headers are correct.
When this tool is useful
- Validate indexing directives on PDFs, docs, images, or feeds where HTML meta tags are not available.
- QA CDN, proxy, or server-level header rules after migrations or infrastructure changes.
- Inspect redirect chains when a final asset appears indexable or blocked unexpectedly.
Practical tips
- Check both the first response and the final response in redirect flows because directives may change between hops.
- Use X-Robots-Tag for non-HTML assets and meta robots for HTML pages when you need layered crawl controls.
- Keep header directives explicit and minimal to avoid ambiguous signals like index plus noindex.
Examples you can test
These examples show the kind of real input and reviewed output this tool is designed to support. Use them as a starting point before pasting your own production content, then compare the output with the destination system that will use the result. The goal is not only to produce a value, but to make the input assumptions, output format, and review step clear enough that the result can be trusted in a real workflow.
Check a PDF header
Example input
URL to a PDF resource
Expected output
Detected X-Robots-Tag directives from response headers
Header-level robots directives are important for non-HTML files that cannot use meta tags.
Inspect redirect headers
Example input
URL with one redirect before final response
Expected output
Header directives at each hop
Intermediate redirects can carry directives that are easy to miss.
Validation checklist
Run through these checks before copying the result into a CMS, codebase, spreadsheet, campaign, support ticket, or production document. Small formatting differences, unit assumptions, hidden whitespace, and platform-specific rules are common sources of mistakes in quick browser tools, so the final review should happen in the same context where the output will be used.
- Check every response in a redirect chain.
- Use X-Robots-Tag for non-HTML resources when needed.
- Avoid broad noindex headers that accidentally affect important assets.
Why people use this tool
X-Robots-Tag headers are invisible in page source and browser developer tools' Elements panel, making them the most commonly overlooked crawl directive. A CDN rule or reverse proxy can inject a noindex header that silently removes an entire class of assets from search results without any visible change to the page itself. Auditing these headers is essential during migrations, CDN configuration changes, and any infrastructure work that touches response header policies.
Related search intents
x-robots-tag checker, http robots header validator, noindex header check, robots header seo tool.