About this tool
Inspect X-Robots-Tag headers on live responses before publishing or migrating assets so PDFs, media files, and documents do not send conflicting indexing directives.
The X-Robots-Tag Checker inspects HTTP response headers to surface indexing directives that operate outside of HTML markup, covering PDFs, images, feeds, and other non-HTML resources that cannot carry meta tags. It follows redirect chains and reports directives at each hop so you can see whether an intermediate proxy or CDN layer is injecting unexpected noindex or nofollow signals. This is the only reliable way to audit crawl directives on assets that lack a document head.
- Fetches URL responses with redirect-hop visibility and records X-Robots-Tag header values per hop.
- Parses final response directives, flags conflicts like index with noindex, and checks preview directive values.
- Helps QA robots controls on non-HTML assets where meta robots tags are unavailable.
How to use X-Robots Checker
Provide a URL for any resource type and the tool issues a request, captures every X-Robots-Tag header in the response (including across redirects), and parses the directives for each user-agent scope present. It highlights conflicts between hops, overly broad noindex rules that may block assets you intend to index, and missing directives on resources you want to exclude. Adjust your server, CDN, or proxy configuration based on the findings and re-check to confirm the headers are correct.
When this tool is useful
- Validate indexing directives on PDFs, docs, images, or feeds where HTML meta tags are not available.
- QA CDN, proxy, or server-level header rules after migrations or infrastructure changes.
- Inspect redirect chains when a final asset appears indexable or blocked unexpectedly.
Practical tips
- Check both the first response and the final response in redirect flows because directives may change between hops.
- Use X-Robots-Tag for non-HTML assets and meta robots for HTML pages when you need layered crawl controls.
- Keep header directives explicit and minimal to avoid ambiguous signals like index plus noindex.
Why people use this tool
X-Robots-Tag headers are invisible in page source and browser developer tools' Elements panel, making them the most commonly overlooked crawl directive. A CDN rule or reverse proxy can inject a noindex header that silently removes an entire class of assets from search results without any visible change to the page itself. Auditing these headers is essential during migrations, CDN configuration changes, and any infrastructure work that touches response header policies.
Related search intents
x-robots-tag checker, http robots header validator, noindex header check, robots header seo tool.