What does the tool do?
Cross-check robots.txt, sitemap declarations, sitemap XML, and sampled page indexability.
Crawl architecture
Robots + Sitemap Cross Checker
Read robots.txt and sitemap XML together, then find blocks, redirects, noindex, broken responses, and canonical mismatches in a bounded sample.
Interpret the tool correctly
Method, data, and limitations
How does it work?
The checker reads robots.txt, follows a bounded sitemap index, then inspects only the configured sample limit.
What data is analyzed?
Robots directives, sitemap XML locations, HTTP status, redirects, meta robots, and canonical tags.
How should results be interpreted?
Sitemap URLs should be reachable, indexable, unblocked, and self-canonical.
Limitations
- — The public scan is deliberately bounded.
- — JavaScript-rendered directives are not evaluated.
- — Large crawls belong in the queue-backed audit workflow.
Privacy and data retention
- — Only the origin you submit is contacted, from our server, within the configured crawl limit.
- — We store the tool key, the origin, the score, and how many URLs were sampled — not the crawled documents.
- — Credential-like query parameters are redacted before the run is recorded.
- — Your IP address is stored only as an irreversible hash for abuse protection.
- — Results are not published, are not indexed, and never appear in our XML sitemap.