How to use it
Pick a ready template in simple mode, or define rule groups one by one in advanced mode. Copy the output or download it as .txt.
CRY Tools
Robots.txt Generator
Build crawler rules with correct syntax: multiple user-agent groups, allow and disallow rules, and sitemap declarations.
robots.txt
This tool runs entirely in your browser; your data is never sent to our server.
Examples
Blocking /admin for Googlebot while declaring your sitemap URL.
Use cases
New site launches, protecting staging, crawl budget management, e-commerce filter pages.
Privacy and data handling
Rules are generated in your browser; the paths and sitemap URL you type are never sent to our server or stored.
What this tool checks
Produces a robots.txt with Allow and Disallow rules grouped by user-agent, plus sitemap directives.
How to read the result
robots.txt controls crawling, not indexing. A disallowed address can still be indexed if something links to it - and without a description, because the content could not be read. To keep a page out of the index the right tool is a noindex meta tag, which requires the page to be crawlable in order to be seen. Using both together is a common and self-defeating mistake.
Common mistakes
The common mistake is listing a directory you want kept private: robots.txt is public, so this announces the address. The second is blocking CSS and JavaScript, which prevents the page from being rendered correctly. The third is placing the file anywhere other than the domain root, where it has no effect.
What this tool cannot see
The tool generates the file. It does not upload it or test your existing rules; verify the behaviour after publishing.
Frequently asked questions
Does Disallow remove my page from the index?
No. Disallow blocks crawling, it does not control indexing. Use noindex to keep a page out of the index.
Where must robots.txt live?
Only at the domain root: example.com/robots.txt. A file in a subdirectory is ignored.
Is the sitemap line required?
Not required, but useful - declaring your sitemap speeds up discovery.