Robots.txt Generator
Create a valid robots.txt file in seconds, right in your browser. Set rules per crawler, add your sitemap, and block AI bots if you need to.
Configuration
Generated robots.txt
How it works
The robots.txt file must be at the root of your website (https://yourdomain.com/robots.txt). Crawlers read it before browsing your site and follow its Allow and Disallow instructions.
Add as many rules as you need, include your sitemap URL, and, if you wish, block AI crawlers. The code on the right updates instantly and you can copy or download it directly.
Frequently Asked Questions
It's a text file at the root of your website that tells crawlers which parts of the site they can or cannot visit. It helps save crawl budget, protect internal sections, and point to the location of your sitemap.
It must be placed at the root of your domain, i.e. at https://yourdomain.com/robots.txt. If it's in any other location, search engines won't find it.
Disallow rules for AI crawlers (GPTBot, CCBot, ClaudeBot, Google-Extended, Bytespider) can stop compliant bots from downloading your content to train models. It's a voluntary rule, not an actual technical block.
Yes, once uploaded you can check its validity in Google Search Console, in the crawl settings section, or with other online robots.txt testing tools.
Other Tools
Need a full technical audit?
Robots.txt is just the start. We review speed, indexing, structure, and much more.
Request a Free Audit