Robots.txt Generator
Build a clean, standards-compliant robots.txt file in a few clicks. Pick your platform, choose what to block — including AI training crawlers — and copy or download the result. No email required.
Build your rules
Your robots.txt
https://yoursite.com/robots.txtWhat is a robots.txt file?
robots.txt is a plain-text file that lives at the root of your domain and tells search-engine crawlers which parts of your site they may and may not request. Used well, it keeps low-value pages (admin areas, internal search, carts) out of crawl budgets and out of the index.
The rules in plain English
- User-agent — which crawler the rules apply to.
*means all of them. - Disallow — a path the crawler should not request.
- Allow — an exception that re-permits a path inside a disallowed folder.
- Sitemap — points crawlers to your XML sitemap so they discover pages faster.
- Crawl-delay — asks (some) bots to wait between requests. Googlebot ignores it; set crawl rate in Search Console instead.
Common mistakes to avoid
Never use Disallow to hide a page you don't want indexed — blocked pages can still appear in results without a snippet. Use a noindex meta tag for that. And remember robots.txt is public: anyone can read yoursite.com/robots.txt, so don't list secret paths there.
Want us to audit your crawl setup, indexation and site architecture end to end? That's part of our technical SEO service — or run a quick free SEO audit first.