Build a valid robots.txt file with allow and disallow rules, crawl-delay, and sitemap references — for every search engine or specific bots. Live output as you click, then copy or download. 100% in your browser — nothing is uploaded or stored.
Tip: robots.txt controls crawling, not indexing. To keep a page out of search, use a noindex robots meta tag as well.
robots.txt in your site's root directory: https://yoursite.com/robots.txt
🔒 Everything runs in your browser. Your input is never uploaded, logged or stored.
The Robots.txt Generator builds a valid robots.txt file that tells search-engine crawlers which parts of your site they may or may not request. Choose a quick preset or set per-bot allow and disallow rules, add an optional crawl-delay, and include your XML sitemap reference — all with live output you can copy or download in one click.
It is part of the Crawl & Indexing group in our free SEO Toolkit. After generating your file, confirm a rule blocks the right URL with the Robots.txt Tester, and remember that to keep a page out of search results you also need the Meta Robots Tag Generator. Everything runs in your browser — your input is never uploaded or stored.
Generate correct robots.txt syntax without memorising the directives.
Start from Allow All, Block All or a WordPress template, then fine-tune the rules to fit your site.
Target all crawlers or specific bots like Googlebot, Bingbot, GPTBot and more with their own rules.
Add an optional crawl-delay and link your XML sitemap so crawlers find your pages efficiently.
Copy the file or download a ready-made robots.txt to upload to your site's root directory.
From preset to a ready file in under a minute.
Choose Allow All, Block All, WordPress or Custom as your starting point.
Choose the user-agent and list the paths to disallow or allow, one per line.
Optionally add a crawl-delay and your sitemap URL so crawlers find your pages.
Copy or download the file and place it at yoursite.com/robots.txt.
A robots.txt file is a plain-text file placed in the root of your website that gives instructions to search-engine crawlers about which URLs they may request. It uses a simple set of directives — User-agent, Disallow, Allow, Crawl-delay and Sitemap. This generator builds those directives for you from a visual form, so you avoid syntax mistakes that could accidentally block your whole site. It is part of the Crawl & Indexing group.
| Directive | What It Does | Example |
|---|---|---|
| User-agent | Names the crawler the rules apply to | User-agent: * |
| Disallow | Blocks a path from being crawled | Disallow: /admin/ |
| Allow | Permits a path inside a blocked folder | Allow: /admin/ajax.php |
| Crawl-delay | Asks crawlers to wait between requests | Crawl-delay: 10 |
| Sitemap | Points crawlers to your XML sitemap | Sitemap: …/sitemap.xml |
An empty Disallow: line means "allow everything", while Disallow: / blocks the entire site — a single slash makes a big difference, which is why a generator helps.
This is the most common robots.txt mistake. Blocking a URL in robots.txt stops crawlers from reading it, but the URL can still appear in search results without a description if other pages link to it. To truly keep a page out of the index, allow it to be crawled and add a noindex meta robots tag, or use an X-Robots-Tag header for non-HTML files.
Keep crawlers away from low-value pages so they spend more time on the content that matters.
Discourage crawling of admin, cart and internal paths that should not appear in search.
The Sitemap directive helps engines discover all your important URLs quickly.
Set different rules for search bots, image bots or AI crawlers like GPTBot.
It must live in your site's root so it is reachable at https://yoursite.com/robots.txt. A file in a subfolder will not be read. On WordPress you can add it via your SEO plugin or by uploading the file to the root directory.
Reputable search engines like Google and Bing follow it, but it is a request, not a hard barrier. Malicious bots can ignore it. For genuinely private content, use server-side protection such as authentication, not robots.txt alone.
The generator only creates text. Review the output before replacing a live file, and test it with the Robots.txt Tester. A wrong rule — especially Disallow: / — can deindex a site, so always check before uploading.
No. The file is generated locally in your browser with JavaScript. Nothing you enter is uploaded, logged or stored, so it is safe to draft rules for unpublished or staging sites.
Block /wp-admin/ while allowing admin-ajax.php, the standard WordPress pattern.
Keep cart, checkout and account pages out of crawl paths to focus on products.
Allow or block AI bots like GPTBot and CCBot depending on your content policy.
Use Block All to keep a development site out of search until it is ready.
Use the Robots.txt Generator with these tools from the SEO Toolkit: verify rules with the Robots.txt Tester, build your sitemap with the XML Sitemap Generator, control page indexing with the Meta Robots Tag Generator and X-Robots-Tag Generator, set redirects with the .htaccess Redirect Generator, and complete your head tags with the Meta Tag Generator.
A clean, valid robots.txt ready for your site's root.
# Example for a WordPress site User-agent: * Allow: /wp-admin/admin-ajax.php Disallow: /wp-admin/ Crawl-delay: 10 Sitemap: https://yoursite.com/sitemap.xml
Anyone who manages how search engines crawl their website.
Everything about generating and using robots.txt.
It builds a valid robots.txt file from a visual form. You pick a preset or set per-bot allow and disallow rules, add an optional crawl-delay and sitemap reference, and the correct syntax is generated live. You then copy or download the file and place it in your site's root directory.
Yes, completely. There is no cost, no sign-up and no limit on how many files you generate. The tool runs entirely in your browser, so there are no server costs and nothing to pay for.
It must be in your website's root so it is reachable at https://yoursite.com/robots.txt. A file placed in a subfolder will be ignored by crawlers. On WordPress, you can add it through your SEO plugin or by uploading the file to the root.
Yes. All processing happens locally in your browser with JavaScript. Nothing you enter — paths, bot names or sitemap URLs — is uploaded, logged or stored. This makes it safe to draft rules for staging or unpublished sites.
No. Robots.txt controls crawling, not indexing. A blocked URL can still appear in results without a description if it is linked elsewhere. To keep a page out of the index, allow crawling and add a noindex meta robots tag instead.
A single Disallow: / blocks crawlers from your entire site, which can remove it from search. Use it only for staging sites you want hidden. The generator includes a Block All preset for this, but always double-check before uploading to a live site.
Reputable crawlers like Googlebot and Bingbot respect it, but it is a request rather than a hard barrier. Malicious or non-compliant bots can ignore it. For truly private content, use server-side protection such as a password, not robots.txt alone.
Yes. Select the specific user-agent — such as GPTBot or CCBot — and disallow the paths you want to protect. This lets you control whether AI training crawlers can access your content while keeping search bots allowed.
Crawl-delay asks crawlers to wait a set number of seconds between requests, which can ease load on small servers. Google ignores it, but some other engines honour it. Use it only if your server struggles with crawl traffic.
After creating the file, use the Robots.txt Tester in this toolkit to check whether a specific URL is allowed or blocked by your rules. Google Search Console also offers a robots.txt report once the file is live on your site.
Yes. The generator is fully responsive, so you can build a robots.txt file from a phone or tablet. The presets, fields and the copy and download buttons all work the same as on desktop.
Convert, compress, and resize images in multiple formats — JPG, PNG, WebP, ICO, and more.
Calculate in-hand salary, CTC breakup, tax deductions, and more for Indian employees.
Convert length, weight, temperature, speed, volume, and 200+ measurement units instantly.
Count words, convert case, generate lorem ipsum, find & replace text, and more writing utilities.
Build a valid robots.txt with the right crawl rules in seconds — free, private and instant in your browser.
🤖 Open the Generator ← All Crawl & Indexing Tools