Paste your robots.txt and test whether any URL is allowed or blocked for a given crawler — using Google's longest-match rule. See exactly which line decides the verdict. 100% in your browser — nothing is uploaded or stored.
This applies the standard robots.txt matching used by Google, including * wildcards and $ end-anchors and longest-match precedence. Build rules with the Robots.txt Generator.
🔒 Everything runs in your browser. Your robots.txt is never uploaded, logged or stored.
The Robots.txt Tester lets you paste your robots.txt and check whether any URL is allowed or blocked for a specific crawler — before you publish. It applies the same matching logic Google uses, including * wildcards, $ end-anchors and the longest-match-wins rule, and it shows you the exact line that decides each verdict, so you never have to guess.
It is part of the Robots & Directives Helpers group in our free SEO Toolkit, and it is the perfect partner to the Robots.txt Generator — build your file there, then verify it here. To control indexing rather than crawling, use the Meta Robots Tag Generator, and for non-HTML files the X-Robots-Tag Generator. Everything runs in your browser — your robots.txt is never uploaded or stored.
Check exactly which URLs your robots.txt allows or blocks.
Instantly see whether a URL is allowed or blocked, with no ambiguity.
Tells you the exact Allow or Disallow line that decides the result.
Test as Googlebot, Bingbot, GPTBot and more — each can have different rules.
Correctly handles * wildcards, $ end-anchors and longest-match precedence.
From robots.txt to a clear verdict in seconds.
Drop in your robots.txt content, or load the sample to explore.
Type the URL or path you want to check.
Choose the crawler to test as, then click Test URL.
See allowed or blocked, plus the rule responsible.
A clear verdict with the deciding rule — build files in the Robots.txt Generator.
/admin/settingsBlocked for Googlebot by Disallow: /admin//admin/public/pageAllowed — longer Allow: /admin/public/ wins over the Disallow/files/guide.pdfBlocked by Disallow: /*.pdf$The tester simulates how a search engine reads your robots.txt. You paste the file, enter a URL and choose a crawler, and it tells you whether that URL would be allowed or blocked — and which rule decides. It follows the matching behaviour Google documents, so the result reflects what real crawlers would do. It is part of the Robots & Directives Helpers group.
Crawlers do not simply read rules top to bottom. They find every Allow and Disallow rule that matches the URL, then apply the most specific one — the rule with the longest path. If an Allow and a Disallow are equally specific, the Allow wins. Understanding this is the key to debugging why a URL is or is not blocked.
| Symbol | Meaning | Example |
|---|---|---|
| * | Matches any sequence of characters | Disallow: /*.pdf |
| $ | Anchors the match to the URL end | Disallow: /*.pdf$ |
| Prefix | A path matches anything starting with it | /cart blocks /cart and /cartoon |
| Longest match | The most specific rule wins | Allow: /admin/public/ beats Disallow: /admin/ |
It controls whether a crawler may fetch a URL, not whether it appears in search.
To keep a page out of results, use a noindex meta robots tag, not a Disallow.
A blocked page can still be indexed without a snippet if other sites link to it.
To deindex, allow crawling so the noindex tag can be read by the crawler.
It follows Google's robots.txt specification: path prefixes, the * wildcard for any characters, the $ anchor for the end of the URL, and longest-match precedence where the most specific rule wins and Allow beats Disallow on an exact tie. This mirrors how Googlebot evaluates a URL.
Because robots.txt controls crawling, not indexing. If other pages link to a blocked URL, Google may index it without a description. To truly remove it, allow crawling and add a noindex meta robots tag so the crawler can read and obey it.
Yes. robots.txt groups rules by user-agent. A crawler uses the most specific group that names it, falling back to the wildcard group. This tester lets you pick the bot so you can confirm each one gets the rules you expect, including AI crawlers like GPTBot.
No. Everything is parsed and tested locally in your browser with JavaScript. Nothing you paste is uploaded, logged or stored, so the tool is safe for staging files and confidential client configurations.
Make sure your new site is not accidentally blocking all crawlers.
Work out why a specific page is or is not being crawled.
Confirm cart and filter URLs are blocked while products stay crawlable.
Verify that bots like GPTBot are allowed or blocked as you intend.
Use the Robots.txt Tester with these tools from the SEO Toolkit: build the file with the Robots.txt Generator, control indexing with the Meta Robots Tag Generator, send directives for files with the X-Robots-Tag Generator, list URLs in the XML Sitemap Generator, set the canonical with the Canonical Tag Generator, and audit a page with the Meta Tag Analyzer.
Anyone who needs to be sure their crawl rules work.
Everything about testing your robots.txt rules.
It simulates how a search engine reads your robots.txt. You paste the file, enter a URL and pick a crawler, and it tells you whether that URL is allowed or blocked, plus the exact Allow or Disallow rule that decides the verdict, using Google's matching logic.
Yes, completely. There is no cost, no sign-up and no limit on how many URLs you test. The tool runs entirely in your browser, so there are no server costs and nothing to pay for.
Crawlers find every matching Allow and Disallow rule, then apply the most specific one — the rule with the longest path. If an Allow and a Disallow are equally specific, the Allow wins. The tester follows exactly this longest-match logic.
Yes. Everything is parsed and tested locally in your browser with JavaScript. Nothing you paste is uploaded, logged or stored, which makes the tool safe for staging files and confidential client configurations.
Not necessarily. robots.txt controls crawling, not indexing. A blocked URL can still appear in results without a description if other pages link to it. To remove a page, allow crawling and add a noindex meta robots tag.
Yes. It correctly handles the * wildcard, which matches any sequence of characters, and the $ anchor, which ties the match to the end of the URL. It also applies prefix matching and longest-match precedence like real crawlers.
Yes. You can test as Googlebot, Googlebot-Image, Bingbot, GPTBot or the wildcard user-agent. Because robots.txt can set different rules per bot, this lets you confirm each crawler, including AI bots, gets the rules you intend.
It lives at the root of your domain, at yourdomain.com/robots.txt. Open that URL, copy the contents, and paste them into the tester. You can also paste a draft you are working on before you upload it.
Test with the crawler you care about. For Google search, use Googlebot. For image search, use Googlebot-Image. To check AI crawler access, use GPTBot. The wildcard option shows what any unnamed bot would see.
Often a more specific Allow rule overrides a broader Disallow, since the longest match wins. The tester shows the deciding rule, so you can see whether an Allow is taking precedence and adjust your robots.txt accordingly.
Yes. The tester is fully responsive, so you can paste robots.txt and check URLs from a phone or tablet. Entering the URL, choosing the user-agent and reading the verdict all work the same as on desktop.
Convert, compress, and resize images in multiple formats — JPG, PNG, WebP, ICO, and more.
Calculate in-hand salary, CTC breakup, tax deductions, and more for Indian employees.
Convert length, weight, temperature, speed, volume, and 200+ measurement units instantly.
Count words, convert case, generate lorem ipsum, find & replace text, and more writing utilities.
Check whether any URL is crawlable for any bot — free, private and instant in your browser.
🤖 Open the Tester ← All Robots & Directives Tools