Free tool
Robots.txt Tester
Test whether a path is allowed for a chosen crawler, using live or pasted rules.
The check could not be completed
Robots.txt results
Rules applied
| Line | Rule | Path pattern | Matches |
|---|
Sitemaps declared
Lines that will be ignored
Generate a robots.txt draft
Builds a plain-text draft in your browser. Crawl-delay is included only when you ask for it, and not every crawler honours it the same way.
One path per line.
Google does not use Crawl-delay. Bing and some others may. Prefer rate limits in Search Console when available.
This is a diagnostic. Searqo applies the matching rules described in the robots exclusion standard, including the * and $ wildcards, but a given crawler may interpret an unusual file differently. robots.txt also controls crawling, not indexing: a blocked URL can still appear in results if other pages link to it.
What This Tool Does
The robots.txt tester takes a set of rules — either fetched from a live site or pasted in — and tells you whether a given path would be allowed or disallowed for a chosen crawler, showing which rule decided the outcome.
How to Use It
Either enter a domain to load its robots.txt, or paste rules directly. Then choose a user agent, enter the path to test, and run the check.
How the Tool Calculates or Retrieves Data
Rules are grouped by user-agent, and the group that matches your chosen crawler most specifically is used. Within that group the tool applies the most specific matching rule, measured by the length of the path pattern, and Allow wins when an Allow and a Disallow are equally specific. The * and $ wildcards are supported.
This mirrors the behaviour described in the robots exclusion standard and the documented behaviour of major crawlers.
Understanding Your Results
The verdict tells you whether crawling is permitted, and the matched rule tells you why. Remember that disallowing a URL prevents crawling, not indexing: a blocked page can still appear in results if other pages link to it. Use a noindex directive, which requires the page to be crawlable, when you want it out of the index.
Frequently Asked Questions
No. It stops the page being crawled, but the URL can still be indexed from links elsewhere. To remove a page, allow crawling and add a noindex directive, or restrict access properly.
The most specific path pattern wins. If an Allow and a Disallow are the same length, the Allow is applied.