Crawl and indexing
robots.txt URL Tester
Paste your robots.txt and a list of URLs to see which are blocked for which crawler, and which line decides it.
Runs in your browser; nothing is uploaded. · Free, no account · 3 min
The site the robots.txt belongs to. Paths are tested against it.
Use the robots.txt token, such as Googlebot or GPTBot. * tests the catch-all group.
Importing a saved run into your account is coming soon. Until then, your last runs stay in this browser only, and you can download your files.
How to use
- Paste the contents of your robots.txt, or load a .txt file.
- Enter your site address and the crawler tokens to test, such as Googlebot or GPTBot.
- Paste the URLs or paths to check, one per line. Arabic paths are fine.
- Read the table: each row shows allowed or blocked and the line number of the rule that decided it.
What it checks and how it works
The tester applies the matching rules of the Robots Exclusion Protocol (RFC 9309): a crawler uses only the most specific group that names it, the longest matching path wins, and Allow beats Disallow when the match lengths tie. Rules are case-sensitive, as Google's documentation states. Matching runs in your browser with the robots-parser package.
A URL with no matching rule is allowed. Google does not support the crawl-delay field, so the tool reports it as information only. The tool reads the file you paste: it does not fetch your live robots.txt, so paste the version your server returns.
Sources
- RFC 9309: Robots Exclusion Protocol opened 29 Oct 2026
- Google Search Central: How Google interprets the robots.txt specification opened 29 Oct 2026
Related tools
AI Crawler robots.txt Builder
Choose which AI crawlers may read your site, and get a robots.txt that follows each vendor's own documentation.
AI Crawler User-Agent Lookup
Paste user-agent strings or log lines to identify AI crawlers, and see what each one is for.
Sitemap Validator
Paste a sitemap.xml or a sitemap index and see every protocol problem, with the line of the URL it affects.
Meta Robots and X-Robots-Tag Generator
Build a meta robots tag or X-Robots-Tag header from Google's documented rules, and see conflicts before you publish.
Questions
About this tool
Under RFC 9309 a crawler obeys only the most specific group that names it. If a group names GPTBot, GPTBot ignores the User-agent: * group entirely. The tester shows which line decided each result so you can see which group applied.
No. Nothing is fetched and nothing is uploaded. You paste the file, so you can test a change before you publish it. Open yoursite.com/robots.txt in a browser and paste what you see.
Paths are percent-encoded as UTF-8 before matching, the way a crawler requests them, so a rule written with the encoded form and a URL typed in Arabic match each other.
Get started today
Find out where your brand stands in AI. Free.
Run a free AI visibility analysis on your domain. No credit card, results in minutes. See what AI engines say about your brand today.
Brands already measuring their AI visibility with Zaher.
