robots.txt Rule Tester
Test crawler groups, Allow/Disallow rules and URL paths locally
Rules and paths stay in this browser. Nothing is uploaded, saved or requested from the target site.
Matches User-agent, Allow and Disallow using RFC 9309. Supports * and a final $. Sitemap entries are listed, not fetched.
Use only letters, hyphens or underscores in the product token, not a full User-Agent string. A full URL contributes only its path and query.
Paste rules, then choose Test locally.
Applicable crawler groups
Rule that determines the result
Parsing notes
Sitemap declarations (not fetched)
How to read this
This only simulates path matching in the pasted text. It does not fetch the live robots.txt or prove that a search engine has crawled, indexed or kept anything private. Crawlers may differ.
Up to 256 KiB of text, 1000 rules and 200 groups; excess is rejected.