Utility tools
Robots.txt Tester
Parse robots.txt rules, inspect sitemap directives, and test crawler access.
Robots.txt
Results
Fetch or parse robots.txt to inspect directives.
Access Test
Choose a user agent and URL, then run Test URL.
Directive Groups
Directive groups will appear here.
Test robots.txt rules before search crawlers find surprises
The robots.txt tester fetches or parses robots.txt rules, lists sitemap directives, and checks whether a user agent can access a URL. Use it for SEO audits, site launches, staging protection, and crawl-budget reviews.
Testing a specific user agent and path helps catch broad disallow rules, missing sitemap references, and accidental crawler blocks.