Utility tools
Robots.txt Tester
Parse robots.txt rules, inspect sitemap directives, and test crawler access.
Paste robots.txt directly, or enter a site URL to fetch its /robots.txt file. External fetches try direct access, the ViWeb Cloudflare proxy, then the ViWeb utils proxy.
Robots.txt
Results
Fetch or parse robots.txt to inspect directives.
Access Test
Choose a user agent and URL, then run Test URL.
Directive Groups
Directive groups will appear here.