robots.txt Tester

A single wrong line in robots.txt can accidentally block your entire site from search engines — or fail to block the pages you meant to hide. This tester removes the guesswork. Fetch any site's live robots.txt or paste your own draft, then enter a URL and a crawler to see instantly whether it is allowed or blocked, and exactly which rule made the decision. Crucially, it implements the real matching logic search engines use: the most specific user-agent group applies, the longest matching path wins, Allow breaks ties, and wildcards (*) and end-anchors ($) are fully supported. That means the verdict here matches what Googlebot would actually do — not a naive prefix check that gets edge cases wrong.

robots.txt problems are silent and expensive: pages quietly vanish from search, or private areas stay crawlable. This tool helps you catch both. Test important URLs against your rules, check how different crawlers — including AI bots like GPTBot and CCBot — are treated, and verify Allow exceptions inside a Disallowed folder work as intended. Because the editor is live, you can draft and refine rules with confidence before pushing them to production. The fetch runs through a secure server endpoint (browsers cannot read another domain's robots.txt directly), and it also surfaces any sitemap declarations found in the file. Whether you are launching a site, running an SEO audit, or debugging a sudden traffic drop, it is the fastest way to know exactly what your robots.txt really does.