Free Robots.txt Tester
Fetch any site's robots.txt, catch dead rules and typos, and test whether a URL is crawlable, for Googlebot, Bingbot, or the AI crawlers behind ChatGPT, Claude and Perplexity.
Why test your robots.txt?
One wrong character in robots.txt can silently remove your site from Google. A typo turns a rule into a no-op, an overbroad pattern blocks pages you need indexed, and a misconfigured server error can stop crawling entirely. Google retired its own tester, so most people only find out from a traffic drop.
The AI crawler question
Robots.txt is also where you decide whether GPTBot, ClaudeBot and PerplexityBot may read your site. Many sites blocked them by default in 2023 and are now invisible to AI assistants their customers ask for recommendations. Test what your file actually says before assuming.
FAQ
What does robots.txt actually do?
It tells crawlers which URLs they may fetch. It controls crawling, not indexing: a page blocked in robots.txt can still appear in Google if other sites link to it. To keep a page out of the index, use a noindex meta tag and let crawlers reach it.
Where did Google's robots.txt tester go?
Google retired the standalone tester from Search Console in 2023 and replaced it with a report that only covers your own verified properties. This tool tests any public site's robots.txt against Google's documented parsing rules.
Should I block AI crawlers like GPTBot and ClaudeBot?
It is a trade-off. Blocking them keeps your content out of AI training and AI answers; allowing them means assistants like ChatGPT can read, cite and recommend your site. If you sell something, being recommendable is usually worth more than the content protection.
Why is my page still indexed after I blocked it in robots.txt?
Blocking a URL stops crawling, not indexing. Google can index a blocked URL from links alone, showing it with no description. Worse, a robots.txt block prevents Google from ever seeing a noindex tag on that page. Unblock it, add noindex, and the page drops out properly.