robots.txt Tester for Googlebot and AI Crawlers
Google retired its own robots.txt Tester. Paste your file, enter a path, and see which crawlers are allowed or blocked, including the AI crawlers that decide whether ChatGPT and Perplexity can cite you.


How to Test robots.txt Rules
Paste your robots.txt
Open yourdomain.com/robots.txt in a browser and paste the whole file.
Enter a path
Use the path of the page you care about, such as /blog/ or /services/aeo-audit/.
Read the verdict
Each crawler shows Allowed or Blocked, plus the exact rule that decided it.


How Crawlers Read robots.txt Rules
A crawler follows the group whose User-agent line matches its name most specifically, and falls back to User-agent: * only if no named group exists. Inside that group, the rule with the longest matching path wins, and when an Allow and a Disallow rule are equally long, Allow wins. This tester follows the same logic Google documents in its robots.txt specification, including the * and $ wildcards.
Two AI-specific points matter. Google-Extended only controls whether your content is used for Gemini model training; it does not remove you from Google AI Overviews, which use normal Googlebot crawling. And blocking OAI-SearchBot keeps you out of ChatGPT search results, while GPTBot only concerns model training.
robots.txt is not the only thing that blocks AI crawlers. Cloudflare bot settings, a firewall rule or a security plugin can block them even when robots.txt allows them. If the tester says Allowed but AI engines still never cite you, see how to fix blocked AI crawlers.


robots.txt tester FAQs
Does Google still have a robots.txt tester?
Google retired the standalone robots.txt Tester in Search Console in late 2023 and replaced it with the robots.txt report, which shows the files Google fetched and any parsing errors. It does not let you test a path against a rule set, which is what this tool does.
Should I block GPTBot?
Blocking GPTBot stops OpenAI from using your pages for model training. It does not by itself remove you from ChatGPT search, which uses OAI-SearchBot. If you want to be cited in ChatGPT answers, keep OAI-SearchBot allowed.
Does blocking Google-Extended remove me from AI Overviews?
No. Google-Extended only controls use of your content for Gemini training and grounding in some Gemini products. AI Overviews and AI Mode are part of Google Search and rely on Googlebot.
Is my robots.txt sent anywhere?
No. The test runs entirely in your browser. Nothing you paste is uploaded.




Related tools and services
Make AI recommend your business, starting this week
Hire an AEO expert who works from your data, not guesses.