Free tool, no sign-up

robots.txt Tester for Googlebot and AI Crawlers

Google retired its own robots.txt Tester. Paste your file, enter a path, and see which crawlers are allowed or blocked, including the AI crawlers that decide whether ChatGPT and Perplexity can cite you.

Close-up of a woman's hands typing on a laptop beside a mixing console in a recording studio.
Paste the whole file so no rule is missed.
Example robots.txt with a GPTBot block, and verdicts for six crawlers on /blog/ with the rule that decided each.
A single named group can block one crawler while every other bot stays allowed.

How to Test robots.txt Rules

1

Paste your robots.txt

Open yourdomain.com/robots.txt in a browser and paste the whole file.

2

Enter a path

Use the path of the page you care about, such as /blog/ or /services/aeo-audit/.

3

Read the verdict

Each crawler shows Allowed or Blocked, plus the exact rule that decided it.

An auto detailing owner works on a laptop on a rolling tool cart in his garage office.
Test the exact page paths that matter to your business.
Table of five example site paths and whether AI crawlers should reach each one, from service pages to the admin area.
Test the paths you want AI engines to quote, not only the home page.

How Crawlers Read robots.txt Rules

A crawler follows the group whose User-agent line matches its name most specifically, and falls back to User-agent: * only if no named group exists. Inside that group, the rule with the longest matching path wins, and when an Allow and a Disallow rule are equally long, Allow wins. This tester follows the same logic Google documents in its robots.txt specification, including the * and $ wildcards.

Two AI-specific points matter. Google-Extended only controls whether your content is used for Gemini model training; it does not remove you from Google AI Overviews, which use normal Googlebot crawling. And blocking OAI-SearchBot keeps you out of ChatGPT search results, while GPTBot only concerns model training.

robots.txt is not the only thing that blocks AI crawlers. Cloudflare bot settings, a firewall rule or a security plugin can block them even when robots.txt allows them. If the tester says Allowed but AI engines still never cite you, see how to fix blocked AI crawlers.

A boutique hotel owner and her IT contractor discuss a result on a laptop in the lobby.
Each crawler gets its own verdict and the rule that decided it.
Example where Allow /blog/ and Disallow /blog/drafts/ both match, showing the longer rule deciding each path.
The most specific rule wins, which is why one extra line can change a verdict.

robots.txt tester FAQs

Does Google still have a robots.txt tester?

Google retired the standalone robots.txt Tester in Search Console in late 2023 and replaced it with the robots.txt report, which shows the files Google fetched and any parsing errors. It does not let you test a path against a rule set, which is what this tool does.

Should I block GPTBot?

Blocking GPTBot stops OpenAI from using your pages for model training. It does not by itself remove you from ChatGPT search, which uses OAI-SearchBot. If you want to be cited in ChatGPT answers, keep OAI-SearchBot allowed.

Does blocking Google-Extended remove me from AI Overviews?

No. Google-Extended only controls use of your content for Gemini training and grounding in some Gemini products. AI Overviews and AI Mode are part of Google Search and rely on Googlebot.

Is my robots.txt sent anywhere?

No. The test runs entirely in your browser. Nothing you paste is uploaded.

A woman reviews a laptop at a desk by tall windows in a converted warehouse office.
One stray line can block a crawler from a whole section of a site.
Table of five robots.txt tokens, from GPTBot to Googlebot, and what blocking each one removes.
Blocking a training crawler and blocking a search crawler have very different costs.
A stone supplier stands among large granite slabs in his warehouse, phone in hand, looking relieved.
Search crawlers and AI crawlers follow their own rules in the same file.
Example of OAI-SearchBot allowed by robots.txt but stopped by a CDN challenge before reaching later gates.
An Allowed verdict here does not rule out a block at the CDN or firewall.

Make AI recommend your business, starting this week

Hire an AEO expert who works from your data, not guesses.