Skip to content
RookAtlas

Is your site open to AI crawlers?

Enter a full URL or paste a robots.txt. See the exact rule that allows or blocks the path and query, then review each AI crawler's documented purpose: search, grounding, user requests or training.

Tests the exact path and query; the #fragment is ignored. In live mode we fetch that origin's robots.txt, not the target page. Nothing is stored.

Before a check runs, Cloudflare Turnstile confirms you're a person; it reads signals from your browser and your IP address for that. Cloudflare's privacy addendum

How the AI crawler checker works

It reads the robots.txt at the root of your site, or the text you paste, and asks, for every AI crawler its maker documents, the question that crawler asks: may I fetch this path and query? Choose a crawler to see its winning rule. A group that names that crawler replaces the wildcard group. URL fragments are ignored. Live mode reads robots.txt from the entered URL's origin; it does not fetch the target page.

How to read the result

  • Search and grounding crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Meta-WebIndexer, Google-Extended) decide whether answers can find and cite your pages. A block here is the one to fix.
  • User fetchers (ChatGPT-User, Claude-User, Perplexity-User, Meta-ExternalFetcher) load a page because a person asked about it. Some say they may not follow robots.txt at all.
  • Training crawlers (GPTBot, ClaudeBot, Meta-ExternalAgent) only collect training data. Blocking them is a choice, and it doesn't take you out of answers that search the web.

Review the winning rule before changing it. An appended Allow may lose to a longer rule, and private paths should remain private. To draft a file, use the AI robots.txt generator.

A path prefix can block more than a directory

Disallow: /app also matches /apple-icon.png. To restrict the app itself, test rules such as /app$, /app? and /app/ against both private and public URLs. Wildcards and end anchors matter, and query parameters are part of the match. For missing Google icons, run the favicon checker too.

What robots.txt can't tell you

A firewall or CDN in front of your site can turn a crawler away before it reads robots.txt, and pages that only build their text with JavaScript leave most AI crawlers with little to read. The AI page check looks at a page itself.

Questions

Should I block AI crawlers?

Block the ones that only collect training data if you don't want your pages in future models; that doesn't take you out of AI answers. Blocking the ones that fetch for search or for a person, such as OAI-SearchBot, ChatGPT-User, Claude-SearchBot or PerplexityBot, keeps your pages out of the answers they power.

Which crawlers are checked?

Those whose makers document a robots.txt token: Google, OpenAI, Anthropic, Perplexity and Meta. xAI documents no crawler for Grok, so Grok isn't listed. Some user-triggered fetchers say they may not follow robots.txt; they are marked as such.

How is a crawler's verdict decided?

The groups naming the crawler apply, the * group only when none does, and the longest matching rule decides, with Allow winning a tie. The test includes the URL path and query but ignores its fragment. Server errors and HTTP 429 produce an unknown result because crawlers may pause or use a cached file.

Can I check a file before it goes live?

Yes: select Pasted robots.txt, enter a full URL and choose a crawler. The pasted rules are applied to that URL without fetching the target page. The public tool still uses a human verification check.

Read more

RookAtlas opens soon.

RookAtlas asks your buyers' questions on eight AI engines and shows how often you're named, next to whom and from which sources, with a 95% interval on visibility and citation rates.