Skip to content
RookAtlas
A glossy white gateway with three lanes on ultramarine: two rounded crawler capsules pass through, the third lane is closed by a glass panel with a shield.

Which AI crawlers to allow in robots.txt: GPTBot, OAI-SearchBot and the rest

What each AI crawler does, from OAI-SearchBot and GPTBot to ClaudeBot and Google-Extended, which ones decide whether answers cite you, and a robots.txt to start from.

6 min read

Every AI engine that cites the web sends crawlers to read it, and most makers now run more than one. One finds pages for answers, one fetches a page when a person asks about it, and one collects training data. In a server log they look alike, but blocking each one costs something different.

This guide lists the crawlers each maker documents, what each is for, and a robots.txt that keeps you in AI answers while leaving the training question to you.

On this page
  1. 01Three jobs, three kinds of crawler
  2. 02The crawlers each maker documents
  3. OpenAI
  4. Anthropic
  5. Perplexity
  6. Google
  7. Meta
  8. xAI
  9. 03A robots.txt that keeps you in the answers
  10. 04Check the layer above robots.txt
  11. 05What blocking training crawlers costs
  12. 06How to check your site

Three jobs, three kinds of crawler

Each maker splits its crawling by purpose, and the purpose is what matters for visibility:

  • Search crawlers build the index an engine searches when it writes an answer. Block one and the engine has little of yours to cite.
  • User fetchers load a page because a person asked about it, for example by pasting a link into a chat. Several makers say these may not follow robots.txt, since a person asked.
  • Training crawlers collect text to train future models. Blocking them doesn't remove you from answers that search the web.

So "should I block AI crawlers?" is really three questions. Most sites that want to be recommended should answer the first two with yes and treat the third as a business decision.

The crawlers each maker documents

These are the tokens the makers publish themselves. Community lists name many more, with behaviour nobody has confirmed; it's safer to write rules for documented tokens than for guesses.

OpenAI

  • OAI-SearchBot finds pages for ChatGPT search answers. OpenAI says sites that block it won't appear in ChatGPT search answers, apart from navigational links, and that a change takes about a day.
  • ChatGPT-User fetches a page when a ChatGPT user's request needs it. OpenAI says robots.txt rules may not apply to it.
  • GPTBot collects training data. It's independent of OAI-SearchBot: blocking it opts out of training only.
  • OAI-AdsBot checks the landing pages of ads submitted to ChatGPT; it has nothing to do with organic answers.

Anthropic

  • Claude-SearchBot crawls to improve the quality of Claude search results.
  • Claude-User fetches a page when someone asks Claude a question that needs it.
  • ClaudeBot collects training data.

Anthropic says all three respect robots.txt, including the non-standard Crawl-delay line.

Perplexity

  • PerplexityBot builds the index Perplexity answers from. Perplexity says it isn't used to train models, and blocking it makes your pages unlikely to appear.
  • Perplexity-User fetches pages for people's requests, and Perplexity says it generally ignores robots.txt.

Google

  • Googlebot crawls for Google Search, which Google AI Overviews and Google AI Mode answer from. A page Googlebot can't crawl can't be a supporting link there; blocking it also removes you from Search itself.
  • Google-Extended isn't a crawler but a robots.txt token. It decides whether Google may use your pages to train Gemini models and to ground answers in the Gemini app. It doesn't affect Google Search.

That second point is the trap: many sites block Google-Extended to opt out of training and, without meaning to, drop out of Gemini's grounded answers too.

Meta

Meta documents three crawlers for Meta AI. Its assistant Muse cites the web, so these are the ones that matter there:

  • Meta-WebIndexer crawls to improve Meta AI's search results; Meta says allowing it helps cite and link your pages in its AI answers.
  • Meta-ExternalAgent crawls for training and for indexing content directly.
  • Meta-ExternalFetcher fetches pages for a user's request and, Meta says, may bypass robots.txt.

xAI

xAI documents no crawler for Grok, so robots.txt has no Grok token to allow or block. Being mentioned on the pages Grok cites is the more reliable lever.

A robots.txt that keeps you in the answers

A starting point that lets every documented search and user crawler in and opts out of training, which you can drop if you're happy to be trained on:

# Search and user crawlers: let them read the site
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Meta-WebIndexer
Disallow: /account/
Allow: /

# Training only: your choice
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Meta-ExternalAgent
Disallow: /

# Everyone else
User-agent: *
Disallow: /account/

Two details make or break a file like this:

  1. A crawler obeys only the group that names it. Under the robots.txt standard (RFC 9309) a crawler follows the most specific group matching its name and ignores the * group. If your * group keeps crawlers out of /account/, repeat that line in every named group, as above, or the named crawlers may go where you didn't intend.
  2. Leave Googlebot and Google-Extended alone unless you mean it. Googlebot falls under the * group here; Google-Extended isn't listed, so Gemini may still ground answers in your pages.

Check the layer above robots.txt

robots.txt is a request a crawler reads. A CDN or firewall in front of your site can turn a crawler away before it reads anything, and then your robots.txt doesn't matter.

Cloudflare is the common case. Since July 2025 it has blocked AI crawlers by default on new domains, and since July 2026 it sorts them into search, agent and training bots that can be allowed or blocked separately, on every plan. If your site sits behind Cloudflare or another CDN, open its bot settings and check that search crawlers are allowed there too.

Google adds one more switch. Search Console's "Search generative AI" control decides whether your site may appear in AI Overviews, AI Mode and Discover's generative AI features. Sites are included by default, and Google says a site must be included to be eligible; it's worth a look if someone on your team has visited that setting.

What blocking training crawlers costs

Answers that search the web don't depend on training crawlers. Answers that don't search lean on what a model learned in training, and some engines answer many questions without searching at all. Opting out of training doesn't remove you from search-based answers, but it does change what the next models learn about you.

No published study has measured that effect on recommendations, so treat it as a licensing and business decision, not a ranking one. Whatever you choose, choose it per crawler, with the trade-off in view.

How to check your site

  1. Open yoursite.com/robots.txt and find the group each crawler above falls under, remembering that a named group replaces the * group.
  2. Check your CDN's or firewall's bot settings for blocked search or user crawlers.
  3. Look for the crawlers in your server or CDN logs. OpenAI, Anthropic and Perplexity publish the IP ranges their crawlers use, so you can tell a real crawler from someone borrowing its name.
  4. Recheck after a redesign or a CDN change, which is when these rules most often break.
Free tool, no sign-upAI crawler checkerTest a full URL, including its query, against live or pasted robots.txt. See the winning rule for Googlebot and AI crawlers, with search and training roles explained.Run the free check

Questions

Does blocking GPTBot remove me from ChatGPT?

No. GPTBot collects training data only. Whether ChatGPT's search answers can show your pages depends on OAI-SearchBot, which OpenAI documents as a separate crawler.

What is Google-Extended?

A robots.txt token Google reads, not a crawler. Blocking it opts out of training Gemini models and also keeps your pages out of grounded answers in the Gemini app. It doesn't affect Google Search, AI Overviews or AI Mode.

How long does a robots.txt change take to apply?

OpenAI says about a day for OAI-SearchBot, and Perplexity says up to 24 hours. Search engines also cache robots.txt, so allow a day or two before reading anything into the results.

Do AI crawlers obey robots.txt?

The search and training crawlers their makers document say they do. The user fetchers are different: OpenAI, Perplexity and Meta say theirs may not follow robots.txt, because a person asked for the page.

Sources

  1. OpenAI: overview of OpenAI crawlers developers.openai.com
  2. Anthropic: does Anthropic crawl data from the web? support.claude.com
  3. Perplexity: Perplexity crawlers docs.perplexity.ai
  4. Google: AI features and your website developers.google.com
  5. Google: common crawlers (Google-Extended) developers.google.com
  6. Meta: web crawlers developers.facebook.com
  7. RFC 9309: Robots Exclusion Protocol rfc-editor.org
  8. Cloudflare: AI crawlers blocked by default (July 2025) cloudflare.com
  9. Cloudflare: search, agent and training bot controls (July 2026) blog.cloudflare.com
  10. Google Search Console Help: Search generative AI control support.google.com
All articles

RookAtlas opens soon.

RookAtlas asks your buyers' questions on eight AI engines and shows how often you're named, next to whom and from which sources, with a 95% interval on visibility and citation rates.