Write a robots.txt for AI crawlers
Choose which AI crawlers may read your site, one by one or with a preset, add the paths no crawler should touch, and copy a robots.txt written the way crawlers actually read it.
- Google-ExtendedGroundingA robots.txt token, not a crawler: it decides whether Gemini apps may ground answers in your pages, and whether Google may train on them.
OpenAI
- OAI-SearchBotSearchFinds pages for ChatGPT's search answers.
- ChatGPT-UserUser requestsFetches a page when a ChatGPT user's request needs it. Its maker says it may not follow robots.txt.
- GPTBotTrainingCollects training data for OpenAI's models.
Anthropic
- Claude-SearchBotSearchCrawls to improve the quality of Claude's search results.
- Claude-UserUser requestsFetches a page when someone asks Claude a question that needs it.
- ClaudeBotTrainingCollects training data for Anthropic's models.
Perplexity
- PerplexityBotSearchBuilds Perplexity's search index; it isn't used for training.
- Perplexity-UserUser requestsFetches a page when a Perplexity user's request needs it. Its maker says it may not follow robots.txt.
Meta
- Meta-WebIndexerSearchCrawls to improve Meta AI's search results; Meta says allowing it helps cite and link your pages in its answers.
- Meta-ExternalFetcherUser requestsFetches a page when a user's request to Meta's AI needs it. Its maker says it may not follow robots.txt.
- Meta-ExternalAgentTrainingCrawls for training Meta's AI models and for indexing content directly.
Googlebot isn't listed: blocking it would take the site out of Google Search, not only its AI features, so it follows the rules for every crawler.
Everything happens in your browser: nothing you choose or type is sent anywhere.
Your robots.txt
9 of 12 AI crawlers allowed. Put it at the root of your site, as /robots.txt.
# robots.txt with rules for each documented AI crawler # Made with the RookAtlas AI robots.txt generator # AI crawlers allowed (grounding, search, fetches for a person) User-agent: Google-Extended User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: Claude-SearchBot User-agent: Claude-User User-agent: PerplexityBot User-agent: Perplexity-User User-agent: Meta-WebIndexer User-agent: Meta-ExternalFetcher Allow: / # AI crawlers blocked (training) User-agent: GPTBot User-agent: ClaudeBot User-agent: Meta-ExternalAgent Disallow: / # Every other crawler, Googlebot included User-agent: * Allow: /
How to use the AI robots.txt generator
- Start from a preset. In AI answers, out of training allows every crawler that finds or fetches pages for answers and blocks the ones that only collect training data, which is where most sites land.
- Change any crawler to Allow or Block. Each row says what the crawler does, so you can see what a block costs before you make it.
- Add the paths no crawler should fetch, such as an account area or a cart, and your sitemap.
- Copy or download the file, put it at the root of your site as /robots.txt, then check the live file with the AI crawler checker.
Three kinds of AI crawler
Every maker that cites the web splits its crawling by purpose, and the purpose decides what a block costs. Search and grounding crawlers build what an engine searches when it writes an answer: OAI-SearchBot for ChatGPT, Claude-SearchBot for Claude, PerplexityBot, Meta-WebIndexer, and Google-Extended for the Gemini app. User fetchers load a page because a person asked about it. Training crawlers collect text for future models and nothing else.
The guide to AI crawlers in robots.txt goes through each maker's crawlers with its documentation.
How the file is written
Allowed AI crawlers share one group, blocked ones another, and every other crawler, Googlebot included, falls under User-agent: *. Your private paths go into every group that allows, because a crawler named in a group ignores the rest of the file. Comments say which group is which, so the next person to edit it can tell.
robots.txt is a request crawlers read, not a lock. Three user fetchers say they may not follow it, and a CDN or firewall in front of your site can turn a crawler away before it reads the file at all. If your site is behind Cloudflare, check its AI crawler settings as well.
Questions
Does blocking GPTBot remove my site from ChatGPT?
No. GPTBot only collects training data. Whether ChatGPT's search answers can show your pages depends on OAI-SearchBot, which OpenAI documents as a separate crawler. This generator keeps the two apart.
Should I block AI training crawlers?
It's a business and licensing choice, not a ranking one. Blocking GPTBot, ClaudeBot or Meta-ExternalAgent opts you out of training their makers' future models; it doesn't remove you from answers that search the web. No published study has measured what it does to recommendations.
What happens if I block Google-Extended?
Google stops using your pages to train Gemini models and to ground answers in the Gemini app. Google Search, AI Overviews and AI Mode are unaffected; they follow Googlebot.
Why are my private paths repeated in every group?
Under the robots.txt standard (RFC 9309) a crawler named in a group follows only that group and ignores the rules for everyone else. Without the repeat, an AI crawler you allow could read a path you meant to keep out.
Why isn't Grok in the list?
xAI documents no crawler for Grok, so there is no robots.txt token to allow or block. The list holds only crawlers their makers document.
