An audit of what AI crawlers can read
If an engine's crawler can't fetch or read your page, no content fix will help. Check every documented AI crawler against your robots.txt, your page rules and your HTML.
- All eight AI engines on every paid prompt run
- A 95% interval on visibility and citation rates
- Every run priced before it starts
Opening soon: sign-up opens with live measurement, after our provider pilot.
AI-access audit of jbl.com, free and on demand
- OAI-SearchBot in robots.txtAllowedOfficial
- GPTBot in robots.txtAllowedOfficial
- PerplexityBot in robots.txtAllowedOfficial
- ClaudeBot in robots.txtAllowedOfficial
- Google-Extended in robots.txtAllowedOfficial
Real results, read from jbl.com/robots.txt on Nov 13, 2026: it blocks none of the engines' crawlers.
RookAtlas's AI-access audit checks your site against the crawlers each engine's maker documents: what robots.txt allows each one, whether noindex or snippet limits keep a page out of AI features, how much text the HTML holds before JavaScript, and which of your cited pages now answer 404 or haven't been updated in over a year. Every finding carries its evidence grade and source.
Every documented AI crawler, by maker
Search crawlers, training crawlers and user-triggered fetchers do different jobs. The audit groups them by maker, says what each fetches for, and what blocking it costs.
- robots.txt read with an RFC 9309 parser
- Blocked training crawlers read as a choice, not an error
- The lines to change, per crawler
- OAI-SearchBotSearchAllowed
- ChatGPT-UserUser requestsAllowed
- GPTBotTrainingBlocked, a choice
- Claude-SearchBotSearchAllowed
- Claude-UserUser requestsAllowed
- ClaudeBotTrainingAllowed
- PerplexityBotSearchBlocked
- Perplexity-UserUser requestsAllowed
- GooglebotSearchAllowed
- Google-ExtendedGroundingAllowed
What a crawler reads before JavaScript
Many AI crawlers don't run JavaScript. The audit reads your page's served HTML, measures the text in it, and finds where the first full answer starts.
- Words and scripts in the served HTML
- Title, description and headings
- noindex, nosnippet and max-snippet
- Title and descriptionPresent
- One h1Present
- noindex, nosnippet, max-snippetNone
- Main text before JavaScriptThin: render it on the server
Cited pages that broke or went stale
Your own pages that answers cite are read after every run. A 404, a soft 404 or a date over a year old becomes an action, most cited first.
- 404, 410 and soft 404s
- Pages whose own date is over a year old
- Re-read weekly, sooner after a failed read
- yourbrand.com/pricing-2024Answers 404Cited in 7Vendor data
- yourbrand.com/products/travel-headphonesSoft 404: lands on the home pageCited in 3Vendor data
- yourbrand.com/blog/noise-cancelling-guideDated Mar 2, 2025, over a year oldCited in 4Correlation
How it works
- Step 1
Run the audit
On demand, free. It reads robots.txt, your key pages and their HTML.
- Step 2
Read the findings
Each with its evidence grade, the documentation behind it and the fix.
- Step 3
Keep it fixed
Cited pages are re-read after runs, and a CI check catches robots.txt changes before deploy.
What's inside
AI crawler rules
robots.txt for every documented AI crawler, grouped by maker.
Page directives
noindex, nosnippet and max-snippet that limit AI features.
Text before JavaScript
How much a non-rendering crawler can read.
Broken cited pages
Your cited pages answering 404, 410 or a soft 404.
Stale cited pages
Your cited pages dated over a year ago.
CI check
A GitHub Actions step that judges robots.txt before it ships.
What it can't do
The audit reads what a crawler is told and served. It can't see an engine's own logs, a firewall rule that turns bots away by address, or whether an engine chose to use a page it could read.
Works with
All featuresQuestions
Prices, credits and plans are on the pricing page .
Should I block AI training crawlers?
That's your choice, and the audit treats it as one. Blocking a training crawler keeps your pages out of model training; blocking a search crawler keeps you out of that engine's answers. The audit shows which is which for each maker.
Does the audit cost credits?
No. It runs on demand on every plan, Free included.
Can I try it without an account?
Yes: the free AI page check and AI crawler checker run the same rules on one page or one robots.txt, no sign-up.