Skip to content
RookAtlas
On every plan, Free included

An audit of what AI crawlers can read

If an engine's crawler can't fetch or read your page, no content fix will help. Check every documented AI crawler against your robots.txt, your page rules and your HTML.

  • All eight AI engines on every paid prompt run
  • A 95% interval on visibility and citation rates
  • Every run priced before it starts

Opening soon: sign-up opens with live measurement, after our provider pilot.

AI-access auditSite audit

AI-access audit of jbl.com, free and on demand

  • OAI-SearchBot in robots.txtAllowedOfficial
  • GPTBot in robots.txtAllowedOfficial
  • PerplexityBot in robots.txtAllowedOfficial
  • ClaudeBot in robots.txtAllowedOfficial
  • Google-Extended in robots.txtAllowedOfficial

Real results, read from jbl.com/robots.txt on Nov 13, 2026: it blocks none of the engines' crawlers.

RookAtlas's AI-access audit checks your site against the crawlers each engine's maker documents: what robots.txt allows each one, whether noindex or snippet limits keep a page out of AI features, how much text the HTML holds before JavaScript, and which of your cited pages now answer 404 or haven't been updated in over a year. Every finding carries its evidence grade and source.

Measured on eight engines

  • ChatGPT
  • Gemini
  • Perplexity
  • AI Overviews
  • AI Mode
  • Claude
  • Grok
  • Muse
01

Every documented AI crawler, by maker

Search crawlers, training crawlers and user-triggered fetchers do different jobs. The audit groups them by maker, says what each fetches for, and what blocking it costs.

  • robots.txt read with an RFC 9309 parser
  • Blocked training crawlers read as a choice, not an error
  • The lines to change, per crawler
AI crawlers by maker
OpenAI
  • OAI-SearchBotSearchAllowed
  • ChatGPT-UserUser requestsAllowed
  • GPTBotTrainingBlocked, a choice
Anthropic
  • Claude-SearchBotSearchAllowed
  • Claude-UserUser requestsAllowed
  • ClaudeBotTrainingAllowed
Perplexity
  • PerplexityBotSearchBlocked
  • Perplexity-UserUser requestsAllowed
Google
  • GooglebotSearchAllowed
  • Google-ExtendedGroundingAllowed
02

What a crawler reads before JavaScript

Many AI crawlers don't run JavaScript. The audit reads your page's served HTML, measures the text in it, and finds where the first full answer starts.

  • Words and scripts in the served HTML
  • Title, description and headings
  • noindex, nosnippet and max-snippet
What a crawler reads before JavaScript
Words in the served HTML18014 scripts, no text in the main element
Words after JavaScript1,240what a browser shows
The page, top to bottom, as a crawler reads itReadable text: 15% No full answer found
  • Title and descriptionPresent
  • One h1Present
  • noindex, nosnippet, max-snippetNone
  • Main text before JavaScriptThin: render it on the server
03

Cited pages that broke or went stale

Your own pages that answers cite are read after every run. A 404, a soft 404 or a date over a year old becomes an action, most cited first.

  • 404, 410 and soft 404s
  • Pages whose own date is over a year old
  • Re-read weekly, sooner after a failed read
Cited pages that need a look
  • yourbrand.com/pricing-2024Answers 404Cited in 7Vendor data
  • yourbrand.com/products/travel-headphonesSoft 404: lands on the home pageCited in 3Vendor data
  • yourbrand.com/blog/noise-cancelling-guideDated Mar 2, 2025, over a year oldCited in 4Correlation

How it works

  1. Step 1

    Run the audit

    On demand, free. It reads robots.txt, your key pages and their HTML.

  2. Step 2

    Read the findings

    Each with its evidence grade, the documentation behind it and the fix.

  3. Step 3

    Keep it fixed

    Cited pages are re-read after runs, and a CI check catches robots.txt changes before deploy.

What's inside

  • AI crawler rules

    robots.txt for every documented AI crawler, grouped by maker.

  • Page directives

    noindex, nosnippet and max-snippet that limit AI features.

  • Text before JavaScript

    How much a non-rendering crawler can read.

  • Broken cited pages

    Your cited pages answering 404, 410 or a soft 404.

  • Stale cited pages

    Your cited pages dated over a year ago.

  • CI check

    A GitHub Actions step that judges robots.txt before it ships.

What it can't do

The audit reads what a crawler is told and served. It can't see an engine's own logs, a firewall rule that turns bots away by address, or whether an engine chose to use a page it could read.

All features

Questions

Prices, credits and plans are on the pricing page .

Should I block AI training crawlers?

That's your choice, and the audit treats it as one. Blocking a training crawler keeps your pages out of model training; blocking a search crawler keeps you out of that engine's answers. The audit shows which is which for each maker.

Does the audit cost credits?

No. It runs on demand on every plan, Free included.

Can I try it without an account?

Yes: the free AI page check and AI crawler checker run the same rules on one page or one robots.txt, no sign-up.

Pick a plan. Every engine is included.

AI access audit is on every plan, Free included. One credit runs a prompt on all eight engines (Free's five monthly credits run on ChatGPT, Gemini and AI Mode), and every run shows its price before it starts. Compare the plans and work out your credits.

Free tool, usable today: AI crawler checker

Opening soon: sign-up opens with live measurement, after our provider pilot.

Plans from
$36/mo
Engines on every paid prompt run
8
Credits per prompt
1