Skip to content
RookAtlas
A glossy browser window on ultramarine whose lower half is an empty dashed outline under a glass magnifying glass.

Do AI crawlers run JavaScript? What GPTBot, ClaudeBot and PerplexityBot actually read

Most AI crawlers read the HTML your server sends and never run its scripts. What that hides from ChatGPT, Claude and Perplexity, how to check a page, and how to fix it.

4 min read

A page can look complete in your browser and be nearly empty to an AI crawler. Browsers run a page's JavaScript and draw what it builds; most AI crawlers take the HTML the server sends and stop there.

If the words that answer a buyer's question only appear after a script runs, an engine that searches the web may find your page and still have nothing on it to quote.

On this page
  1. 01What the crawler logs show
  2. 02What goes missing
  3. 03How to check a page
  4. 04How to fix it
  5. 05Google renders, later

What the crawler logs show

The most detailed public evidence comes from Vercel and MERJ, who analysed crawler traffic across Vercel's network and published the results in December 2024. Their finding was blunt: none of the major AI crawlers they saw rendered JavaScript. That covered OpenAI's OAI-SearchBot, ChatGPT-User and GPTBot, Anthropic's ClaudeBot, Meta's Meta-ExternalAgent, ByteDance's Bytespider and PerplexityBot.

Some of them do download JavaScript files (11.5% of ChatGPT's fetches and 23.8% of Claude's were scripts) but they don't execute them. Google is the exception: Gemini uses Google's crawling infrastructure, which renders pages much as Googlebot does for Search.

Crawlers change, and these numbers are a snapshot. But a page built to work without its scripts doesn't depend on any crawler's rendering, now or later.

In a browser

Your brandyourbrand.com

Noise cancelling headphones for long flights

Up to 35 hours with noise cancelling on, a case that folds flat and free 30-day returns, from $129.

Earbuds$129On-ear$179Over-ear$249

Without JavaScript

<html>
  <head>
    <title>Your brand</title>
  </head>
  <body>
    <div id="root"></div>
    <script src="/app.js"></script>
  </body>
</html>
Rendering example: the same page in a browser and in the initial HTML a crawler receives.

What goes missing

A crawler that doesn't run scripts sees the HTML your server returns. Anything a script adds afterwards is invisible to it. On a typical marketing or product site that can include:

  • The whole page, when a single-page app sends an empty shell and builds everything in the browser.
  • Prices, plans and specifications fetched from an API after the page loads.
  • Reviews, FAQs and comparison tables injected by widgets.
  • Text behind tabs and accordions that loads only when someone clicks.
  • Titles, descriptions and canonical tags set by a script rather than by the server.

Vercel and MERJ add one nuance: data already present in the first response, such as JSON embedded in the page, may still be read, because models can interpret it. Don't count on it; plain HTML is the form every crawler reads.

How to check a page

You don't need special tools to see what a non-rendering crawler gets:

  1. View the source. In your browser, open view-source (Ctrl+U or Cmd+Option+U) rather than the inspector, which shows the page after scripts ran. Search for the sentence that answers your buyers' question.
  2. Turn JavaScript off. In your browser's developer tools, disable JavaScript and reload. What's left is roughly what these crawlers read.
  3. Fetch it from the command line. Run curl -sL https://yoursite.com/page and search the output for that same sentence.
  4. Check status codes and redirects. In the same logs, a third of ChatGPT's and Claude's fetches landed on pages that returned 404, and ChatGPT spent a further 14% following redirects. Stable URLs and direct redirects waste fewer of a crawler's visits.

Sending a request with a crawler's user agent doesn't prove what that crawler sees: CDNs and firewalls check where a request comes from, so they may treat an imitation differently from the real crawler.

How to fix it

The fix is to have the server send the content, not to write a second page for crawlers.

  • Render on the server or ahead of time. Server rendering and static generation, which frameworks such as Next.js, Nuxt and Astro provide, put the full text into the first response. A client-rendered app can often pre-render its public pages and keep client rendering for the parts behind sign-in.
  • Put the answer near the top of the HTML. A heading that asks the question and a first paragraph that answers it are the easiest parts of a page to quote.
  • Ship facts as text. Prices, plan limits and specifications belong in the markup, not only in an API response or an image.
  • Set metadata on the server. Titles, descriptions, canonical tags and any robots directives should be in the first response.
  • Serve everyone the same page. Showing crawlers content that people don't see is cloaking under Google's spam policies, and it breaks the moment a crawler changes how it identifies itself.

Google renders, later

Googlebot does run JavaScript, so pages built in the browser can reach Google's index and, through it, Google AI Overviews, Google AI Mode and Gemini. Google describes this as crawling, then rendering, then indexing, and rendering can wait until resources allow.

That makes Google the forgiving case rather than the model to build for. A page that works without its scripts is ready for every engine at once: ChatGPT, Claude and Perplexity included.

Free tool, no sign-upAI page checkCheck any public page the way AI crawlers read it: before JavaScript, through robots.txt and the page's own directives. See the first answer they find and what to fix, free.Run the free check

Questions

Does Googlebot run JavaScript?

Yes. Google renders pages before indexing them, though rendering can be delayed. Gemini uses the same infrastructure. The AI crawlers from OpenAI, Anthropic, Perplexity and Meta that Vercel and MERJ observed did not.

If my page is server-rendered, will AI engines cite it?

Not by itself. Server rendering makes sure a crawler can read the page. Whether an answer names you depends on the question, the sources an engine trusts and your competitors, which only measuring many answers shows.

Do I need an llms.txt file instead?

No. Google says Search, including its AI features, doesn't use llms.txt or other special AI files. Making the page itself readable is what every crawler relies on.

Is pre-rendering a page only for crawlers allowed?

Serving crawlers the same content people see, rendered ahead of time, is fine. Serving them different content is cloaking under Google's spam policies. The safe route is one page for everyone, rendered on the server.

Sources

  1. Vercel and MERJ: the rise of the AI crawler (December 2024) vercel.com
  2. Google: understand the JavaScript SEO basics developers.google.com
  3. Google: AI features and your website developers.google.com
  4. Google: optimizing for generative AI in Google Search developers.google.com
  5. Google: spam policies for Google web search developers.google.com
All articles

RookAtlas opens soon.

RookAtlas asks your buyers' questions on eight AI engines and shows how often you're named, next to whom and from which sources, with a 95% interval on visibility and citation rates.