JavaScript SEO for AI crawlers: make content visible to every bot

If ChatGPT, Claude or Perplexity describe your business vaguely, get your prices wrong or never cite your pages, the cause may be a JavaScript SEO problem: how your site is built rather than what it says. Googlebot runs JavaScript before it indexes a page. The most detailed published test of AI crawlers, by Vercel and MERJ in December 2024, found that the crawlers behind ChatGPT, Claude, Perplexity and Meta's AI downloaded JavaScript files but did not run them, so anything scripts added after the page loaded never reached those systems.

This guide is for owners of React, Vue, Wix, Webflow, Shopify and headless sites, and for their developers. You will learn to see a page the way a non-rendering bot does in about five minutes, what must be in the initial HTML, how to choose between server-side rendering, static generation and prerendering, and how to stop the problem coming back. Crawler behaviour and Google's documentation were checked in October 2026.

Two terms run through the guide. The initial HTML (or raw HTML) is the document your server sends before any JavaScript runs. The rendered HTML is the page after a browser, or a rendering crawler such as Googlebot, has run the scripts. JavaScript SEO is the work of making sure the content that matters is in the version each crawler actually reads.

Why JavaScript affects AI crawlers differently from Googlebot

Googlebot renders pages: it runs your JavaScript in a headless Chromium browser and indexes the result. The AI crawlers measured so far do not. They read the initial HTML, so text, links and tags that scripts add later are missing from what they collect.

Google's JavaScript SEO basics describe three phases: crawling, rendering and indexing. Pages that return a 200 status code are queued for rendering and, once Google's resources allow, a headless Chromium executes the JavaScript. The same page still recommends server-side rendering or prerendering, because it makes a site faster for users and crawlers, and because not all bots can run JavaScript.

What the Vercel and MERJ study found

The rise of the AI crawler, published by Vercel with MERJ in December 2024, analysed crawler traffic across Vercel's hosting network and tested which bots execute JavaScript. None of the major AI crawlers did: OpenAI's OAI-SearchBot, ChatGPT-User and GPTBot, Anthropic's ClaudeBot, Meta-ExternalAgent, ByteDance's Bytespider and PerplexityBot.

They did download script files. JavaScript made up 11.50% of ChatGPT's fetches and 23.84% of Claude's, but the files were never executed. Google's Gemini, which uses Googlebot's infrastructure, and Applebot both rendered pages. The volumes are no longer small: in one month GPTBot made 569 million requests across Vercel's network and Claude made 370 million, together about 20% of Googlebot's 4.5 billion.

How much weight to put on it

Treat the study as strong but dated evidence. It comes from a hosting company that sells server-rendering frameworks, it measured one network, and the data is from late 2024. We found no newer published measurement and no statement from OpenAI, Anthropic or Perplexity that their crawlers now render JavaScript (checked October 2026). OpenAI's overview of its crawlers, for example, explains what each bot is for without promising script execution.

The safe working assumption is that AI crawlers read only your initial HTML until an operator documents otherwise. Vercel adds that data embedded in the initial HTML, such as JSON, may still be indexed, but plain visible HTML is the only version every system can read.

Why AI Overviews and Copilot behave differently

Google's AI Overviews and AI Mode draw on Google's Search index, so they inherit Googlebot's rendering. Google's guide to generative AI features says a page needs to be indexed and eligible to show with a snippet, with no additional technical requirements. Microsoft Copilot grounds its answers in Bing, which is more cautious: Bing's Webmaster Guidelines warn against hiding critical content behind client-side rendering and say content that cannot be reliably rendered may not be indexed or selected for grounding results.

Which crawlers run JavaScript: our evidence table

Method: in October 2026 we compared what each operator documents with the one named measurement (Vercel and MERJ, December 2024) and recorded the source behind each answer. Where nothing is published, the table says so.

The pattern is simple. If your content is client-rendered, Google can usually still see it, Bing may or may not, and ChatGPT search, Claude and Perplexity almost certainly do not. Our guide to blocking or allowing AI crawlers explains each bot's token and job.

Test what bots see: raw HTML vs the rendered page

To see what a non-rendering crawler sees, look at the initial HTML instead of the page in your browser. Four free checks, from a two-minute look to Google's own rendering test, cover almost every case.

The page on your screen is rendered, and so is the Elements panel in DevTools ("inspect element"), which is why content can look present to a developer when bots never receive it. View source shows the initial HTML. First, though, confirm the bots are allowed in at all: run the page through the free AI Crawler Checker to see, crawler by crawler, whether robots.txt lets ChatGPT search, Perplexity, Claude and Google fetch it.

The four checks

1. View the source. In Chrome, type view-source: before the page address, or press Ctrl+U. Use Find to search for the main heading, the first sentence of your answer, a price or key fact, your phone number and the text of one internal link. A phrase missing from the source is missing for non-rendering crawlers.
2. Turn JavaScript off. Open DevTools, open the Command Menu, type "JavaScript", choose Disable JavaScript and reload. What remains is roughly what an AI crawler gets; blank sections, spinners and "Loading" messages mark content that depends on scripts. JavaScript stays off only while DevTools is open.
3. Fetch the page with a bot's user agent. In a terminal, run curl -s -A "GPTBot" https://www.example.com/teeth-whitening/ | grep -c "teeth whitening". A result of 0 means the phrase is not in the HTML that user agent receives. Repeat with OAI-SearchBot and PerplexityBot. This tests what your server, CDN or firewall sends to that user agent, so it catches bot blocking and bot-specific pages; it does not imitate the bot itself.
4. Check Google's rendered version. In Search Console, inspect the URL, choose Test live URL, then View tested page. According to Google's URL Inspection tool help, this view includes a screenshot, the page's HTML, HTTP headers and JavaScript console output, and Google's own JavaScript guides point to the same tool for checking rendered HTML. Content that appears here but not in view source is visible to Google and invisible to most AI crawlers.

Also run curl -s -o /dev/null -w "%{http_code}" https://www.example.com/no-such-page. It should print 404. If it prints 200, your site returns soft 404s, a single-page-app problem covered later in this guide.

Score it: the five-phrase visibility test

To compare pages and track them over time, turn the first check into a score. This is our own method, not a published standard, and it needs nothing but a browser.

1. Pick one URL per template: home, service or product, blog post, location and category page.
2. For each, list five phrases an AI answer would need: the main heading, the opening answer sentence, one hard fact (a price, opening hours or a specification), the business name with phone number, and the anchor text of one internal link.
3. Score a point for each phrase you find in view source.
4. Read the score: 5 means the template is visible to every crawler; 3 or 4 means specific widgets need fixing; 0 to 2 means the template needs a rendering fix.

Example (hypothetical): a dental clinic in Pune runs its site as a React single-page app. View source on its teeth-whitening page shows an empty container and a few script tags, so it scores 0 out of 5, while URL Inspection shows the full page with prices and reviews. Google can index the page, but ChatGPT search and Perplexity are working from almost nothing.

Where our free tools fit

Our free tools fetch the HTML your server returns without running JavaScript, so they read a page roughly as a non-rendering crawler does. If the AI Extractability Checker finds almost no text on a page that looks full in your browser, suspect rendering before you rewrite anything. The Website SEO Audit crawls up to 50 pages from your sitemap and robots.txt; missing titles, empty headings or thin pages across one template usually share this cause.

What must be in the initial HTML

Anything you want an AI system to read, quote, cite or follow must be in the initial HTML: the main content, the links to your other pages and the tags that describe the page. JavaScript can still add interactivity on top.

Google clarified three rules for JavaScript sites in December 2025, recorded in its log of Search documentation updates. On 15 December it said Google may skip rendering when it finds a noindex tag, so JavaScript that removes noindex may not work. On 17 December it said to keep the canonical in the original HTML and not change it with JavaScript, or, if that is impossible, to leave it out of the HTML and set it only with JavaScript. On 18 December it noted that pages returning a non-200 status code, such as 404 pages, may not be rendered at all.

Write the content itself so it can be lifted cleanly. Our guide to getting cited in AI search covers answer-first writing, and the on-page SEO checklist covers titles, headings and descriptions.

Keep the important parts near the top

Server rendering can backfire if it bloats the HTML. Google's March 2026 post Inside Googlebot says Googlebot currently fetches up to 2MB per HTML URL, headers included; anything beyond that is not fetched, rendered or indexed, while JavaScript and CSS files have separate limits. Google suggests keeping inline base64 images and large inline CSS or JavaScript out of the HTML, and placing meta tags, canonical tags and structured data near the top.

Frameworks that embed large blocks of hydration data in the page should check their biggest templates and keep the main content above that data. We found no published size limits for AI crawlers, so the same discipline is a sensible default for them too.

Rendering options compared: CSR, SSR, static generation and prerendering

Server-side rendering and static generation put the full content in the initial HTML, so every crawler sees it. Client-side rendering leaves the initial HTML nearly empty, so only rendering crawlers such as Googlebot see the content. Hydration and prerendering sit in between, depending on how they are set up.

• Client-side rendering (CSR) is when the server sends a near-empty HTML shell and the browser builds the page with JavaScript.
• Server-side rendering (SSR) is when the server builds the complete HTML for each request.
• Static site generation (SSG) builds every page's HTML in advance. Incremental static regeneration (ISR) rebuilds individual pages on a schedule or on demand.
• Hydration is the step in which JavaScript attaches to server-rendered HTML in the browser to make it interactive.
• Prerendering runs a client-side app ahead of time and saves the HTML it produces. Dynamic rendering serves that saved HTML only to bots.

The table is our assessment, built from Google's and Bing's documentation and the Vercel and MERJ results; the effort column assumes a small site with a developer available.

Dynamic rendering is a stopgap, not a destination

Google's page on dynamic rendering as a workaround calls it a workaround and not a recommended solution, because it adds complexity and resource demands, and recommends server-side rendering, static rendering or hydration instead. For AI crawlers it has a further weakness: it works only for the user agents you list, so each new bot must be added by hand. If you need it while you rebuild, keep the bot version identical in content to what users see.

Framework defaults worth knowing

• React. The React team deprecated Create React App for new apps on 14 February 2025 and recommends frameworks such as Next.js, React Router and Expo. A React or Vue app built with a plain bundler such as Vite renders in the browser unless you add prerendering or a framework.
• Next.js. Pages can render on the server or at build time, but components can opt out with ssr: false, and data fetched in the browser after load is not in the initial HTML. Metadata can also be streamed, which is covered in the metadata section below.
• Nuxt and Angular. Nuxt renders on the server by default, and Angular offers server-rendered and prerendered output through its SSR package. Both let you switch individual routes to client-only rendering, which is exactly where content goes missing.

Google notes that server-side rendering and prerendering also make sites faster. The checks in our post on website speed checks show whether yours improved after the change.

Decision table: which fix to choose

Choose the fix by platform and by how much content is missing from the initial HTML. Most small sites do not need a rebuild: they need a few widgets replaced, or their public pages prerendered.

How we built it: we mapped common site set-ups against the rendering options above and ranked the fixes by how much missing content each restores for the effort involved. It is a prioritisation framework, not a measured result.

Fix order for a small site

1. Allow the bots in robots.txt and at your CDN or firewall.
2. Get the main content of every important template into the initial HTML.
3. Make every navigation and in-text link a real <a href> element pointing at a real URL.
4. Put the title, meta description, canonical and robots tags in the initial HTML.
5. Move JSON-LD into the initial HTML, and make missing pages return 404.
6. Add the raw-HTML check to your release process.

What not to do

• Do not serve bots different content from what people see.
• Do not block JavaScript or CSS files in robots.txt; Google's URL Inspection help notes that blocked resources can change how Google renders the page.
• Do not treat Google's rendered view as proof that GPTBot saw the same thing, and do not rebuild a whole site when three widgets are the problem.

Website builders and CMSs: what to check

Wix, Shopify themes and WordPress build pages on the server, so their core pages usually pass. The gaps come from what you add: apps, plugins, embeds and widgets that load content with JavaScript after the page arrives.

Two questions settle most builder sites: is the text typed into the platform's own elements or pulled in from an embed, and does each app print its content into the page or draw it afterwards? If you cannot change a widget, copy its key content into native text, such as your five most common questions with their answers, so every crawler gets it while the widget stays for visitors.

Links, lazy loading and content behind clicks

Crawlers discover pages through real links and read content that is present without interaction. Click-handler navigation, content that loads on scroll or click, and views that change only after a # in the URL all hide pages, and AI crawlers are less forgiving than Googlebot.

Use real links

Google's link best practices say Google can only crawl a link that is an <a> element with an href attribute, and cannot reliably extract URLs from links that work through script events. Google can follow links that JavaScript inserts, because it finds them after rendering; a non-rendering crawler cannot. Compare <a href="/teeth-whitening/">Teeth whitening</a>, which every crawler can follow, with <span onclick="goTo('/teeth-whitening/')">, which none can rely on.

Single-page apps should also give every view its own URL. Google's JavaScript SEO basics say you cannot rely on URL fragments such as /#/services to work with Googlebot, and recommend the History API instead.

Lazy loading and infinite scroll

Google's guide to fixing lazy-loaded content says content should load when it becomes visible in the viewport, without relying on user actions such as scrolling or clicking. For infinite scroll it asks for a persistent, unique URL for each chunk, with the address bar updated through the History API as each chunk becomes the main content.

AI crawlers set a higher bar, because they never run the script that fills the viewport. An image with the HTML loading="lazy" attribute keeps its URL and alt text in the initial HTML; text that a script inserts on scroll does not. Lazy-load images and media, not the text that answers the question.

Tabs, accordions and "read more"

Text in tabs and accordions is in the initial HTML if it is only hidden with CSS, and invisible to non-rendering crawlers if a script fetches it on click. The HTML <details> and <summary> elements give you an accordion with no script at all. Check by searching view source for a sentence from inside a closed tab.

Soft 404s in single-page apps

Single-page apps often answer every URL with a 200 status code and draw an error message in the browser, so error pages can be indexed as real pages. Google's JavaScript SEO basics suggest redirecting to a URL where the server returns a 404, or adding a noindex tag to error pages. Because Google may skip rendering non-200 pages, a real 404 from the server is the cleanest signal for every crawler.

Structured data and metadata in JavaScript apps

Put titles, meta descriptions, canonical tags and JSON-LD structured data in the initial HTML. Google can read most of them after rendering, but non-rendering crawlers cannot, and Google treats some JavaScript changes to these tags as unreliable.

Structured data added with JavaScript

Google's page on generating structured data with JavaScript names Google Tag Manager and custom JavaScript as the common methods, and Google can read JSON-LD injected this way. A crawler that does not run JavaScript never sees it, so move Tag Manager JSON-LD into the page template.

Keep expectations realistic. Google's generative AI guide says no special schema.org markup is needed to appear in AI Overviews or AI Mode. Structured data earns its place through the rich results that remain and by describing your business unambiguously; our schema markup guide covers which types are still worth adding.

Titles, descriptions and social tags

Many single-page apps set the title and meta description with a script once the route loads. Googlebot sees the result after rendering; a non-rendering crawler sees the placeholder in the initial HTML, often the same title on every page. Open Graph tags have the same problem: the Next.js documentation gives Facebook's link-preview crawler as an example of a bot that cannot execute JavaScript.

The Next.js streaming metadata setting

Since version 15.2, Next.js can stream metadata from generateMetadata after the first part of the page has been sent. For bots that match its htmlLimitedBots setting, it waits for the metadata before sending the page instead; the documented default list includes Google crawlers, Bingbot, Twitterbot and Slackbot. If your bot test shows a missing or misplaced title for GPTBot or OAI-SearchBot, add the AI crawler tokens to htmlLimitedBots in next.config.js. The setting replaces the default list, so include any default bots you still need.

Monitor rendering after each release

Rendering breaks quietly. A new plugin, a framework upgrade or a CDN rule can empty your initial HTML without changing how the page looks in a browser, so check the raw HTML on every release and review crawler activity every month.

1. Keep a key URL list: one URL per template plus your ten most valuable pages, with five phrases for each.
2. Check after every deploy: fetch each URL with curl and confirm the phrases, title, canonical and status code. A developer can script this to fail the release when a phrase disappears.
3. Re-run URL Inspection monthly on each template to confirm Google still renders the full page.
4. Read your logs monthly: upload an access log to the free AI Bot Log Analyzer to see which pages GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot fetched. Important pages that never appear, or that return errors, are your next fixes.
5. Crawl quarterly with the Website SEO Audit and compare titles, headings and word counts template by template.
6. Watch the outcome: AI Overviews and AI Mode impressions in Search Console's generative AI performance report, and referrals from chatgpt.com and perplexity.ai in your analytics.

Logs also show waste. Vercel and MERJ found that 34.82% of ChatGPT's crawler fetches and 34.16% of Claude's hit 404 pages, against 8.22% for Googlebot. The study could not say exactly why, so keep redirects in place when URLs change and limit your sitemap to live, canonical URLs.

Once the text is reliably in the HTML, make it more quotable. The AI Answer Snippet Extractor scores every paragraph out of 100 as a quotable snippet, which shows which sections to tighten first. For the wider crawl, index and speed checks, work through our technical SEO audit checklist.

If you would rather not scope a rendering change alone, our technical SEO service covers rendering audits and fixes, and you can get in touch with the URLs you have already tested.