AI Crawlers Do Not Run JavaScript

The short answer

AI crawlers do not run JavaScript. When GPTBot, ClaudeBot or PerplexityBot requests a page, it performs a plain HTTP GET, takes the bytes the server returns, and parses them as text. There is no browser, no DOM construction, no hydration and no waiting for XHR to resolve. Joint analysis by Vercel and MERJ, across a network serving 569 million GPTBot requests and 370 million ClaudeBot requests in a single month, found no evidence of JavaScript execution by any major AI crawler. The crawlers do sometimes download script files — GPTBot fetched JavaScript in 11.50% of requests, ClaudeBot in 23.84% — and then never run them.

Google is the exception, and it is the reason so many teams have not noticed the problem. Googlebot operates a headless Chromium rendering service, and Gemini is grounded on that infrastructure. So a client-side-rendered parametric catalog can rank perfectly well in Google Search while being completely absent from ChatGPT, Claude and Perplexity. Those are not the same audience any more: Forrester's 2026 buyer research found that 94% of business buyers now use AI during their buying process, and Adobe measured AI-referred traffic to retail sites growing 393% year on year in Q1 2026.

What GPTBot actually receives

It helps to be precise about the two documents involved, because most catalog teams only ever look at the second one.

The raw HTML is the byte stream the origin server writes to the response. It is what curl prints, what view-source: shows, and what a text-only HTTP client parses.

The rendered DOM is what exists in memory after the browser has parsed that HTML, downloaded and executed every script, resolved every fetch() call, and applied every mutation. It is what your browser devtools Elements panel shows.

In a modern SPA catalog these two documents have almost nothing in common. The raw HTML is a shell: a <div id="root">, a preload manifest and a hundred kilobytes of bundle references. Every meaningful string — part number, package, tolerance, operating temperature range, RoHS status, lifecycle status, stock, price break — arrives later, from an API call the crawler will never make.

An AI crawler sees your raw HTML and nothing else. If a specification is not in the bytes the server returns, that specification does not exist as far as the model is concerned.

Which clients render, and which do not

Every "No" in that table is measured behaviour from the Vercel and MERJ dataset, not inference. Every "Yes" is documented by the operator: Google publishes its rendering pipeline, and Apple's own crawler documentation states that Applebot "may render the content of your website within a browser". Anthropic's newer Claude-SearchBot and Claude-User agents post-date the study and were not measured separately; no operator has announced a rendering pipeline for them since.

The practical consequence is asymmetric in an unhelpful way. Rendering is the expensive part of crawling, and the operators who skipped it are precisely the ones now sitting between your product and your buyer.

Why parametric catalogs fail worse than everything else

Adobe's April 2026 analysis scored retail pages for machine readability and found product detail pages at 66% — the lowest of any page type, below homepages (75%), category pages (74%) and even FAQ pages (80%). That ordering is not an accident. The pages carrying the most structured, most valuable, most decision-relevant data are the ones built with the most JavaScript.

Five patterns account for most of the damage:

1. 01Filter state lives in the client. Your parametric selector is a React component reading from a local store. There is no server-rendered URL for "0603, 100nF, X7R, 50V", so there is nothing for a crawler to fetch, and nothing to cite.
2. 02The specifications table is an API response. The page shell renders, then a request to /api/product/attributes populates the table. The crawler stops at the shell.
3. 03Price and availability are deliberately client-side. Reasonable for caching. Fatal for machine visibility, because a model that cannot see stock will not recommend the part.
4. 04Tabs and accordions defer their content. Datasheets, application notes, compliance certificates and CAD links are commonly loaded on click. A crawler never clicks.
5. 05Infinite scroll replaces pagination. Product 51 onwards has no crawlable URL at all.

Partsgraph audited 984 distributor and manufacturer domains worldwide — North America, Europe and Asia — in August 2026. The median AI-visibility score was 50 out of 100, and 70% of the cohort graded D or F. Roughly half could not serve a single readable catalog page to a standard non-browser client. Electronics distribution was the weakest segment, with a median of 34.

How do I test whether AI crawlers can read my catalog?

Do this against your own origin. Testing a competitor's site by spoofing a crawler user-agent produces false negatives, because enterprise bot managers verify crawlers by source IP, not by the user-agent string — a spoofed header from an unrecognised address gets challenged regardless of what robots.txt says.

1. 01Fetch the raw HTML. Pick your single most valuable product page.

curl -sS --compressed -o page.html -w "status=%{http_code} bytes=%{size_download}" "https://example.com/products/part-number"

1. 01Look for the part number in the bytes. Not in the browser — in the file.

grep -c "MPN-12345" page.html

If this returns 0, no AI crawler can identify the page as being about that part.

1. 01Look for three parametric values you would expect an engineer to filter on.

grep -oiE "operating temperature|tolerance|package|rohs|lifecycle" page.html | sort | uniq -c

1. 01Count the structured data blocks.

grep -c "application/ld+json" page.html

Zero means no machine-readable product record. One or more means you should extract it and validate that it actually carries sku, mpn, gtin, brand, offers and your key additionalProperty entries, rather than just a breadcrumb trail.

1. 01Compare against the rendered page. Open the same URL in a browser with JavaScript disabled. What survives is roughly what a crawler gets. The gap between that and the normal page is your invisibility.

1. 01Check the whole path, not one page. Repeat for a category page, a search-results page, a datasheet link and a CAD download. A crawler that can read a product page but cannot reach it from a crawlable category listing is still stuck.

A page that passes will show the part number dozens of times, the parametric labels present as text, and at least one JSON-LD block. For reference, a well-built product page typically returns 60–200 KB of HTML containing the full specification table; a broken SPA shell returns 4–15 KB containing nothing but bundle references.

How do you fix it without rewriting the site?

You do not need to re-platform. You need the bytes at the origin to contain the facts. There are four routes, and they are not mutually exclusive.

Server-side rendering of product and category routes is the durable answer. If you are already on a framework that supports it, moving the catalog routes to server components or static generation with incremental revalidation is usually a matter of weeks, not quarters, because the data layer already exists — it is just being called from the wrong side.

Prerendering is the honest stopgap. Render each product page once at build or on a schedule, cache the HTML at the edge, and serve it to everyone. Serve the same document to humans and machines; cloaking a different page to crawlers is both a policy risk and, in practice, self-defeating, because the model will quote whichever version it received.

Server-rendered JSON-LD is the highest-leverage low-effort change. Put it in the HTML the server emits, not in a tag manager.

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "Product",
  "sku": "MPN-12345",
  "mpn": "MPN-12345",
  "name": "100 nF 50 V X7R 0603 ceramic capacitor",
  "brand": { "@type": "Brand", "name": "Example Components" },
  "additionalProperty": [
    { "@type": "PropertyValue", "name": "Capacitance", "value": "100 nF" },
    { "@type": "PropertyValue", "name": "Voltage rating", "value": "50 V" },
    { "@type": "PropertyValue", "name": "Dielectric", "value": "X7R" },
    { "@type": "PropertyValue", "name": "Operating temperature", "value": "-55 to +125 C" }
  ],
  "offers": {
    "@type": "Offer",
    "priceCurrency": "USD",
    "price": "0.042",
    "availability": "https://schema.org/InStock"
  }
}
</script>

A markdown mirror is the cheapest way to make a page unambiguous. Serve a plain-text equivalent of every product page at a stable, linked URL, with the specification table rendered as a markdown table. Text-only clients parse it perfectly, it costs almost nothing to generate from the same data source, and it gives you a canonical machine surface that does not drift when the front end is redesigned. Note that llms.txt is not this: as of 2026 no major AI operator has committed to reading it, and Google's Search Relations team has explicitly declined to endorse it. Mirrors work because they are ordinary linked pages.

An overlay puts all of the above in front of the existing site rather than inside it. A sidecar service or edge worker serves server-rendered agent-readable pages, JSON-LD, markdown mirrors and a hosted MCP endpoint on your own domain, reading from your existing PIM. Nothing about the customer-facing site changes. This is the approach Partsgraph takes, and it is also what a growing number of manufacturers are building in-house: Microchip shipped a free public MCP server for its product data in November 2025, and ECIA's TrustedParts launched an inventory AI agent service in June 2026.

What to do first, in order

1. 01Run the six-step test above on your top ten products and record the byte counts.
2. 02Server-render JSON-LD on product pages. It is the smallest change with the largest measurable effect.
3. 03Make every specification, datasheet link and CAD link present in the initial HTML, even if the interactive version stays client-side.
4. 04Give every parametric filter combination that matters a real, crawlable, server-rendered URL.
5. 05Publish a markdown mirror per product and link it from the page.
6. 06Only then worry about rankings, prompts and answer share. Retrieval precedes citation; there is nothing to optimise until the content exists in the response body.

The mechanism here is not subtle and it is not disputed. A model can only cite what it received, and what it received is your raw HTML. Everything else is a rendering step that never happened.

Want to know where your catalog stands? The free Partsgraph AI-visibility grader runs the checks in this article against your own domain at [/audit](/audit).