# Why AI Cannot Read Your Datasheets

> Bot walls, image-only scans, viewer-only documents and separate doc hosts make datasheets invisible to AI. Test yours in three commands, then fix them.

**Published:** 2026-08-09  
**Category:** Technical · **Tags:** datasheets, PDF, AI crawlers, product data, technical documentation, OCR  
**Canonical:** https://partsgraph.ai/blog/why-ai-cannot-read-your-datasheets

## In short

**Why can't AI assistants read my product datasheets?**

Four things break datasheets for AI: the PDF is blocked by a bot wall or login before a crawler can fetch it; the file is an image-only scan with no text layer, so extraction returns zero characters; the document is trapped in a JavaScript viewer with no direct file URL; or it sits on a separate documentation host whose robots.txt says no. Any one of these makes the definitive specification for your part unreachable, and the datasheet is where the specification actually lives — the web page only ever carries a summary.

---

## The short answer

A datasheet becomes invisible to AI in one of four ways, and they are all mundane. The file is blocked before it is ever fetched, by a bot rule, a login or an interstitial. The file is a scanned image with no text layer, so an extractor pulls out zero characters. The document is only reachable through a JavaScript viewer with no direct URL. Or the PDF lives on a separate documentation host — `docs.`, `media.`, a DAM, a CDN — whose own robots.txt or WAF says no, regardless of how permissive your main site is.

None of these are exotic. All four are common, and any one of them is sufficient. The consequence is worth stating plainly:

> The datasheet is the specification of record. The product page is a summary of it. When AI can read the summary but not the source, it answers engineering questions from the summary — and it gets them wrong.

## Why datasheets matter more than web pages here

A product page carries perhaps eight to twenty attributes: package, primary rating, a lifecycle flag, price, stock. A datasheet carries hundreds: absolute maximum ratings, electrical characteristics across temperature, timing diagrams, pin descriptions, thermal resistance figures, recommended footprints, moisture sensitivity level, ordering part number decoders. Almost every question an engineer actually asks an assistant — *can I run this at 5.5 V*, *what is the quiescent current at 85 degrees*, *is the pinout compatible with the part I am replacing* — is answered in the datasheet and nowhere else.

That matters more now than it did, because the questions are being asked to assistants rather than typed into your search box. Forrester found in January 2026 that **94% of business buyers now use AI during their buying process**. And the pages carrying the most technical depth are the ones performing worst: Adobe's April 2026 analysis of retail machine-readability scored product detail pages at **66%, the lowest of any page type**, below homepages at 75% and FAQ pages at 80%.

When the underlying data is not reachable, the failure is not silence — it is confident error. OpenAI wound down ChatGPT's Instant Checkout in March 2026 with roughly 30 merchants live, and inaccurate product data was central to why: OpenAI had been scraping retail sites for product information, and inventory, shipping and pricing data were frequently wrong. That is the same failure mode as a wrong maximum rating, with lower stakes.

## The five ways a datasheet becomes unreadable

| Failure | What a crawler sees | How to spot it |
|---|---|---|
| Bot wall | 403, 429, or an HTML challenge page instead of a PDF | Response `content-type` is `text/html`, not `application/pdf` |
| Image-only scan | A valid PDF containing no extractable text | Text extraction returns 0 characters; no embedded fonts |
| Viewer-only document | A page of JavaScript with no file URL | The PDF loads from a blob or an authenticated stream |
| Separate host policy | A clean 200 on the product page, a block on the document | The document host has its own robots.txt and WAF |
| Ephemeral or gated URL | A signed link that expires, or a form gate | The link works in your browser and 403s from `curl` |

The fifth is the most self-inflicted. Short-lived signed URLs, "register to download" gates and one-time tokens are all invisible walls: a crawler has no session, no cookies and no form-filling behaviour. And because AI crawlers do not run JavaScript, any download link that is written into the DOM by a script simply does not exist from their side of the wire.

## How do I check whether my datasheets are readable?

Three commands settle it. Run them against your own documents.

1. **Check the fetch.** Does the URL return a PDF to a plain, non-browser client?

```
curl -sSIL "https://example.com/datasheets/part-12345.pdf" | grep -iE "^HTTP|content-type|content-length"
```

You want a final `200` and `content-type: application/pdf`. A `content-type: text/html` means you were handed a challenge or a login page dressed as a download.

2. **Download it and confirm it is really a PDF.**

```
curl -sSL -o ds.pdf "https://example.com/datasheets/part-12345.pdf" -w "status=%{http_code} type=%{content_type} bytes=%{size_download}"
head -c 5 ds.pdf
```

The first five bytes must be `%PDF-`. If they are `<!DOC`, you downloaded an error page.

3. **Test the text layer.** This is the one nobody runs.

```
pdftotext -q ds.pdf - | tr -d '[:space:]' | wc -c
pdffonts ds.pdf
```

## How can I tell whether a PDF has a text layer?

Two numbers, and the gap between them is unmistakable. We ran both tests against a real, current manufacturer datasheet — Texas Instruments' LM358, a 4.15 MB PDF — and against a rasterised copy of the same document, which is exactly what a scanned legacy datasheet looks like to a machine.

| Test | Real datasheet | Image-only scan |
|---|---|---|
| `pdftotext` characters extracted | 95,895 | 0 |
| `pdffonts` embedded font rows | 114 | 0 |
| Visually identical to a human | Yes | Yes |
| Usable by an AI assistant | Yes | No |

That is the whole diagnostic. **Zero extractable characters and zero embedded fonts means there is no text in the file — only a picture of text.** The document renders perfectly for a person and contributes literally nothing to a model. A scanned 1990s datasheet for a part still in production is, to every AI system on the internet, a blank page.

Two refinements worth knowing. A low but non-zero character count — say a few hundred characters on a forty-page document — usually means a scanned body with a text header, which is just as unusable. And in `pdffonts` output, the `uni` column matters: fonts subset without a ToUnicode map can extract as mojibake even though characters are technically present. Spot-check the first 200 characters of extracted text rather than trusting the count alone.

To sweep a whole library, loop it:

```
while read -r url; do
  code=$(curl -sSL -o /tmp/d.pdf -w "%{http_code}" --max-time 30 "$url")
  chars=$(pdftotext -q /tmp/d.pdf - 2>/dev/null | tr -d '[:space:]' | wc -c)
  echo "$code $chars $url"
done < datasheet-urls.txt
```

Sort the output by character count ascending. Everything at the top of that list is a hole in your product data.

## The separate-host trap

This one catches large organisations almost universally, because it is a consequence of doing things properly. Documents get moved to a DAM, a documentation subdomain or a CDN, each of which is a distinct origin with its own configuration — and often its own owner.

Under RFC 9309, robots.txt is scoped to a single origin: scheme, host and port. `https://www.example.com/robots.txt` governs nothing at `https://docs.example.com/` and nothing at `https://cdn.example-media.net/`. If your main site welcomes AI crawlers and your document host was stood up from a default template with a blanket `Disallow: /`, or sits behind a bot manager configured to challenge everything non-human, then your entire technical library is dark while your marketing site is wide open.

Check every origin that serves a document:

```
for h in www.example.com docs.example.com cdn.example-media.net; do
  echo "--- $h"
  curl -sS --max-time 15 "https://$h/robots.txt" | head -30
done
```

## What good looks like

A correctly published datasheet is unremarkable, which is the point. Texas Instruments serves the LM358 datasheet at a stable, guessable URL under `/lit/ds/`; its robots.txt places no restriction on that path; the file returns `200` with `content-type: application/pdf` to a non-browser client identifying itself as GPTBot; and the PDF carries a full text layer with embedded, Unicode-mapped fonts. No login, no viewer, no signed URL, no interstitial. Any extraction pipeline in the world can read it.

The direction of travel in the components industry is towards making this easier still. Microchip published a free, no-authentication MCP server for its product data — specifications, datasheets, inventory, pricing and lead times — in November 2025, and ECIA's TrustedParts launched an inventory AI agent service in June 2026 exposing authorised distributor availability to Copilot, ChatGPT and Claude. Both are the same recognition: a document that a machine has to fight for is a document that loses.

## How do you fix it?

In order of return on effort:

1. **Unblock the document paths.** Allow AI crawlers on `/datasheets/*`, `/lit/*`, `/documents/*` and equivalent, at the WAF as well as in robots.txt. These two are separate switches and both must be set.
2. **Remove download gates from public technical documents.** If a datasheet is on your public site, it is already public. A registration form only stops the machines that would have recommended you.
3. **OCR every image-only PDF and republish with a text layer.** Prioritise by part revenue, not by document age. Modern OCR on a clean 300 dpi scan is close to lossless on body text; verify the parameter tables by hand.
4. **Give every document a stable, direct, linkable URL.** No blobs, no viewers, no expiring tokens, no `?token=` query strings.
5. **Publish an HTML or markdown twin of each datasheet.** Specification tables as real tables. This is the single most effective change, because it removes all dependence on PDF layout heuristics and gives you a page you can also monitor for accuracy.
6. **Cross-link the document from the product page in server-rendered HTML**, so a crawler that reaches the page can reach the document without executing anything.
7. **Expose the extracted parameters as structured data** — JSON-LD on the page, and ideally a queryable endpoint — so an agent can filter on them rather than re-parsing a PDF every time.

Partsgraph audited 984 distributor and manufacturer domains worldwide, across North America, Europe and Asia, in August 2026. The median AI-visibility score was 50 out of 100, and 70% of the cohort graded D or F. The document layer was consistently the weakest component of the score: 38% could not serve a single readable catalog page to a standard non-browser client, and document access failed more often than page access did.

The uncomfortable part is that none of this shows up in your analytics. A blocked crawler does not file a bug report, a bad OCR does not throw an exception, and a model that cannot read your datasheet does not tell the buyer it could not read your datasheet. It just recommends a competitor whose PDF opened.

*You can check your own document library against these tests with the free Partsgraph AI-visibility grader at [/audit](/audit).*

## Common questions

### Can AI crawlers read PDFs at all?

They can fetch them, and PDF text extraction is a standard part of ingestion pipelines. What they cannot do is invent text that is not in the file. Extraction reads the text layer; if the PDF is a scanned image with no text layer, extraction returns nothing and the document contributes nothing.

### How do I tell whether a PDF has a text layer?

Run pdftotext against the file and count the characters it returns, then run pdffonts and count the rows. A healthy datasheet returns tens of thousands of characters and a long list of embedded fonts. An image-only scan returns zero characters and zero fonts.

### My datasheets are on a separate documentation subdomain. Does that matter?

Yes. Under RFC 9309, robots.txt is scoped to a single origin — scheme, host and port. A permissive robots.txt on www.example.com grants nothing on docs.example.com or on your CDN hostname. Each origin serving documents needs its own policy, and each is separately subject to your WAF rules.

### Should I put datasheet content on the web page as HTML?

Yes, and it is usually the highest-return fix. An HTML or markdown twin of each datasheet — specification tables, absolute maximum ratings, pin descriptions, ordering information — is trivially parseable, cacheable and citable, and it removes any dependence on how well a PDF extractor handles your layout.

### Will blocking bots on my document host protect my IP?

It mostly protects you from being recommended. Datasheets are already mirrored across aggregators, so blocking the crawler rarely removes the content from the training corpus; it just means the version the model sees is a third party's stale copy rather than your current revision. If control is the goal, serve the authoritative document and monitor accuracy.

### Do multi-column layouts and tables survive extraction?

Imperfectly. Two-column datasheet layouts often interleave when extracted linearly, and merged-cell parameter tables lose their row structure. Tagged PDFs with a correct reading order help considerably, but an HTML or markdown mirror of the same tables removes the ambiguity entirely.

### How many datasheets should I test?

Test your top 20 by traffic first, then sweep the whole library with a script. In our experience the failures cluster by vintage and by authoring tool, so a sample that only covers recent parts will look far healthier than the library actually is.

## Sources

1. [Adobe Digital Insights, AI traffic and retail machine-readability, April 2026](https://business.adobe.com/blog/ai-traffic-surge-retail-sites-not-machine-readable)
2. [eCommerceNews, Adobe machine-readability scores by page type, April 2026](https://e-commerce.news/story/adobe-says-ai-retail-traffic-surges-as-readability-lags)
3. [CNBC, OpenAI revamps shopping in ChatGPT after Instant Checkout, March 2026](https://www.cnbc.com/2026/03/24/openai-revamps-shopping-experience-in-chatgpt-after-instant-checkout.html)
4. [Forrester, The State Of Business Buying, 2026 (January 2026)](https://www.forrester.com/blogs/state-of-business-buying-2026/)
5. [Vercel and MERJ, The rise of the AI crawler, December 2024](https://vercel.com/blog/the-rise-of-the-ai-crawler)
6. [RFC 9309, Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html)
7. [Microchip Technology, MCP Server press release, November 2025](https://ir.microchip.com/news-events/press-releases/detail/1344/microchip-technology-unveils-model-context-protocol-mcp-server-to-power-ai-driven-product-data-access)
8. [ECIA, TrustedParts.com launches Inventory AI Agent Service, June 2026](https://www.ecianow.org/2026/06/11/trustedparts-com-launches-inventory-ai-agent-service/)
9. [Texas Instruments, LM358 datasheet (used as a worked example)](https://www.ti.com/lit/ds/symlink/lm358.pdf)

---

Partsgraph — the agent-ready parts data layer. Free AI-visibility grader: https://partsgraph.ai/audit