# The Agentic Web and Industrial Product Data

> The web is splitting into a human surface and a machine surface. Crawl-vs-refer economics, Cloudflare's toll gates, signed agents, and what to do about it.

**Published:** 2026-08-09  
**Category:** Market · **Tags:** agentic web, crawlers, Cloudflare, Web Bot Auth, llms.txt, product data  
**Canonical:** https://partsgraph.ai/blog/the-agentic-web-and-industrial-data

## In short

**How is the agentic web changing what industrial manufacturers must publish?**

The web is separating into a human surface optimised for browsers and a machine surface optimised for agents, and the two are being governed differently. Automated traffic already exceeds human traffic — Imperva measured 53% of web traffic as automated in 2025 — and infrastructure providers are now metering it: from 15 September 2026 Cloudflare blocks AI training and agent crawlers by default on ad-bearing pages for new and free-tier sites, with pay-per-crawl and cryptographically signed agent identity as the emerging alternatives to blanket blocking. For a manufacturer the practical consequence is that being crawlable is no longer the same as being reachable, and the companies that become accurate and reachable first become the default answer.

---

## The web now has two audiences, and only one of them has eyes

For thirty years, publishing on the web meant publishing for a person holding a mouse. Everything downstream — layout, JavaScript frameworks, cookie banners, analytics, ad inventory — assumed a human at the other end.

That assumption is now false more often than it is true. Imperva's 2026 Bad Bot Report found automated traffic accounted for 53% of all web traffic in 2025, up from 51%, with human traffic down to 47% and still falling. Cloudflare made the same observation when announcing its 2026 crawler policy: the majority of internet traffic is non-human.

The interesting consequence is not the ratio. It is that the two audiences are now being *governed separately* — different access rules, different pricing, different identity requirements. The web is bifurcating into a human surface and a machine surface, and industrial manufacturers have spent no time at all thinking about the second one.

## Why did the economics break?

Search worked because it was a trade. A crawler took your page and returned visitors. Publishers accepted the deal because the exchange rate was favourable.

AI retrieval broke the exchange rate. Cloudflare began publishing crawl-to-refer ratios — pages crawled per referral sent — on Radar in 2025, and the numbers were not close to the search norm of a couple of crawls or fewer per visitor. In a late-June 2025 sample, Anthropic's crawler sat at roughly 70,900 pages crawled per referral. Cloudflare also reported in 2026 that more than half of AI crawl traffic is spent re-fetching pages that have not changed.

Meanwhile the human side of the trade thinned out independently. SparkToro measured 68% of US Google searches ending without a click in early 2026, with roughly 276 clicks per 1,000 searches reaching the open web, down from 374 in 2024.

So publishers faced rising extraction and falling compensation, and did the predictable thing.

**But note carefully that this logic is specific to businesses whose content is the product.** A manufacturer's datasheet is not monetised by pageviews. It is marketing collateral for a physical thing you sell at a margin. If an assistant reads your entire catalog 70,000 times and sends one engineer who specifies your part into a 500,000-unit programme, you have won. The publisher's crawl-to-refer complaint, imported unexamined into an industrial context, produces exactly the wrong policy.

## What the toll gates actually do

| Mechanism | What it is | Status | Relevance to a manufacturer |
| --- | --- | --- | --- |
| Purpose-based categorisation | Cloudflare splits AI traffic into Search, Agent and Training, each separately controllable | Live; new defaults from 15 September 2026 | High — you should decide your own policy per category rather than inherit one |
| Default blocking on ad pages | Training and Agent crawlers blocked by default on ad-bearing pages; mixed-use crawlers blocked entirely there | From 15 September 2026, for new customers, new sites and existing free-tier sites | Indirect — your catalog is unlikely to carry ads, but the trade press discussing your parts does |
| Pay-per-crawl | HTTP 402 responses quoting a price, with `crawler-max-price`, `crawler-exact-price` and `crawler-charged` headers, settled at the edge | Private beta | Low today; a signal of where metering is heading |
| Web Bot Auth / signed agents | Cryptographic agent identity via HTTP Message Signatures and a `Signature-Agent` header, replacing user-agent strings | Shipped by Cloudflare in August 2025; progressing through the IETF | **High** — this is the precondition for serving entitled data to a known agent |

The last row is the one that matters commercially, and it is the one nobody in industrial supply is discussing.

Everything an industrial supplier actually sells on — contract pricing, allocated stock, customer-specific lead times, approved-alternate lists — is data you cannot publish openly. Today that means agents get the public list price and a generic availability string, which is to say they get something wrong. Verifiable agent identity is what turns "we cannot expose that" into "we can expose that to *this* agent, acting for *this* customer, and log it." That is not a crawler-blocking story. It is a channel-enablement story that happens to run on the same plumbing.

## The llms.txt lesson

It is worth dwelling on llms.txt, because it is the perfect illustration of how this market misallocates effort.

The proposal was reasonable: a plain-text file telling AI systems where your good content is. It was widely adopted. It was written up in every marketing newsletter. Ahrefs then analysed 137,210 domains and found that **97% of published llms.txt files received zero requests in May 2026**. Most of the fetches that did occur came from SEO audit tools rather than AI systems, and the study found no evidence of any citation lift in ChatGPT, Perplexity or AI Overviews.

Meanwhile the same dataset showed AI bots making up 19.5% of all requests, with agentic infrastructure the largest AI category at 10.5%. The machines were absolutely there. They were just fetching your actual pages — the ones many manufacturers render client-side and therefore serve empty.

The lesson generalises beyond one file. **Adoption of a convention is not the same as consumption of it.** The reliable test is not whether a standard has momentum in the discourse; it is whether real clients fetch it. Two things currently pass that test: server-rendered HTML, and Model Context Protocol endpoints — MCP having reached roughly 97 million monthly SDK downloads by March 2026 and moved to neutral governance under the Linux Foundation's Agentic AI Foundation in December 2025, with AWS, Anthropic, Block, Bloomberg, Cloudflare, Google, Microsoft and OpenAI as platinum members.

## The regulatory current runs the same way

There is a second force pushing industrial data into machine-readable form, and it has nothing to do with AI.

Commission Implementing Regulation (EU) 2026/1778, adopted on 16 July 2026, laid down the operating arrangements for the Digital Product Passport registry established under the Ecodesign for Sustainable Products Regulation — structure, identity verification, registration, proof of registration, logging and retention. The Commission launched the registry days later. Construction products run on a separate track under the revised Construction Products Regulation, on a longer timetable.

Strip the compliance language and the requirement is: structured, machine-readable product data, with verified identity and provenance, retrievable by a third party. That is very nearly a description of an agent-ready catalog. Any manufacturer treating DPP compliance and AI readability as two unrelated programmes with two budgets is building the same asset twice.

## What a manufacturer should actually do

1. **Test what a machine sees.** Fetch your own catalog pages with a plain non-browser client. When Partsgraph audited 984 distributor and manufacturer domains worldwide in August 2026, 38% returned no readable catalog page to a standard non-browser client, and the median AI-visibility score was 50 out of 100. Client-side rendering is the usual culprit and it is invisible in every normal analytics view.
2. **Write an explicit crawler policy and publish it.** Decide, per category, what you allow: search indexing, retrieval for answers, agent access, training. In the same audit, only 19% of the 984 domains published an explicit AI-crawler policy naming the major bots — meaning four in five are running whatever default their CDN vendor chooses for them, and as edge defaults tighten, silence resolves to "no".
3. **Check your CDN defaults before 15 September 2026.** If you are on a free tier or spinning up new properties, the defaults change under you. This is a ten-minute task that nobody has diarised.
4. **Do not block indiscriminately.** Copying a publisher's blocking posture optimises for revenue you do not earn and forfeits distribution you do need.
5. **Serve the durable data statically and the volatile data live.** Parametrics, compliance status and lifecycle can be server-rendered and structured. Stock, lead time and contract price cannot be crawled correctly at any refresh rate and belong behind a queryable endpoint.
6. **Prepare for identity, not just access.** Signed agents will make entitlement-aware responses possible. The suppliers who have already normalised their pricing and availability data will be able to use that; the ones who have not will spend that window doing data cleanup.
7. **Monitor accuracy, not just visibility.** A confidently wrong lead time does more commercial damage than an absent one, and neither shows up in a rankings report.

## Why being first matters more here than usual

There is a specific reason to move early, and it is not the ordinary land-grab argument.

Assistants converge. When a model finds a source that answers a class of question correctly and cheaply — one that parses without ambiguity, resolves part numbers consistently and does not contradict itself between pages — that source becomes the path of least resistance for the whole class. It gets retrieved more, cited more, and reinforced in the next round of training and in the retrieval indexes built on top.

That is not a ranking you can buy back later with a bigger budget, because the mechanism is not an auction. It is a default. And industrial defaults are unusually sticky, because they are laundered through documents that persist: an approved-vendor list, a standard specification, a template bill of materials. A part that becomes the default answer in 2026 is still in the template in 2031.

The corollary is uncomfortable for anyone waiting for the picture to clarify. The cost of being early is a data project. The cost of being late is not a data project done later — it is a data project done later against a competitor who is already the default answer.

*The free grader at [/audit](/audit) shows exactly what a machine reads from your catalog today, and where the gaps are.*

## Common questions

### What changes on 15 September 2026?

Cloudflare begins categorising AI crawler traffic as Search, Agent or Training rather than treating it as one class. From that date, Training and Agent crawlers are blocked by default on pages that display advertising, and mixed-use crawlers that will not declare their purpose per request are blocked entirely on those pages. The defaults apply to new customers, new sites created by existing customers, and all existing free-tier sites; paying customers can override them in security settings beforehand.

### Does this affect a manufacturer's product catalog?

Usually not directly, because the defaults key off ad-bearing pages and most manufacturer catalogs carry no advertising. It matters for two indirect reasons. First, the trade publications, distributor content and technical forums where your parts are discussed are frequently ad-supported, so third-party coverage of your products may become less available to assistants. Second, it establishes the precedent that crawler access is a metered, per-purpose permission — and that model will spread.

### What is crawl-to-refer, and why does it matter?

It is the number of pages an AI company crawls for every visitor it sends back. Cloudflare began publishing these ratios on Radar in 2025, with Anthropic's crawler at roughly 70,900 pages crawled per referral in one late-June 2025 sample, against a legacy search norm of a couple of crawls or fewer per visitor sent. The ratio is why publishers started blocking. For a manufacturer the inversion is important: you are not selling advertising against traffic, so a high crawl-to-refer ratio is not a cost to you — it is distribution you are not paying for.

### Should I publish an llms.txt file?

It costs almost nothing and does almost nothing. Ahrefs analysed 137,210 domains and found 97% of published llms.txt files received zero requests in May 2026, with most of the fetches that did occur coming from SEO tools rather than AI systems, and no measurable citation lift. Publish one if it is free; never let it substitute for server-rendered pages, structured data, feeds or an endpoint.

### What is Web Bot Auth and why should a manufacturer care?

It is a way for an automated client to cryptographically prove its identity using HTTP Message Signatures, an Ed25519 key per agent and a Signature-Agent header, rather than relying on an easily spoofed user-agent string. Cloudflare shipped signed agents in August 2025 with an initial cohort including ChatGPT agent and Block's Goose, and the work is now progressing through the IETF. It matters because it is the precondition for serving different data to different agents — which is how contract pricing and customer-specific availability eventually reach agents safely.

### Is blocking AI crawlers a sensible default for an industrial supplier?

Almost never. Blocking makes sense when your content is the product and traffic is the revenue — publishing, media, research. For a manufacturer or distributor, the catalog is marketing collateral for a physical product, and being absent from an assistant's answer costs a specification. The correct posture is selective: open the catalog, publish an explicit crawler policy, and meter or authenticate the things that are genuinely commercially sensitive, such as contract pricing.

### How does EU regulation interact with this?

It pushes in the same direction from a different motive. Commission Implementing Regulation (EU) 2026/1778 of 16 July 2026 set the operating rules for the EU Digital Product Passport registry under ESPR, which the Commission launched days later. Construction products are being handled separately under the revised Construction Products Regulation on a later timetable. The compliance requirement is structured, machine-readable product data with verified identity and provenance — which is substantially the same asset the agentic web requires. Manufacturers building it twice, once for Brussels and once for the assistants, are wasting the budget.

## Sources

1. [TechCrunch, Cloudflare's new policy pushes AI companies to pay for publishers' content (1 July 2026)](https://techcrunch.com/2026/07/01/cloudflares-new-policy-pushes-ai-companies-to-pay-for-publishers-content/)
2. [Help Net Security, Cloudflare changes AI crawler access rules (2 July 2026)](https://www.helpnetsecurity.com/2026/07/02/cloudflare-ai-crawler-controls/)
3. [Cloudflare, Introducing pay per crawl](https://blog.cloudflare.com/introducing-pay-per-crawl/)
4. [Cloudflare, The crawl before the fall of referrals: crawl-to-refer ratios on Radar](https://blog.cloudflare.com/ai-search-crawl-refer-ratio-on-radar/)
5. [Cloudflare, The age of agents: cryptographically recognizing agent traffic (28 August 2025)](https://blog.cloudflare.com/signed-agents/)
6. [IETF, HTTP Message Signatures for automated traffic architecture (Web Bot Auth)](https://datatracker.ietf.org/doc/html/draft-meunier-web-bot-auth-architecture)
7. [Ahrefs, We analysed 137,000 sites: 97% of llms.txt files never get read (May 2026)](https://ahrefs.com/blog/llmstxt-study/)
8. [Imperva / Thales, 2026 Bad Bot Report: Bots in the Agentic Age](https://www.imperva.com/blog/bad-bot-report-2026-bots-agentic-age/)
9. [SparkToro, In 2026, Less than One Third of Google Searches Still Send a Click](https://sparktoro.com/blog/in-2026-less-than-one-third-of-google-searches-still-send-a-click/)
10. [Model Context Protocol, MCP joins the Agentic AI Foundation (9 December 2025)](https://blog.modelcontextprotocol.io/posts/2025-12-09-mcp-joins-agentic-ai-foundation/)
11. [EUR-Lex, Commission Implementing Regulation (EU) 2026/1778 on the digital product passport registry (16 July 2026)](https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=OJ%3AL_202601778)

---

Partsgraph — the agent-ready parts data layer. Free AI-visibility grader: https://partsgraph.ai/audit