robots.txt for AI Crawlers: The 2026 Guide

The short answer

Stop thinking about "AI bots" as one thing. There are three classes with completely different economics, and a single Disallow line applied to all of them is almost always the wrong trade.

A training crawler is asking to learn from your catalog. A search indexer is asking to be able to cite it. A user-triggered fetcher is a customer, right now, asking a question about your product. Blocking the first is a licensing decision. Blocking the third is hanging up the phone.

Forrester found that 94% of business buyers used AI during their most recent buying process. A meaningful share of the fetches hitting your origin with an AI user-agent are not scraping — they are someone with a bill of materials open, mid-decision. Your robots.txt should distinguish between them, and most do not.

Every AI user-agent that matters in 2026

Five points in that table are routinely misunderstood or missed, and they are worth stating explicitly.

`Google-Extended` and `Applebot-Extended` are not crawlers. Neither one fetches anything. They are permission tokens: Google-Extended controls whether content already crawled by Googlebot may be used to train Gemini and to ground its answers, and Apple's documentation is explicit that Applebot-Extended "does not crawl webpages" and is "only used to determine how to use the data crawled by the Applebot user agent". Disallowing Applebot-Extended does not remove you from Siri, Spotlight or Safari results.

Anthropic is the outlier on user-triggered fetches. OpenAI states that for ChatGPT-User, "because these actions are initiated by a user, robots.txt rules may not apply". Perplexity says Perplexity-User "generally ignores robots.txt rules". Google's documentation says of its user-triggered fetchers: "because the fetch was requested by a user, these fetchers generally ignore robots.txt rules". Amazon says Amzn-User "may not follow all robots.txt directives". Anthropic, by contrast, documents that its bots honour robots.txt and names Claude-User as a token site owners can use to control user-initiated requests. If you disallow the whole set, Claude is the one that will actually stop — which is the opposite of what most people intend.

Amazon and Meta each run a split fleet. Beyond Amazonbot, Amazon documents Amzn-SearchBot (search experiences including Alexa and Rufus) and Amzn-User (live user actions), both of which it says do not crawl for generative AI training. Meta runs Meta-ExternalAgent for training and indexing and Meta-ExternalFetcher for user-requested fetches, the latter of which may bypass robots.txt.

Bytespider is a policy problem, not a robots.txt problem. ByteDance states that it respects robots.txt; site operators widely report otherwise. If you want it stopped, stop it at the edge and do not rely on the file.

A note on `Gemini-Deep-Research`: Google's published crawler documentation does not yet list this token among its common crawlers or user-triggered fetchers, but it is in active use and major distributors already address it by name — Digi-Key's robots.txt explicitly allows both Google-Extended and Gemini-Deep-Research, alongside GPTBot, ChatGPT-User, OAI-SearchBot, PerplexityBot, Perplexity-User, ClaudeBot, Claude-User and Claude-SearchBot. Include it; an unrecognised token in robots.txt is harmless.

What does blocking each class actually cost?

For a manufacturer or distributor whose catalog is already public and already mirrored across aggregators, blocking the training class is mostly symbolic: the data reaches the models regardless, via distributor listings and Common Crawl. What changes is whose version of your data the model holds. Blocking the other two classes has no upside at all for a public catalog.

Copy-pasteable robots.txt

Recipe A: maximum agent visibility

The right default for a manufacturer or distributor whose catalog is public and whose commercial interest is being found and cited accurately.

# --- Answer engines and user-triggered fetchers ---
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Gemini-Deep-Research
User-agent: Amzn-SearchBot
Disallow: /account/
Disallow: /cart/
Disallow: /checkout/
Allow: /

# --- Training crawlers and permission tokens ---
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Meta-ExternalAgent
User-agent: Amazonbot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /account/
Disallow: /cart/
Disallow: /checkout/
Allow: /

# --- Everything else ---
User-agent: *
Disallow: /account/
Disallow: /cart/
Disallow: /checkout/

Sitemap: https://www.example.com/sitemap.xml
Sitemap: https://www.example.com/sitemap-products.xml

Recipe B: allow retrieval, refuse training

For organisations that have made a deliberate decision not to contribute to model training but still want to be found and cited. Note what this costs: disallowing Google-Extended removes your content from Gemini grounding as well as training.

User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Gemini-Deep-Research
Allow: /

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Bytespider
User-agent: Meta-ExternalAgent
User-agent: Amazonbot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

User-agent: *
Disallow: /account/

Recipe C: catalog open, everything else closed

Useful when the catalog and document library are the assets you want surfaced, and the rest of the estate is noise.

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Gemini-Deep-Research
Disallow: /
Allow: /products/
Allow: /catalog/
Allow: /datasheets/
Allow: /documents/
Allow: /sitemap-products.xml

Under RFC 9309 the longest matching path rule wins, so the Allow lines override the blanket Disallow: / for those prefixes. Verify the result rather than assuming it — parser behaviour on edge cases still varies.

Expressing "no" in a way that carries: Content Signals

robots.txt controls access. It says nothing about use once the bytes have been served. Cloudflare's Content Signals Policy, launched 24 September 2025, adds a machine-readable usage preference to the same file, with three signals: search (building a search index and providing search results), ai-input (inputting content into one or more AI models) and ai-train (training or fine-tuning AI models).

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Disallow: /account/

Cloudflare applied search=yes, ai-train=no to over 3.8 million domains that had enabled its managed robots.txt, deliberately leaving ai-input unset rather than guessing. It is a stated preference, not an enforcement mechanism — but it is a clear, dated, machine-readable statement of intent, which is worth having.

Why silence is becoming "no"

For twenty years, an absent robots.txt rule meant "yes". That default is being withdrawn at the infrastructure layer, and manufacturers who have never touched the file are being opted out by their CDN.

• 1 July 2025 — Cloudflare, which sits in front of roughly a fifth of the web, became the first major infrastructure provider to block AI crawlers by default, asking every newly onboarded domain up front whether to permit them.
• 24 September 2025 — the Content Signals Policy shipped, with ai-train=no applied by default to 3.8 million managed domains.
• 1 July 2026 — Cloudflare announced that from 15 September 2026, on new domains, new sites of existing customers and all existing free-tier customers, the Training and Agent crawler categories are blocked by default on pages that display ads. Search stays allowed. Multi-purpose crawlers that blend Search with Training inherit the most restrictive rule.

Read that last one precisely, because it is widely misquoted. It is scoped to ad-monetised pages, which most manufacturer catalogs are not. The change that already affects you is the 2025 one: if your site was onboarded to a modern CDN recently, or sits on a free tier, you may already be refusing AI crawlers without anyone having decided to. Check before you assume.

robots.txt is a request. Your WAF is the enforcement.

This is where most well-intentioned configurations fail. robots.txt is a voluntary protocol read by compliant clients. Your bot manager is a hard gate that classifies by verified source IP, and default rule sets routinely challenge or block anything that does not look like a browser. A crawler that reads Allow: / and then receives a JavaScript challenge has been blocked, whatever the file says.

Every major operator publishes its ranges so you can allowlist them properly:

To check whether a specific address in your logs is genuinely OpenAI:

python3 - <<'EOF'
import ipaddress, json, urllib.request
probe = ipaddress.ip_address("132.196.86.5")
data = json.load(urllib.request.urlopen("https://openai.com/gptbot.json"))
nets = [ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix"))
        for p in data["prefixes"]]
print(any(probe in n for n in nets if n.version == probe.version))
EOF

Three edge rules cover most of it:

1. 01Allowlist the verified ranges before your bot-score rule fires, scoped to public catalog and document paths only.
2. 02Exempt those paths from JavaScript challenges and rate limits. A non-browser client cannot solve a challenge, and a crawler indexing 40,000 SKUs will trip a rate limit tuned for humans.
3. 03Do not allowlist by user-agent string. It is a free-text header. Match on IP, then optionally on user-agent.

Remember that each origin is configured separately. Your product site, your documentation subdomain and your CDN hostname each need their own robots.txt and their own WAF policy.

How do I verify it worked from my logs?

Configuration you have not verified in logs is a hypothesis. Allow a few days before judging the result: operators cache robots.txt, and Meta's documentation warns that changes can take up to 24 hours to take effect, while Amazon says its crawlers may use a copy cached within the last 30 days. Then:

grep -aiE "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|Claude-SearchBot|PerplexityBot|Perplexity-User|Gemini-Deep-Research|Applebot|Amazonbot|Bytespider|CCBot|Meta-External" access.log | wc -l

Then break it down by agent and status code — this is the number that matters:

awk '/GPTBot/ {print $9}' access.log | sort | uniq -c | sort -rn

What you are looking for:

1. 01Non-zero volume per agent. Zero requests from any AI agent over 72 hours on a public catalog means you are blocked upstream, not merely unpopular.
2. 02A 2xx-dominated status distribution. A wall of 403s means the WAF is overriding robots.txt. A wall of 404s means your sitemap or internal links are pointing at URLs that no longer exist.
3. 03Document paths appearing, not just product paths. If /datasheets/ never shows up, your technical library is invisible.
4. 04Response sizes that make sense. Repeated 200s of 4–15 KB on product URLs usually means crawlers are receiving an empty client-rendered shell — technically allowed, practically useless.
5. 05The user-triggered agents appearing at all. ChatGPT-User, Claude-User and Perplexity-User traffic is the closest thing you have to a real-time signal of buyer interest. It is worth reporting on separately.

Common mistakes

• Blocking user-triggered fetchers by accident, in a blanket rule aimed at training crawlers.
• Setting robots.txt correctly and leaving the WAF untouched. These are two switches and both must be set.
• Forgetting the documentation subdomain, the DAM and the CDN, each of which is a separate origin under RFC 9309.
• Blocking crawlers from CSS and JavaScript assets. It does not matter to text-only crawlers, but Googlebot and Applebot both render, and starving them of assets degrades what they see.
• Assuming an aggregator carries your data accurately. It carries whatever it scraped, whenever it scraped it.
• Treating llms.txt as a substitute. No major operator has committed to reading it, and logs generally show it is never requested.

Partsgraph audited 984 distributor and manufacturer domains worldwide, across North America, Europe and Asia, in August 2026. Only 19% published an explicit AI-crawler policy of any kind — and among those that did, more than twice as many blocked a major AI crawler as invited one. None advertised an MCP endpoint. Median AI-visibility score: 50 out of 100. The gap is not competitive yet — which is precisely why a well-formed robots.txt is still one of the highest-leverage hours of work available to a product-data team.

Check what AI crawlers can actually reach on your domain with the free Partsgraph grader at [/audit](/audit).