← Research

We Audited AI-Crawler Policies at 46 Top Crypto Sites. 98% Let Every Bot Through.

By Ben · Published July 29, 2026 · 6 min read

There’s a piece of advice repeated in every GEO deck this year: audit your robots.txt — you’re probably blocking the AI crawlers that feed answer engines, and you don’t even know it.

It’s good advice. But is it true of the companies that actually matter in a category? We decided to stop asserting it and measure it. We pulled the live robots.txt from 46 leading crypto and web3 sites — the major exchanges, wallets, L1s and L2s, data providers, and on-ramps — and ran each one through the same parser behind our AI Crawler Audit, checking 13 named AI user-agents against RFC-9309 precedence rules.

Here’s what we found — and why it changes who should actually be worried.

The headline number

Of the 46 sites, 43 returned a readable robots.txt (three sat behind bot protection that blocked even a plain file request — a finding in itself). Across those 43:

  • 1 site (2%) blocks any AI crawler at all.
  • 0 sites block the search-and-answer crawlers specifically — OAI-SearchBot, PerplexityBot, ChatGPT-User, Claude-User, Perplexity-User — the ones that build live AI answers.
  • The remaining ~98% allow every AI bot we checked, training and search alike.

The top of the crypto market has, quietly and almost universally, decided to let the machines read everything.

Why this is the opposite of the common warning

The “you’re accidentally blocking AI” narrative comes from the early-2025 wave, when sites reflexively pasted Disallow: / blocks for GPTBot and friends to keep their content out of training data — and caught the search crawlers in the same net by mistake.

The leading crypto brands are past that. Whether by deliberate strategy or by never having blocked in the first place, they’ve concluded that visibility in AI answers is worth more than withholding their content from a training run. For a category where the buying decision increasingly starts with “ask an AI which exchange is safe,” that’s rational.

So if you run a top-20 exchange, the accidental-block problem probably isn’t yours. The advice is aimed at the wrong target.

So who is blocking — and how it happens invisibly

Two groups still have the problem, and neither shows up in a benchmark of category leaders:

  1. The long tail. Smaller and older sites — the challenger exchange, the mid-tier protocol, the project that shipped its robots.txt in 2023 and never revisited it. These are exactly the sites competing for the same AI citations as the incumbents, and a stale block is a self-inflicted wound.

  2. The managed layer you didn’t write. This is the sneaky one. CDN and WAF providers now ship one-click “block AI bots” features that inject Disallow rules into your served robots.txt at the edge — rules that never appear in your repo. You can hand-write a perfectly welcoming robots.txt, deploy it, and still serve a file that blocks GPTBot, ClaudeBot, and CCBot, because a toggle in a dashboard you forgot about is prepending its own block.

We know that one first-hand: it happened on this very site. Our hand-written file welcomed every crawler; our CDN’s managed rule was quietly injecting a block above it. Our own audit tool flagged us. That’s the point — the block you need to worry about is the one you didn’t author.

What to actually do

  • Verify the served file, not the source. Fetch your live robots.txt over HTTPS and read what’s actually returned — not what’s in your repo. (Our crawler audit does exactly this.)
  • Check the managed layer. If you’re on Cloudflare, Fastly, or similar, look for an “AI bots” or “AI scrapers” toggle. It may be blocking on your behalf.
  • Separate the two decisions. Blocking training crawlers (GPTBot, CCBot) is a defensible content-licensing choice. Blocking search crawlers (OAI-SearchBot, PerplexityBot) removes you from AI answers — almost always by accident.

The incumbents already got this right. The opportunity — as usual in crypto — belongs to whoever in the challenger set stops guessing and checks.

Method: live robots.txt fetched for 46 leading crypto/web3 domains on July 29, 2026; 43 returned a readable file. User-agent policies evaluated for 13 named AI crawlers using RFC-9309 group precedence. Data and analysis are Zion Labs’.

Zion Labs researches how AI answer engines choose what to cite, and builds the tools and monitoring to help brands become the source they trust. Try the free tools or book a free audit.