--- title: "We Audited AI-Crawler Policies at 46 Top Crypto Sites. 98% Let Every Bot Through." description: "The standard GEO advice is 'check your robots.txt, you're probably blocking AI crawlers.' We ran the numbers on the biggest crypto and web3 sites. The accidental-block panic is aimed at the wrong target." published: 2026-07-29 updated: 2026-07-29 author: "Zion Labs" canonical: https://www.zionlabs.io/research/crypto-ai-crawler-benchmark source: Zion Labs research --- # We Audited AI-Crawler Policies at 46 Top Crypto Sites. 98% Let Every Bot Through. > **This was our pilot.** We've since expanded it into the [Crypto & Fintech AI-Readiness Index](/research/crypto-fintech-ai-readiness-index): 146 sites across both verticals, refreshed quarterly. Start there for the current data; this piece remains as the original 46-site study. There's a piece of advice repeated in every GEO deck this year: *audit your robots.txt. You're probably blocking the AI crawlers that feed answer engines, and you don't even know it.* It's good advice. But is it true of the companies that actually matter in a category? We decided to stop asserting it and measure it. We pulled the live `robots.txt` from **46 leading crypto and web3 sites** (the major exchanges, wallets, L1s and L2s, data providers, and on-ramps) and ran each one through the same parser behind our [AI Crawler Audit](/tools/ai-crawler-audit), checking 13 named AI user-agents against RFC-9309 precedence rules. Here's what we found, and why it changes who should actually be worried. ## The headline number Of the 46 sites, **43 returned a readable `robots.txt`** (three sat behind bot protection that blocked even a plain file request, a finding in itself). Across those 43: - **1 site (2%)** blocks any AI crawler at all. - **0 sites** block the search-and-answer crawlers specifically: `OAI-SearchBot`, `PerplexityBot`, `ChatGPT-User`, `Claude-User` and `Perplexity-User`, the ones that build live AI answers. - The remaining **~98%** allow every AI bot we checked, training and search alike. The top of the crypto market has, quietly and almost universally, decided to let the machines read everything. ## Why this is the opposite of the common warning The "you're accidentally blocking AI" narrative comes from the early-2025 wave, when sites reflexively pasted `Disallow: /` blocks for `GPTBot` and friends to keep their content out of training data, and caught the *search* crawlers in the same net by mistake. The leading crypto brands are past that. Whether by deliberate strategy or by never having blocked in the first place, they've concluded that visibility in AI answers is worth more than withholding their content from a training run. For a category where the buying decision increasingly starts with "ask an AI which exchange is safe," that's rational. So if you run a top-20 exchange, the accidental-block problem probably isn't yours. **The advice is aimed at the wrong target.** ## So who *is* blocking, and how it happens invisibly Two groups still have the problem, and neither shows up in a benchmark of category leaders: 1. **The long tail.** Smaller and older sites: the challenger exchange, the mid-tier protocol, the project that shipped its `robots.txt` in 2023 and never revisited it. These are exactly the sites competing for the same AI citations as the incumbents, and a stale block is a self-inflicted wound. 2. **The managed layer you didn't write.** This is the sneaky one. CDN and WAF providers now ship one-click "block AI bots" features that **inject `Disallow` rules into your served `robots.txt` at the edge**, rules that never appear in your repo. You can hand-write a perfectly welcoming `robots.txt`, deploy it, and still serve a file that blocks `GPTBot`, `ClaudeBot`, and `CCBot`, because a toggle in a dashboard you forgot about is prepending its own block. We know that one first-hand: it happened on this very site. Our hand-written file welcomed every crawler; our CDN's managed rule was quietly injecting a block above it. Our own audit tool flagged us. That's the point: **the block you need to worry about is the one you didn't author.** ## What to actually do - **Verify the served file, not the source.** Fetch your live `robots.txt` over HTTPS and read what's actually returned, not what's in your repo. (Our [crawler audit](/tools/ai-crawler-audit) does exactly this.) - **Check the managed layer.** If you're on Cloudflare, Fastly, or similar, look for an "AI bots" or "AI scrapers" toggle. It may be blocking on your behalf. - **Separate the two decisions.** Blocking *training* crawlers (`GPTBot`, `CCBot`) is a defensible content-licensing choice. Blocking *search* crawlers (`OAI-SearchBot`, `PerplexityBot`) removes you from AI answers, almost always by accident. The incumbents already got this right. The opportunity, as usual in crypto, belongs to whoever in the challenger set stops guessing and checks. --- *Method: live `robots.txt` fetched for 46 leading crypto/web3 domains on July 29, 2026; 43 returned a readable file. User-agent policies evaluated for 13 named AI crawlers using RFC-9309 group precedence. Data and analysis are Zion Labs'.*