← Research

Everyone's Generating llms.txt Files. Almost No AI Reads Them.

By Ben · Published July 29, 2026 · 6 min read

There’s a new file everyone in SEO is generating. Point a tool at your sitemap, get a tidy llms.txt, drop it at your root, and — the pitch goes — the AI models will finally understand your site.

The data says otherwise. llms.txt has real adoption and almost no real usage. Before you spend a sprint on it, here’s what’s actually true in 2026 — and where that effort belongs instead.

We have a stake in saying this plainly: we built an llms.txt generator, saw this data, and deleted it. A tool that implies a file boosts your AI visibility, when it doesn’t, is exactly the cargo-culting we exist to cut through.

TL;DR

  • ~10% of sites have adopted llms.txt — a real footprint after 18 months of hype.
  • ~0.1% of AI crawler traffic ever requests it. In one study, 84 of 62,100 AI bot visits touched the file.
  • No major AI vendor — Google, OpenAI, Anthropic, Meta, Mistral — has committed to reading it in production.
  • Google said no on the record. John Mueller compared it to the discredited keywords meta tag.
  • Ship one as a courtesy if you like. Don’t count it as an AI-visibility lever.

What llms.txt is supposed to do

llms.txt is a proposed standard: a single Markdown file at your site root that tells large language models what your site is and points them at your most important pages, in plain language. Think of it as a curated map for AI — the way robots.txt is a set of rules for crawlers.

It’s a genuinely nice idea. The problem is whether anything on the other end is reading the map.

The adoption-vs-usage gap

Adoption is real. A SE Ranking study of 300,000 domains found a 10.13% adoption rate — roughly one site in ten, after a year and a half of industry conversation.

Usage is not. When OtterlyAI looked at actual crawler behavior, only 84 of 62,100 AI bot visits requested the llms.txt file — about 0.1% of AI crawler traffic. The search and answer crawlers that build AI responses — GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended — overwhelmingly skip the file and crawl your HTML directly, the same as any other page.

A file on 10% of sites that receives 0.1% of the traffic isn’t an operational control. It’s a gesture.

What the AI companies actually say

The vendors have been clear, mostly by omission:

  • Google confirmed in mid-2025 that it doesn’t support llms.txt and isn’t planning to — John Mueller compared it to the keywords meta tag, a signal engines abandoned two decades ago because it was trivially gamed.
  • OpenAI’s crawler documentation governs its bots through robots.txt and never mentions llms.txt.
  • As of 2026, no major LLM vendor has documented that their crawlers or retrieval systems consume llms.txt from external sites.

Could that change? Sure. Standards sometimes win late. But you optimize for the engines that exist, not the ones a spec hopes to summon.

So why did it take off?

Because it’s easy, it’s automatable, and it feels like progress. A generator turns your sitemap into a file in seconds. It’s a clean deliverable to screenshot in a report. And “we’ve implemented llms.txt” sounds like AI-readiness.

None of that is the same as an AI model reading it. Effort you can measure is not the same as effort that moves the outcome.

Where that effort actually belongs

The levers that decide whether AI cites you are unglamorous and well-documented:

  1. Don’t block the answer crawlers by accident. The single most common own-goal is a robots.txt that blocks OAI-SearchBot or PerplexityBot — the crawlers that feed live AI answers. (Check yours with our AI crawler audit.)
  2. Make your content survive retrieval. AI answers are assembled from chunks, not whole pages. Each passage has to stand alone, name its subject, and carry a date. (See where yours falls apart in the chunk previewer and answerability score.)
  3. Be a resolvable entity. If a model can’t tie your brand to a known thing, it won’t recommend you with confidence. (Check with our entity check.)

Every one of those is measured, documented, and acted on by the engines today. llms.txt is none of those things.

The honest verdict

Publishing an llms.txt costs you almost nothing, and a handful of niche tools do read it, so there’s no harm in shipping one as a courtesy. Keep it hand-written and accurate if you do.

Just don’t file it under “AI visibility.” The map is only useful if someone’s driving by it — and right now, almost no one is.

Figures attributed inline to their original publishers (SE Ranking, OtterlyAI, Google/Search Engine coverage). Analysis is Zion Labs’.

Zion Labs researches how AI answer engines choose what to cite, and builds the tools and monitoring to help brands become the source they trust. Try the free tools or book a free audit.