97% of llms.txt files are never read—but I put one on my website anyway

The paradox that sums up the whole story

Between June 2025 and May 2026, the number of sites publishing an llms.txt file rose from 4,088 to 36,120, according to Originality.ai’s tracking of over 3 million domains. That’s an 8.8-fold increase in twelve months. During the same period, Ahrefs analyzed the server logs of 137,000 domains and found that 97% of these files received no requests in May 2026. Not a single one. And among the meager 3% that received traffic, the top requesters aren’t even AI systems: they’re SEO tools (21.7% of requests), checking to see if the file exists so they can check a box in an audit report. GPTBot accounts for 4.51% of requests, ClaudeBot for 0.80%.

In other words: tens of thousands of sites are writing a letter that almost no one opens. And yet, I’ve put one on my site. Not out of naivety—but by design. To understand the calculation, you first need to understand what this file really is, who actually uses it, and why both sides of the debate are each half-right.

llms.txt in Two Minutes: A Markdown Summary for Machines

The proposal comes from Jeremy Howard, co-founder of Answer.AI (and the guy behind fast.ai), published on September 3, 2024, on llmstxt.org. The initial observation is sound: when a language model visits your page, it receives your entire HTML—the navigation, scripts, cookie banners, and footer. This is noise that wastes its context window. The idea: to provide a condensed, structured version in Markdown at the root of the site (/llms.txt) that AI systems can consume directly.

The format is intentionally simple. An H1 heading, a blockquote summarizing the site, followed by H2 sections containing lists of links in the format [name](url): descriptive note. An “Optional” section can group together content that can be omitted when context is limited. The spec also includes a more robust counterpart, llms-full.txt, which embeds the full content of the pages rather than just links—Anthropic publishes both for its developer documentation, and this file is growing at an even more staggering rate: ×107 over the same period, from 23 to 2,463 sites.

That’s it. No exotic markup, no JSON schema, no validation. A text file that a human can read and a machine prefers.

Who publishes it—and who actually reads it

The list of adopters reads like a directory of developer tools: Anthropic, Stripe, Cloudflare, Zapier, Vercel, Supabase, Cursor. Cloudflare takes the concept a step further by publishing separate pairs of files for each of its product lines. And Mintlify, the documentation platform, enabled automatic generation for all its client sites in November 2024—thousands of llms.txt files appeared overnight, which, incidentally, boosts adoption statistics more than any single decision.

But look who’s missing from the list: ordinary companies. As of March 2026, only 7.4% of Fortune 500 companies had an llms.txt file—compared to 92.8% for robots.txt. And most importantly, look who isn’t reading it: the crawlers from major AI providers. None—not OpenAI, Anthropic, Google, or Perplexity—has officially announced that it uses this file. John Mueller of Google put it bluntly on Reddit in the spring of 2025: to the best of his knowledge, no AI service uses llms.txt; server logs confirm this, and the file reminds him of the “keywords” meta tag—an unverified self-declaration of what the site claims to be. “Why not just check the site directly?” ”

On this specific point, Mueller is right, and Ahrefs’ data proves it. If you add an llms.txt file hoping that ChatGPT will suddenly start citing your site in its responses, you’re going to be disappointed. That’s not how response engines build their citations—I’ve detailed what really matters in my article on AI citability, and llms.txt isn’t at the top of the list.

The real “reader” isn’t the crawler you think it is

So why are Stripe and Anthropic bothering with it? Because the debate is misguided. Everyone views llms.txt as a crawl tool—a signal for indexing bots that come in droves. Its real use in 2026 lies elsewhere: on-demand consultation, right in the middle of a work session.

When a developer types @docs in Cursor, when Claude Code fetches documentation for a library, when an MCP server exposes documentation to an agent—these tools consume llms.txt. Not through periodic crawling, which would leave massive traces in the logs, but through ad-hoc fetching, at the exact moment a human asks a question. That’s exactly why the list of adopters is dominated by developer-focused products: their documentation is queried by code agents thousands of times a day, and a clean Markdown index tangibly improves the quality of the responses these agents provide about their product.

The practical conclusion I draw from this: the value of llms.txt depends entirely on who is querying your content via an intermediary agent. API documentation or SaaS product documentation? The file really does its job. A small business showcase site in Sherbrooke? No one will be asking for your llms.txt this week. But—and this is where my assessment differs from the prevailing cynicism—the cost of production is so low that the risk-reward ratio remains in my favor.

How I Serve It on My Site (and the Mistake I Made)

My site runs on Next.js with Supabase. Instead of a static file, I serve /llms.txt via a route handler that queries the database with every request—so published blog posts are automatically added to it:

// app/llms.txt/route.js
export async function GET() {
  const { data: articles } = await supabase
    .from("blog_articles")
    .select("title, slug")
    .eq("published", true)
    .order("id", { ascending: false });

  const list = articles
    .map((a, i) => `${i + 1}. ${a.title} — /blog/${a.slug}`)
    .join("\n");

  return new Response(
    `# Pascal Potvin — Designer & Digital Creator\n\n## Blog\n\n${list}`,
    { headers: { "Content-Type": "text/plain; charset=utf-8" } }
  );
}

Twenty lines, zero dependencies, and the file can’t misrepresent the site’s status since it’s generated from the same database as the pages.

The file’s content follows the spec’s structure: who I am, my services with their URLs, my tools, and the list of articles. No superlatives, no marketing copy—just facts and links. A bot reading this file should be able to correctly answer “What services does Pascal Potvin offer?” without loading a single other page.

The pitfall: turning it into a “meta keywords 2.0”

Mueller’s comparison to the meta keywords tag is more than just a jab—it’s a prophecy about how this file might meet its end. The meta keywords tag didn’t die because the idea was bad; it died because websites lied in it—on a massive scale—until search engines stopped reading it. The same fate awaits llms.txt if people treat it as a tool for manipulation.

And the industry is already at it. Ever since “GEO” became the buzzword of 2026, I’ve seen offers from agencies billing “llms.txt optimization” as a premium service, with the implicit promise of better citations in ChatGPT. Let’s be clear: there is no data—zero—showing that an llms.txt file improves the frequency with which response engines cite it. The Ahrefs study from June 2026 literally shows the opposite: the targeted systems don’t even ask for the file. Paying to “optimize” a document that no one reads is the pinnacle of selling hot air.

Three simple rules to stay on the right side of things: only write in your llms.txt what is verifiable on the site itself; don’t stuff it with keywords—the models read natural language, not keyword density; and don’t spend any money or more than an hour on it. It’s a technical best-practice file, not a strategy.

My verdict: a 20-minute gamble

Here’s the full breakdown. Cost: about 20 minutes if your content is already structured, zero dollars, zero maintenance if you generate it dynamically. Current benefit: marginal for a site like mine—a few IDE agents and assistants that query it on demand. Potential benefit: If one of the major providers decides tomorrow to officially adopt the format (the demand exists: “llms.txt” is searched for 17,000 times a month worldwide, according to Ahrefs in July 2026), sites that are already set up will have a head start. Risk: none, as long as the file tells the truth.

An asymmetric bet at virtually no cost—it’s worth taking. But it doesn’t replace the real work: a fast site, factual and up-to-date content, clean structured data, and authority built link by link. The llms.txt file is the doorknob; the house behind it still has to be worth visiting.