We Serve an llms.txt. AI Crawlers Hit Our Pages 151 Times in 14 Days and Never Once Asked for It

Google's John Mueller said something specific about llms.txt that almost nobody acted on:

AFAIK, none of the AI services have said they're using LLMs.TXT (and you can tell when you look at your server logs that they don't even check for it). To me, it's comparable to the keywords meta tag, this is what a site-owner claims their site is about.

That is a testable claim, and it names the test. So we ran it on ourselves.

We serve a full llms.txt at saaslinks.net/llms.txt. It is 30KB, hand-written, and has been live since 17 June 2026. We pulled 14 days of raw nginx access logs and counted every request to it, then counted what the AI crawlers fetched instead.

Key takeaways

• Across the 14-day window, 18 named AI crawler families made requests to our server.
• Those crawlers fetched /robots.txt 723 times and /llms.txt zero times.
• /llms.txt was requested 13 times in total. Every single request came from a human browser, a tech-profiling crawler, a data broker, or our own curl checks. None came from an AI crawler.
• AI crawlers were not ignoring us. They made 151 requests to our own pages, on all 15 calendar dates the window touches.
• OpenAI accounted for 90% of that AI crawl traffic. Google-Extended fetched none of our pages at all.
• What they did fetch is the useful finding: pricing benchmarks, curated lists, and vetting explainers. Data-dense reference pages, not our sales pages.

Why bother measuring this ourselves

Three larger studies already point the same way, and we are not claiming to have found something new in direction:

• Ahrefs analyzed 137,000 domains using their own bot analytics and found 97% of published llms.txt files receive zero requests.
• SE Ranking analyzed roughly 300,000 domains and found no statistically significant correlation between having an llms.txt file and how often a domain is cited in AI answers.
• Otterly recorded 62,100 AI bot visits over 90 days; 84 of them targeted llms.txt, or 0.1%.

What those studies cannot show you is one site's complete picture: every AI crawler that visited, every file it asked for, and what it read instead. Aggregate percentages tell you the base rate. A single full log tells you the mechanism. This post is the second thing, and it is small on purpose.

Method

Window. 20 July 2026 00:01 UTC to 3 August 2026 06:58 UTC. 14 days, spanning 15 calendar dates. 147,180 parsed requests on the host.

Crawler identification. We matched user-agent strings for 18 AI-specific crawlers: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, anthropic-ai, PerplexityBot, Perplexity-User, Google-Extended, Amazonbot, meta-externalagent, Bytespider, CCBot, DuckAssistBot, MistralAI-User, YouBot, cohere-ai. We counted Googlebot, bingbot and Applebot separately as classic search crawlers.

Attribution. This is the part that needs disclosing. Our server runs several sites behind one shared nginx access log, and the default log format does not record the Host header. So we could not split historical traffic by hostname. We handled it two ways:

1. For llms.txt requests, attribution does not matter. We counted every request to /llms.txt across the entire host and found zero from AI crawlers. Zero across all sites is necessarily zero for this one.
2. For AI crawl volume on our pages, we counted only requests whose path exactly matches one of the 125 URLs in our published sitemap, and we excluded / because it is ambiguous across sites. That makes 151 a conservative floor, not a total.

The caching wrinkle, which cuts in our favor. Our /robots.txt is served from the Cloudflare edge cache (cf-cache-status: HIT), so a large share of robots.txt fetches never reach origin and never appear in these logs. Our /llms.txt is DYNAMIC and never edge-cached, so every llms.txt request reaches origin and is logged. The comparison below therefore undercounts robots.txt and counts llms.txt completely. The real gap is wider than the numbers show.

Finding 1: not one AI crawler asked for llms.txt

Thirteen requests to /llms.txt in 14 days. Here is every one of them.

Date	Status	Requester	Category
23 Jul	200	Dataprovider.com	Data broker
25 Jul	200	BuiltWith/1.4	Tech-profiling crawler
26 Jul	200	Chrome-Lighthouse	Audit tool
27 Jul	200	curl/8.7.1 (×3)	Our own checks
27 Jul	200	PipericBot/1.0	Unclassified bot
31 Jul	200	BuiltWith/1.4	Tech-profiling crawler
2 Aug	404	TheWebReport/1.0	Unclassified bot
2 Aug	301, 404	iPhone Safari	Human
3 Aug	200	Chrome-Lighthouse	Audit tool
3 Aug	200	curl/8.7.1	Our own check

Four of the thirteen were us. The rest were tech profilers, a data broker, two unclassified bots, and one person on an iPhone. The 404 and 301 responses belong to other sites on the same host that do not serve the file, which is itself a small reminder of how few sites bother.

Not one request from OpenAI, Anthropic, Perplexity, Google, Meta, Amazon, Mistral, Cohere, You.com, or Common Crawl.

Finding 2: the same crawlers fetch robots.txt constantly

The obvious objection is that AI crawlers might simply not fetch root-level files. They do.

Crawler	Requests (host-wide)	robots.txt	llms.txt
Bytespider (ByteDance)	1,932	28	0
ChatGPT-User (OpenAI)	1,134	0	0
ClaudeBot (Anthropic)	797	319	0
OAI-SearchBot (OpenAI)	586	256	0
Claude-SearchBot (Anthropic)	376	96	0
GPTBot (OpenAI)	170	0	0
PerplexityBot	126	12	0
CCBot (Common Crawl)	110	1	0
Amazonbot	29	0	0
Claude-User (Anthropic)	22	8	0
Google-Extended	16	0	0
meta-externalagent (Meta)	14	0	0
anthropic-ai	12	0	0
Perplexity-User	12	0	0
cohere-ai	10	0	0
DuckAssistBot	3	2	0
YouBot	3	1	0
MistralAI-User	1	0	0
Total	5,353	723	0

723 to nil, with robots.txt undercounted because of edge caching.

These crawlers know how to request a text file from a domain root. They request one of them on a schedule. They have never requested the other. That is the whole finding, and it is not ambiguous.

Finding 3: AI crawlers are reading us, just not that file

This is the part that makes the result meaningful rather than trivial. A zero would prove nothing if AI systems ignored the site altogether. They do not.

Restricted to paths we can confirm are ours, AI crawlers made 151 requests across 15 calendar dates:

Crawler	Requests to our pages	Share
ChatGPT-User (OpenAI, user-triggered fetch)	82	54.3%
OAI-SearchBot (OpenAI, search index)	54	35.8%
Claude-User (Anthropic, user-triggered fetch)	7	4.6%
Bytespider (ByteDance)	4	2.6%
ClaudeBot (Anthropic, training)	2	1.3%
CCBot (Common Crawl)	1	0.7%
YouBot (You.com)	1	0.7%

Two things stand out.

OpenAI is 90% of AI crawling on this site (136 of 151). If you are prioritizing work on AI visibility with finite time, that ratio is worth knowing. It also means a single vendor's behavior dominates most people's "AI traffic" numbers.

Google-Extended fetched none of our pages. It made 16 requests host-wide and zero to our confirmed URLs. Google-Extended is a control token rather than a crawler in the usual sense, so this is expected, but it is worth saying plainly because plenty of advice treats it as something you optimize for.

Finding 4: what they actually came for

The pages AI crawlers requested most:

Requests	Page
23	Guest post pricing benchmarks by DR and traffic
11	Foundational backlinks list
8	Parasite SEO and site reputation abuse
7	How to choose a link building agency
7	Best SaaS link building agencies
6	/blog index
5	Is buying backlinks safe?
5	Web 2.0 sites list
4	Domain Rating checker
4	Pricing

There is a clear shape here. The most-fetched pages are the ones carrying numbers, named entities, and curated lists: a pricing benchmark table, two vetted lists, an agency comparison, a definitional risk explainer. These are the pages an assistant would pull from to answer "what does a guest post cost at DR 50" or "which link building agencies serve SaaS."

Our sales pages barely register. The reference pages do all the work.

That is a more actionable takeaway than anything llms.txt offers: if you want to be quotable, publish things that are worth quoting. Our linkable assets guide and statistics pages breakdown cover how to build them deliberately.

What Google actually says

Google's official guide to optimizing for generative AI features, published 15 May 2026 and last updated 10 July 2026, has a mythbusting section. On machine-readable files it is unambiguous:

You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them.

The same guide states that "optimizing for generative AI search is optimizing for the search experience, and thus still SEO." It also dismisses content chunking and AI-specific rewriting in the same section.

Note the precise scope: that statement covers Google Search and its generative AI features. It is not a statement about ChatGPT, Claude, or Perplexity. Our logs are what covers those, and they say the same thing.

So should you delete your llms.txt?

No, and we are not deleting ours. That is not a contradiction, so let us be exact about the reasoning.

What the evidence supports:

• No AI crawler we observed requests the file. Four independent datasets now agree, including this one.
• Therefore llms.txt cannot currently be a driver of AI visibility or citations.
• Therefore any agency selling llms.txt implementation as an AI-visibility service is selling something with no measured effect. That is the honest version of the r/SEO complaint about it.

What the evidence does not support:

• It does not prove the file will never be used. Adoption of a proposed standard can lag its publication by years, and robots.txt itself was a convention long before it was ubiquitous.
• It does not prove the file is harmful. Ours costs one route file and no maintenance beyond keeping it accurate.
• It does not mean documentation-style llms.txt files are pointless for their other use: some developer-tool teams use them so that humans and coding assistants can paste a clean overview of a product's docs. That is a real use case with a real audience, and it has nothing to do with search rankings.

So the position that follows from the data: keep it if it costs you nothing, never pay for it as an AI-visibility tactic, and never count it in a forecast. Treat it the way you would treat a nicely formatted sitemap for a search engine that has not launched yet.

Limitations

We would rather state these than have them pointed out.

• One site, 14 days, modest volume. 151 AI-crawler requests to our pages is a small sample. A larger or more heavily crawled site would produce a firmer denominator.
• Shared log, no Host header. We attributed by exact sitemap path and excluded /, so 151 is a floor. The llms.txt zero is unaffected by this, because it was counted host-wide.
• User agents are self-reported. A crawler can spoof or omit its identity. An AI system fetching our pages under a browser user-agent string would not appear in the AI counts.
• Cloudflare sits in front. If the edge blocked an AI crawler outright, we would not see it at origin. We observed 18 AI crawler families reaching origin, so there is no wholesale block, but we cannot rule out partial filtering.
• This measures requests, not citations. It shows what crawlers fetched. It does not measure whether any of it produced a mention in an AI answer. Those are separate questions, and we cover the citation side in the websites that influence AI answers.
• Requests are not endorsements. A crawler fetching a page says the page was retrievable and plausibly relevant. It does not say the model used it.

The bottom line

Mueller's claim was that you can tell from your server logs that AI services do not check for llms.txt. We checked our server logs. He is right, at least for the 18 crawler families that visited us over 14 days: 723 robots.txt fetches, zero llms.txt fetches, and 151 requests to actual pages in the same period.

The file is not a scam and it is not a ranking factor. It is a proposed convention that the systems it was proposed for have not adopted. Keep it if it is free, drop it from your forecasts, and spend the effort on the pages the crawlers are demonstrably already reading.

If you want the same treatment applied to paid links rather than AI crawlers, our study of 3,790 hand-built backlinks re-crawled every URL months after placement to see what survived.