What Content-Type should llms.txt be served with?

text/plain; charset=utf-8 is the safe default, because the file is a .txt and every server already knows that mapping. text/markdown; charset=utf-8 is equally acceptable and signals the payload better; Stripe's public file uses it. What breaks clients is text/html, application/octet-stream, or any type that makes the parser treat your Markdown as a page. That failure is almost always a routing mistake, not an encoding one: a framework catch-all, a rewrite rule, or a static host serving a directory index.

Does llms.txt need UTF-8 and a byte-order mark?

UTF-8 is what the spec assumes, and the practical reason is tokenization: mixed or legacy encodings produce mojibake in the middle of a description that an agent may quote verbatim. A BOM is explicitly optional in the spec, and both choices exist in the wild — llmstxt.org's own file starts with the two bytes for a bare '#'. If you do ship a BOM, ship a single valid one; a doubled or truncated BOM shows up as a stray character before your H1, which validators flag as a missing title. Verify with three bytes of hex, not your editor.

How should llms.txt be cached?

Longer than your pages, shorter than your policy. The file changes rarely, so a shared-cache TTL in the range of an hour to a day is normal: Vercel serves its own file with s-maxage=3600 plus stale-while-revalidate=86400, and Vercel and Cloudflare both pair a short must-revalidate policy with an ETag. The wrong setting is a year-long immutable cache, because a reorganisation can leave agents reading a retired URL map for months. If you cannot revalidate cheaply, prefer a moderate TTL and a predictable purge step.

GEO · October 2026

How to Serve llms.txt

A correct Markdown file behind the wrong response headers is not a correct llms.txt. Here is what the spec asks the server to do, what eight live files actually return right now, and the delivery check that catches the two failures no validator can see from the outside.

The Spec's Server-Side Requirements Are Short

The proposal at llmstxt.org is mostly about the file's contents, and it deliberately says little about transport. Three things it does state. The file is Markdown, placed at /llms.txt or at any subpath such as /docs/llms.txt, and where more than one file applies an agent should use the most specific one. An optional byte-order mark is allowed. And the recommended way to advertise the file is a link relation — rel="describedby" pointing at the llms.txt that covers a page, rel="alternate" type="text/markdown" pointing at a Markdown twin — delivered either as an HTML link element or as an HTTP Link: response header, the header form being the one that works for non-HTML resources.

The discovery mechanics build on that: paths and Link headers get a client to the file, and the server's response is what decides whether the client believes what it found. Two of the three properties that matter — status code and content type — are invisible in the file itself.

Content-Type: text/plain or text/markdown

Send text/plain; charset=utf-8 and you are done, because that is what every static host maps for a .txt extension. Send text/markdown; charset=utf-8 and you are still correct, and slightly more precise for humans reading response headers. Third-party validation guidance accepts both and treats text/html or application/octet-stream as a misconfiguration, on the reasonable grounds that an agent handed a page of HTML markup has to guess where your index went.

Which means a 200 is not the pass condition on its own. The common failure is a routing one: a framework catch-all route, a rewrite that sends unknown paths to the homepage, or a host that answers a directory index for /llms.txt/ while the real file was never exported. The file looks perfect in your repository and the response is HTML with a 200 status. That is a subset of what the usual "it does not work" symptoms turn out to be.

What Eight Live llms.txt Files Return

Headers fetched on October 4, 2026 with curl and a normal desktop user agent. The point of the table is the variety: all of these are deliberate implementations, and four of them make different calls on the same two decisions.

URL	Status	Content-Type	Caching
developers.cloudflare.com/llms.txt	200	text/plain; charset=utf-8	public, max-age=0, must-revalidate + ETag
docs.stripe.com/llms.txt	200	text/markdown; charset=utf-8	varies on Accept, plus a Link: rel="service-meta" header
vercel.com/llms.txt	200	text/plain; charset=utf-8	s-maxage=3600, stale-while-revalidate=86400 + ETag
supabase.com/llms.txt	200	text/plain; charset=utf-8	cache-control: public only — no explicit TTL
llmstxt.org/llms.txt	200	text/plain; charset=utf-8	max-age=600 + weak ETag, last modified Sep 24
www.checklyhq.com/llms.txt	200	text/plain; charset=utf-8	public, max-age=0, must-revalidate + ETag
docs.anthropic.com/llms.txt	301	text/html (redirect body)	client must follow the redirect
openai.com/llms.txt	403	text/html	same 403 for curl, Googlebot and GPTBot agents

Two rows deserve a comment. The 403 is an edge verdict, not a user-agent one — the same response came back for the default curl agent, a Googlebot-style agent and a GPTBot-style agent, so there is no polite way to ask for that file. And the 301 matters because a plain curl -s without -L shows you the redirect body, which looks like an HTML page. Anyone debugging with the wrong flag concludes the file is being served as HTML.

Encoding: UTF-8, a BOM, and Three Bytes of Hex

UTF-8 is what the spec assumes, and it is not cosmetic: a file saved in a legacy encoding puts mojibake in the middle of a description that an agent may quote back verbatim. The BOM is explicitly optional, and the reference implementation ships without one — the first bytes of llmstxt.org's own file are 0x23 0x20, a bare hash and space. Check your own file the same way an agent receives it, rather than trusting an editor's status bar:

# first four bytes: watch for efbbbf (a single BOM) or doubled bytes
curl -s https://example.com/llms.txt | head -c 4 | xxd

# the header the client will actually see
curl -sI https://example.com/llms.txt | grep -i content-type

Caching: Longer Than Your Pages, Shorter Than Your Policy

An index file changes when your site structure changes, which is nothing like your publishing cadence, so it can be cached harder than a page. It should not be cached forever. A shared TTL of an hour to a day covers almost every case, and pairing a modest TTL with an ETag keeps revalidation cheap enough that you can afford to be conservative — max-age=0, must-revalidate plus an ETag, the pattern both Cloudflare's docs and Checkly use, is the conservative end of that scale and still avoids sending the body when nothing changed.

Do not mark it immutable for a year. A reorganisation or a domain migration would leave agents reading a retired URL map long after the old paths 404.
A bare cache-control: public with no TTL is not a policy. It lets intermediaries pick their own heuristic freshness, which is how a file ends up stale in some networks and fresh in others.
Purge on rename, not on edit. The change that breaks clients is a removed or moved page, so hook the cache purge to whatever writes the file rather than remembering to do it later.
Keep it under a size you can revalidate. If the file has grown past the sensible token budget, the delivery problem is really a content problem.

The Delivery Check: Five Commands Before You Move On

Run this against production, not against your local preview, because every failure described above is introduced by the host rather than by the file:

# 1. status and type only (HEAD is cheap but its content-length can read 0)
curl -sI https://example.com/llms.txt | head -5

# 2. follow redirects, then look at the status you land on
curl -sIL -o /dev/null -w "%{http_code} %{url_effective}\n" https://example.com/llms.txt

# 3. confirm the body starts with your H1, not with <!doctype html>
curl -s https://example.com/llms.txt | head -n 3

# 4. header bytes, to rule out a BOM surprise
curl -s https://example.com/llms.txt | head -c 4 | xxd

# 5. what a shared cache is allowed to keep
curl -sI https://example.com/llms.txt | grep -iE "cache-control|etag|last-modified"

A file that passes all five is delivered as well as it can be. That is worth doing, and worth being honest about: correct delivery is a precondition, not a ranking lever. Published audits — SE Ranking's scan of roughly 300,000 domains, Ahrefs' log study of 137,210 domains with a valid file — found no citation correlation and a large share of files never requested at all. The reason to fix your headers is that a client which does fetch the file gets the Markdown you wrote instead of an HTML challenge page. Generate a clean version with the llms.txt generator, re-check structure with the llms.txt checker, then verify the response the way a crawler sees it.