Should an open-source project publish an llms.txt file?

On the evidence, open-source and developer documentation is the one vertical where the format actually shipped: in our October 14, 2026 check of 41 project and docs hosts, 18 served a real plain-text file — 44%, against 3% in the same week's nonprofit sample. The reason is structural rather than cultural. Documentation sites already generate Markdown sources at build time, already publish versioned page trees, and already have a reason to hand an agent the install path and the API surface instead of letting it scrape a sidebar. If your project has docs at a stable URL, the file is mostly a build step, not a writing project.

Should llms.txt point at versioned URLs like /docs/2.4/ or at stable ones?

Stable ones, and version only where the version is the thing being asked about. Every release that rewrites a path invalidates every listed URL, and a file full of 404s is worse than no file because it teaches an agent that your documentation is broken. Our own fetch caught the shape of this problem from the other side: docs.djangoproject.com answers /llms.txt with a 301 to /llms.txt/ and then a 302 to /en/6.1/ — the versioned documentation root, served as 54 KB of HTML rather than a file. List the unversioned entry points (getting started, reference, guides) and, if you keep old majors alive, one line per major pointing at its landing page. Then re-run the link check on every docs deploy.

Do you need Markdown versions of every documentation page?

Not every page, but it is the cheapest win available to a documentation site. Of the 18 real files we read, twelve list at least one .md target and six list none. react.dev and nodejs.org pair every one of their links with a .md twin (179 of 179 and 70 of 70), vite.dev sends 47 of 49 and lets two point at MDN because MDN has no Markdown twin, and grafana.com sends 91 of 92. The six files with zero .md targets send an agent back into rendered HTML, where navigation, banners and client-side rendering sit between the crawler and the text. If your generator emits .md files already, listing them costs one line per page.

Open Source · October 2026

llms.txt for Open Source Projects

Developer documentation is where llms.txt is not an experiment. We fetched the path from 41 project and docs hosts and 18 answered with a real file. Here is what they contain, where the other 23 go wrong, and the version worth generating from your own docs build.

What 41 Developer Docs Serve Today

We requested /llms.txt from this server on October 14, 2026 with a GPTBot-style user agent and a 20-second timeout, across framework, runtime, database, library and tool documentation domains. Eighteen served a real plain-text file — 44% of the sample, an order of magnitude above the 3% we measured across 67 nonprofit domains the day before.

Response	Count	Examples
`200` with a real text file	18	angular.dev, bun.sh, clickhouse.com, duckdb.org, hono.dev, kotlinlang.org, nodejs.org, nuxt.com, ollama.com, react.dev, redis.io, svelte.dev, supabase.com, tanstack.com, vite.dev, vuejs.org, grafana.com, www.mongodb.com
Redirect, then 404	4	astro.build, swift.org, www.postgresql.org, www.rust-lang.org
Redirect, then documentation HTML	1	docs.djangoproject.com lands on `/en/6.1/` — 54,108 bytes of `text/html`
Redirect, then a real file on another host	1	prisma.io (308 to `www.prisma.io/llms.txt`, 4,448 bytes `text/plain`)
`200`, but HTML behind it	3	deno.com serves 58,366 bytes of its own page; www.docker.com and pytorch.org each serve a byte-identical 3,038-byte bot challenge containing the words “Client Challenge”
`404`	13	laravel.com, rubyonrails.org, spring.io, tailwindcss.com, kubernetes.io, www.elastic.co, go.dev, flutter.dev, www.tensorflow.org, huggingface.co, fastapi.tiangolo.com, neovim.io, docs.python.org
`403`	1	sqlite.org — a 14-byte `text/plain` “Access denied” body

Read the sizes before the status codes: spring.io answers the path with a 349,416-byte HTML error page, fastapi.tiangolo.com 91,075 bytes and laravel.com 86,135. A crawler that asked for an index gets navigation chrome instead, while docs.python.org at least keeps its 404 to 146 bytes.

The Redirect Trap: Six Chains, Four Dead Ends

Four of the six chains in this sample end somewhere that is not a file. A redirect is still reachable in principle, per the discovery guide — but it is also a promise about where the file lives, and these four break it.

Host	First hop	Where it ends
astro.build	`302` → docs.astro.build/llms.txt	`404`
www.rust-lang.org	`301` → rust-lang.org/llms.txt	`404`, 9,297 bytes of HTML
swift.org	`302` → www.swift.org/llms.txt	`404`, 146 bytes
www.postgresql.org	`301` → /llms.txt/ (trailing slash)	`404`, 4,223 bytes
docs.djangoproject.com	`301` → /llms.txt/ then `302` → /en/6.1/	`200`, 54,108 bytes of `text/html`
prisma.io	`308` → www.prisma.io/llms.txt	`200`, 4,448 bytes of `text/plain`

Two failures repeat. The host swap: rust-lang.org and swift.org redirect between the apex and the www host, and neither target carries the file. The trailing slash: postgresql.org answers it with a 404 and Django falls through to a locale-prefixed version root. Prisma is the one chain that works, and it is the pattern to copy — one canonical host, one 308, one file.

1.7 KB to 103 KB: Two Schools of Structure

The 18 files total 391,647 bytes, averaging 21,758 each, and the spread is 62×. At one end, svelte.dev ships 1,676 bytes in 23 lines and ollama.com 1,786; at the other, clickhouse.com serves 103,391 bytes and nuxt.com 57,285. Two thousand seven hundred links later, the more useful split is organisational: some projects write a sectioned index with prose, others emit a flat table of contents.

Host	Bytes	Lines	H2	Links	.md targets
svelte.dev	1,676	23	3	7	0
ollama.com	1,786	29	4	10	0
supabase.com	3,242	55	4	42	30
duckdb.org	3,371	37	4	18	2
vite.dev	3,528	84	1	49	47
hono.dev	5,927	105	3	94	0
angular.dev	6,745	136	16	88	1
vuejs.org	7,415	153	1	94	94
www.mongodb.com	10,385	114	10	64	7
grafana.com	13,751	135	9	92	91
tanstack.com	13,775	64	7	81	0
react.dev	14,476	251	2	179	179
nodejs.org	15,010	76	1	70	70
bun.sh	33,834	330	2	322	319
redis.io	46,852	248	11	199	173
kotlinlang.org	49,198	501	0	499	0
nuxt.com	57,285	386	8	342	337
clickhouse.com	103,391	499	7	452	0

Kotlin is the extreme flat case: 501 lines, an H1, then 499 links with no heading between them, so there is no section an agent can cite by name. ClickHouse is the sectioned case at the same size — 452 links across seven H2s — while Angular runs 16 sections over 136 lines and Redis splits 248 lines into eleven. Sectioning is what makes a large file navigable, which is why the size discussion is really a sectioning discussion.

Markdown Twins, Optional Sections and llms-full

Across the 18 files we counted 2,702 Markdown links, 1,350 pointing at a .md target — half. Twelve files list at least one; six list none. The pairing is cleanest where the generator already emits both formats: react.dev sends 179 of 179 targets to .md, nodejs.org 70 of 70, grafana.com 91 of 92, bun.sh 319 of 322. Vite's two exceptions point at MDN, which is the right way to break the rule.

Vue does something worth copying for a different reason: its file writes relative Markdown paths and keeps the heading anchor in the link text, as in [Introduction {#introduction}](/guide/introduction.md). Relative targets survive a host migration, and the anchor says where inside the page the topic starts.

Formal adoption of the spec's optional parts is thinner than adoption of the path. Only seven of the 18 files carry an ## Optional section (bun.sh, clickhouse.com, duckdb.org, hono.dev, nuxt.com, ollama.com, tanstack.com), and six mention llms-full at all (grafana.com, hono.dev, nuxt.com, svelte.dev, supabase.com, tanstack.com). Svelte's 1,676-byte file is the most disciplined version of that idea here: a “Documentation Sets” section offering llms-small.txt, llms-medium.txt and llms-full.txt with one sentence each on what was removed, plus per-package files for Svelte, SvelteKit, the CLI and the AI tooling — a working answer to the tens-of-megabytes problem the full file otherwise creates.

Content-Type and Delivery: 18 of 18 Say text/plain

Every one of the 18 files is served as text/plain. Seventeen add a charset parameter (utf-8, uppercase UTF-8 on vuejs.org and vite.dev) and grafana.com sends the bare type. None uses text/markdown, which the content-type guide recommends where it is available — developer docs converge on the conservative choice. Delivery is where they can still fail: two hosts in this sample answered the path with the same 3,038-byte challenge page that three nonprofit domains returned the day before.

# 1. One request, no redirects — a 3xx here is where two thirds of this sample fails
curl -sI https://example.dev/llms.txt | grep -iE "^(HTTP|content-type|location)"

# 2. Follow the chain and read the first line: an H1, or <!doctype html>?
curl -sL https://example.dev/llms.txt | head -3

# 3. Every listed target should answer 200, .md twins included
curl -s https://example.dev/llms.txt | grep -oE "https?://[^)]+" | while read u; do
  printf "%s %s\n" "$(curl -s -o /dev/null -w '%{http_code}' -L "$u")" "$u"
done | grep -v "^200"

What an Open-Source Project Should Publish

Documentation sites win here because they can generate the file. The sections that earn their place are the ones an agent cannot infer from a repository listing or a sidebar:

Identity and licence. What the project is, which runtime or platform it targets, which licence governs reuse, and which version the docs describe — the block that stops an assistant describing your last major as current.
Getting started. Install, first run, the handful of commands a new user types. Version-stable, always.
Guides and reference. Two sections, not one: conceptual pages separated from the API surface.
Examples and templates. Runnable starters, official bindings, framework integrations.
Releases and migrations. Changelog, upgrade guide, deprecation policy, supported-version matrix — linked, not pasted into the file.
Community and contribution. Contributing guide, code of conduct, issue tracker, security policy.
Optional. Roadmap, benchmarks, comparisons — everything a reader can safely skip.

# ExampleProject

> ExampleProject is an open-source HTTP framework for TypeScript runtime X, MIT licensed.
> The documentation below describes version 3.x, the current major. Pages are also available
> as Markdown files by appending .md to any documentation URL.

## Getting started
- [Installation](https://example.dev/docs/install): Requirements, the install command and a first run.
- [Quick start](https://example.dev/docs/quickstart): A minimal server, a route and a deployment.
- [Upgrade guide](https://example.dev/docs/upgrade): Moving from 2.x to 3.x, with breaking changes.

## Guides
- [Routing](https://example.dev/docs/routing.md): Path parameters, wildcards and route groups.
- [Middleware](https://example.dev/docs/middleware.md): The request lifecycle and where each hook runs.
- [Testing](https://example.dev/docs/testing.md): Unit and integration testing against the framework.

## Reference
- [API reference](https://example.dev/docs/api.md): Exported functions, types and options for the current major.
- [Configuration](https://example.dev/docs/config.md): Every setting, its default and its environment variable.
- [Errors](https://example.dev/docs/errors.md): Error codes, causes and fixes.

## Releases
- [Changelog](https://example.dev/docs/changelog.md): Released versions, newest first.
- [Support policy](https://example.dev/docs/support): Which majors receive security fixes and until when.

## Community
- [Contributing](https://example.dev/CONTRIBUTING.md): Setup, test commands and review expectations.
- [Code of conduct](https://example.dev/CODE_OF_CONDUCT.md): Reporting and enforcement.
- [GitHub discussions](https://github.com/example/example/discussions): Questions and proposals.

## Optional
- [Benchmarks](https://example.dev/benchmarks.md): Methodology and results.
- [Roadmap](https://example.dev/roadmap.md): Planned work, unscheduled.

Build the first version from your docs tree with the llms.txt generator, then run the draft through the checker for the summary slot and section order. For an OSS project the file belongs in the docs build, not a hand-edited file at the repository root — the moment it is a build artifact, the link check, the .md twins and the version rewrite ride along with the pipeline you already run.

Status codes, redirect chains, content types, byte sizes, line counts, section counts and link counts referenced were captured from this server on October 14, 2026 using a GPTBot-style user agent with redirect following. Edge bot rules, generated files and documentation paths change without notice, so re-check your own paths rather than treating this table as current configuration for any project named.