Do government websites need an llms.txt file?

No law or federal standard requires one, and we found none in the US federal estate: across 26 government hosts fetched on October 9, 2026, zero served a real plain-text file. The argument for publishing one anyway is the same as for any site — a public agency publishes services, eligibility rules, forms, statistics and policy, and an assistant asked about any of them will answer from whatever it can retrieve. A curated index tells it which page is authoritative. The two public-sector files we did find, federalregister.gov and nsw.gov.au, both spend most of their length on machine-readable entry points rather than prose.

Why did most government hosts block our fetch instead of returning 404?

Because public-sector infrastructure runs heavy edge filtering and cannot tell an AI agent from a scraper. Eight of the 26 hosts we tested on October 9, 2026 never let us see the path: six answered 403 with an Akamai 'Access Denied' page (cdc.gov, usda.gov, transportation.gov, ssa.gov, dhs.gov, dol.gov), nih.gov answered 403 with a Cloudflare 'Just a moment...' challenge, and fda.gov answered 404 only after a redirect to an abuse-detection apology page. The same edge rule blocked the robots.txt fetch on seven of the 25 files we tried to read. If you publish an llms.txt behind that filter, allow the path explicitly or the file is unreadable.

Should an agency's llms.txt list eligibility rules, fees or processing times?

Link them, never transcribe them. Fees, thresholds and processing times are the fastest-rotting content on a government site, and a quoted figure is exactly the kind of stale number an assistant repeats with confidence. List the eligibility page, the fee schedule page and the form itself, and let the agent fetch the live values. The same rule applies to emergency information: an llms.txt is a static index, so never point it at anything that changes by the minute — link the alert or warning channel instead.

Public Sector · October 2026

llms.txt for Government and Public-Sector Sites

Public agencies are the most-cited content on the web and the least indexed for agents. Here is what 45 government hosts actually serve today, the seven buckets a public-sector file needs, and the one cross-check that catches a file nobody can fetch.

What 45 Government Hosts Serve Today

We fetched /llms.txt from this server on October 9, 2026 with a generic user agent, following up to three redirects: a first pass over 26 national and federal domains, then a second pass over 19 more public-sector hosts — state, provincial, municipal, health-service and statistics domains — plus one university host as a control.

Response	Count	Examples
`200`, `text/plain` — a real file	2	federalregister.gov (2.7 KB), nsw.gov.au (17.2 KB)
Blocked at the edge	12	cdc.gov, nih.gov, usda.gov, transportation.gov, ssa.gov, dhs.gov, dol.gov, fda.gov, sec.gov, bls.gov, dta.gov.au, mygov.in
`404`	28	usa.gov, irs.gov, nasa.gov, noaa.gov, weather.gov, epa.gov, energy.gov, va.gov, census.gov, gsa.gov, nps.gov, ftc.gov, nist.gov, digital.gov, gov.uk, europa.eu, whitehouse.gov, ai.gov, uspto.gov, usds.gov, nhs.uk, gov.scot, qld.gov.au, ontario.ca, gov.ie, govt.nz, data.gov, cms.gov
`200`, but no file behind it	2	australia.gov.au (redirected to a myGov portal page), govinfo.gov (64 KB HTML)
Connection failed at the HTTP/2 layer	1	canada.ca

Zero files in the first pass, and both real files in the entire sample sit outside the US federal estate. Read the failure modes closely, because two of them are worse than a missing file: a 200 with a 64 KB HTML page behind it (govinfo.gov) will make a client parse a rendered page as Markdown, and an edge block is invisible to your own verification from inside the firewall.

Why the Public Sector Sits at Zero

Three forces, none of them laziness. First, there is no mandate: nothing in federal policy requires the file, and the Web Content Accessibility Guidelines obligations that do apply — Section 508 for federal agencies, the Plain Writing Act of 2010 for how they write, and the ADA Title II rule requiring WCAG 2.1 AA conformance for state and local government sites — say nothing about machine-readable indexes. Second, procurement: an agency site ships through a CMS release process, an edge filter and a change board, so a new root file is a ticket, not a commit. Third, incentive: web teams are measured on search performance and accessibility audits, not on whether a chatbot can find the eligibility page.

The two files that do exist came from teams that treat machine access as a service, and that is the useful lesson. Not one of the 26 first-pass hosts told an agent where its authoritative content lives — while a large share of them run public APIs, open-data portals and statistics endpoints that an agent would use if it knew about them.

The Two Files Worth Copying

federalregister.gov runs 2,741 bytes and 22 lines: an H1, a long prose summary of what the HTML edition is, a ## Public Inspection section, then ## Core Resources whose three links point at JSON API endpoints rather than HTML pages, and finally ## Guidance for AI Agents — rate limiting (use the API for bulk harvests, respect 429 and retry-after), the document types to expect, and a citation convention: reference by document number or volume/page. That guidance block is the most interesting thing any government file in our sample does, and it has one defect worth noticing: the agencies link closes its parenthesis on the next line, so the URL is malformed for a strict parser.

nsw.gov.au runs 17,247 bytes over 189 lines: H1, summary blockquote, then eighteen H2 sections separated by horizontal rules — arts and culture, business, community services, driving and transport, education, emergency, employment, energy and so on — each a list of links with one-line descriptions. Two caveats. The entire file is wrapped in a ```markdown fence, which is not the spec's convention and will be read as content by parsers that do not strip it. And it links dated pages such as /nsw-youth-week-2025, which is how a curated file starts producing 404s a year later.

The Seven Buckets a Public-Sector File Needs

Jurisdiction and identity. Agency name, the level of government, and the geography served — the line that stops an assistant answering a national question with a state rule.
Services and transactions. Renew a licence, register a business, apply for a permit, pay a fee, book an appointment. The transactional pages are the ones assistants get asked about most.
Eligibility and benefits. Programme pages, eligibility checkers, and the forms themselves — linked, never transcribed. Thresholds change with every budget cycle.
Law, rules and policy. The statute or regulation, the current guidance, the published consultation, and the plain-language explainer. Give the agent the hierarchy so it quotes the right one.
Data and statistics. The open-data portal, the statistics release calendar, the API documentation, the public dataset index. This is where government genuinely leads every other vertical.
Records, contact and transparency. Access-to-information or FOIA request route, public records register, procurement notices, organisation contact pages, accessibility statement.
The machine-readable tail. Sitemap index, robots.txt, any OpenAPI document, and the language or translated-content hubs. This is the section an ## Optional heading is meant for.

What Must Never Appear

Personal data of any kind: benefit case files, licence numbers, tax records, complaint files, staff directories with direct numbers or inboxes.
Authenticated paths — login walls, filing portals, payment flows, case-status dashboards. Point at the landing page, never the transaction URL.
Anything live or emergency-related. A static index cannot stay current, so link the alert channel rather than describing it.
Draft, embargoed or internal policy, procurement evaluation material and unreleased consultation documents.
Third-party content the agency republishes under licence — syndicated statistics, licensed imagery, embedded maps.

The categories and the triage questions are the same as any other site; the agency-specific version is the disclosure regime. See what not to include in llms.txt for the full leak list.

The robots.txt Cross-Check Nobody Runs

We also fetched robots.txt from the 25 hosts that would answer. Seven blocked that fetch too, three returned 404, and of the fifteen we could read, exactly one named an AI crawler: GOV.UK, which scopes meta-externalagent to a single Disallow: /search/all* line with a comment explaining that the crawler causes timeouts on its search backend. No government robots.txt in our sample mentioned GPTBot, ClaudeBot, PerplexityBot, Google-Extended or CCBot. AI access policy for public-sector sites is effectively unwritten — which means the file is the whole policy, and the file has to be reachable.

That is the failure mode to design out: a WAF rule that answers 403 to a non-browser user agent means your llms.txt exists for humans reading it in a browser and nobody else. The crawler token reference and the two ready-to-paste policies are in robots.txt for AI crawlers.

Publishing and Verifying It

On most agency stacks the realistic path is a static file at the document root, published with the same pipeline as robots.txt. Where the estate spans subdomains — an agency root plus programme subdomains — remember the rule that a file covers the URLs under its own path and the most specific one wins, so a root index plus a /programmes/llms.txt for the large programme is a valid split rather than a duplicate. Then verify from outside the network:

# 1. Status, content type, byte count — 403 or text/html here ends the exercise
curl -sIL -A "Mozilla/5.0" https://www.example.gov/llms.txt | grep -Ei "HTTP/|content-type|content-length"

# 2. Confirm the body is Markdown, not a WAF page or an HTML 404 shell
curl -sL -A "Mozilla/5.0" https://www.example.gov/llms.txt | head -5

# 3. Every link in the file must answer 200 for a plain-text client too
curl -sL https://www.example.gov/llms.txt | grep -oE "https://[^) ]+" | sort -u | \
  while read u; do printf "%s %s\n" "$(curl -s -o /dev/null -w '%{http_code}' -m 10 "$u")" "$u"; done

Paste the file into the validator for structure, link and duplicate checks before it goes near a change board, or import your sitemap into the generator and delete everything that is not a durable hub — a public site's sitemap holds years of press releases and expired consultations. Serving it correctly once it exists is a separate problem, covered in content type, encoding and caching.

The Short Version

Two of 45 government hosts publish a real llms.txt, and both of them point agents at machine-readable entry points rather than prose. A public-sector file is a small curated index — jurisdiction, services, eligibility, law, data, records, and a machine-readable tail — with no personal data, no authenticated paths and no numbers that change with the next budget. Then allow the path at the edge, because an unreachable file is indistinguishable from no file at all.