Is publishing an llms.txt file itself a security risk?

The file is not the risk — what you put in it is. /llms.txt is served publicly, without authentication, at a predictable path, so anything written there is readable by any crawler, scraper, directory or competitor that requests it. A careful file that lists only your public pages exposes nothing you had not already published. A file that lists a staging host, an internal search endpoint or a preview URL turns an obscure path into a curated, machine-readable pointer.

Should noindex or password-protected pages be listed in llms.txt?

No. llms.txt does not override robots.txt directives, noindex tags, WAF rules or authentication — the spec describes it as context for agents, not as a permission system. Listing a protected page only advertises that the path exists and is worth fetching. If a page is blocked for crawlers, keep it out of the file entirely rather than listing it with a note.

What should I do if a private URL already leaked into llms.txt?

Remove the line, then assume the URL was already fetched and copied somewhere: directories and archives keep snapshots. Rotate any credential, token or signed URL that ever appeared in the file, since those must be treated as disclosed the moment they became public. If the path was only obscure rather than protected, expect scanning traffic on it now, and either put real access control in front of it or retire it with a 410.

Content security · September 2026

What Not to Include in llms.txt

Most llms.txt advice is about what to add. This one is about what to leave out — because /llms.txt is a public file at a predictable path, and every entry in it is a disclosure decision you make once and cannot quietly reverse.

A Public File, Served Without Asking

An llms.txt file sits at the site root or at any subpath and covers the URLs under that path. It is fetched with a plain GET request — no login, no session, no referrer. That is the point of the convention: an agent working for a user should be able to pull a concise map of the site in one request.

The same property makes it a disclosure surface. Anything written there is readable by crawlers that respect nothing in particular, by SEO tools, by directories that index llms.txt files, and by anyone who fetches the path on a hunch. The spec is explicit that this is not a permission system: robots.txt declares what access is acceptable, while llms.txt is consulted on demand to help an agent understand content. It grants nothing and withholds nothing.

So the test for a line in llms.txt is not "is this link useful to a model?" It is "am I comfortable with this URL being public, quotable and indexed for good?"

Five Categories That Should Never Appear

Preview, staging and branch URLs. Deploy previews, *.staging hosts, branch aliases and temporary campaign domains. They are usually unlisted but unprotected, they drift, and they often carry work-in-progress claims that contradict your live site.
Private application and admin routes. Dashboards, client portals, internal search endpoints, ticket queues, /admin and /internal paths. Even without credentials in the URL, naming these paths in a curated file is a roadmap for automated probing.
Credentials, tokens and signed URLs. API keys, ?token= query strings, unlisted share links with embedded secrets, webhook URLs, private bucket paths. Time-limited links are worse: the file outlives the signature.
Customer, employee or partner data. Gated case studies, named-account pricing, private reports, anything under NDA. This is where OWASP's risk register maps directly — sensitive information disclosure is a first-class entry in the LLM Top 10, and a public index is one of the simplest ways to publish something you meant to keep internal.
Unreleased roadmap and embargoed pricing. A page about a product not yet announced, next quarter's pricing, a migration guide for a plan you have not deprecated yet. Agents will read it, and assistants will summarise it to customers before your launch email goes out.

If a page fits any of those five, it stays out of the file — however useful it would be as context. The broader curation problem is covered in llms.txt mistakes.

Why "It's Only a URL" Underestimates the Damage

Three things happen to a URL once it appears in a public llms.txt file:

It gets syndicated. Community directories collect and re-publish llms.txt files, so your entry is copied off your domain within days. Removing it later does not remove the copies.
It gets fetched by more than agents. Link checkers, monitors and scrapers treat the file as a link inventory and walk every entry, including the ones you hoped nobody would notice.
It becomes quotable context. A note like "internal pricing model, subject to change" travels with the link. Assistants repeat what the file says.

There is a second-order effect too: a file that names your internal structure — campaign naming, roadmap shape, which accounts get special pricing — is competitive intelligence even when each URL alone is harmless.

A Three-Question Triage for Borderline Pages

Most decisions are obvious at the extremes. For the ones in the middle — a partner case study, a comparison page, a docs section behind a login — run three questions in order:

Question	If the answer is no
Can it be fetched with no login?	Leave it out; llms.txt will not unlock it.
Would a competitor reading it be fine?	Leave it out; the file is public by design.
Would you want it quoted in an AI answer?	Leave it out; anything listed is context an assistant may repeat.

One related trap: a page that is disallowed in robots.txt should not be listed either. Cross-checking the two files against each other is covered in llms.txt vs robots.txt, and the short version is that a URL blocked for crawlers but listed for agents produces contradictory signals.

Pre-Publish Checklist

Run these two scans against the file before it ships. They catch the patterns that leak in through generators and copy-paste from sitemaps:

# Flag previews, staging hosts, admin paths and anything with a query string
grep -inE "staging|preview|localhost|127\.0\.0\.1|\.internal|admin|portal|token=|api_key|secret" llms.txt

# Print every distinct URL so you can eyeball domains and paths
grep -oE "https?://[^) ]+" llms.txt | sort -u

Then confirm three things by hand: every link points at a host you control, no entry carries a query string, and nothing in the notes would be awkward in a press release. If a generator produced the file, this pass matters more, not less — auto-generated files inherit whatever the source inventory contained.

If Something Already Leaked

Do these in order:

Rotate first, edit second. Any credential, token or signed link that appeared in the file is disclosed. Reset it before you touch the markup.
Remove the line and redeploy so the live file stops advertising the path. Then purge the CDN cache — a stale copy at the edge keeps serving the old file.
Assume the URL is known. Watch logs for traffic against it in the following weeks. If it was only obscure rather than protected, either add real access control or retire it with a 410.
Add the pattern to your build check so the same class of path cannot come back on the next deploy.

When you are done, re-check the file structurally — the llms.txt checker confirms every link resolves and every entry follows the spec. If you are rebuilding rather than editing, the generator drafts one from your sitemap; this article is then your review pass, because the generator knows nothing about which pages are meant to stay quiet.