How do AI crawlers know my llms.txt exists?

Mostly because they try the path. The specification defines the filename and where it lives — /llms.txt at the site root, or at any subpath such as /docs/llms.txt — and recommends link relations so a client can find it from a page. It defines no registry and no discovery protocol beyond that. In practice a crawler that already requests /robots.txt on every domain can request /llms.txt the same way, and an agent that lands on a page carrying a describedby link can follow it. Everything else — search engines indexing the file, directories listing it, a sitemap mentioning it — is a hint to a human or a third-party tool rather than a path into an answer engine.

Should I add llms.txt to my sitemap or robots.txt?

Treat both as hints, and check the second one carefully. Sitemap directives are specified for XML sitemaps, so adding a plain-text file to a Sitemap line is non-standard and there is no public evidence an engine consumes it. The robots.txt risk runs the other way: robots.txt is an exclusion protocol, and a broad Disallow: / or a pattern such as Disallow: /*.txt blocks the fetch of the very file you published. Confirm with a crawler user agent that /llms.txt still returns 200.

Do llms.txt directories help with AI visibility?

Not in the way people hope. The three directories the llmstxt.org proposal lists — llmstxt.site, directory.llmstxt.cloud and llmstxthub.com — are community-maintained indexes of public files. They are useful because people browse them, link to them and study what other sites publish, and they give you a second place your file is referenced from. They are not a distribution channel into ChatGPT, Claude or Perplexity, and a listing has no measurable effect on citations.

GEO · September 2026

How AI Crawlers Discover Your llms.txt

Most llms.txt advice covers what to write. Almost none covers how the file gets found — which is the part that decides whether writing it was worth an afternoon. Here are the channels that actually exist, what each one is good for, and how to prove yours is reachable.

Publishing Is Not Discovery

A published file is a file at a URL. Discovery is the separate question of how a client that has never visited your site learns that the URL exists. The v2 proposal is explicit about the first and deliberately thin about the second: it fixes the filename, fixes the locations, and recommends one mechanism for pointing at the file from elsewhere. The v2 changes are worth reading for the spec layer; this article is about the plumbing underneath it.

The failure mode is silent: a valid file that no client ever requests looks identical in your editor to one that is working, and only access logs or crawler measurement tell them apart.

Channel 1 — The Path Convention, Root and Subpath

The strongest thing your file has going for it is that its location is predictable. The spec places llms.txt at /llms.txt or at any path within the site, and states that a file covers the URLs under its path — with the most specific file winning where several apply. That gives a client two cheap moves: ask the root of a domain, and ask the subpath it is already working in.

Root file — /llms.txt. Covers the whole site by convention. This is the one to ship first, and the one a client is most likely to guess.
Subpath file — for example /docs/llms.txt for a documentation hub, or /locations/denver/llms.txt for a branch. Covers only the pages beneath it, and takes precedence there.

A subpath file is not separate from the root file; it is a routing decision inside it. The root file should link to the subpath file the same way it links to pages, so a client that reads the root learns that a narrower map exists. Sites whose documentation dwarfs the rest of the site are the usual case — the root file stays small and points at the dense part of the site.

Channel 2 — Link Relations: describedby and alternate

This is the only discovery mechanism the specification actively recommends, and it is the one most sites skip because it needs a server or template change rather than a text file.

Two link relations do the work. rel="describedby" points at the llms.txt file that covers the page; rel="alternate" with type="text/markdown" points at the clean markdown version of that page. They can be expressed as HTML link elements in the head, or as an HTTP Link response header.

Link: </llms.txt>; rel="describedby", </docs/quickstart.md>; rel="alternate"; type="text/markdown"

The header form is the interesting one, and the spec calls it out: a header can be added in web server or CDN configuration without touching a single page, and it works for non-HTML responses such as the markdown files themselves. A Cloudflare Worker or an nginx add_header directive covering text/html and text/markdown responses is usually the cheapest implementation. The markdown-twin half of this pattern is covered in Markdown for Agents vs llms.txt.

Channel 3 — robots.txt and Sitemaps: Hints, and One Way to Break Yourself

robots.txt is an exclusion protocol, not a discovery one, and conflating the two is where people get hurt. There is one real discovery signal worth knowing and one real hazard.

Move	What it is worth
Leaving `/llms.txt` crawlable	Essential. A broad `Disallow: /` or a pattern like `Disallow: /*.txt` matches the file and kills the fetch
A second `Sitemap:` line in robots.txt pointing at `/llms.txt`	A non-standard hint. Sitemap directives are defined for XML sitemaps; cheap to add, no verified consumer
Listing `/llms.txt` in `sitemap.xml`	Mostly redundant — it puts the file in search indexes, not in front of an agent
Allow rules for `GPTBot`, `ClaudeBot`, `PerplexityBot`	Indirect but real: a blocked crawler never reaches any file, llms.txt included

If you have not audited the file in a while, the 2026 crawler reference lists the tokens worth knowing. The delivery failures that look like discovery failures — wrong content type, redirect chains, WAF blocks — are catalogued separately in llms.txt not working.

Channel 4 — Directories, Platforms and the Human Layer

The proposal itself keeps a short list of directories that index public llms.txt files: llmstxt.site, directory.llmstxt.cloud and llmstxthub.com. None of them routes your file into a model, so treating a listing as an AI-visibility tactic is wishful. What it does give you is a second referencing page and a place to see what comparable sites publish — which is how most llms.txt conventions actually spread.

Two platform forces matter more. Hosted documentation products now generate a file for every site they serve, so a growing share of the web has one by default rather than by choice. And Chrome's Lighthouse ships llms.txt as part of its agentic browsing audits, turning the file from a blog-post suggestion into a checkbox in a tool developers already run. Neither guarantees a fetch, but both raise the odds that the file a client looks for will be there.

The Four-Command Reachability Check

Whatever channel someone arrives by, the file has to survive the fetch. Four commands settle it:

# 1. status, content type, redirects
curl -sI https://example.com/llms.txt

# 2. the way a crawler sees it
curl -sA "GPTBot/1.2" https://example.com/llms.txt | head -5

# 3. is robots.txt blocking it
curl -s https://example.com/robots.txt | grep -i llms

# 4. does the header form exist on a real page
curl -sI https://example.com/docs/ | grep -i "^link:"

Expected results: 200 with text/plain or text/markdown and no Location header; a body that starts with an H1 and the summary blockquote; nothing in robots.txt that matches the path; and an optional Link header if you implemented Channel 2. Then stop optimising discovery and look at consumption: a widely cited Ahrefs study published in 2026 found that of roughly 38,000 domains with a valid llms.txt, 97% received no requests for it at all in a month. Discovery is worth getting right precisely because so little of it is happening by itself — but the file has to be aimed at the clients that do fetch it, which is a question about which engines read llms.txt, not about directories.

If you are starting from nothing, generate a first draft with the llms.txt generator, publish it at the root, then run the production URL through the llms.txt checker before you spend an afternoon on link headers.