Can I trust an auto-generated llms.txt without editing it?

For a small site, usually yes as a first draft. A crawler can read your titles and meta descriptions and group pages by URL path, which is most of the syntax. What it cannot know is intent: whether your pricing page matters more than your blog, whether a page is thin or duplicated, or whether an area of the site is scheduled for retirement. The syntax survives untouched; the curation is where you add value. Plan on five minutes of deleting and reordering after any import.

Does the generator upload llms.txt to my website?

No. The generator builds the text and lets you copy or download it as llms.txt. Publishing is your job, and it depends on your stack: a static file in public/, static/ or an equivalent folder, a route handler, or a build-time template. The file has to live at your domain root so it resolves at https://yourdomain.com/llms.txt. If you are not sure how your platform serves root files, the platform guides in this blog cover the common frameworks one by one.

How often should I regenerate my llms.txt?

Regenerate when the site changes shape, not when it changes volume. Adding a handful of blog posts does not require a new file if your file links to the blog index rather than individual articles. New product areas, renamed sections, a docs restructure or a new language all do. A practical trigger is any release that adds or removes a top-level section, plus a quarterly pass that re-runs the validator and checks that no listed URL 404s.

Workflow · September 2026

llms.txt Generator and Validator Workflow

Generating the file is the easy half. The half that decides whether anything reads it is importing the right pages, cutting the ones a crawler cannot judge, and proving the published URL actually serves Markdown. Six steps, in order.

Why a Workflow Beats a One-Off Generate

Most bad llms.txt files are not malformed — they are unedited imports. A crawl dumps four hundred URLs into a list, the list becomes the file, and the result reads like a sitemap with marketing copy attached. The spec wants the opposite: a short curated index, a summary line, and page-level descriptions that tell a client what question each link answers. So the process is really two jobs, mechanical then editorial, and they are worth keeping separate.

Step 1: Decide What the File Is For

Before opening any tool, write down the five to fifteen pages you would hand a new developer, journalist or buyer. That list is the file. It is usually the homepage, the product or service overview, the docs or guide entry point, pricing, the case-study index, the blog index and a contact or booking page. Everything else is Optional-section material at best.

The reason to do this first is that the import step will overwrite your judgment with whatever order the crawler finds. If you already know the target list, you are deleting and reordering instead of deciding from scratch.

Step 2: Import Pages — Three Inputs, Three Trade-offs

The llms.txt generator accepts pages three ways. Pick based on how much structure your site already exposes.

Input	What it gives you	Use it when
Auto-crawl	Real page titles, meta descriptions, links grouped by section	You have no sitemap, or want titles the page itself declares
sitemap.xml import	Every indexed URL, instantly	Your sitemap is curated and tidy — it becomes the raw list
Manual	Exactly the URLs you type, nothing else	Small sites, or a file you intend to hand-maintain

For larger sites, the crawl path is the better default because it answers a question a sitemap cannot: what does this page actually call itself, and what does its meta description say? Those two strings are what your link lines will carry, so pulling them from the live page beats inventing them.

Step 3: Cut What a Crawler Cannot Judge

Every import needs the same deletion pass. Remove paginated archives, tag and author pages, legal boilerplate, thank-you and confirmation screens, and any near-duplicate location page beyond your top markets. Then apply the security filter: if a URL is not something you would print on a poster, it does not belong in a public file — staging hosts, admin routes and signed links especially. That check has its own triage list.

One structural decision belongs here too. If a dense subtree exists — a documentation set, a resource library — consider keeping the root file short and publishing a subpath file such as /docs/llms.txt, then linking it from the root. Clients use the most specific file that covers the URL they are working in, so this is routing rather than duplication.

Step 4: Write the Two Prose Slots

Only two parts of the file are prose: the H1 site name and the blockquote summary underneath it. Both get read in full, and the summary is the sentence an assistant is most likely to reuse. Keep it to one or two lines, name what the site is and who it is for, and skip the adjective stack. The per-link annotation follows the same rule — one short clause naming the question the page answers, not a restatement of its title. The annotation rules cover the phrasing that gets lifted versus the phrasing that gets skipped.

Step 5: Validate the Text Before You Deploy

The validator reads the file text and returns a 0–100 AI-readiness score plus an issue report split into errors and warnings. The errors are the mechanical ones that break spec compliance: a missing or empty H1, an H1 that is not the first line, more than one H1, unrecognized lines that are not link list items, links with no display text, and URLs that are not absolute. Warnings flag the softer stuff — structure and readability choices worth fixing but not fatal.

Two habits make this step useful rather than ceremonial. Paste the file text, not a screenshot of your CMS editor, because trailing whitespace and smart quotes survive the copy-paste and show up as issues. And treat a score above 80 as permission to deploy rather than proof the file is good — the validator cannot know that pricing is missing from a site that sells software. It scores the text; the three-layer validation approach adds the delivery checks the text cannot cover.

Step 6: Deploy, Then Prove It With Three Commands

Publish the file at your domain root so it resolves at /llms.txt, then verify what a crawler actually receives:

# 1. status code and final URL (watch for redirects)
curl -sIL -o /dev/null -w "%{http_code} %{url_effective}\n" https://example.com/llms.txt

# 2. content type and size — expect text/plain or text/markdown
curl -sI https://example.com/llms.txt | grep -i -E "content-type|content-length"

# 3. the links your file lists, fetched at scale
curl -s https://example.com/llms.txt | grep -oE "https://[^)]+" | while read u; do
  printf "%s %s\n" "$(curl -s -o /dev/null -w '%{http_code}' "$u")" "$u"
done | sort | uniq -c | head

A 200 with text/html means a rewrite is answering for the path and the file is never being read as Markdown — a delivery bug, not a content one. Dead links in the third command are the other common failure, and they tend to arrive in batches after a site migration.

When to Generate at Build Time Instead

The manual workflow above is right for sites that change shape a few times a year. If your pages live in a repository and every deploy is a build, generating the file at build time removes the drift entirely: the list is derived from the same source as the site, so renamed URLs cannot leave a stale link behind. That is a plugin or a small script in most frameworks, and it pairs well with a link check that fails the build instead of waiting for the quarterly review.

Start with the generator to get a correct first draft in about ten minutes, run it through the checker, ship it, and only automate once the curated list has stopped changing.