llms.txt Guides & Examples
Everything you need to ship a spec-compliant llms.txt — from the exact format to real-world examples.
llms.txt for Forums and Community Platforms: What 34 UGC Sites Serve (2026)
We fetched /llms.txt from 34 forum, community and UGC hosts on October 17, 2026: two serve a real plain-text file, one answers 200 with HTML behind the path, twenty return 403 — thirteen of those a byte-identical 295-byte Discourse body naming the platform's own error code — six return 404 and five never answer at all. Inside the two live files: 4,555 bytes, 17 links (ten of them root-relative), zero thread URLs, dev.to's ai_disclosure_level write contract and OpenWrt's Discourse MCP pointer.
Read article →llms.txt for Insurance Sites: What 40 Insurers Serve and the YMYL Rules They Skip (2026)
We fetched /llms.txt from 40 insurance, insurtech, brokerage and reinsurance domains on October 16, 2026: nine serve a real plain-text or Markdown file, six block the fetch with a 403 from the edge, twenty return 404, three answer 200 with HTML behind the path and two never complete the connection. Inside the nine live files: 114,768 bytes, 66 H2 sections, 67 links written in a form Markdown cannot parse, not one .md twin — and zero files that restrict what an AI assistant may state about price, eligibility or claims.
Read article →llms.txt for Marketplaces and Directories: What 49 Two-Sided Platforms Serve (2026)
We fetched /llms.txt from 49 marketplace, classifieds, freelance and directory domains on October 15, 2026: six serve a real plain-text file, 25 block the request at the edge — 14 of those with one of two Cloudflare challenge pages, 4,544–4,554 bytes for one and 5,501–5,593 for the other — four answer 200 with HTML behind the path, ten return 404, two answer 202 with zero bytes and one returns 429. Inside the six live files: 90,718 bytes, 215 Markdown links, not one .md twin, a single Optional section, and the buyer/seller routing blocks that only two of six platforms publish.
Read article →llms.txt for Open Source Projects: What 41 Developer Docs Serve (2026)
We fetched /llms.txt from 41 open-source project and developer documentation hosts on October 14, 2026: 18 serve a real plain-text file, six answer with a redirect and four of those chains end in a 404, three answer 200 with HTML and no file behind it, 13 return 404 and one returns 403. Inside the 18 files: 391,647 bytes total, 2,702 links of which 1,350 point at .md twins, an H2 count ranging from zero to sixteen, and only seven files carrying an Optional section.
Read article →llms.txt for Nonprofits and Charities: What 67 Organizations Actually Publish (2026)
We fetched /llms.txt from 67 nonprofit, charity, foundation and NGO domains on October 13, 2026: two served a real file (wri.org 10.3 KB, conservation.org 2.6 KB), twelve blocked the fetch at the edge, 45 returned 404 and five answered 200 with no file behind it — three of them a byte-identical 3,038-byte bot challenge. The seven buckets a nonprofit file needs, why impact totals must stay behind a link, the donor-data exclusions, the robots.txt cross-check that found one AI crawler token in twelve files, and a three-command delivery check.
Read article →Does llms.txt Work on JavaScript-Rendered Sites? AI Crawlers and Empty Pages (2026)
We fetched /llms.txt from 12 product domains with the GPTBot user agent on October 12, 2026: six serve a real plain-text file, five return 404 with a 400-byte error body, and Trello answers the path with 160,977 bytes of its own HTML — a body containing the string "llms" zero times. Probing 54 URLs listed inside nine real files, 53 returned readable text because they point at .md twins; the one failure was a listed page returning 404 to four user agents. Plus 14 app and dashboard routes, six of which answered 200 OK with under 60 words.
Read article →llms-full.txt Size and Structure: The Full File Nobody Specified (2026)
We measured eight live llms-full.txt files on October 11, 2026: 67,649,033 bytes at developers.cloudflare.com, 43,107,672 at platform.claude.com, 4,893,417 at nuxt.com and 432,841 at vite.dev, with gzip cutting only four to six times. What the v2 spec says about the file (nothing), the per-section split Cloudflare ships (910 KB for one product), the four of six index files that never mention their own full file, and three curl commands that report size and encoding.
Read article →llms.txt for Professional Services: Law Firms, Accountants and Consultants (2026)
We fetched /llms.txt from 34 law, accounting and consulting domains on October 10, 2026: four served a real plain-text file, six blocked the fetch at the edge and one throttled it, one answered 200 with a 404 page behind it, twenty-one returned 404 and one never completed the TLS handshake. The seven buckets a practice needs, why client matters, fee language and portal paths must stay behind a link, the AI-usage clause Baker Tilly ships, the robots.txt served as text/javascript, and a three-command delivery check.
Read article →llms.txt for Government Sites: What a .gov Domain Should Publish (2026)
We fetched /llms.txt from 45 government hosts on October 9, 2026: two served a real plain-text file and both are outside the US federal estate (federalregister.gov, nsw.gov.au), twelve blocked the fetch at the edge, twenty-eight returned 404 and two answered 200 with an HTML page behind it. The seven buckets a public-sector file needs, the agent-guidance block the Federal Register writes, what must never appear, and the robots.txt cross-check that found one AI crawler token in 25 government files.
Read article →llms.txt for Real Estate Sites: Portals, Brokerages and Listings (2026)
We fetched /llms.txt from 25 real estate, portal, brokerage and proptech domains on October 8, 2026: one served a real plain-text file, fourteen blocked the fetch at the edge (Zillow, Trulia, Apartments.com, Zoopla, LoopNet and others), seven returned 404 and two answered 200 with an HTML error page behind it. The seven buckets a property site needs, why individual listings and prices must stay behind a link, the robots.txt cross-check nobody runs, and a three-command delivery check.
Read article →llms.txt for Travel Sites: Hotels, Airlines and OTAs (2026)
We fetched /llms.txt from 21 travel, hotel, airline and OTA domains on October 6, 2026: four served a real plain-text file, nine returned 404, four blocked the fetch at the edge and two answered 200 with no file behind it. The six buckets a travel site needs, why prices and availability must stay behind a link, the instruction block Expedia and KAYAK ship, the two bloat cases (219 KB and 477 KB), and a three-command delivery check.
Read article →llms.txt for Finance Sites: Banks, Brokerages and Fintech (2026)
We fetched /llms.txt from 19 banking, brokerage, payments and fintech domains on October 6, 2026: four served a real plain-text file, nine returned 404, four blocked the fetch at the edge and two answered 200 with an HTML page behind it. The six buckets a regulated site needs, why rates and fees belong behind a link rather than in the file, the root-file plus /docs/llms.txt pattern, and a three-command delivery check.
Read article →llms.txt for Hospitals and Health Systems: What a Medical Site Should Publish (2026)
We fetched /llms.txt from 25 hosts across 24 health systems on October 5, 2026: four hosts served a real Markdown file, one answered 200 with a 333 KB HTML page, eight returned an edge 403 and twelve a 404. The six buckets a hospital file needs, the instruction block Penn Medicine writes before its links, what must never appear (portals, protocols, PHI), and a three-command delivery check.
Read article →How to Serve llms.txt: Content-Type, Encoding and Caching (2026)
Writing a valid file is half the job — the server has to hand it over correctly. What Content-Type to send (text/plain vs text/markdown), why UTF-8 and BOM matter and how to check both in hex, what eight live llms.txt files returned on October 4, 2026 (including a 403 that ignores every user agent and a 301 that looks like HTML), how to set caching so the map never goes stale, and the five-command delivery check.
Read article →llms.txt for Universities: What a .edu Site Should Publish (2026)
We fetched /llms.txt across 33 university, library and online-learning domains on October 3, 2026: three served a real Markdown file, four answered 200 OK without serving one, six blocked the fetch at the edge. The three failure modes (a JS challenge dressed as 200, a CMS catch-all, a WAF masking the real status), the six buckets a university file needs, subpath splits for libraries and research, what must never appear, and a three-command verification pass.
Read article →llms.txt for News Publishers: Sections, Paywalls and AI Access (2026)
Newsrooms are the vertical where llms.txt is most often missing: 30 major publisher domains answered 404 or an edge block for /llms.txt on October 2, 2026, and 17 of 18 news robots.txt files we parsed disallow ClaudeBot. Why the absent file is usually deliberate, the six buckets a publisher index should hold instead of article URLs, how to handle paywalls and syndicated copies, and a three-command verification pass.
Read article →llms.txt for E-commerce: Product, Category and Policy Pages (2026)
A store is the site type where llms.txt gets exported instead of curated. The six page buckets every shop needs, why you document a product URL pattern instead of listing thousands of SKUs, what Target, Shopify, Etsy and Zalando actually publish (fetched October 1, 2026), how price and stock rot the file, where product feeds and the agentic checkout protocols take over, and a three-command verification pass.
Read article →llms.txt for SaaS: Pricing, Docs and Changelog Pages (2026)
A SaaS site indexes two audiences at once — buyers and builders. The eight page buckets every SaaS llms.txt needs, how Vercel, Supabase and Resend structure theirs (curl-verified in September 2026), why pricing should be a link rather than a number, when to split docs into /docs/llms.txt, what should never appear, and a three-command verification pass.
Read article →llms.txt Generator and Validator Workflow: The 6-Step Method (2026)
Generating the file is the easy half. Import pages from a sitemap or an auto-crawl, delete what the crawler cannot judge (archives, near-duplicates, staging routes), write the two prose slots that get quoted verbatim, score the file in the validator before deploying it, then prove the published URL serves Markdown with three curl commands — plus when to generate at build time instead.
Read article →How Long Should an llms.txt Be? Size Limits and Token Budgets (2026)
The spec sets no byte limit, so every published ceiling comes from somewhere else. What third-party guidance actually recommends, how to convert bytes to tokens in two commands, a size-to-verdict table from under 5KB to over 100KB, the five decisions that bloat a file, when to split into subpath files instead of deleting, and the adoption data that decides whether any of it is worth the afternoon.
Read article →How to Get Cited by AI Search Engines: The 2026 Citation Playbook
Citations are a retrieval outcome, not a writing contest. The three OpenAI crawlers and what each one controls, which index each engine searches (Bing-fed vs Google vs Perplexity's own crawl), the four habits that make a paragraph liftable verbatim, what published audits say does not move citation share, where llms.txt helps and where the hype breaks down, and a 30-day check with copyable commands.
Read article →llms.txt for API Documentation: What to List and How to Structure It (2026)
API references are the one vertical where llms.txt earns its keep. The five buckets every API file needs — auth, resource groups, errors, operations and machine-readable targets — a copyable thirteen-link example, how to point agents at your OpenAPI spec, Markdown twins and MCP server, the page-vs-endpoint size rule, the instruction slot before the first H2, and a four-command check that catches dead targets.
Read article →How AI Crawlers Discover Your llms.txt: Link Relations, Subpaths and Directories (2026)
Publishing the file is step one; discovery is the plumbing. The channels that actually move a client toward /llms.txt — the path convention and subpath files such as /docs/llms.txt, rel="describedby" and rel="alternate" as HTML or HTTP Link headers, what robots.txt can and cannot do for discovery, the three directories llmstxt.org lists, and a four-command reachability check.
Read article →llms.txt for Local Businesses: Service Pages, NAP Data and Multi-Location Files (2026)
A local business has ten pages and one job for llms.txt: telling an assistant what you do, where you cover, and how to book. Which service, location, pricing and credentials pages to list and in what order, how to write a summary blockquote that reads like a business card, when a branch earns its own /locations/<city>/llms.txt file, the maintenance trap around hours and prices, and a complete copyable example.
Read article →How to Write llms.txt Descriptions: Summary Blockquote and Link Annotations (2026)
The H1, the blockquote summary and the text after each colon are the only prose in an llms.txt file — and the parts most likely to be quoted verbatim. How to write a summary an engine can lift, per-link descriptions that name the question each page answers, the token budget each slot gets, and a before/after rewrite of a documentation section.
Read article →What Not to Include in llms.txt: Confidential Pages, Staging URLs and Leaks (2026)
llms.txt is a public file, so every line is a disclosure decision. The five categories that should never appear — staging hosts, admin routes, tokens and signed URLs, customer data, unreleased roadmap — a three-question triage for borderline pages, two grep scans to run before you ship, and the clean-up order if a private path already leaked.
Read article →llms.txt and Schema.org: Do You Need Both? (2026)
llms.txt is a site-level index that tells an agent which pages are worth reading; Schema.org JSON-LD is page-level labeling that tells a parser what each page means. What the v2 spec actually says about referencing structured data, what Google's May 2026 FAQ deprecation and June AI-guide mythbusting changed, the four consistency rules that stop the two files contradicting each other, and a copyable matching pair.
Read article →How to Validate an llms.txt File: Spec, Links and Delivery (2026)
Validation reads the text of your file, not the response that delivers it — which is why a clean validator run can still leave you with a file nobody reads. The three-layer check: structure and the only element the spec requires, a curl loop that catches 404s, redirects and duplicates, and the path, content-type, status and cache tests that catch the rest.
Read article →ai.txt vs llms.txt: Which AI File Should You Ship in 2026?
ai.txt declares permissions — may this media be mined or used for training? llms.txt declares content — here is what the site contains. How the two files differ, where the W3C TDM Reservation Protocol, the TDM-Reservation header and noai fit, the December 2025 German ruling that made machine-readable opt-outs the standard, and the four-signal stack worth shipping.
Read article →Markdown for Agents vs llms.txt: Content Negotiation or Curated Index? (2026)
llms.txt is a curated index that tells agents what exists; Markdown for Agents is HTTP content negotiation that changes the format one page arrives in. The Feb 2026 data on which agents send Accept: text/markdown, how Cloudflare and Vercel implement the delivery layer, and how Markdown twins + rel="alternate" connect the two.
Read article →AGENTS.md vs llms.txt: Two Files, Two Readers (2026)
Both are plain Markdown conventions and both get called "the file that tells AI about your project" — but one is read inside your repository and the other is fetched from your website. Side-by-side comparison, what belongs in each, the new link relations in llms.txt v2, the Claude Code holdout and the documented imports that fix it, plus what the ETH Zurich study of AGENTS.md files actually found.
Read article →llms.txt Maintenance: How to Keep Your File From Going Stale (2026)
A published llms.txt decays quietly: renamed URLs, deleted pages, a summary describing a product you repositioned months ago. A three-layer maintenance system — a link check that fails your build, a split between what you generate and what you curate, and a quarterly review list you can finish in ten minutes.
Read article →llms.txt Not Working? 7 Delivery-Level Fixes (Content-Type, Redirects, WAF, 2026)
Seven delivery-level reasons a published llms.txt never gets read — a catch-all rewrite answering text/html, rewrite rules that map the path onto robots.txt, redirect chains, case-sensitive paths, WAF and robots.txt blocks, files that never shipped in the build, and a stale edge cache — each with the curl command that proves which one you have.
Read article →llms.txt for Multilingual Websites: One File or One Per Language? (2026)
A file covers the URLs under its path and the most specific one wins — that single spec rule is what makes per-language llms.txt files work. Three structures (single root file, root index plus /en/ and /de/ files, root plus Optional section), how to keep it aligned with hreflang and canonicals, a curl-verified table of what Stripe, Cloudflare, Shopify, GitLab and Wikipedia actually ship on September 14, 2026, and a five-minute verification pass.
Read article →llms.txt vs MCP: The Read Layer vs the Action Layer for AI Agents (2026)
llms.txt answers "what is on this site?", MCP answers "what can I do here?" — and the 2026-07-28 MCP revision made the action layer stateless. How the two protocols differ, why MCP has no discovery of its own, where WebMCP and Lighthouse's Agentic Browsing audits fit, and the five-step adoption order for the next quarter.
Read article →How to Measure Whether AI Engines Read Your llms.txt (Logs, GA4 and Cloudflare, 2026)
A three-layer measurement framework for llms.txt: count direct /llms.txt fetches in your access logs, track AI crawler hits on the pages your file lists, and read AI referral sessions in the GA4 AI Assistant channel (added May 13, 2026) plus Cloudflare AI Crawl Control — with the exact log commands, the two GA4 caveats, and a 30-day benchmark table.
Read article →robots.txt for AI Crawlers in 2026: Allow, Block or Feed GPTBot, ClaudeBot and PerplexityBot
Training, search-index and user-triggered: the three classes of AI crawler decide what blocking actually costs you. The 2026 token reference for GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended and CCBot, two ready-to-paste robots.txt policies, Cloudflare Content Signals, and where your llms.txt file takes over.
Read article →llms.txt Case Studies 2026: Inside the Files of Vue.js, Nuxt, Cloudflare, OpenAI and Anthropic
Six production llms.txt files, curl-verified on September 10, 2026: how Vue.js, Nuxt, Cloudflare Developers, OpenAI and Anthropic structure their files — hierarchical tables of contents, markdown twins, subpath hubs and llms-full.txt — plus the five patterns worth copying.
Read article →How to Add llms.txt to Nuxt (public/, Server Routes, nuxt-llms, 2026)
Nuxt ships no llms.txt output, but its public/ directory serves a static file at the site root in 60 seconds, a server/routes handler generates one per request, and NuxtLabs' nuxt-llms module prerenders llms.txt and llms-full.txt at every build — with Nuxt Content 3.2+ integration. Includes nuxt.com's own file, curl-verified.
Read article →How to Add llms.txt to Shopify: Replace the Default File (2026)
Shopify now auto-generates an llms.txt for every store at yourstore.com/llms.txt — but the default reads like Shopify's pitch, not yours. Override it with a templates/llms.txt.liquid theme file (the robots.txt.liquid mechanism, confirmed on the Shopify developer community): what to include, exact admin clicks, a ready-to-copy store sample and a curl check.
Read article →How to Add llms.txt to Jekyll (GitHub Pages, 2026)
Jekyll has no built-in llms.txt support, but its root-file rules make it easy: a static file in 60 seconds, a Liquid template that re-lists every post at every build (no plugin, so GitHub Pages accepts it), or the jekyll-llms-output plugin for llms.txt + llms-full.txt. Includes real config, the GitHub Pages plugin whitelist caveat and a curl check of a live Jekyll site.
Read article →How to Add llms.txt to VitePress (Static or Auto-Generated, 2026)
No native llms.txt support in VitePress, but public/ ships a static file in 60 seconds and vitepress-plugin-llms — used in production by Vue.js, Vite and Vitest — generates llms.txt, llms-full.txt and per-page Markdown at every build. Real config plus a curl check.
Read article →How to Add llms.txt to MkDocs (Material for MkDocs, 2026)
MkDocs has no built-in llms.txt support, but docs/ files are copied verbatim to the site root and the mkdocs-llmstxt plugin generates a spec-compliant file at every build — with .md page versions, llms-full.txt and Read the Docs subpath support. Includes real mkdocs.yml config and a curl check.
Read article →How to Add llms.txt to Hugo (Static or Auto-Generated, 2026)
Hugo has no built-in llms.txt support as of 2026, but its static/ folder and custom output formats make it one of the easiest SSGs to add one: a hand-written file in 60 seconds, or a file auto-generated from your content at build time. Includes real hugo.toml config and a curl check.
Read article →How to Add llms.txt to Docusaurus (Docs Sites, 2026)
No official plugin yet, but docs sites are llms.txt's best use case: ship a static file in static/ in 60 seconds, or generate llms.txt and llms-full.txt from your docs with the docusaurus-plugin-llms community plugin — plus a checklist and curl verification.
Read article →How to Add llms.txt to an Astro Site (Static or Generated, 2026)
Astro has no llms.txt file convention, but public/ files and src/pages endpoints make it trivial: a static public/llms.txt, a build-time .ts endpoint, or the astro-slop integration — with examples and a curl check.
Read article →How to Add llms.txt to Next.js (App Router, 2026)
No native llms.txt convention in Next.js yet — here is how to ship one today: a static public/llms.txt file, a dynamic app/llms.txt route handler, or ISR, plus verification.
Read article →Which AI Engines Actually Read llms.txt? (2026 Crawler Data)
Two years of crawler audits show 97% of llms.txt files get zero requests — here is the evidence from 500M+ AI bot requests, Google's on-the-record stance, and the realistic playbook.
Read article →6 llms.txt Mistakes That Make AI Engines Ignore Your File
Placement, format, links, curation, tone and metrics: the six errors that get your llms.txt skipped — plus a five-minute verification checklist.
Read article →llms.txt v2: What Changed in the 2026 Spec Update
The first major revision of the llmstxt.org spec: two markdown URL forms, officially defined subpath files, rel=alternate and rel=describedby link relations, and what happened to llms_txt2ctx.
Read article →llms.txt Best Practices 2026: The 9 Rules for a File AI Engines Trust
Format, curation, and maintenance rules that separate an llms.txt agents actually use from one they ignore — with a complete reference example.
Read article →Does llms.txt Affect SEO? What Google Actually Says (2026)
Google says llms.txt has no effect on rankings. Here is the evidence, the nuance, and where the file genuinely pays off for AI visibility.
Read article →llms.txt vs sitemap.xml: Why AI Engines Need a Different File
sitemap.xml lists every URL for search engines; llms.txt curates the pages that matter for AI agents. When you need both and how they work together.
Read article →How to Add llms.txt to Cloudflare Pages (Astro, React, Plain HTML)
Publish llms.txt on Cloudflare Pages in 5 minutes — static file, dashboard upload, or a Pages Function for dynamic generation — plus verification.
Read article →llms.txt vs llms-full.txt: What's the Difference and Which Do You Need?
The curated index vs the full Markdown dump: what each is for, when you need llms-full.txt, and how to link them correctly.
Read article →The Complete llms.txt Format Guide
Every element of the llmstxt.org v2 spec, element-by-element, with a valid end-to-end example and the mistakes people make.
Read article →How to Create an llms.txt File: Step-by-Step Guide
Create your llms.txt in 10 minutes: pick your 5–15 best pages, format them per the spec, publish at your site root, and validate.
Read article →llms.txt Examples: Real Files for Every Site Type
Complete llms.txt files for SaaS docs, e-commerce, blogs and corporate sites — copy the structure that fits your site.
Read article →llms.txt vs robots.txt vs sitemap.xml: What's the Difference?
What each root-level file does, who reads it, and how the three work together for AI engines and search crawlers.
Read article →How to Add llms.txt to WordPress (2026 Guide)
Three ways to ship llms.txt on WordPress — upload the file, use a plugin, or add it with code — plus the pitfalls that break it.
Read article →What is llms.txt? The Complete Guide
What llms.txt is, why it matters for AI engines like ChatGPT and Claude, and how it fits with robots.txt and sitemap.xml.
Read article →Need it done for you?
Paste your sitemap.xml into the generator and get a spec-compliant llms.txt in seconds.
Open the Generator →