# LLMs.txt Generator — Full Site Content > Complete text content of llmstxtgenerator.dev in a single Markdown file: free llms.txt generator and validator tools, plus every guide, tutorial, comparison and troubleshooting article. Generated automatically from the deployed pages. > Source: https://llmstxtgenerator.dev/llms-full.txt · Companion index: https://llmstxtgenerator.dev/llms.txt ## Contents - [LLMs.txt Generator — Create AI-Ready Website Files for Free](https://llmstxtgenerator.dev/) - [LLMs.txt Generator — Create Spec-Compliant llms.txt Files](https://llmstxtgenerator.dev/generator/) - [LLMs.txt Validator — Check & Score Your llms.txt File](https://llmstxtgenerator.dev/checker/) - [Blog — llms.txt Guides, Tutorials & Examples](https://llmstxtgenerator.dev/blog/) - [What is llms.txt? — The Complete Guide for Developers & Site Owners](https://llmstxtgenerator.dev/guide/) - [LLMs-full.txt Generator — Full Site Content From Your Sitemap](https://llmstxtgenerator.dev/llms-full-txt-generator/) - [AGENTS.md vs llms.txt: Two Files, Two Readers (2026)](https://llmstxtgenerator.dev/blog/agents-md-vs-llms-txt/) - [How to Get Cited by AI Search Engines: The 2026 Citation Playbook](https://llmstxtgenerator.dev/blog/ai-search-citations/) - [ai.txt vs llms.txt: Which AI File Should You Ship in 2026?](https://llmstxtgenerator.dev/blog/ai-txt-vs-llms-txt/) - [Does llms.txt Affect SEO? What Google Actually Says (2026)](https://llmstxtgenerator.dev/blog/does-llms-txt-affect-seo/) - [How to Create an llms.txt File: Step-by-Step Guide](https://llmstxtgenerator.dev/blog/how-to-create-llms-txt/) - [How to Validate an llms.txt File: Spec, Links and Delivery (2026)](https://llmstxtgenerator.dev/blog/how-to-validate-llms-txt/) - [llms-full.txt Size and Structure: How to Build the Full File Without Breaking a Context Window (2026)](https://llmstxtgenerator.dev/blog/llms-full-txt-size-and-structure/) - [llms.txt and Schema.org Structured Data: Do You Need Both? (2026)](https://llmstxtgenerator.dev/blog/llms-txt-and-schema-org/) - [How to Add llms.txt to an Astro Site (Static or Generated, 2026)](https://llmstxtgenerator.dev/blog/llms-txt-astro/) - [llms.txt Best Practices 2026: The 9 Rules for a File AI Engines Trust](https://llmstxtgenerator.dev/blog/llms-txt-best-practices/) - [llms.txt Case Studies 2026: Inside the Files of Vue.js, Nuxt, Cloudflare, OpenAI and Anthropic](https://llmstxtgenerator.dev/blog/llms-txt-case-studies/) - [How to Add llms.txt to Cloudflare Pages (Astro, React, Plain HTML)](https://llmstxtgenerator.dev/blog/llms-txt-cloudflare-pages/) - [How to Serve llms.txt: Content-Type, Encoding and Caching (2026)](https://llmstxtgenerator.dev/blog/llms-txt-content-type-and-caching/) - [How to Write llms.txt Descriptions: Summary Blockquote and Link Annotations (2026)](https://llmstxtgenerator.dev/blog/llms-txt-descriptions/) - [How AI Crawlers Discover Your llms.txt: Link Relations, Subpaths and Directories (2026)](https://llmstxtgenerator.dev/blog/llms-txt-discovery/) - [How to Add llms.txt to Docusaurus (Docs Sites, 2026)](https://llmstxtgenerator.dev/blog/llms-txt-docusaurus/) - [llms.txt Examples: Real Files for Every Site Type](https://llmstxtgenerator.dev/blog/llms-txt-examples/) - [llms.txt for API Documentation: What to List and How to Structure It (2026)](https://llmstxtgenerator.dev/blog/llms-txt-for-api-docs/) - [llms.txt for E-commerce: Product, Category and Policy Pages (2026)](https://llmstxtgenerator.dev/blog/llms-txt-for-ecommerce/) - [llms.txt for Finance Sites: Banks, Brokerages and Fintech (2026)](https://llmstxtgenerator.dev/blog/llms-txt-for-finance/) - [llms.txt for Forums and Community Platforms: What 34 UGC Sites Serve (2026)](https://llmstxtgenerator.dev/blog/llms-txt-for-forums-and-communities/) - [llms.txt for Government Sites: What a .gov Domain Should Publish (2026)](https://llmstxtgenerator.dev/blog/llms-txt-for-government-sites/) - [llms.txt for Hospitals and Health Systems: What a Medical Site Should Publish (2026)](https://llmstxtgenerator.dev/blog/llms-txt-for-hospitals/) - [llms.txt for Insurance Sites: What 40 Insurers Serve and the YMYL Rules They Skip (2026)](https://llmstxtgenerator.dev/blog/llms-txt-for-insurance/) - [llms.txt for Local Businesses: Service Pages, NAP Data and Multi-Location Files (2026)](https://llmstxtgenerator.dev/blog/llms-txt-for-local-business/) - [llms.txt for Marketplaces and Directories: What 49 Two-Sided Platforms Serve (2026)](https://llmstxtgenerator.dev/blog/llms-txt-for-marketplaces/) - [llms.txt for News Publishers: Sections, Paywalls and AI Access (2026)](https://llmstxtgenerator.dev/blog/llms-txt-for-news-publishers/) - [llms.txt for Nonprofits and Charities: What 67 Organizations Actually Publish (2026)](https://llmstxtgenerator.dev/blog/llms-txt-for-nonprofits/) - [llms.txt for Open Source Projects: What 41 Developer Docs Serve (2026)](https://llmstxtgenerator.dev/blog/llms-txt-for-open-source-projects/) - [llms.txt for Professional Services: Law Firms, Accountants and Consultants (2026)](https://llmstxtgenerator.dev/blog/llms-txt-for-professional-services/) - [llms.txt for Real Estate Sites: Portals, Brokerages and Listings (2026)](https://llmstxtgenerator.dev/blog/llms-txt-for-real-estate/) - [llms.txt for SaaS: Pricing, Docs and Changelog Pages (2026)](https://llmstxtgenerator.dev/blog/llms-txt-for-saas/) - [llms.txt for Travel Sites: Hotels, Airlines and OTAs (2026)](https://llmstxtgenerator.dev/blog/llms-txt-for-travel-sites/) - [llms.txt for Universities: What a .edu Site Should Publish (2026)](https://llmstxtgenerator.dev/blog/llms-txt-for-universities/) - [llms.txt Format Guide — Complete Specification & Structure](https://llmstxtgenerator.dev/blog/llms-txt-format/) - [llms.txt Generator and Validator Workflow: The 6-Step Method (2026)](https://llmstxtgenerator.dev/blog/llms-txt-generator-workflow/) - [How to Add llms.txt to Hugo (Static or Auto-Generated, 2026)](https://llmstxtgenerator.dev/blog/llms-txt-hugo/) - [Does llms.txt Work on JavaScript-Rendered Sites? AI Crawlers and Empty Pages (2026)](https://llmstxtgenerator.dev/blog/llms-txt-javascript-rendering/) - [How to Add llms.txt to Jekyll (GitHub Pages, 2026)](https://llmstxtgenerator.dev/blog/llms-txt-jekyll/) - [llms.txt Maintenance: How to Keep Your File From Going Stale (2026)](https://llmstxtgenerator.dev/blog/llms-txt-maintenance/) - [6 llms.txt Mistakes That Make AI Engines Ignore Your File](https://llmstxtgenerator.dev/blog/llms-txt-mistakes/) - [How to Add llms.txt to MkDocs (Material for MkDocs, 2026)](https://llmstxtgenerator.dev/blog/llms-txt-mkdocs/) - [llms.txt for Multilingual Websites: One File or One Per Language? (2026)](https://llmstxtgenerator.dev/blog/llms-txt-multilingual-sites/) - [How to Add llms.txt to Next.js (App Router, 2026)](https://llmstxtgenerator.dev/blog/llms-txt-nextjs/) - [llms.txt Not Working? 7 Delivery-Level Fixes (Content-Type, Redirects, WAF, 2026)](https://llmstxtgenerator.dev/blog/llms-txt-not-working/) - [How to Add llms.txt to Nuxt (public/, Server Routes, nuxt-llms, 2026)](https://llmstxtgenerator.dev/blog/llms-txt-nuxt/) - [How to Add llms.txt to Shopify: Replace the Default File (2026)](https://llmstxtgenerator.dev/blog/llms-txt-shopify/) - [How Long Should an llms.txt Be? Size Limits and Token Budgets (2026)](https://llmstxtgenerator.dev/blog/llms-txt-size-limit/) - [llms.txt v2: What Changed in the 2026 Spec Update](https://llmstxtgenerator.dev/blog/llms-txt-v2/) - [How to Add llms.txt to VitePress (Static or Auto-Generated, 2026)](https://llmstxtgenerator.dev/blog/llms-txt-vitepress/) - [llms.txt vs llms-full.txt: What's the Difference and Which Do You Need?](https://llmstxtgenerator.dev/blog/llms-txt-vs-llms-full-txt/) - [llms.txt vs MCP: The Read Layer vs the Action Layer for AI Agents (2026)](https://llmstxtgenerator.dev/blog/llms-txt-vs-mcp/) - [llms.txt vs robots.txt vs sitemap.xml: What's the Difference?](https://llmstxtgenerator.dev/blog/llms-txt-vs-robots-txt/) - [llms.txt vs sitemap.xml: Why AI Engines Need a Different File](https://llmstxtgenerator.dev/blog/llms-txt-vs-sitemap-xml/) - [What Not to Include in llms.txt: Confidential Pages, Staging URLs and Leaks (2026)](https://llmstxtgenerator.dev/blog/llms-txt-what-not-to-include/) - [How to Add llms.txt to WordPress (2026 Guide)](https://llmstxtgenerator.dev/blog/llms-txt-wordpress/) - [Markdown for Agents vs llms.txt: Content Negotiation or Curated Index? (2026)](https://llmstxtgenerator.dev/blog/markdown-for-agents-vs-llms-txt/) - [How to Measure Whether AI Engines Read Your llms.txt (Logs, GA4 and Cloudflare, 2026)](https://llmstxtgenerator.dev/blog/measure-llms-txt-traffic/) - [robots.txt for AI Crawlers in 2026: Allow, Block or Feed GPTBot, ClaudeBot and PerplexityBot](https://llmstxtgenerator.dev/blog/robots-txt-ai-crawlers/) - [Which AI Engines Actually Read llms.txt? (2026 Crawler Data)](https://llmstxtgenerator.dev/blog/which-ai-engines-read-llms-txt/) --- # LLMs.txt Generator — Create AI-Ready Website Files for Free > Source: https://llmstxtgenerator.dev/ Free · No sign-up · Spec-compliant (llmstxt.org) # Make your website AI-ready in 60 seconds **llms.txt** is the file that tells AI engines — ChatGPT, Claude, Perplexity, Gemini — what your site is about, the same way robots.txt guides search crawlers. Generate yours free, validate it, and deploy in minutes. [Generate your llms.txt](/generator) [Validate an existing file](/checker) No account needed · Your content never leaves your browser [ ### Generator Enter your site info, import URLs from a sitemap, and generate a clean llms.txt — titles auto-derived from URLs. Try it now →](/generator)[ ### Validator Paste or upload your file. Get a 0–100 AI-readiness score with a detailed, line-by-line issue report. Check a file →](/checker)[ ### Learn the spec What llms.txt is, why it matters for Generative Engine Optimization (GEO), and how to deploy it. Read the guide →](/guide) ## Learn llms.txt inside out From the exact format to real-world examples — everything you need to ship a spec-perfect file. [ ### 📐 The Complete Format Guide Every element of the llmstxt.org v2 spec, element-by-element, with a valid end-to-end example and common mistakes. Read the spec →](/blog/llms-txt-format/)[ ### 📚 Real-World Examples Complete llms.txt files for SaaS docs, e-commerce, blogs and corporate sites — copy the structure that fits you. See examples →](/blog/llms-txt-examples/)[ ### ⚖️ vs robots.txt & sitemap.xml Three root files, three different jobs. A 9-dimension comparison and how they work together. Compare the files →](/blog/llms-txt-vs-robots-txt/)[ ### 🛠️ Step-by-Step Tutorial Create and deploy your llms.txt in 10 minutes — with per-platform instructions for WordPress, Next.js, Hugo and more. Start the tutorial →](/blog/how-to-create-llms-txt/)[ ### 🔌 WordPress Guide The largest CMS on the web — here's how to add llms.txt via upload, plugins (Yoast, Rank Math) or a child-theme snippet. WordPress how-to →](/blog/llms-txt-wordpress/)[ ### 📖 What is llms.txt? Start here if you're new: what llms.txt is, why AI engines need it, and the basics of GEO. Beginner's guide →](/guide) ## What llms.txt looks like A single Markdown file at your site root. That's it — no schema, no XML, no server config. ``` # Acme Docs > Developer documentation for the Acme API, > covering REST, GraphQL, SDKs and best practices. ## Important links - [Home](https://acme.com/): Documentation home - [Getting Started](https://acme.com/getting-started): 5-minute guide - [API Reference](https://acme.com/api): Full REST reference - [GraphQL](https://acme.com/graphql): GraphQL endpoint & schema ``` ## Why it matters ### AI is the new front door More and more users get answers from ChatGPT, Claude and Perplexity instead of Google. Without llms.txt, AI agents crawl your whole site and guess what matters — often missing your best content. ### A content map for LLMs llms.txt gives AI engines a curated, structured summary of your pages so they can understand and cite you accurately. That's the foundation of Generative Engine Optimization (GEO). ### Deploys in minutes Generate → upload to your site root → done. Same effort as robots.txt, with zero dependencies or configuration. ## llms.txt vs robots.txt vs sitemap.xml Three tiny files, three very different jobs. You probably want all three. File Audience What it does Format `robots.txt` Search engine crawlers Allows or blocks crawling of specific paths Plain text rules `sitemap.xml` Search engine crawlers Lists every URL for indexing XML llms.txt AI engines (ChatGPT, Claude, Perplexity…) Structured content summary for LLM understanding & citation Markdown ## Give AI engines a map of your site Generate a spec-compliant llms.txt for your website in under a minute — free, forever. [Start generating →](/generator) [Validate first](/checker) --- # LLMs.txt Generator — Create Spec-Compliant llms.txt Files > Source: https://llmstxtgenerator.dev/generator/ # LLMs.txt Generator Generate a file that follows the [llmstxt.org](https://llmstxt.org) specification, so ChatGPT, Claude, Perplexity and other AI engines can understand your website. ## Site details Site name \* (H1, the only required element) Tagline — optional, recommended. One sentence that tells AI what your site is. llms-full.txt URL — optional. Reference a full-documentation version (great for docs sites). ## Page links Add URLs one by one, auto-crawl your whole site, or import from your sitemap. Auto-crawl & auto-pick — enter your homepage URL; we discover your pages, pull real titles & descriptions, and pick the most important ones, grouped by section Import from sitemap.xml — paste your sitemap URL to extract every page automatically Add a link No links yet — add a URL above or import a sitemap. ## Your llms.txt ✓ Valid Markdown · spec-compliant **Deployment:** save the file as `llms.txt` and upload it to your site root (next to `robots.txt`). Verify it's live at `https://your-domain.com/llms.txt`, then run it through the [validator](/checker) for a full score. --- # LLMs.txt Validator — Check & Score Your llms.txt File > Source: https://llmstxtgenerator.dev/checker/ # LLMs.txt Validator Paste or upload your `llms.txt` and check it against the [llmstxt.org](https://llmstxt.org) specification — with a **0–100 AI-readiness score** and a detailed issue report. Paste your llms.txt content or Upload file — AI-readiness score ## Issue report Errors break spec compliance; warnings are recommendations for better AI readability. --- # Blog — llms.txt Guides, Tutorials & Examples > Source: https://llmstxtgenerator.dev/blog/ Blog & Guides # llms.txt Guides & Examples Everything you need to ship a spec-compliant llms.txt — from the exact format to real-world examples. [ Community 2026-10-17 · 8 min ## llms.txt for Forums and Community Platforms: What 34 UGC Sites Serve (2026) We fetched /llms.txt from 34 forum, community and UGC hosts on October 17, 2026: two serve a real plain-text file, one answers 200 with HTML behind the path, twenty return 403 — thirteen of those a byte-identical 295-byte Discourse body naming the platform's own error code — six return 404 and five never answer at all. Inside the two live files: 4,555 bytes, 17 links (ten of them root-relative), zero thread URLs, dev.to's ai\_disclosure\_level write contract and OpenWrt's Discourse MCP pointer. Read article →](/blog/llms-txt-for-forums-and-communities/)[ Insurance 2026-10-16 · 8 min ## llms.txt for Insurance Sites: What 40 Insurers Serve and the YMYL Rules They Skip (2026) We fetched /llms.txt from 40 insurance, insurtech, brokerage and reinsurance domains on October 16, 2026: nine serve a real plain-text or Markdown file, six block the fetch with a 403 from the edge, twenty return 404, three answer 200 with HTML behind the path and two never complete the connection. Inside the nine live files: 114,768 bytes, 66 H2 sections, 67 links written in a form Markdown cannot parse, not one .md twin — and zero files that restrict what an AI assistant may state about price, eligibility or claims. Read article →](/blog/llms-txt-for-insurance/)[ Marketplace 2026-10-15 · 8 min ## llms.txt for Marketplaces and Directories: What 49 Two-Sided Platforms Serve (2026) We fetched /llms.txt from 49 marketplace, classifieds, freelance and directory domains on October 15, 2026: six serve a real plain-text file, 25 block the request at the edge — 14 of those with one of two Cloudflare challenge pages, 4,544–4,554 bytes for one and 5,501–5,593 for the other — four answer 200 with HTML behind the path, ten return 404, two answer 202 with zero bytes and one returns 429. Inside the six live files: 90,718 bytes, 215 Markdown links, not one .md twin, a single Optional section, and the buyer/seller routing blocks that only two of six platforms publish. Read article →](/blog/llms-txt-for-marketplaces/)[ Open Source 2026-10-14 · 8 min ## llms.txt for Open Source Projects: What 41 Developer Docs Serve (2026) We fetched /llms.txt from 41 open-source project and developer documentation hosts on October 14, 2026: 18 serve a real plain-text file, six answer with a redirect and four of those chains end in a 404, three answer 200 with HTML and no file behind it, 13 return 404 and one returns 403. Inside the 18 files: 391,647 bytes total, 2,702 links of which 1,350 point at .md twins, an H2 count ranging from zero to sixteen, and only seven files carrying an Optional section. Read article →](/blog/llms-txt-for-open-source-projects/)[ Nonprofit 2026-10-13 · 8 min ## llms.txt for Nonprofits and Charities: What 67 Organizations Actually Publish (2026) We fetched /llms.txt from 67 nonprofit, charity, foundation and NGO domains on October 13, 2026: two served a real file (wri.org 10.3 KB, conservation.org 2.6 KB), twelve blocked the fetch at the edge, 45 returned 404 and five answered 200 with no file behind it — three of them a byte-identical 3,038-byte bot challenge. The seven buckets a nonprofit file needs, why impact totals must stay behind a link, the donor-data exclusions, the robots.txt cross-check that found one AI crawler token in twelve files, and a three-command delivery check. Read article →](/blog/llms-txt-for-nonprofits/)[ GEO 2026-10-12 · 8 min ## Does llms.txt Work on JavaScript-Rendered Sites? AI Crawlers and Empty Pages (2026) We fetched /llms.txt from 12 product domains with the GPTBot user agent on October 12, 2026: six serve a real plain-text file, five return 404 with a 400-byte error body, and Trello answers the path with 160,977 bytes of its own HTML — a body containing the string "llms" zero times. Probing 54 URLs listed inside nine real files, 53 returned readable text because they point at .md twins; the one failure was a listed page returning 404 to four user agents. Plus 14 app and dashboard routes, six of which answered 200 OK with under 60 words. Read article →](/blog/llms-txt-javascript-rendering/)[ Format & Spec 2026-10-11 · 8 min ## llms-full.txt Size and Structure: The Full File Nobody Specified (2026) We measured eight live llms-full.txt files on October 11, 2026: 67,649,033 bytes at developers.cloudflare.com, 43,107,672 at platform.claude.com, 4,893,417 at nuxt.com and 432,841 at vite.dev, with gzip cutting only four to six times. What the v2 spec says about the file (nothing), the per-section split Cloudflare ships (910 KB for one product), the four of six index files that never mention their own full file, and three curl commands that report size and encoding. Read article →](/blog/llms-full-txt-size-and-structure/)[ Professional Services 2026-10-10 · 8 min ## llms.txt for Professional Services: Law Firms, Accountants and Consultants (2026) We fetched /llms.txt from 34 law, accounting and consulting domains on October 10, 2026: four served a real plain-text file, six blocked the fetch at the edge and one throttled it, one answered 200 with a 404 page behind it, twenty-one returned 404 and one never completed the TLS handshake. The seven buckets a practice needs, why client matters, fee language and portal paths must stay behind a link, the AI-usage clause Baker Tilly ships, the robots.txt served as text/javascript, and a three-command delivery check. Read article →](/blog/llms-txt-for-professional-services/)[ Public Sector 2026-10-09 · 8 min ## llms.txt for Government Sites: What a .gov Domain Should Publish (2026) We fetched /llms.txt from 45 government hosts on October 9, 2026: two served a real plain-text file and both are outside the US federal estate (federalregister.gov, nsw.gov.au), twelve blocked the fetch at the edge, twenty-eight returned 404 and two answered 200 with an HTML page behind it. The seven buckets a public-sector file needs, the agent-guidance block the Federal Register writes, what must never appear, and the robots.txt cross-check that found one AI crawler token in 25 government files. Read article →](/blog/llms-txt-for-government-sites/)[ Real Estate 2026-10-08 · 8 min ## llms.txt for Real Estate Sites: Portals, Brokerages and Listings (2026) We fetched /llms.txt from 25 real estate, portal, brokerage and proptech domains on October 8, 2026: one served a real plain-text file, fourteen blocked the fetch at the edge (Zillow, Trulia, Apartments.com, Zoopla, LoopNet and others), seven returned 404 and two answered 200 with an HTML error page behind it. The seven buckets a property site needs, why individual listings and prices must stay behind a link, the robots.txt cross-check nobody runs, and a three-command delivery check. Read article →](/blog/llms-txt-for-real-estate/)[ Travel 2026-10-07 · 8 min ## llms.txt for Travel Sites: Hotels, Airlines and OTAs (2026) We fetched /llms.txt from 21 travel, hotel, airline and OTA domains on October 6, 2026: four served a real plain-text file, nine returned 404, four blocked the fetch at the edge and two answered 200 with no file behind it. The six buckets a travel site needs, why prices and availability must stay behind a link, the instruction block Expedia and KAYAK ship, the two bloat cases (219 KB and 477 KB), and a three-command delivery check. Read article →](/blog/llms-txt-for-travel-sites/)[ Finance 2026-10-06 · 8 min ## llms.txt for Finance Sites: Banks, Brokerages and Fintech (2026) We fetched /llms.txt from 19 banking, brokerage, payments and fintech domains on October 6, 2026: four served a real plain-text file, nine returned 404, four blocked the fetch at the edge and two answered 200 with an HTML page behind it. The six buckets a regulated site needs, why rates and fees belong behind a link rather than in the file, the root-file plus /docs/llms.txt pattern, and a three-command delivery check. Read article →](/blog/llms-txt-for-finance/)[ Healthcare 2026-10-05 · 8 min ## llms.txt for Hospitals and Health Systems: What a Medical Site Should Publish (2026) We fetched /llms.txt from 25 hosts across 24 health systems on October 5, 2026: four hosts served a real Markdown file, one answered 200 with a 333 KB HTML page, eight returned an edge 403 and twelve a 404. The six buckets a hospital file needs, the instruction block Penn Medicine writes before its links, what must never appear (portals, protocols, PHI), and a three-command delivery check. Read article →](/blog/llms-txt-for-hospitals/)[ GEO 2026-10-04 · 7 min ## How to Serve llms.txt: Content-Type, Encoding and Caching (2026) Writing a valid file is half the job — the server has to hand it over correctly. What Content-Type to send (text/plain vs text/markdown), why UTF-8 and BOM matter and how to check both in hex, what eight live llms.txt files returned on October 4, 2026 (including a 403 that ignores every user agent and a 301 that looks like HTML), how to set caching so the map never goes stale, and the five-command delivery check. Read article →](/blog/llms-txt-content-type-and-caching/)[ Education 2026-10-03 · 8 min ## llms.txt for Universities: What a .edu Site Should Publish (2026) We fetched /llms.txt across 33 university, library and online-learning domains on October 3, 2026: three served a real Markdown file, four answered 200 OK without serving one, six blocked the fetch at the edge. The three failure modes (a JS challenge dressed as 200, a CMS catch-all, a WAF masking the real status), the six buckets a university file needs, subpath splits for libraries and research, what must never appear, and a three-command verification pass. Read article →](/blog/llms-txt-for-universities/)[ Publishing 2026-10-02 · 8 min ## llms.txt for News Publishers: Sections, Paywalls and AI Access (2026) Newsrooms are the vertical where llms.txt is most often missing: 30 major publisher domains answered 404 or an edge block for /llms.txt on October 2, 2026, and 17 of 18 news robots.txt files we parsed disallow ClaudeBot. Why the absent file is usually deliberate, the six buckets a publisher index should hold instead of article URLs, how to handle paywalls and syndicated copies, and a three-command verification pass. Read article →](/blog/llms-txt-for-news-publishers/)[ E-commerce 2026-10-01 · 8 min ## llms.txt for E-commerce: Product, Category and Policy Pages (2026) A store is the site type where llms.txt gets exported instead of curated. The six page buckets every shop needs, why you document a product URL pattern instead of listing thousands of SKUs, what Target, Shopify, Etsy and Zalando actually publish (fetched October 1, 2026), how price and stock rot the file, where product feeds and the agentic checkout protocols take over, and a three-command verification pass. Read article →](/blog/llms-txt-for-ecommerce/)[ SaaS 2026-09-30 · 8 min ## llms.txt for SaaS: Pricing, Docs and Changelog Pages (2026) A SaaS site indexes two audiences at once — buyers and builders. The eight page buckets every SaaS llms.txt needs, how Vercel, Supabase and Resend structure theirs (curl-verified in September 2026), why pricing should be a link rather than a number, when to split docs into /docs/llms.txt, what should never appear, and a three-command verification pass. Read article →](/blog/llms-txt-for-saas/)[ Workflow 2026-09-29 · 7 min ## llms.txt Generator and Validator Workflow: The 6-Step Method (2026) Generating the file is the easy half. Import pages from a sitemap or an auto-crawl, delete what the crawler cannot judge (archives, near-duplicates, staging routes), write the two prose slots that get quoted verbatim, score the file in the validator before deploying it, then prove the published URL serves Markdown with three curl commands — plus when to generate at build time instead. Read article →](/blog/llms-txt-generator-workflow/)[ GEO 2026-09-28 · 7 min ## How Long Should an llms.txt Be? Size Limits and Token Budgets (2026) The spec sets no byte limit, so every published ceiling comes from somewhere else. What third-party guidance actually recommends, how to convert bytes to tokens in two commands, a size-to-verdict table from under 5KB to over 100KB, the five decisions that bloat a file, when to split into subpath files instead of deleting, and the adoption data that decides whether any of it is worth the afternoon. Read article →](/blog/llms-txt-size-limit/)[ GEO 2026-09-27 · 8 min ## How to Get Cited by AI Search Engines: The 2026 Citation Playbook Citations are a retrieval outcome, not a writing contest. The three OpenAI crawlers and what each one controls, which index each engine searches (Bing-fed vs Google vs Perplexity's own crawl), the four habits that make a paragraph liftable verbatim, what published audits say does not move citation share, where llms.txt helps and where the hype breaks down, and a 30-day check with copyable commands. Read article →](/blog/ai-search-citations/)[ API Docs 2026-09-26 · 8 min ## llms.txt for API Documentation: What to List and How to Structure It (2026) API references are the one vertical where llms.txt earns its keep. The five buckets every API file needs — auth, resource groups, errors, operations and machine-readable targets — a copyable thirteen-link example, how to point agents at your OpenAPI spec, Markdown twins and MCP server, the page-vs-endpoint size rule, the instruction slot before the first H2, and a four-command check that catches dead targets. Read article →](/blog/llms-txt-for-api-docs/)[ GEO 2026-09-25 · 7 min ## How AI Crawlers Discover Your llms.txt: Link Relations, Subpaths and Directories (2026) Publishing the file is step one; discovery is the plumbing. The channels that actually move a client toward /llms.txt — the path convention and subpath files such as /docs/llms.txt, rel="describedby" and rel="alternate" as HTML or HTTP Link headers, what robots.txt can and cannot do for discovery, the three directories llmstxt.org lists, and a four-command reachability check. Read article →](/blog/llms-txt-discovery/)[ Local SEO 2026-09-24 · 7 min ## llms.txt for Local Businesses: Service Pages, NAP Data and Multi-Location Files (2026) A local business has ten pages and one job for llms.txt: telling an assistant what you do, where you cover, and how to book. Which service, location, pricing and credentials pages to list and in what order, how to write a summary blockquote that reads like a business card, when a branch earns its own /locations//llms.txt file, the maintenance trap around hours and prices, and a complete copyable example. Read article →](/blog/llms-txt-for-local-business/)[ Writing 2026-09-23 · 7 min ## How to Write llms.txt Descriptions: Summary Blockquote and Link Annotations (2026) The H1, the blockquote summary and the text after each colon are the only prose in an llms.txt file — and the parts most likely to be quoted verbatim. How to write a summary an engine can lift, per-link descriptions that name the question each page answers, the token budget each slot gets, and a before/after rewrite of a documentation section. Read article →](/blog/llms-txt-descriptions/)[ Content Security 2026-09-22 · 7 min ## What Not to Include in llms.txt: Confidential Pages, Staging URLs and Leaks (2026) llms.txt is a public file, so every line is a disclosure decision. The five categories that should never appear — staging hosts, admin routes, tokens and signed URLs, customer data, unreleased roadmap — a three-question triage for borderline pages, two grep scans to run before you ship, and the clean-up order if a private path already leaked. Read article →](/blog/llms-txt-what-not-to-include/)[ GEO 2026-09-21 · 8 min ## llms.txt and Schema.org: Do You Need Both? (2026) llms.txt is a site-level index that tells an agent which pages are worth reading; Schema.org JSON-LD is page-level labeling that tells a parser what each page means. What the v2 spec actually says about referencing structured data, what Google's May 2026 FAQ deprecation and June AI-guide mythbusting changed, the four consistency rules that stop the two files contradicting each other, and a copyable matching pair. Read article →](/blog/llms-txt-and-schema-org/)[ Validation 2026-09-20 · 7 min ## How to Validate an llms.txt File: Spec, Links and Delivery (2026) Validation reads the text of your file, not the response that delivers it — which is why a clean validator run can still leave you with a file nobody reads. The three-layer check: structure and the only element the spec requires, a curl loop that catches 404s, redirects and duplicates, and the path, content-type, status and cache tests that catch the rest. Read article →](/blog/how-to-validate-llms-txt/)[ AI Policy 2026-09-19 · 7 min ## ai.txt vs llms.txt: Which AI File Should You Ship in 2026? ai.txt declares permissions — may this media be mined or used for training? llms.txt declares content — here is what the site contains. How the two files differ, where the W3C TDM Reservation Protocol, the TDM-Reservation header and noai fit, the December 2025 German ruling that made machine-readable opt-outs the standard, and the four-signal stack worth shipping. Read article →](/blog/ai-txt-vs-llms-txt/)[ AI Agents 2026-09-18 · 8 min ## Markdown for Agents vs llms.txt: Content Negotiation or Curated Index? (2026) llms.txt is a curated index that tells agents what exists; Markdown for Agents is HTTP content negotiation that changes the format one page arrives in. The Feb 2026 data on which agents send Accept: text/markdown, how Cloudflare and Vercel implement the delivery layer, and how Markdown twins + rel="alternate" connect the two. Read article →](/blog/markdown-for-agents-vs-llms-txt/)[ AI Agents 2026-09-17 · 8 min ## AGENTS.md vs llms.txt: Two Files, Two Readers (2026) Both are plain Markdown conventions and both get called "the file that tells AI about your project" — but one is read inside your repository and the other is fetched from your website. Side-by-side comparison, what belongs in each, the new link relations in llms.txt v2, the Claude Code holdout and the documented imports that fix it, plus what the ETH Zurich study of AGENTS.md files actually found. Read article →](/blog/agents-md-vs-llms-txt/)[ Maintenance 2026-09-16 · 7 min ## llms.txt Maintenance: How to Keep Your File From Going Stale (2026) A published llms.txt decays quietly: renamed URLs, deleted pages, a summary describing a product you repositioned months ago. A three-layer maintenance system — a link check that fails your build, a split between what you generate and what you curate, and a quarterly review list you can finish in ten minutes. Read article →](/blog/llms-txt-maintenance/)[ Troubleshooting 2026-09-15 · 7 min ## llms.txt Not Working? 7 Delivery-Level Fixes (Content-Type, Redirects, WAF, 2026) Seven delivery-level reasons a published llms.txt never gets read — a catch-all rewrite answering text/html, rewrite rules that map the path onto robots.txt, redirect chains, case-sensitive paths, WAF and robots.txt blocks, files that never shipped in the build, and a stale edge cache — each with the curl command that proves which one you have. Read article →](/blog/llms-txt-not-working/)[ Global SEO 2026-09-14 · 7 min ## llms.txt for Multilingual Websites: One File or One Per Language? (2026) A file covers the URLs under its path and the most specific one wins — that single spec rule is what makes per-language llms.txt files work. Three structures (single root file, root index plus /en/ and /de/ files, root plus Optional section), how to keep it aligned with hreflang and canonicals, a curl-verified table of what Stripe, Cloudflare, Shopify, GitLab and Wikipedia actually ship on September 14, 2026, and a five-minute verification pass. Read article →](/blog/llms-txt-multilingual-sites/)[ AI Agents 2026-09-13 · 7 min ## llms.txt vs MCP: The Read Layer vs the Action Layer for AI Agents (2026) llms.txt answers "what is on this site?", MCP answers "what can I do here?" — and the 2026-07-28 MCP revision made the action layer stateless. How the two protocols differ, why MCP has no discovery of its own, where WebMCP and Lighthouse's Agentic Browsing audits fit, and the five-step adoption order for the next quarter. Read article →](/blog/llms-txt-vs-mcp/)[ Measurement 2026-09-12 · 8 min ## How to Measure Whether AI Engines Read Your llms.txt (Logs, GA4 and Cloudflare, 2026) A three-layer measurement framework for llms.txt: count direct /llms.txt fetches in your access logs, track AI crawler hits on the pages your file lists, and read AI referral sessions in the GA4 AI Assistant channel (added May 13, 2026) plus Cloudflare AI Crawl Control — with the exact log commands, the two GA4 caveats, and a 30-day benchmark table. Read article →](/blog/measure-llms-txt-traffic/)[ AI Crawlers 2026-09-11 · 7 min ## robots.txt for AI Crawlers in 2026: Allow, Block or Feed GPTBot, ClaudeBot and PerplexityBot Training, search-index and user-triggered: the three classes of AI crawler decide what blocking actually costs you. The 2026 token reference for GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended and CCBot, two ready-to-paste robots.txt policies, Cloudflare Content Signals, and where your llms.txt file takes over. Read article →](/blog/robots-txt-ai-crawlers/)[ Case Studies 2026-09-10 · 7 min ## llms.txt Case Studies 2026: Inside the Files of Vue.js, Nuxt, Cloudflare, OpenAI and Anthropic Six production llms.txt files, curl-verified on September 10, 2026: how Vue.js, Nuxt, Cloudflare Developers, OpenAI and Anthropic structure their files — hierarchical tables of contents, markdown twins, subpath hubs and llms-full.txt — plus the five patterns worth copying. Read article →](/blog/llms-txt-case-studies/)[ Tutorial 2026-09-09 · 7 min ## How to Add llms.txt to Nuxt (public/, Server Routes, nuxt-llms, 2026) Nuxt ships no llms.txt output, but its public/ directory serves a static file at the site root in 60 seconds, a server/routes handler generates one per request, and NuxtLabs' nuxt-llms module prerenders llms.txt and llms-full.txt at every build — with Nuxt Content 3.2+ integration. Includes nuxt.com's own file, curl-verified. Read article →](/blog/llms-txt-nuxt/)[ E-commerce 2026-09-08 · 6 min ## How to Add llms.txt to Shopify: Replace the Default File (2026) Shopify now auto-generates an llms.txt for every store at yourstore.com/llms.txt — but the default reads like Shopify's pitch, not yours. Override it with a templates/llms.txt.liquid theme file (the robots.txt.liquid mechanism, confirmed on the Shopify developer community): what to include, exact admin clicks, a ready-to-copy store sample and a curl check. Read article →](/blog/llms-txt-shopify/)[ Tutorial 2026-09-07 · 7 min ## How to Add llms.txt to Jekyll (GitHub Pages, 2026) Jekyll has no built-in llms.txt support, but its root-file rules make it easy: a static file in 60 seconds, a Liquid template that re-lists every post at every build (no plugin, so GitHub Pages accepts it), or the jekyll-llms-output plugin for llms.txt + llms-full.txt. Includes real config, the GitHub Pages plugin whitelist caveat and a curl check of a live Jekyll site. Read article →](/blog/llms-txt-jekyll/)[ Tutorial 2026-09-06 · 7 min ## How to Add llms.txt to VitePress (Static or Auto-Generated, 2026) No native llms.txt support in VitePress, but public/ ships a static file in 60 seconds and vitepress-plugin-llms — used in production by Vue.js, Vite and Vitest — generates llms.txt, llms-full.txt and per-page Markdown at every build. Real config plus a curl check. Read article →](/blog/llms-txt-vitepress/)[ Tutorial 2026-09-05 · 6 min ## How to Add llms.txt to MkDocs (Material for MkDocs, 2026) MkDocs has no built-in llms.txt support, but docs/ files are copied verbatim to the site root and the mkdocs-llmstxt plugin generates a spec-compliant file at every build — with .md page versions, llms-full.txt and Read the Docs subpath support. Includes real mkdocs.yml config and a curl check. Read article →](/blog/llms-txt-mkdocs/)[ Tutorial 2026-09-04 · 7 min ## How to Add llms.txt to Hugo (Static or Auto-Generated, 2026) Hugo has no built-in llms.txt support as of 2026, but its static/ folder and custom output formats make it one of the easiest SSGs to add one: a hand-written file in 60 seconds, or a file auto-generated from your content at build time. Includes real hugo.toml config and a curl check. Read article →](/blog/llms-txt-hugo/)[ Tutorial 2026-09-03 · 6 min ## How to Add llms.txt to Docusaurus (Docs Sites, 2026) No official plugin yet, but docs sites are llms.txt's best use case: ship a static file in static/ in 60 seconds, or generate llms.txt and llms-full.txt from your docs with the docusaurus-plugin-llms community plugin — plus a checklist and curl verification. Read article →](/blog/llms-txt-docusaurus/)[ Tutorial 2026-09-02 · 6 min ## How to Add llms.txt to an Astro Site (Static or Generated, 2026) Astro has no llms.txt file convention, but public/ files and src/pages endpoints make it trivial: a static public/llms.txt, a build-time .ts endpoint, or the astro-slop integration — with examples and a curl check. Read article →](/blog/llms-txt-astro/)[ Tutorial 2026-09-01 · 7 min ## How to Add llms.txt to Next.js (App Router, 2026) No native llms.txt convention in Next.js yet — here is how to ship one today: a static public/llms.txt file, a dynamic app/llms.txt route handler, or ISR, plus verification. Read article →](/blog/llms-txt-nextjs/)[ Research 2026-08-31 · 7 min ## Which AI Engines Actually Read llms.txt? (2026 Crawler Data) Two years of crawler audits show 97% of llms.txt files get zero requests — here is the evidence from 500M+ AI bot requests, Google's on-the-record stance, and the realistic playbook. Read article →](/blog/which-ai-engines-read-llms-txt/)[ Guide 2026-08-30 · 6 min ## 6 llms.txt Mistakes That Make AI Engines Ignore Your File Placement, format, links, curation, tone and metrics: the six errors that get your llms.txt skipped — plus a five-minute verification checklist. Read article →](/blog/llms-txt-mistakes/)[ Format & Spec 2026-08-29 · 7 min ## llms.txt v2: What Changed in the 2026 Spec Update The first major revision of the llmstxt.org spec: two markdown URL forms, officially defined subpath files, rel=alternate and rel=describedby link relations, and what happened to llms\_txt2ctx. Read article →](/blog/llms-txt-v2/)[ Guide 2026-08-27 · 6 min ## llms.txt Best Practices 2026: The 9 Rules for a File AI Engines Trust Format, curation, and maintenance rules that separate an llms.txt agents actually use from one they ignore — with a complete reference example. Read article →](/blog/llms-txt-best-practices/)[ SEO 2026-08-27 · 5 min ## Does llms.txt Affect SEO? What Google Actually Says (2026) Google says llms.txt has no effect on rankings. Here is the evidence, the nuance, and where the file genuinely pays off for AI visibility. Read article →](/blog/does-llms-txt-affect-seo/)[ Explainer 2026-08-27 · 5 min ## llms.txt vs sitemap.xml: Why AI Engines Need a Different File sitemap.xml lists every URL for search engines; llms.txt curates the pages that matter for AI agents. When you need both and how they work together. Read article →](/blog/llms-txt-vs-sitemap-xml/)[ Tutorial 2026-08-27 · 5 min ## How to Add llms.txt to Cloudflare Pages (Astro, React, Plain HTML) Publish llms.txt on Cloudflare Pages in 5 minutes — static file, dashboard upload, or a Pages Function for dynamic generation — plus verification. Read article →](/blog/llms-txt-cloudflare-pages/)[ Explainer 2026-08-27 · 5 min ## llms.txt vs llms-full.txt: What's the Difference and Which Do You Need? The curated index vs the full Markdown dump: what each is for, when you need llms-full.txt, and how to link them correctly. Read article →](/blog/llms-txt-vs-llms-full-txt/)[ Format & Spec 2026-08-15 · 9 min ## The Complete llms.txt Format Guide Every element of the llmstxt.org v2 spec, element-by-element, with a valid end-to-end example and the mistakes people make. Read article →](/blog/llms-txt-format/)[ Tutorial 2026-08-15 · 8 min ## How to Create an llms.txt File: Step-by-Step Guide Create your llms.txt in 10 minutes: pick your 5–15 best pages, format them per the spec, publish at your site root, and validate. Read article →](/blog/how-to-create-llms-txt/)[ Examples 2026-08-15 · 7 min ## llms.txt Examples: Real Files for Every Site Type Complete llms.txt files for SaaS docs, e-commerce, blogs and corporate sites — copy the structure that fits your site. Read article →](/blog/llms-txt-examples/)[ Explainer 2026-08-15 · 6 min ## llms.txt vs robots.txt vs sitemap.xml: What's the Difference? What each root-level file does, who reads it, and how the three work together for AI engines and search crawlers. Read article →](/blog/llms-txt-vs-robots-txt/)[ WordPress 2026-08-15 · 6 min ## How to Add llms.txt to WordPress (2026 Guide) Three ways to ship llms.txt on WordPress — upload the file, use a plugin, or add it with code — plus the pitfalls that break it. Read article →](/blog/llms-txt-wordpress/)[ Beginner Guide 2026-08-15 · 4 min ## What is llms.txt? The Complete Guide What llms.txt is, why it matters for AI engines like ChatGPT and Claude, and how it fits with robots.txt and sitemap.xml. Read article →](/guide/) ## Need it done for you? Paste your sitemap.xml into the generator and get a spec-compliant llms.txt in seconds. [Open the Generator →](/generator) --- # What is llms.txt? — The Complete Guide for Developers & Site Owners > Source: https://llmstxtgenerator.dev/guide/ Complete guide · Updated for llmstxt.org v2 # What is llms.txt? The complete guide llms.txt is the "content map" that AI engines read to understand your website. This guide explains what it is, why it matters, and how to create and deploy it in minutes. ## What is llms.txt? **llms.txt** is a Markdown file placed in the root of your website, proposed in 2024 by AI researcher [Jeremy Howard](https://en.wikipedia.org/wiki/Jeremy_Howard_\(entrepreneur\)) as an open convention for large language models (LLMs). Just as robots.txt gives search-engine crawlers directions, llms.txt gives AI engines a **structured summary and index** of your site's content — so they can understand and cite it accurately instead of guessing. The canonical specification lives at [llmstxt.org](https://llmstxt.org). ## Why llms.txt matters now AI is becoming a primary entry point to the web. Users increasingly ask ChatGPT, Claude, Perplexity and Gemini questions instead of typing keywords into Google. These AI engines need an efficient, reliable way to understand your site: - **Without llms.txt:** AI agents crawl your entire site, guess at what's important, and may miss or misread your core content. - **With llms.txt:** AI reads your curated content map directly and cites your pages accurately — the foundation of **Generative Engine Optimization (GEO)**. ## The llms.txt format An llms.txt file is deliberately simple — plain Markdown. A minimal valid file looks like this: ``` # Site Name > One-sentence description of the site (optional). ## Important links - [Page title](https://your-site.com/page): Short description - [Another page](https://your-site.com/other): Another description ``` The rules per the llmstxt.org spec: - **The H1 heading (`# Site Name`) is the only mandatory element.** Everything else is optional. - Links must be **absolute URLs** (`https://…`) — no relative paths. - Group links under **H2 headings** (`## Section`); use Markdown list items for links. - Optionally add a **blockquote tagline** (`> …`) describing the site. - For documentation-heavy sites, you can reference a full version at `/llms-full.txt`. ## llms.txt vs robots.txt vs sitemap.xml They complement each other — most sites should have all three. File Audience Purpose `robots.txt` Search engine crawlers Allows / blocks crawling `sitemap.xml` Search engine crawlers Lists all URLs for indexing llms.txt AI engines (ChatGPT, Claude, Perplexity…) Structured content summary for LLM understanding & citation ## How to create an llms.txt 1. Use [our free generator](/generator) — enter your site name, tagline and URLs (or import a sitemap) and it produces a spec-compliant file instantly. 2. Download the file and upload it to your website root, next to `robots.txt`. 3. Verify it's publicly accessible at `https://your-domain.com/llms.txt`. 4. Run it through the [validator](/checker) to confirm the structure and get an AI-readiness score. 5. Update it whenever your site structure changes significantly. ## Frequently asked questions ### Does llms.txt affect my Google ranking? No, not directly. Traditional search engines like Google do not use llms.txt. Its value is in how AI engines (ChatGPT, Claude, Perplexity, etc.) understand and cite your content — the GEO channel. ### How often should I update it? Whenever your site structure changes in a meaningful way — new sections, renamed pages, removed content. For most sites, reviewing it monthly is plenty. ### Is llms.txt an official standard? Not yet formally. It's a community-driven open convention that isn't standardized by the W3C or IETF, but adoption is growing quickly — tools, frameworks and AI products increasingly recognize it. ### Where should the file live? In the root of your website, exactly like `robots.txt`. The canonical URL is `https://your-domain.com/llms.txt`. AI engines look for it there first. ## Keep reading [ ### 📐 The Complete Format Guide Every element of the v2 spec explained, with a valid end-to-end example. ](/blog/llms-txt-format/)[ ### 📚 Real-World Examples Copy-paste llms.txt files for SaaS, e-commerce, blogs and corporate sites. ](/blog/llms-txt-examples/) ## Ready to get started? Generate your first llms.txt in under a minute — free, no sign-up. [⚡ Generate your llms.txt](/generator) --- # LLMs-full.txt Generator — Full Site Content From Your Sitemap > Source: https://llmstxtgenerator.dev/llms-full-txt-generator/ Paid tool · Full-content knowledge package # LLMs-full.txt Generator Give us your `sitemap.xml` and we crawl your pages, extract the real text and build a single **llms-full.txt** knowledge file that ChatGPT, Claude, Perplexity and Gemini can read directly — not just a list of links. ## Step 1 — Scan your sitemap We read your sitemap, list every page with its real title, and show how much content we can extract. ## Step 2 — Choose pages Content mode Summary — ~300 words per page (lean file) Full content — up to 2,000 words per page ## Step 3 — Preview & download ### Unlock the full file — one-time $1 The preview shows the first 2 pages. Unlock to download the complete llms-full.txt with every page you selected. License key — sent to you after purchase ## What is llms-full.txt? An **llms-full.txt** file is the deep-content companion to a standard [llms.txt](/blog/llms-txt-vs-llms-full-txt/). Where llms.txt gives AI engines a curated list of links, llms-full.txt contains the actual text of your pages — articles, docs, product copy — in one Markdown file. Models can then answer questions about your site accurately without crawling every URL. llms.txt (free) llms-full.txt (this tool) **Content** Titles, links, short descriptions Full page text as Markdown **Typical size** 2–10 KB 100 KB – 2 MB **Best for** Discoverability Answer accuracy and depth **Recommended** Every site Docs, blogs, content-heavy sites ## How the generator works 1. **Scan** — we fetch your sitemap.xml (including sitemap index files) and list every page with its real title and content size. 2. **Select** — untick anything that shouldn't be in the AI knowledge file: tag pages, legal boilerplate, thin pages. 3. **Generate** — we crawl each selected page, strip navigation, footers and scripts, and convert the main content to clean Markdown. 4. **Publish** — download the file, upload it to your site root as `/llms-full.txt`, and link it from your llms.txt. ## Already have an llms.txt? Validate it first with our free [llms.txt validator](/checker/) (AI Readiness Score, line-level fixes), or build the index file with the free [llms.txt generator](/generator/) — it auto-crawls your site or imports your sitemap. Our own site publishes both files: [/llms.txt](/llms.txt) and [/llms-full.txt](/llms-full.txt). Privacy note: we fetch the pages you submit server-side to build your file. Submitted URLs are processed for that request only and are not stored. See our [Privacy Policy](/privacy-policy/). --- # AGENTS.md vs llms.txt: Two Files, Two Readers (2026) > Source: https://llmstxtgenerator.dev/blog/agents-md-vs-llms-txt/ Comparison · AI agents · Updated 2026 # AGENTS.md vs llms.txt: Two Files, Two Readers (2026) Both are plain Markdown, both are conventions rather than enforced standards, and both are frequently described as "the file that tells AI about your project." They have almost nothing else in common. One is read inside your repository, the other is fetched from your website. ## The Short Answer AGENTS.md llms.txt Reader Coding agents — OpenAI Codex, Jules, Cursor, Aider, Zed, Warp and others External AI crawlers, chat assistants with search, IDE agents browsing docs Lives in Repository root, plus nested copies per package Site root, or any path such as `/docs/llms.txt` Transport A git checkout A plain HTTP GET, cacheable at the edge Contains Build and test commands, code style, boundaries, PR rules Site name, one-line summary, curated links to the pages that matter Governance Open format, stewarded by the Agentic AI Foundation under the Linux Foundation Community proposal by Jeremy Howard, documented at llmstxt.org ## AGENTS.md Is a Repository Contract The [agents.md](https://agents.md/) project defines AGENTS.md as a "README for agents" — the extra, sometimes detailed context that would clutter a README written for humans: exact build steps, test invocations, conventions that differ from tool defaults, and the files an agent should never touch. The site reports adoption across more than 60,000 open-source projects. Real files look like real engineering notes. The root AGENTS.md in [openai/codex](https://github.com/openai/codex/blob/main/AGENTS.md) runs about 320 lines and is mostly Rust-specific constraints and sandbox warnings. Apache Airflow's copy opens with an SPDX license header before its first instruction. Shape, not length, is what carries over: ``` # AGENTS.md ## Setup commands - Install deps: `pnpm install` - Run tests: `pnpm test` ## Code style - TypeScript strict mode - Use functional patterns where possible ## Boundaries - Never edit files under `src/generated/` - Do not modify the sandbox environment-variable checks ``` In a monorepo, place a second AGENTS.md inside each package. Agents read the nearest file in the directory tree, so the closest one takes precedence and every subproject can ship tailored instructions. ## llms.txt Is a Site Contract The [llms.txt](https://llmstxt.org/) proposal was updated to v2 in August 2026, and the v2 text is explicit about placement: the file can sit at the site root or at any path, covering the pages under that path. It also recommends publishing a clean Markdown twin for each page — either with `.md` appended (`page.html.md`) or with the extension replaced (`page.md`) — so the links inside your file point at content an agent can read without stripping HTML. ``` # Acme Docs > API documentation for Acme's payments platform, covering charges, > refunds and webhooks. ## Guides - [Quickstart](https://docs.acme.com/quickstart.md): first charge in 5 minutes - [Refunds](https://docs.acme.com/refunds.md): partial and full refund flows ## Optional - [Changelog](https://docs.acme.com/changelog.md) ``` The same v2 document leans on HTTP to help clients find both artefacts: a `rel="alternate" type="text/markdown"` link points at the Markdown version of a page, and `rel="describedby"` points at the llms.txt file that covers it — expressed either as HTML elements or as a server-level `Link:` response header, which requires no page changes. Chrome's Lighthouse agentic browsing checks now audit sites for the file, and the labs themselves ship one: OpenAI, Anthropic and Gemini all publish llms.txt for their developer documentation. ## The Discovery Difference llms.txt has two discovery channels: the `/llms.txt` convention and the `describedby` link relation. AGENTS.md has exactly one — the folder it sits in. There is no registry, no HTTP header, no sitemap entry. That single fact explains the nesting rule, the per-package files, and why the format's own documentation stresses that the closest file wins. ## The Claude Code Exception One major agent still does not read AGENTS.md natively. Anthropic's memory documentation states it plainly: "Claude Code reads CLAUDE.md, not AGENTS.md." The documented fixes are cheap, so a repository does not have to choose between tools: - Create a `CLAUDE.md` that imports it — a line containing `@AGENTS.md` expands the file into Claude's context at session start, and Claude-specific rules can follow below it. - Or symlink instead of duplicating: `ln -s AGENTS.md CLAUDE.md` (on Windows a symlink needs Administrator rights or Developer Mode, so use the import). - Or run `/init` once, which reads AGENTS.md when `CLAUDE_CODE_NEW_INIT=1` is set, and `/import` (Claude Code v2.1.213+) which copies a supported agent's configuration into Claude Code. Two size limits are worth knowing because they are enforcement, not advice: the docs target under 200 lines per memory file and say Claude Code skips a file over 4 MiB. Imported files still load at launch, so splitting into imports organises a file without shrinking the context it consumes. ## What the Evidence Says About Context Files An ETH Zurich and LogicStar.ai preprint, ["On the Impact of AGENTS.md Files on the Efficiency of AI Coding Agents"](https://arxiv.org/abs/2601.20404) (revised June 2026), measured generated and developer-written context files against real GitHub pull requests. The result is unromantic: context files consistently increased the number of steps agents needed, developer-written files produced a marginal gain in task success, and LLM-generated files were marginally negative. The practical rule that follows is the same one that keeps llms.txt useful: write down only what the reader cannot infer. For a repository that means exact commands, non-obvious constraints and past failures. For a site it means a short summary and a curated list of the handful of pages that answer questions — not a mirror of your sitemap. Curation is the whole product in both cases. ## Which One Do You Actually Need? - **Public docs site plus a repo:** both files, and keep them separate — never paste an AGENTS.md into llms.txt or the reverse. - **Marketing site, no code:** llms.txt only. Draft it from your sitemap with the [generator](/generator) and validate before publishing with the [checker](/checker). - **Private repository, no public site:** AGENTS.md alone. There is nothing for external crawlers to read. - **Open-source project with hosted docs:** both — AGENTS.md for contributors' agents, llms.txt and its Markdown twins for everyone else's. Related reading: how llms.txt differs from the action layer in [llms.txt vs MCP](/blog/llms-txt-vs-mcp/), the [nine rules for a file AI engines trust](/blog/llms-txt-best-practices/), and which assistants actually fetch it, per [which AI engines read llms.txt](/blog/which-ai-engines-read-llms-txt/). --- # How to Get Cited by AI Search Engines: The 2026 Citation Playbook > Source: https://llmstxtgenerator.dev/blog/ai-search-citations/ GEO · September 2026 # How to Get Cited by AI Search Engines A citation is what happens when a model decides one page is the most useful thing it read this second. That decision is made upstream of your prose: in crawl eligibility, in the index the engine searches, and in whether a single paragraph of yours can be lifted without edits. ## Eligibility Comes Before Content Every AI answer pipeline has a door, and most sites are accidentally holding it shut. OpenAI publishes three user agents that do three different jobs, and confusing them is the most common technical error in this space: - **OAI-SearchBot** — builds the index behind ChatGPT's search and citation layer. Block it and you have removed yourself from consideration. - **GPTBot** — collects training data. Allowing it shapes whether your brand is part of the model's baseline knowledge, not whether you get cited today. - **ChatGPT-User** — fetches a specific URL live when a user's request triggers a visit. Block it and the model cannot open the page it was about to quote. Perplexity uses `PerplexityBot` and `Perplexity-User` in the same two roles, and Microsoft's `bingbot` plus IndexNow govern Copilot. The full allow/block decision table is in [robots.txt for AI crawlers](/blog/robots-txt-ai-crawlers/). ## Which Index Does Each Engine Actually Read? "I rank on Google" is not a retrieval strategy. Each assistant queries a different corpus, and a page missing from that corpus is invisible regardless of quality. Engine Retrieval layer Practical consequence ChatGPT Bing-fed index plus OAI-SearchBot Bing indexation is a prerequisite; verify in Bing Webmaster Tools Google AI Overviews Google's own index Classic technical SEO still applies; top-10 pages dominate citations Perplexity Its own real-time crawl Freshness and forum/news-adjacent sources carry unusual weight Claude Third-party search backend Being present in general web indexes matters more than any single file Microsoft Copilot Bing plus IndexNow Push updated URLs through IndexNow instead of waiting for a recrawl A Seer Interactive analysis of SearchGPT citations found roughly 87% matched Bing's top ten results, and a June 2025 Ahrefs study reported only around 14% overlap between the top sources cited by Google AI Overviews, ChatGPT and Perplexity. Treat every engine as a separate channel with its own index, not as one audience. The engine-by-engine fetcher audit in [which AI engines read llms.txt](/blog/which-ai-engines-read-llms-txt/) makes the same point from the crawler side. ## Write Passages a Model Can Lift Verbatim Answers are assembled from short spans, so the unit of optimisation is the paragraph, not the page. Four habits separate quotable pages from pages that get read but never mentioned: - **Answer in the first 40–60 words.** State the conclusion before the context. A model scanning for evidence needs a self-contained span, not a build-up. - **Keep one idea per paragraph,** with the entity named in full. "It reduces latency" is unciteable outside its page; "Cloudflare's content negotiation reduces latency" survives extraction. - **Make tables and lists carry data.** Comparisons, limits, thresholds and prices are the spans most often reproduced, because they are hard to paraphrase without loss. - **Ship numbers inside the sentence,** not only inside an image or a chart. If the figure is not in the text, it is not in the corpus. One striking data point cuts against the intuition that higher rankings always win: a June 2025 Semrush study found that nearly 90% of pages ChatGPT cited were ranking 21 or lower in classic search. Retrievability and quotability beat position, which is why passage formatting is worth more than another round of keyword tuning. ## Corroboration: The Part You Cannot Write Yourself Citation selection is biased toward sources the model has seen agree with each other. Aggregated audits consistently find that the cited mix for ChatGPT is dominated by brand pages, news and media, community threads and encyclopedia entries — a large share of which are not owned by the brand being discussed. The practical implication is unglamorous: digital PR, a coherent entity story, and an honest presence in the communities where your topic is debated do more for citation share than a rewrite of your own homepage. Entity consistency across the web — the same name, description and category everywhere — is what lets an engine decide your page and that third-party mention are about the same thing. That is the same alignment problem the [llms.txt and Schema.org](/blog/llms-txt-and-schema-org/) combination solves at the file level. ## Where llms.txt Fits — and Where It Does Not Be sceptical of anyone selling llms.txt as a citation lever. Controlled experiments that added the file and measured citation share reported no reliable lift, and crawler audits keep finding that the large majority of published llms.txt files are never requested. What the file genuinely does is two smaller jobs: it gives agent-style clients an accurate map of what your site contains, and it shortens the path from a fetched index to the right page. That makes it a hygiene item, not a strategy. Publish a short, accurate file, keep the descriptions specific enough to be quoted, and spend the remaining effort on the three things that move citations: crawl eligibility, presence in the engine's own index, and extractable paragraphs. The wording rules for the summary and link annotations are covered in [how to write llms.txt descriptions](/blog/llms-txt-descriptions/), and the measurement loop — logs, referrals and prompt tracking — is in [measuring whether AI engines read your file](/blog/measure-llms-txt-traffic/). ## A 30-Day Citation Check Run this in order; each step takes minutes and the fourth is the only one that changes week to week. ``` # 1. Can the citation crawlers reach you at all? curl -s https://example.com/robots.txt | grep -i -E "oai-searchbot|gptbot|perplexitybot" # 2. Is the page in the index ChatGPT actually searches? curl -sI https://example.com/pricing | grep -i -E "x-robots-tag|last-modified" # 3. Does your llms.txt describe the pages you want cited? curl -s https://example.com/llms.txt | head -20 # 4. Baseline twenty buyer prompts monthly and record whether your domain appears ``` Steps one to three fix eligibility and discovery; step four tells you whether any of it worked. Start from a validated file rather than a hand-typed one — the [llms.txt generator](/generator) produces a spec-compliant skeleton, and the [llms.txt checker](/checker) catches the broken links and delivery errors that quietly remove a file from consideration. --- # ai.txt vs llms.txt: Which AI File Should You Ship in 2026? > Source: https://llmstxtgenerator.dev/blog/ai-txt-vs-llms-txt/ Comparison · AI permissions · Updated 2026 # ai.txt vs llms.txt: Which AI File Should You Ship in 2026? Both files sit at your site root, both are plain text, and both get described as "the file that tells AI what to do with my site." They are not interchangeable. ai.txt is a permission file that restricts use; llms.txt is an index that invites reading. Mixing them up is the most common mistake in the AI-files conversation. ## The Short Answer ai.txt llms.txt Declares Permissions: may this content be mined or used for commercial AI training? Content: here is what the site contains and which pages matter Direction Restrictive — an opt-out at the media level Invitational — an opt-in to a better answer Origin Spawning AI, distributed through its API to partners including Hugging Face and Stability AI The Answer.AI proposal published at llmstxt.org Read when At the moment media is downloaded, including downloads triggered by old dataset links When an AI reader chooses to fetch your index before browsing Legal weight Pitched as a machine-readable opt-out aligned with the EU DSM Article 4 TDM exception None. It is a convention, not a standard, and never a rights reservation ## What ai.txt Actually Does ai.txt lives in your root directory and sets machine-readable permissions for **commercial text and data mining**. Spawning's generator exposes five media types — text, images, audio, video and code — each set to block or allow, and the default posture is opt-out. The interesting design decision is the timing. A robots.txt file is read when a crawler walks your site; ai.txt is read when your media is downloaded. That difference matters if your images already sit inside a public dataset: adding a robots.txt today cannot retroactively remove the links, but a permission file checked at download time can affect what happens the next time someone trains from those links. What ai.txt cannot do is reach copies of your work hosted on domains you do not control, and it only binds tools that actually check it. Treat it as a consent layer, not a lock. ## What llms.txt Actually Does llms.txt is a Markdown file at your site root that opens with an H1 site name, an optional blockquote summary, and a list of curated links with one-line descriptions. The 2026 revision of the spec added two Markdown link forms, defined subpath files, and two link relations — `rel="alternate"` and `rel="describedby"`. There is nothing in that spec about permission. llms.txt does not block a crawler, does not reserve rights, and does not change rankings. Its entire job is to shorten the path between an AI reader and the pages that answer questions. If you want the restriction side of the story, that lives in [the robots.txt token reference for AI crawlers](/blog/robots-txt-ai-crawlers/). ## The Legal Layer: TDMRep and Machine-Readable Opt-Outs The reason "just write it in your terms of service" stopped being advice: a German appeals court decision handed down in December 2025 (OLG Hamburg, 5 U 104/24) held that natural-language opt-outs buried in terms of use are insufficient, because a machine cannot read them. What counts is a machine-readable reservation. The W3C Community Group specification for that is the TDM Reservation Protocol. It publishes a JSON file at `/.well-known/tdmrep.json` where a reservation flag of 1 reserves rights under CDSM Article 4(3), and it also defines a `TDM-Reservation` HTTP response header as a header-only fallback: ``` { "tdm-reservation": 1, "tdm-policy": "https://example.com/ai-policy.html" } ``` One property in that file has no robots.txt equivalent: `tdm-policy` points at a URL where the actual policy lives, so the machine-readable signal and the human-readable explanation stay in sync. The community group's own documentation is explicit that robots.txt remains the right place to signal an _opt-in_ for search, and TDMRep is the opt-out that sits beside it — which is exactly what you would expect from a site that wants AI citations but not AI training. ## Where noai and X-Robots-Tag Fit The `noai` and `noimageai` directives, usually delivered as an `X-Robots-Tag` response header, are the third signal and the least formalised. They are not part of an IETF or W3C standard; the TDMRep group notes that `noai` was proposed but does not cover the full range of mining practices publishers want to address. Use them as an extra header on responses, not as your primary reservation. - **robots.txt** (RFC 9309) — crawl and search permissions, named AI tokens such as `GPTBot` or `ClaudeBot`. - **tdmrep.json** — purpose-level rights reservation under CDSM Article 4(3). - **ai.txt** — media-type permissions for commercial mining, read at download time. - **X-Robots-Tag: noai** — supplementary per-response signal, no standard behind it. ## What to Ship: A Decision List - **You want to be cited in AI answers:** llms.txt plus a sitemap. Draft it from your sitemap with the [generator](/generator) and run the result through the [checker](/checker) before publishing. Add nothing about permissions to this file. - **You publish original media or proprietary text:** add ai.txt or tdmrep.json. This is the decision that actually changes what happens to your content, so make it deliberately rather than by default. - **You are EU-facing and commercially minded:** lead with `/.well-known/tdmrep.json` and keep robots.txt as the search opt-in, per the community group's own guidance. - **You run a documentation site:** llms.txt is doing real work here, and permission files are largely irrelevant. Keep the index curated and the [expectations about which engines actually fetch it](/blog/which-ai-engines-read-llms-txt/) realistic. ## Verifying All Four Signals in Five Minutes Every file in this stack is a plain HTTP GET, so verification is one command per file. Replace the host and check the status line and the body — a 200 with an HTML error page is the classic failure mode: ``` curl -sI https://example.com/llms.txt | head -1 curl -sI https://example.com/ai.txt | head -1 curl -s https://example.com/.well-known/tdmrep.json curl -sI https://example.com/ | grep -i 'x-robots-tag\|tdm-reservation' ``` If the first line comes back as `text/html`, your rewrite rules are serving your catch-all page instead of the file — a delivery problem, not a content problem, and the same one that breaks most llms.txt launches. The [robots.txt vs llms.txt breakdown](/blog/llms-txt-vs-robots-txt/) covers which file belongs in which job if you are still deciding what to publish at all. --- # Does llms.txt Affect SEO? What Google Actually Says (2026) > Source: https://llmstxtgenerator.dev/blog/does-llms-txt-affect-seo/ SEO myth-busting · AI visibility · Updated 2026 # Does llms.txt Affect SEO? What Google Actually Says Short answer: **no — not for Google rankings.** But that's not the whole story. Here's the evidence, the nuance, and where llms.txt genuinely pays off in 2026. ## Google's Official Position Google has been consistent: it does not use llms.txt for ranking, indexing, or crawling decisions. In 2026, Google's search advocate John Mueller addressed the file directly, likening it to the `meta keywords` tag — a file that sounds authoritative but is outside Google's systems entirely. The official line is that Google will not treat llms.txt as a ranking signal. So if your goal is purely "higher Google rankings," llms.txt will not move the needle. No amount of H1 optimization or absolute URLs in the file changes Googlebot's behavior. ## Why the Confusion Exists llms.txt sounds like robots.txt, which _does_ affect Google. It sits in the same root directory, it's the same kind of plain-text file, and it's marketed as "the file for AI." The naming and placement create an intuitive leap — "if robots.txt controls crawlers and this is for AI, Google must care" — that the technical reality doesn't support. ## Where llms.txt Actually Matters Google is one consumer of your content. AI engines are increasingly the others: - **ChatGPT / Claude / Perplexity / Gemini browsing** — these products' agents check for llms.txt and use it to decide what to read about your site. When they answer "is X good for Y?", your curated index improves the odds the answer reflects your best content. - **AI-native discovery** — the major AI companies publish their own llms.txt files and have publicly supported the spec, making it an emerging standard for how sites describe themselves to agents. - **Developer tooling** — Chrome's Lighthouse (13.3+) includes an Agentic Browsing audit that checks for llms.txt, and documentation platforms like Mintlify and GitBook generate one automatically. It's becoming a baseline expectation for technical sites. The measurable reality is still modest: third-party analyses (e.g. Limy's study of billions of AI crawler requests) found most AI crawlers fetch HTML pages directly rather than llms.txt, and only about 10% of sites have adopted the file. It's an emerging signal with real but limited current penetration. ## The Honest 2026 Playbook 1. **Don't expect rankings.** Build llms.txt for AI assistants, not for Google. 2. **Do the cheap, real SEO first** — crawlable pages, fast Core Web Vitals, one clear topic per URL, and an accurate sitemap. That's what feeds Google's AI Overviews anyway. 3. **Ship llms.txt anyway** — it takes 10 minutes, costs nothing, and positions your site for the channel that's growing: AI-referenced answers. Use our [generator](/generator) to create it and the [validator](/checker) to keep it correct. 4. **Keep it current** — a stale llms.txt pointing at old pages is worse than none. Regenerate after major content changes. ## The Bottom Line llms.txt will not improve your Google rankings, and anyone selling it as a "Google ranking hack" is wrong. But it's a legitimate, low-cost piece of AI visibility infrastructure — the same way you'd claim your brand on emerging platforms early. Combined with solid [sitemap hygiene](/blog/llms-txt-vs-sitemap-xml/) and genuinely useful content, it makes your site easier for AI engines to represent accurately. That's a bet worth taking even when the payoff is a year out. --- # How to Create an llms.txt File: Step-by-Step Guide > Source: https://llmstxtgenerator.dev/blog/how-to-create-llms-txt/ llms.txt tutorial · 10-minute setup # How to Create an llms.txt File: Step-by-Step Guide An **llms.txt** file tells AI engines — ChatGPT, Claude, Gemini, Perplexity, and coding agents — exactly which pages of your site are worth reading. This llms.txt tutorial walks you from zero to a live, validated file in about **10 minutes**, no matter what your site is built with. ## Why Create an llms.txt File in the First Place? Every day, AI assistants answer questions with content from your site. Without an llms.txt file, those models have to guess what to crawl, what to skip, and which of your pages actually matter. The result: they often land on your marketing pages instead of your documentation, or miss your best content entirely. An llms.txt file fixes that by giving agents a small, curated Markdown index of your most important pages — the LLM equivalent of `robots.txt`, but for what _should_ be read instead of what shouldn't. The payoff is concrete: Chrome's Lighthouse now audits sites for an llms.txt file, documentation platforms like Mintlify and GitBook generate one automatically, and major AI labs publish their own. Sites with a valid file get referenced more accurately — and more often — in AI-generated answers. Creating one is one of the cheapest SEO moves available: a single text file, no plugins, no redesign. Here's the exact process. ## Step 1: Pick Your Site Name and Write a One-Line Summary The first two lines of your file are the most important. Line one is the **H1**: your site or project name, prefixed with a single `#`. Line two is a **blockquote** (a line starting with `>`) summarizing what the site is and who it's for, in one or two sentences. This is the only context an agent gets before deciding whether to read further, so be concrete: what you do, for whom, and what makes you different. ``` # Acme Corp > Acme Corp builds time-tracking software for small teams. Free plan available for up to 10 users. ``` Keep the summary short and factual — a one-sentence pitch works better than marketing fluff. The H1 here is the only element the llmstxt.org v2 specification marks as required, so never skip it. ## Step 2: List Your 5–15 Most Important Pages An llms.txt file is a curated index, not a sitemap. Resist the urge to list everything — a well-chosen 5–15 links beats 200 links that drown agents in noise. Include pages an AI should actually quote: product overviews, documentation, API references, pricing, setup guides. Skip boilerplate, thin blog posts, login pages, and anything behind an auth wall. Write down the full absolute URLs for your candidates: ``` # Candidate pages - https://acme.dev/product.md → Product overview - https://acme.dev/pricing.md → Plans and pricing - https://acme.dev/docs/quickstart.md → Quick start guide - https://acme.dev/docs/api.md → REST API reference - https://acme.dev/blog/ → Skip: low value for AI answers ``` Prefer clean Markdown versions of your pages when they exist (most documentation sites serve `.md` variants of every page — same URL with `.md` appended). Plain HTML pages work too, but Markdown is cheaper for LLMs to parse and less likely to be truncated. ## Step 3: Organize Everything in the llms.txt Format Now turn your list into the spec structure: H1 at the top, blockquote summary, then `##` sections (H2 headings) each containing a Markdown link list. Every link is written as `[text](url): short description` — the description after the colon is optional but recommended, because it tells agents what's behind each link without fetching it. Use a `## Optional` section at the end for secondary links agents can skip when context is short. Here's the complete skeleton: ``` # Acme Corp > Acme Corp builds time-tracking software for small teams. ## Important links - [Product overview](https://acme.dev/product.md): What Acme does and why teams choose it - [Pricing](https://acme.dev/pricing.md): Plans, per-seat pricing, and the free tier ## Documentation - [Quick start](https://acme.dev/docs/quickstart.md): Set up Acme in 10 minutes - [API reference](https://acme.dev/docs/api.md): Complete REST API reference ## Optional - [Changelog](https://acme.dev/changelog.md): Release notes for all versions ``` Three rules keep the file machine-readable: all URLs must be absolute (`https://…`, never `/relative/paths`), there is no comment syntax (a line starting with `#` is a heading, so don't use hash comments), and H2 sections must contain lists of links — not prose. Prose belongs in the free-form notes area between the blockquote and the first `##`. ## Step 4: Generate the File Automatically (or Write It by Hand) If you've already followed Steps 1–3, writing the file by hand takes two minutes — it's plain Markdown, nothing more. That said, hand-writing is exactly where mistakes creep in: missing H1s, relative URLs, sections without descriptions. If you want a spec-perfect file in seconds, paste your site URL into our [free llms.txt generator](/generator) — it crawls your most important pages, builds the formatted file for you, and lets you tweak the section names and descriptions before downloading: ``` # Generated by llmstxtgenerator.dev # Acme Corp > Acme Corp builds time-tracking software for small teams. ## Important links - [Product overview](https://acme.dev/product.md): ... - [Pricing](https://acme.dev/pricing.md): ... ``` Either path produces the same plain-text file. The generator is fastest for large or frequently changing sites; writing by hand gives you full control over tone and link order. For a first file, we recommend the generator, then manual edits — the combination of speed and control is hard to beat. Save the result as `llms.txt` (lowercase, no extension, UTF-8 encoding). ## Step 5: Put the File at Your Website's Root The file must be publicly reachable at `https://your-domain.com/llms.txt`. Exactly where that file lives in your project depends on your platform — here's where to place it for the most common setups: Platform Where to place llms.txt **WordPress** Upload to `public_html/llms.txt` via the file manager or FTP — the web root, not your theme folder. **Next.js** Add `llms.txt` to the `public/` folder; Next.js copies it to the site root on every build. **Hugo** Put the file in `static/llms.txt` — Hugo copies everything in `static/` verbatim to the site root. **Shopify** Upload the file via Files, then create a URL redirect from `/llms.txt` to the uploaded file URL so the standard path works. **Cloudflare Pages** Add `llms.txt` to your build output folder (e.g. `public/` for Astro) — it ships to the site root automatically. On a static host, the deployment step is usually as simple as dropping the file next to your `index.html` and redeploying — for example: ``` cp llms.txt public/llms.txt # then rebuild / redeploy as usual ``` ## Step 6: Verify the File Is Actually Reachable A file that isn't publicly accessible is the same as no file at all — and this is the step most people skip. Open a browser (or an incognito window) and visit `https://your-domain.com/llms.txt`. You should see your plain-text file, not a 404, a redirect, or a login page. From the terminal, the equivalent check is: ``` curl -I https://your-domain.com/llms.txt # Expect: HTTP/2 200 and content-type: text/plain ``` A `200` status with a `text/plain` content type is exactly what you want. If you get a `404`, double-check the file name (it's `llms.txt`, not `llms.txt.txt` or `Llms.txt`) and the deploy path from Step 5. A redirect is acceptable on platforms like Shopify, but a 404 isn't. ## Step 7: Validate the File With the llms.txt Checker The last step is a quality gate. Paste the URL or file contents into the [free llms.txt checker](/checker), which validates the structure against the llmstxt.org v2 specification — required H1, absolute URLs, correctly formed sections — and scores AI-readiness: ``` Checking https://your-domain.com/llms.txt ... ✔ H1 heading present ✔ Blockquote summary found ✔ 3 sections, all with link lists ✔ All 9 links use absolute URLs ✔ 0 errors — structure score: 10/10 ``` Fix anything the checker flags, redeploy, and re-check. Once you see zero errors, you're done: agents will start finding the file the next time they crawl your domain. That's the whole process — seven steps, ten minutes, one small text file. ## Hand-Written or Generated: Which llms.txt Should You Use? Both approaches are valid, and the file format is identical either way — so this is a workflow choice, not a quality one. Hand-writing gives you total control over wording and ordering, which matters if your descriptions need a human editorial touch. The generator wins on speed, completeness, and consistency: it never forgets the H1, never emits a relative URL, and regenerates in seconds when your site changes. The pragmatic pattern is a hybrid: generate an initial file, review and polish the descriptions, then keep the generator around for monthly refreshes. If you're creating files for multiple sites or products, the generator's consistency alone is worth it. ## Common llms.txt Creation Mistakes to Avoid Most problems in hand-written files come down to a handful of recurring mistakes: - **Missing H1.** Starting the file with prose or an H2 means the required title element is gone. Always begin with `# Site Name`. - **Relative URLs.** `- [Docs](/docs)` breaks parsers and agents. Every link must be an absolute `https://…` URL. - **Treating it like a sitemap.** Listing hundreds of pages dilutes the file's value. Curate: 5–15 links that actually help agents answer questions about you. - **Hash comments.** A line like `# TODO: add pricing` is parsed as a heading. There is no comment syntax — use HTML comments (``) in the notes area if you must annotate. - **Links without descriptions.** Bare links force agents to fetch each page to learn what it is. Add a `: short description` to every item. - **Never re-checking.** Links go stale, pages get renamed, sections disappear. Set a monthly reminder to re-run the checker, or regenerate from scratch after any redesign. ## llms.txt Creation FAQ ### How long does it take to create an llms.txt file? Around 10 minutes for a typical site: 2 minutes to pick your site name, summary, and top pages, 2 minutes to format them, 3 minutes to deploy, and 3 minutes to verify and validate. Using the generator cuts the formatting step to seconds. ### Do I still need an llms.txt file if I have a sitemap.xml? Yes — they serve different purposes. A sitemap lists every URL for search-engine crawlers; llms.txt is a curated, prioritized index written in Markdown for LLMs. Agents can consume both, but llms.txt tells them what matters, which is what produces accurate AI answers. ### Can I create an llms.txt file for a WordPress or Shopify site? Absolutely. On WordPress, upload the file to `public_html/`. On Shopify, upload it via Files and add a URL redirect from `/llms.txt`. Both make the standard `https://your-domain.com/llms.txt` URL work — see the placement table in Step 5 for details. ## Keep reading [ ### 📐 The Complete Format Guide Every element of the v2 spec explained, with a valid end-to-end example. ](/blog/llms-txt-format/)[ ### 🔌 llms.txt on WordPress Three ways to ship llms.txt on WordPress, plus the pitfalls that break it. ](/blog/llms-txt-wordpress/) ## Ready to create your llms.txt file? Generate a spec-perfect file in under a minute, then validate it with the checker — free, no sign-up. [⚡ Generate your llms.txt](/generator) [Check my file](/checker) --- # How to Validate an llms.txt File: Spec, Links and Delivery (2026) > Source: https://llmstxtgenerator.dev/blog/how-to-validate-llms-txt/ Validation · September 2026 # How to Validate an llms.txt File: Spec, Links and Delivery Validation is where most llms.txt projects quietly go wrong. A file can be pasted into a validator, come back clean, and still never be read — because the text of the file and the way the file is served are two different problems. Here is the three-layer check worth running on every llms.txt you publish. ## What "Valid" Actually Means The [specification](https://llmstxt.org/) is deliberately short: one H1 with the site name, a blockquote summary, optional free-form details, then H2-delimited lists of links in `name: description` form. Of those, only the H1 is genuinely required. That minimalism is why three different layers of checking exist, and why a file that passes one can fail another. Layer Question it answers How to test it **1\. Structure** Does the file follow the format? Parser / validator, or a markdown lint pass **2\. Link integrity** Does every listed URL resolve to the page you meant? HTTP status check on each link **3\. Delivery** Can a crawler actually fetch it as text? Headers, content type, cache and WAF behaviour ## Layer 1: Structure — the Checks a Parser Can Make These are mechanical and objective. A file passes or it does not: - **Exactly one H1, on the first line.** Multiple H1s, or a heading that arrives after the summary, confuse the parse. - **A blockquote summary.** Optional in the spec, but the single highest-value line in the file — it is the context an assistant reads before deciding whether to keep going. - **Links as list items, with display text and an absolute URL.** `- [Docs](https://example.com/docs): API reference` is valid; a bare path or a bullet with no link is a formatting error. - **No prose in link sections.** Paragraphs between the H2 sections are where parsers report unrecognised lines — put narrative text above the first H2. - **`Optional` used for the right thing.** That H2 is reserved for resources that can be skipped when a consumer is short on context. Filling it with your best pages defeats the point. ``` # Acme Docs > Acme is the developer documentation site for the Acme API, > with REST and GraphQL references, SDKs and migration guides. ## Important links - [Home](https://acme.com/): Documentation home - [Quick start](https://acme.com/getting-started): 5-minute setup - [API reference](https://acme.com/api): Complete REST reference ## Optional - [Changelog](https://acme.com/changelog): Release history ``` If you would rather not eyeball it, the [llms.txt validator](/checker) runs this layer and more: it flags a missing H1, an H1 that is not on the first line, multiple H1s, links with no display text, relative URLs, unrecognised lines and a file with no links at all, then returns a 0–100 AI-readiness score so you can see progress rather than a binary verdict. ## Layer 2: Link Integrity — the Check Nobody Runs Structure checks never touch your URLs. A file can be spec-perfect and full of 404s, redirect chains and pages that moved six months ago — see [llms.txt maintenance](/blog/llms-txt-maintenance/) for why that decay is the normal state of a published file. The audit is a loop: ``` # 1. pull the file curl -s https://example.com/llms.txt -o /tmp/llms.txt -w '%{http_code} %{content_type}\n' # 2. status of every absolute URL in it grep -oE 'https?://[^)]+' /tmp/llms.txt | sort -u | while read -r u; do printf '%s %s\n' "$(curl -s -o /dev/null -w '%{http_code}' -L "$u")" "$u" done # 3. anything still redirecting or gone grep -oE 'https?://[^)]+' /tmp/llms.txt | sort -u | while read -r u; do curl -sI -o /dev/null -w '%{http_code} %{url_effective}\n' "$u" done ``` - A **404** means the entry is worse than nothing: the consumer spent a fetch and learned nothing. - A **301 or 302** where the effective URL differs from the one you listed means you are sending agents through a detour. List the final URL. - **Duplicates** — the same page under `www` and non-`www`, or with tracking parameters — split your authority across entries that look like separate resources. - A **gated or login-walled** link is a dead end. If the page needs a session, describe it in the file but link to the public overview instead. ## Layer 3: Delivery — Where Most Failures Hide The last layer is the one a text validator cannot see, because it never leaves the file. Four checks catch almost everything: - **Path.** The file must be reachable at `/llms.txt` on the host that serves your pages. A copy on a `cdn.` subdomain or a docs subdomain only covers the URLs under that subdomain. - **Content type.** `text/plain` or `text/markdown`. A catch-all rewrite that answers `text/html` is the classic failure, and it usually means an SPA fallback swallowed the path. - **Status.** 200, directly — not a 302 to the homepage. Redirects to marketing pages are counted as missing files by anything doing a strict fetch. - **Content.** Confirm the bytes you published are the bytes being served. Edge caches happily serve the previous version of a file that changed an hour ago. Each of those has a specific curl test and a specific fix — the [delivery-level troubleshooting guide](/blog/llms-txt-not-working/) walks through all seven variants, from WAF rules to case-sensitive paths. ## Make Validation Part of the Build, Not a Launch Task A file validated once at launch is a file that is correct for one day. Put the cheapest layer in CI so a rename or a deleted page fails the build instead of silently rotting in production: ``` - name: Validate llms.txt run: | test -f public/llms.txt || exit 1 head -1 public/llms.txt | grep -qE '^# ' || exit 1 node scripts/check-llms-links.mjs public/llms.txt ``` Keep the link check non-blocking at first while you clean up the existing entries, then make it blocking once the file is green. It is the same rhythm described in the [six llms.txt mistakes](/blog/llms-txt-mistakes/) that get a file ignored — only this time it is enforced by tooling rather than memory. ## What a Validator Cannot Tell You Two things, and both matter more than the mechanical checks: 1. **Whether your summary is any good.** A grammatically valid blockquote can still be a slogan. Rewrite it as a factual description of what the site covers and who it is for. 2. **Whether you listed the right pages.** A perfectly formatted file of your ten least important URLs passes every check and helps nobody. That judgement is editorial, and it is what separates a [trusted file](/blog/llms-txt-best-practices/) from a compliant one. Use the [generator](/generator) to get a structurally valid first draft from your sitemap, then cut it down by hand before publishing. Validation is the last step of that workflow, not the first. --- # llms-full.txt Size and Structure: How to Build the Full File Without Breaking a Context Window (2026) > Source: https://llmstxtgenerator.dev/blog/llms-full-txt-size-and-structure/ Format & Spec · October 2026 # llms-full.txt Size and Structure: The Full File Nobody Specified Everyone references `llms-full.txt`. The specification does not. We fetched the file from eight live documentation sites on **October 11, 2026** and measured every byte: the largest is **67.6 MB** before compression, the smallest still 433 KB, and four of the domains that serve one never mention it in their own `/llms.txt`. ## The Spec Never Defined llms-full.txt The v2 proposal at `llmstxt.org`, last modified **August 10, 2026**, describes the `/llms.txt` file, markdown twins served at the page URL with a `.md` extension, and two link relations — `rel="alternate" type="text/markdown"` for the twin and `rel="describedby"` for the index that covers a page. We searched the page for the string `llms-full` on October 11, 2026: **zero matches**. That absence is deliberate. The spec states the index "stays small enough to fit in context. The detail lives behind the links, and is fetched only when needed." A single file holding every page of a large docs site is the opposite design, and it survives as a convention rather than a requirement — which is exactly why a client that implements only the written spec will never look for one unless your index points at it. The comparison between the curated index and the full dump is in [llms.txt vs llms-full.txt](/blog/llms-txt-vs-llms-full-txt/). ## What Eight Live Full Files Actually Weigh Every request followed redirects and timed out at 30 seconds. Sizes are uncompressed bytes on the wire after decoding: Host and path Bytes gzip transfer Structure signal developers.cloudflare.com/llms-full.txt 67,649,033 11,527,423 29,052 H2 headings; per-section files exist platform.claude.com/docs/llms-full.txt (301 from docs.anthropic.com) 43,107,672 — starts with a "Root URL" page, then full rendered docs nuxt.com/llms-full.txt 4,893,417 1,291,540 3,586 H2 headings ai-sdk.dev/llms-full.txt 4,724,729 — byte-identical to /llms.txt on the same host vitest.dev/llms-full.txt 1,382,071 371,248 215 `url: /` page markers svelte.dev/llms-full.txt 1,261,050 357,726 220 H1 page headings under a SYSTEM preamble vuejs.org/llms-full.txt 953,556 314,771 94 `url: /` page markers vite.dev/llms-full.txt 432,841 127,916 42 `url: /` page markers Four hosts we probed do not serve one at all: **docs.stripe.com** returned 404 with 23,881 bytes of HTML, **www.anthropic.com** returned 404 with 20,584 bytes of an error page, **openai.com** returned 403 with 9,698 bytes, and **docs.astro.build** answered 404 with no content type and a zero-byte body. ## Compression Buys Four to Six Times, Not Enough All six hosts that served a real file sent `content-encoding: gzip` when we offered it. The saving is real and consistent — Cloudflare's 67.6 MB arrives as 11.5 MB, Nuxt's 4.9 MB as 1.29 MB, Vite's 433 KB as 128 KB. On a character-based estimate of bytes divided by four, that still leaves the two largest files at roughly **16.9 million and 10.8 million tokens** respectively, with Nuxt and the AI SDK near 1.2 million each. Treat that arithmetic as planning, not measurement: it is a byte-to-token approximation, not a tokenizer run, and the ratio shifts with code blocks and non-English text. The direction is never in doubt. Any file in the seven-digit token range can only be read in slices, so the file has to be built to be sliced. The size-versus-verdict table for the index file itself is in [llms.txt size limits](/blog/llms-txt-size-limit/). ## Split It the Way Cloudflare Does The 67.6 MB root file is not the only option that host publishes. Its product sections each carry their own full file, and all three we probed returned 200: - `/workers/llms-full.txt` — 5,127,319 bytes - `/learning-paths/llms-full.txt` — 1,841,842 bytes - `/ai-gateway/llms-full.txt` — 910,239 bytes That is the pattern worth copying, and it is the one the spec's subpath rule already supports: an index at a path covers the URLs beneath it, and the most specific file wins. A reader that needs the gateway docs fetches 910 KB instead of 67 MB. Structure it as one file per product or per top-level section, keep the root full file optional, and give each section its own index so the split is discoverable. The same reasoning applies to a markdown-twin pipeline, which is where these files come from — see [llms.txt for API documentation](/blog/llms-txt-for-api-docs/). ## Only Two of Six Index Files Mention Their Own Full File We pulled `/llms.txt` from each of the six hosts that serve a full file and counted references to it: - **nuxt.com** — three mentions, including a prose line: "Every documentation, blog and deploy page is also Markdown" followed by the full-file URL and a note that other versions are listed in full there. - **svelte.dev** — one entry, "Complete documentation", describing it as all Svelte and SvelteKit docs with examples. - **developers.cloudflare.com, vuejs.org, vite.dev, vitest.dev** — zero mentions of `llms-full` in the index file. Nuxt's wording is the standard to aim for, because it tells a reader both _what_ the file is and _when_ to prefer it. And **ai-sdk.dev** shows the failure mode at the other extreme: its `/llms.txt` serves 4,724,729 bytes with the same MD5 (`eebec45d272ac9891b3c5be10912c124`) as `/llms-full.txt` on the same host, in `text/markdown`. The path meant for a curated index is a 1.2-million-token dump, so an agent that politely checks the conventional URL first pays the full price before it learns anything. ## Three Commands Before You Publish One The checks that matter are size, encoding and what the index actually links: ``` # 1. Size and content type of the full file curl -sSL -o /dev/null -w '%{http_code} %{size_download} %{content_type}\n' \ https://www.example.com/llms-full.txt # 2. What gzip actually costs a reader (send Accept-Encoding explicitly) curl -sSL -H 'Accept-Encoding: gzip' -o /dev/null \ -w 'wire=%{size_download} encoding=%{content_type}\n' \ https://www.example.com/llms-full.txt # 3. Does the index point at it, and how many pages are inside? curl -sSL https://www.example.com/llms.txt | grep -ci 'llms-full' grep -c '^url: /' llms-full.txt ``` The third line is the one that finds most problems: either the index is silent and the full file is effectively invisible, or the count is in the thousands and the file should have been split. Build the index in the [generator](/generator), verify paths and link targets in the [checker](/checker), then measure the delivered bytes rather than trusting the file in your repository. The verdict: `llms-full.txt` is a convention nobody wrote down, published at sizes that no context window holds, and referenced by only a third of the sites that pay to serve it. Ship it per section, describe it in one honest line in the index, and let the reader choose the 910 KB slice over the 67 MB whole. --- # llms.txt and Schema.org Structured Data: Do You Need Both? (2026) > Source: https://llmstxtgenerator.dev/blog/llms-txt-and-schema-org/ Structured data · GEO · Updated 2026 # llms.txt and Schema.org: Do You Need Both? Two signals get filed under one label — "machine-readable data for AI" — and then treated as interchangeable. They are not. One is a site-level index an agent reads to decide where to go; the other is page-level vocabulary that tells a parser what a page is. Mixing them up is how teams end up stuffing JSON into a Markdown file. ## The Two Layers, Side by Side llms.txt Schema.org JSON-LD Question it answers Which pages on this site are worth my context window? What is this page, and which entity does it describe? Scope One file covering a site or a path Per page, per entity — usually one or a few nodes Where it lives A Markdown file at the site root or a subpath Inside the HTML of the page it describes Who consumes it Coding agents and assistants with retrieval, plus Lighthouse's agentic browsing audit Search engines and any parser that extracts entities The shorthand: **llms.txt is the map, schema is the labeling system on it.** Neither substitutes for the other, and neither is a ranking tactic — that debate is settled in [does llms.txt affect SEO](/blog/does-llms-txt-affect-seo/). ## What the Spec Actually Says About Structured Data Most "llms.txt plus schema" advice floating around is inference. There is one line of spec to anchor on. In the "Existing standards" section of the v2 proposal (last modified 10 August 2026), Jeremy Howard writes that the file "can also reference structured data markup used on the site, helping LLMs understand how to interpret this information in context." That is a suggestion, not a requirement — and it is deliberately narrow. The proposal is not saying _embed your JSON-LD in llms.txt_. It is saying the index may point at markup so an agent knows what it is looking at. The same v2 text formalises the two link relations that connect files to pages: `rel="alternate" type="text/markdown"` for a page's Markdown twin, and `rel="describedby"` for the llms.txt file that covers it. ## 2026 Sharpened the Division of Labour Two 2026 announcements confused a lot of people into thinking one layer had won. Both are true, and both point the same way. - **7 May 2026 — Google retired the FAQ rich result.** A deprecation notice went up at the top of the FAQ structured data documentation: rich results stopped appearing in Search, Search Console reporting and Rich Results Test support followed in June, and API support ends in August. The schema type itself was not deprecated: FAQPage remains valid, and Google states that unused structured data does not cause problems. - **15 June 2026 — Google's AI optimization guide called out file-based signals.** Its mythbusting section says machine-readable files like llms.txt are not needed to appear in generative AI features and have no effect on Search rankings or AI Overviews. Read together, the message is not "structured data died." It is that the visible reward attached to both shrank while the plumbing stayed useful. Keep the measured reality in view: an Ahrefs study of 137,210 domains found 97% of llms.txt files got no requests at all in May 2026 — see [which AI engines read llms.txt](/blog/which-ai-engines-read-llms-txt/). ## The Consistency Rules That Actually Matter Where the two touch, they should agree. Four rules cover almost every real conflict: 1. **Entity name.** The H1 of your llms.txt should match the `name` in your Organization node, character for character — "Acme Analytics" and "Acme Analytics, Inc." are two entities to a parser. 2. **Canonical URLs.** Every link in llms.txt should be the canonical form — no tracking parameters, no trailing-slash drift. If your schema uses `@id`, reuse those exact URLs. 3. **One summary, one story.** The blockquote summary and the schema `description` should describe the same product the same way. A stale llms.txt promising a free tier your pricing schema dropped is the contradiction that makes an agent hedge. 4. **Only pages that exist.** Schema on a live page promises the URL resolves; so does every link in the index. Broken pointers age both files at once. ## Three Ways to Connect Them 1. **Add a "Machine-readable data" section to llms.txt.** The spec-endorsed move: a short list pointing at sitemap.xml, the llms.txt file itself, and the pages carrying your key structured data. 2. **Seed the header from the same CMS fields that emit JSON-LD.** If site name, one-line description and canonical URL already drive an Organization node, they can drive the H1 and blockquote too. One source of truth, two outputs. 3. **Reuse one curated list.** The 10–50 pages you pick for llms.txt are the same pages worth an `ItemList` node. Curate once. ## A Matching Pair You Can Copy Same entity, two formats, no contradictions: ``` { "@context": "https://schema.org", "@type": "Organization", "name": "Acme Analytics", "url": "https://acme.example/", "description": "Product analytics for small SaaS teams.", "sameAs": ["https://github.com/acme"] } ``` ``` # Acme Analytics > Product analytics for small SaaS teams: event tracking, funnels and retention > reports, with setup docs and an API reference. ## Docs - [Quickstart](https://acme.example/docs/quickstart.md): Install the SDK and send a first event - [API reference](https://acme.example/docs/api.md): Every endpoint and response body ## Machine-readable data - [Sitemap](https://acme.example/sitemap.xml): Every indexable page - [llms.txt](https://acme.example/llms.txt): This file - [Organization details](https://acme.example/about.md): The facts published in our JSON-LD node ``` ## What Not to Do - **Don't embed JSON-LD inside llms.txt.** It is a Markdown format; a wall of JSON makes the file harder for the reader it was built for. Reference markup, don't inline it. - **Don't delete schema expecting a ranking win.** Removing FAQPage markup after May 2026 changes nothing in Search; removing _inaccurate_ markup is worth doing anyway. - **Don't expect markup to make agents fetch llms.txt.** No provider documents such a mechanism, and most published files still see almost no traffic. ## A 20-Minute Checklist 1. Fetch `/llms.txt` and your homepage's JSON-LD. Diff the entity name, canonical URL and description. 2. Fix the H1 so it matches your Organization `name` exactly. 3. Add the "Machine-readable data" section with three or four links. 4. Run the file through the [checker](/checker). Validate the JSON-LD with the Schema Markup Validator — FAQ support left the Rich Results Test in June 2026. 5. Re-check quarterly, or whenever URLs change. Starting fresh? The [generator](/generator) builds a spec-compliant file from your sitemap in seconds. --- # How to Add llms.txt to an Astro Site (Static or Generated, 2026) > Source: https://llmstxtgenerator.dev/blog/llms-txt-astro/ Astro tutorial · Static + generated · 6 minutes # How to Add llms.txt to an Astro Site Astro has no llms.txt file convention — and it is one of the few frameworks that genuinely does not need one. Between the `public/` directory and `src/pages` endpoints, there are three clean ways to ship a spec-compliant `llms.txt`, from a 30-second static file to a fully generated index. ## Why Astro and llms.txt Are a Natural Fit The llms.txt convention exists so AI engines and coding agents can find the handful of pages that matter on your site. Astro sites are usually built from Markdown and frontmatter — exactly the structured content llms.txt wants to point at. On top of that, Astro's default output is static: whatever the build produces is what your CDN serves, which makes a root-level text file trivial. Two Astro behaviors do all the work. First, every file in `public/` is copied verbatim into the output root, so `public/llms.txt` becomes `/llms.txt` automatically. Second, any `.ts` file in `src/pages/` becomes a route, so `src/pages/llms.txt.ts` serves plain text at the same path. This very site — llmstxtgenerator.dev — is an Astro 5 site and serves its own [llms.txt](/llms.txt) from the `public/` directory. ## Method 1: Static File in public/ (30 Seconds) Create `public/llms.txt` with your H1, a blockquote summary, and curated sections of markdown links. This follows the [llmstxt.org v2 spec](/blog/llms-txt-v2/): ``` # Example Docs > Example Docs is the official documentation for the Example platform. > It covers installation, configuration, the API and troubleshooting. ## Getting Started - [Quickstart](https://example.com/docs/quickstart): first project in 5 minutes - [Installation](https://example.com/docs/install): system requirements and setup - [Authentication](https://example.com/docs/auth): API keys and access tokens ## API Reference - [REST API](https://example.com/docs/api): every endpoint, parameter and error - [Webhooks](https://example.com/docs/webhooks): events your server can subscribe to ``` Run `npm run build` and check the output — the file appears at `dist/llms.txt`, ready for any static host. Deploying to Cloudflare Pages, Netlify or GitHub Pages requires no server config at all. Keep this method when your important links rarely change; move to Method 2 when they do. ## Method 2: Generate It at Build Time from a .ts Endpoint For a file that should stay in sync with your site, create `src/pages/llms.txt.ts`. Astro maps the filename to the `/llms.txt` route, and on a static site the endpoint runs during the build — the result is a real file on your CDN, with nothing executed at request time: ``` // src/pages/llms.txt.ts import type { APIRoute } from 'astro'; const pages = [ ['Quickstart', 'https://example.com/docs/quickstart', 'first project in 5 minutes'], ['Authentication', 'https://example.com/docs/auth', 'API keys and access tokens'], ['REST API', 'https://example.com/docs/api', 'every endpoint, parameter and error'], ]; export const GET: APIRoute = () { const lines = [ '# Example Docs', '', '> Example Docs is the official documentation for the Example platform.', '', '## Documentation', '', ]; for (const [name, url, summary] of pages) { lines.push('- ' + name + ': ' + summary + ' (' + url + ')'); } return new Response(lines.join('\n') + '\n', { headers: { 'Content-Type': 'text/plain; charset=utf-8' }, }); }; ``` Two details matter. The `Content-Type` must be `text/plain` — agents and validators expect plain text, not HTML. And if you later add an adapter for server-side rendering, add `export const prerender = true` to keep generating the file at build time instead of on every request. To go further, pull from your content collections instead of a hard-coded array. Import `getCollection` from `astro:content`, map each post's frontmatter title, description and URL into a markdown link line, and join them. Your `llms.txt` then updates itself every time you publish content. ## Method 3: The astro-slop Integration (Maintenance-Free) If you run a large docs or blog site, the community [astro-slop](https://github.com/yaroslav/astro-slop) integration automates the whole pipeline. It generates an `llms.txt` index from your pages, per-page Markdown versions of every route, a companion `llms-full.txt`, and adds content negotiation plus `rel="alternate"` link headers. It is the closest thing to a "just works" llms.txt setup for Astro today — worth a look before you hand-roll Method 2 on a big site. ## Astro-Specific Checklist - **One source of truth** — a `public/llms.txt` file and a `src/pages/llms.txt.ts` route that both target /llms.txt will drift apart. Pick one. - **Absolute URLs only** — `https://example.com/docs/start`, never `/docs/start`. Agents resolve links from the file's location. - **Serve text/plain** — endpoints must set `Content-Type: text/plain`; a validators may reject HTML responses. - **Curate, don't dump** — 10–50 links to canonical docs pages beat 500 links to marketing pages. See the [format guide](/blog/llms-txt-format/) for structure rules. - **Add llms-full.txt for deep docs** — if your reference section is large, link a generated [llms-full.txt](/blog/llms-txt-vs-llms-full-txt/) companion file. - **Keep it fresh** — rebuild and redeploy whenever a listed page changes; stale files erode agent trust. The [best practices guide](/blog/llms-txt-best-practices/) covers maintenance cadence. ## Verify and Deploy After deploying, confirm the route serves plain text with a 200 status: ``` curl -s -o /dev/null -w "%{http_code} %{content_type}\n" https://your-domain.com/llms.txt # Expect: 200 text/plain ``` If you host on Cloudflare Pages, our [Cloudflare Pages deployment guide](/blog/llms-txt-cloudflare-pages/) covers the build configuration. Then paste your URL into the free [llms.txt checker](/checker) to validate the H1, the blockquote summary, absolute URLs and section structure before AI engines start reading it. Build the initial curated file in seconds by running your sitemap through the [free llms.txt generator](/generator), then validate the result with the [checker](/checker) before deploying. A correct, current `llms.txt` is one of the highest-leverage AI visibility changes most Astro sites can make this week. --- # llms.txt Best Practices 2026: The 9 Rules for a File AI Engines Trust > Source: https://llmstxtgenerator.dev/blog/llms-txt-best-practices/ Spec guide · AI readiness · Updated 2026 # llms.txt Best Practices 2026: The 9 Rules for a File AI Engines Trust The llms.txt spec is short — one required H1, a few conventions — but the difference between a file agents use and a file they ignore is in the details. These nine rules cover format, curation, and maintenance. ## 1\. Start With the H1 — It's the Only Required Element The first line of llms.txt must be a single Markdown H1 with your site name. Everything else is optional by the spec — but a file with no H1 fails the very first check an agent (or a validator) runs. ``` # Example Site ``` ## 2\. Write a One-Line Summary as a Blockquote Right under the H1, a `>` blockquote telling agents what the site is and who it's for. This is the context an AI engine uses before reading anything else — make it specific, not marketing-speak. ``` # Example Site > A SaaS API documentation site for developers building with Example SDK. ``` ## 3\. Curate, Don't Dump 5 to 15 links. The whole point of llms.txt is that an agent can read it in milliseconds and know what matters. If your site needs more material, publish [llms-full.txt](/blog/llms-txt-vs-llms-full-txt/) and link to it — don't turn the index into a sitemap. ## 4\. Use Absolute URLs — Always Every link must be a full `https://your-domain.com/…` URL. Relative paths like `/docs/start` or protocol-relative `//example.com/x` break agents that fetch the file directly. The one allowed exception is the relative link to `llms-full.txt` described in the spec. ## 5\. Group Links With H2 Sections Use `##` headings to group related links — Documentation, Blog, Pricing, Support. Agents use these sections to match the user's question to the right part of your site. ``` ## Documentation - [Getting Started](https://example.com/docs/getting-started) - [API Reference](https://example.com/docs/api) ## Blog - [How We Built X](https://example.com/blog/how-we-built-x) ``` ## 6\. Order Links by Importance The spec has no ranking mechanism beyond order — so your most valuable pages go first within each section. First impressions matter for agents the same way they do for humans. ## 7\. Keep Link Text Descriptive Use link text that says what the page is ("Getting Started Guide"), not the URL or generic words ("click here"). Agents quote link text when they cite your site. ## 8\. Maintain It Like Code - Regenerate after major content changes (new flagship pages, removed sections). - Validate after every change — our [free validator](/checker) scores spec compliance and flags broken structure. - Generate from your sitemap with the [generator](/generator) when you add many pages at once. - Serve it as `text/plain` at the site root with no auth, no redirects. ## 9\. Know What It Is — and Isn't llms.txt is an AI-visibility signal, not a [Google ranking factor](/blog/does-llms-txt-affect-seo/). Don't neglect robots.txt, sitemap, and on-page SEO because you have it — run all three together ([see how they compare](/blog/llms-txt-vs-robots-txt/)), and you're covered for search crawlers _and_ AI agents. ## A Complete Reference Example ``` # Example Site > A SaaS API documentation site for developers building with Example SDK. ## Documentation - [Getting Started](https://example.com/docs/getting-started) - [Authentication](https://example.com/docs/auth) - [API Reference](https://example.com/docs/api) ## Blog - [How We Built X](https://example.com/blog/how-we-built-x) - [Pricing Philosophy](https://example.com/blog/pricing) [llms-full.txt](llms-full.txt) ``` That's a file an agent can act on in under a second. Build yours with the [generator](/generator), check it with the [validator](/checker), and keep it honest. --- # llms.txt Case Studies 2026: Inside the Files of Vue.js, Nuxt, Cloudflare, OpenAI and Anthropic > Source: https://llmstxtgenerator.dev/blog/llms-txt-case-studies/ Case studies · Real production files · September 2026 # llms.txt Case Studies: Inside Real Production Files Specs and templates tell you how llms.txt _should_ look. Real files show you how teams actually ship it. We fetched six production llms.txt files with curl on September 10, 2026 and broke down exactly what Vue.js, Nuxt, Cloudflare, OpenAI and Anthropic put in theirs — and what you can copy. ## Six Production Files, Verified Live Every file below was requested over HTTPS during research for this article. All of them responded with HTTP 200 and spec-shaped content: Site File Distinctive pattern Vue.js `/llms.txt` Hierarchical table of contents, every link a .md page Nuxt `/llms.txt` + 4.7 MB llms-full.txt Agent-first: MCP server, sitemap.md, openapi.json pointers Cloudflare Developers `/llms.txt` + per-product files Hub-and-spoke index of subpath llms.txt files OpenAI API docs `/docs/llms.txt` + llms-full.txt Markdown twin for every page, single combined export Anthropic docs `/llms.txt` Multilingual hub with per-language page counts llmstxt.org `/llms.txt` The spec site's own minimalist example ## Vue.js: A Clean Hierarchical Table of Contents The first three lines of `vuejs.org/llms.txt` state the site name and tagline, then the file becomes a pure navigation document — every framework topic grouped under headings like `Getting Started` and `Essentials`, each link pointing at the Markdown version of a page: ``` # Vue.js Vue.js - The Progressive JavaScript Framework ### Getting Started - [Introduction {#introduction}](/guide/introduction.md) ``` Note the details: URLs end in `.md`, and the anchor metadata in brackets matches the HTML page's heading IDs so agents can link both formats. For full depth, Vue.js also serves `llms-full.txt` (about 0.9 MB at the time of testing) — exactly the two-file setup described in our [llms.txt vs llms-full.txt](/blog/llms-txt-vs-llms-full-txt/) guide. ## Nuxt: The Agent-First File Nuxt's file shows where the format is heading. After a one-sentence summary it tells agents that every page has a Markdown twin, then points to an unusually rich set of machine-readable resources: ``` # Nuxt Docs > Nuxt is an open source framework that makes web development intuitive and powerful. Every documentation, blog and deploy page is also Markdown. Append `.md` to a URL... ``` The rest of the file lists `llms-full.txt` — 4,756,574 bytes of single-file documentation, which we downloaded to confirm — plus a `sitemap.md`, a public OpenAPI spec and an MCP server endpoint. For an LLM agent, one llms.txt now opens the door to the whole ecosystem. The file even includes a `When to use this` section that tells the agent which tasks the docs are for — curation as copywriting. ## Cloudflare Developers: A Hub of Subpath Files Cloudflare's documentation is huge, so its root file does not try to list everything. It is a hub: one link per product area, each pointing at that product's _own_ llms.txt one level down: ``` ## Application performance - [DNS](https://developers.cloudflare.com/dns/llms.txt): Deliver excellent performance and reliability to your domain ``` This is the subpath pattern the [llms.txt v2 spec](/blog/llms-txt-v2/) formally defines: a file covers the URLs under its own path, and agents use the most specific one. Cloudflare pairs the hub with a root `llms-full.txt`, giving agents both a browseable directory and a complete dump. ## OpenAI and Anthropic: AI Labs Publish Their Own Files The biggest irony in the llms.txt ecosystem is that the AI labs themselves are early adopters. OpenAI's API docs (`platform.openai.com/docs/llms.txt`, HTTP 200) opens with `# OpenAI API docs`, states that every entry has a Markdown twin, and links a combined single-file export. Anthropic's file goes multilingual, listing page counts per language — English at 700 pages, others at 249 — so an agent knows which translations are complete before fetching anything. Neither file is a ranking hack: crawler studies we covered in [Which AI Engines Actually Read llms.txt?](/blog/which-ai-engines-read-llms-txt/) show most AI bots never request the file. These companies ship it because their docs are Markdown anyway and the file costs nothing to generate. Google's `ai.google.dev/llms.txt` returns 404 — a reminder that adoption is real but not universal. ## What the Best Production Files Have in Common Strip away the branding and five patterns recur in every case study above: - **One-line identity.** H1 with the product name plus a sentence saying what the docs cover. - **Markdown-first links.** Point agents at `.md` versions, not HTML pages. - **A description on every link**, so agents can decide before fetching — Cloudflare's one-liners and Nuxt's task guidance are both this idea. - **Curated H2 groups** of five to forty links, never a URL dump. - **An escape hatch**: llms-full.txt (or subpath files) for sites too big for one page. None of these files is longer than a couple of hundred lines, and all of them are plain Markdown. Structure your file the same way — then check it the way we checked these, with curl, a validator, or both. Generate a spec-compliant draft from your sitemap with the [llms.txt generator](/generator), run it through the [checker](/checker), and publish it at your site root. If you need a refresher on any element, the [complete llms.txt format guide](/blog/llms-txt-format/) covers every field. --- # How to Add llms.txt to Cloudflare Pages (Astro, React, Plain HTML) > Source: https://llmstxtgenerator.dev/blog/llms-txt-cloudflare-pages/ Deployment tutorial · Cloudflare Pages · 5 minutes # How to Add llms.txt to Cloudflare Pages Cloudflare Pages is one of the easiest hosts to serve llms.txt from — you get it at the site root with zero server config. Here are the three ways to do it, from simplest to most flexible. ## Method 1: Static File (Simplest — Works for Any Framework) Every Cloudflare Pages project serves files from a build output directory. Put an `llms.txt` file there and it's automatically live at your domain root. **Astro / SvelteKit / 11ty (static dir):** drop the file in the framework's static folder, which is copied to the output as-is: ``` # Astro: place this at public/llms.txt # (public/ contents are copied to dist/ at build time) ``` **Plain HTML / Vite / React (manual):** add `llms.txt` next to your `index.html` in the output folder (e.g. `dist/llms.txt`), or configure your build tool to copy it there. Build and deploy as usual — either with `wrangler pages deploy` or via your connected Git repository. The file is served at `https://your-domain.com/llms.txt`. ## Method 2: Dashboard Direct Upload If you manage the project through the Cloudflare dashboard (Workers & Pages → your project → **Upload assets**), you can upload an `llms.txt` file directly without a build pipeline at all. This is handy for a quick test or for sites not built from a repo. ## Method 3: Pages Function (Dynamic Generation) When your llms.txt should reflect live data — a fresh sitemap, a changing content list, or a dynamic `llms-full.txt` — generate it with a [Pages Function](https://developers.cloudflare.com/pages/functions/): ``` // functions/llms.txt.js — serves https://your-domain.com/llms.txt export async function onRequest() { const llms = `# Example Site > A concise description of the site. ## Docs - [Getting Started](https://example.com/docs/start)`; return new Response(llms, { headers: { "Content-Type": "text/plain; charset=utf-8" }, }); } ``` The function can fetch your `sitemap.xml`, filter sections, or pull from a CMS — anything you can do in JavaScript. Just keep the response under the platform's limits and serve `text/plain`. ## Verify It's Live After deploying, check from the command line or browser: ``` curl -I https://your-domain.com/llms.txt # Expect: HTTP/2 200, Content-Type: text/plain ``` Then validate the content — run the URL through our free [llms.txt checker](/checker) to confirm the H1, absolute URLs, and Markdown structure are spec-compliant before AI engines start reading it. ## Common Mistakes - **Wrong folder** — the file must end up in the _build output_, not your source root. Check `dist/` (or your output dir) after building. - **Relative URLs** — llms.txt links must be absolute (`https://…`), not `/docs/start`. - **Missing H1** — the file must start with a single `#` heading with your site name; the blockquote summary below it is what agents read for context. - **Caching surprises** — static files on Pages are cached at the edge; if you regenerate via a Function, add `Cache-Control: no-store` (as above) or purge when content changes. Once it's live, add the file to your footer or homepage so human visitors (and the agents that read pages) discover it — and consider adding an [llms-full.txt](/blog/llms-txt-vs-llms-full-txt/) companion for content-heavy sections. --- # How to Serve llms.txt: Content-Type, Encoding and Caching (2026) > Source: https://llmstxtgenerator.dev/blog/llms-txt-content-type-and-caching/ GEO · October 2026 # How to Serve llms.txt A correct Markdown file behind the wrong response headers is not a correct llms.txt. Here is what the spec asks the server to do, what eight live files actually return right now, and the delivery check that catches the two failures no validator can see from the outside. ## The Spec's Server-Side Requirements Are Short The proposal at llmstxt.org is mostly about the file's contents, and it deliberately says little about transport. Three things it does state. The file is Markdown, placed at `/llms.txt` or at any subpath such as `/docs/llms.txt`, and where more than one file applies an agent should use the most specific one. An optional byte-order mark is allowed. And the recommended way to advertise the file is a link relation — `rel="describedby"` pointing at the llms.txt that covers a page, `rel="alternate" type="text/markdown"` pointing at a Markdown twin — delivered either as an HTML link element or as an HTTP `Link:` response header, the header form being the one that works for non-HTML resources. The [discovery mechanics](/blog/llms-txt-discovery/) build on that: paths and Link headers get a client to the file, and the server's response is what decides whether the client believes what it found. Two of the three properties that matter — status code and content type — are invisible in the file itself. ## Content-Type: text/plain or text/markdown Send `text/plain; charset=utf-8` and you are done, because that is what every static host maps for a `.txt` extension. Send `text/markdown; charset=utf-8` and you are still correct, and slightly more precise for humans reading response headers. Third-party validation guidance accepts both and treats `text/html` or `application/octet-stream` as a misconfiguration, on the reasonable grounds that an agent handed a page of HTML markup has to guess where your index went. Which means a `200` is not the pass condition on its own. The common failure is a routing one: a framework catch-all route, a rewrite that sends unknown paths to the homepage, or a host that answers a directory index for `/llms.txt/` while the real file was never exported. The file looks perfect in your repository and the response is HTML with a 200 status. That is a subset of what [the usual "it does not work" symptoms](/blog/llms-txt-not-working/) turn out to be. ## What Eight Live llms.txt Files Return Headers fetched on October 4, 2026 with curl and a normal desktop user agent. The point of the table is the variety: all of these are deliberate implementations, and four of them make different calls on the same two decisions. URL Status Content-Type Caching developers.cloudflare.com/llms.txt 200 text/plain; charset=utf-8 public, max-age=0, must-revalidate + ETag docs.stripe.com/llms.txt 200 text/markdown; charset=utf-8 varies on Accept, plus a Link: rel="service-meta" header vercel.com/llms.txt 200 text/plain; charset=utf-8 s-maxage=3600, stale-while-revalidate=86400 + ETag supabase.com/llms.txt 200 text/plain; charset=utf-8 cache-control: public only — no explicit TTL llmstxt.org/llms.txt 200 text/plain; charset=utf-8 max-age=600 + weak ETag, last modified Sep 24 www.checklyhq.com/llms.txt 200 text/plain; charset=utf-8 public, max-age=0, must-revalidate + ETag docs.anthropic.com/llms.txt 301 text/html (redirect body) client must follow the redirect openai.com/llms.txt 403 text/html same 403 for curl, Googlebot and GPTBot agents Two rows deserve a comment. The 403 is an edge verdict, not a user-agent one — the same response came back for the default curl agent, a Googlebot-style agent and a GPTBot-style agent, so there is no polite way to ask for that file. And the 301 matters because a plain `curl -s` without `-L` shows you the redirect body, which looks like an HTML page. Anyone debugging with the wrong flag concludes the file is being served as HTML. ## Encoding: UTF-8, a BOM, and Three Bytes of Hex UTF-8 is what the spec assumes, and it is not cosmetic: a file saved in a legacy encoding puts mojibake in the middle of a description that an agent may quote back verbatim. The BOM is explicitly optional, and the reference implementation ships without one — the first bytes of llmstxt.org's own file are `0x23 0x20`, a bare hash and space. Check your own file the same way an agent receives it, rather than trusting an editor's status bar: ``` # first four bytes: watch for efbbbf (a single BOM) or doubled bytes curl -s https://example.com/llms.txt | head -c 4 | xxd # the header the client will actually see curl -sI https://example.com/llms.txt | grep -i content-type ``` ## Caching: Longer Than Your Pages, Shorter Than Your Policy An index file changes when your site structure changes, which is nothing like your publishing cadence, so it can be cached harder than a page. It should not be cached forever. A shared TTL of an hour to a day covers almost every case, and pairing a modest TTL with an `ETag` keeps revalidation cheap enough that you can afford to be conservative — `max-age=0, must-revalidate` plus an ETag, the pattern both Cloudflare's docs and Checkly use, is the conservative end of that scale and still avoids sending the body when nothing changed. - **Do not mark it immutable for a year.** A reorganisation or a domain migration would leave agents reading a retired URL map long after the old paths 404. - **A bare `cache-control: public` with no TTL is not a policy.** It lets intermediaries pick their own heuristic freshness, which is how a file ends up stale in some networks and fresh in others. - **Purge on rename, not on edit.** The change that breaks clients is a removed or moved page, so hook the cache purge to whatever writes the file rather than remembering to do it later. - **Keep it under a size you can revalidate.** If the file has grown past the [sensible token budget](/blog/llms-txt-size-limit/), the delivery problem is really a content problem. ## The Delivery Check: Five Commands Before You Move On Run this against production, not against your local preview, because every failure described above is introduced by the host rather than by the file: ``` # 1. status and type only (HEAD is cheap but its content-length can read 0) curl -sI https://example.com/llms.txt | head -5 # 2. follow redirects, then look at the status you land on curl -sIL -o /dev/null -w "%{http_code} %{url_effective}\n" https://example.com/llms.txt # 3. confirm the body starts with your H1, not with curl -s https://example.com/llms.txt | head -n 3 # 4. header bytes, to rule out a BOM surprise curl -s https://example.com/llms.txt | head -c 4 | xxd # 5. what a shared cache is allowed to keep curl -sI https://example.com/llms.txt | grep -iE "cache-control|etag|last-modified" ``` A file that passes all five is delivered as well as it can be. That is worth doing, and worth being honest about: correct delivery is a precondition, not a ranking lever. Published audits — SE Ranking's scan of roughly 300,000 domains, Ahrefs' log study of 137,210 domains with a valid file — found no citation correlation and a large share of files never requested at all. The reason to fix your headers is that a client which does fetch the file gets the Markdown you wrote instead of an HTML challenge page. Generate a clean version with the [llms.txt generator](/generator), re-check structure with the [llms.txt checker](/checker), then verify the response the way a crawler sees it. --- # How to Write llms.txt Descriptions: Summary Blockquote and Link Annotations (2026) > Source: https://llmstxtgenerator.dev/blog/llms-txt-descriptions/ Writing · September 2026 # llms.txt Descriptions: The Lines That Actually Get Quoted Structure tells an engine where to look. Prose decides whether it bothers. These are the only three places in your file where you get to write sentences — and most files waste all three. ## Three Slots, Three Different Jobs The [llmstxt.org](https://llmstxt.org) format is deliberately thin: one H1, an optional blockquote summary, H2 sections of links. Everything else in the file is addresses. That leaves you exactly three slots for language, and each one is read differently. Slot Who reads it What it must do `# Site Name` Anything parsing the file Identify the entity, not the tagline `> Summary` A model building a first impression of the domain Be liftable verbatim into an answer `- [Title](url): text` Retrieval deciding which URL to fetch Name the question the page answers ## The Summary Blockquote: Write It to Be Quoted The blockquote under your H1 is the most reusable string in the file. An agent that has just discovered your domain needs a working model of it in one read, and a summary that reads like a landing page hero gives it nothing to hold: _we help modern teams do more_ is not a fact, and a model with no verifiable claim will simply move on to another source. A summary that survives being quoted answers four things in order: - **What it is.** The category of thing, in the words a person would search for. - **Who it is for.** The audience is often the strongest filter in a retrieval step. - **What is inside.** Docs, API reference, changelog, comparisons — the shape of the site. - **What is distinctive.** One concrete differentiator, ideally a number or a constraint. Keep it to two to five sentences. And write it in the third person: an engine quoting the phrase _we help teams_ has to rewrite the pronoun to place it in an answer, and every rewrite is a chance to lose your framing. `The site is` or the entity name itself travels intact. ## Link Descriptions: Name the Question, Not the Title The text after the colon is the second most valuable real estate in the file and the most commonly automated into sludge. A description that repeats the page title adds nothing a model could not derive; a description that names the specific thing the page covers is what a retrieval step can actually match against a user's question. The examples people cite most often make the same point: _Pricing_ is weak, _plan comparison with feature breakdown and current monthly prices_ is strong. Three patterns to prefer over restating the title: - **Scope lines.** "Covers retention, funnel and cohort analysis — not marketing attribution." - **Question lines.** "Answers whether the free tier includes API access, and what the rate limits are." - **Constraint lines.** "Lists supported SDKs and the minimum runtime version for each." The negation in the first pattern looks odd in marketing copy and reads perfectly to a model: it bounds the page. Descriptions are also the place to disambiguate near-duplicate pages, which is the problem behind most duplicate-link findings in [a structural validation pass](/blog/how-to-validate-llms-txt/). ## Before and After: One Section, Rewritten Same links, same order. Only the descriptions changed: ``` ## Documentation - [Quickstart](https://example.com/docs/quickstart.md): Get started - [Data model](https://example.com/docs/data-model.md): Our data model - [API reference](https://example.com/docs/api.md): API docs ## Documentation - [Quickstart](https://example.com/docs/quickstart.md): Install the SDK, send a first event and verify it in the live explorer - [Data model](https://example.com/docs/data-model.md): How events, users and accounts are keyed, including identity-resolution rules - [API reference](https://example.com/docs/api.md): Every endpoint with request and response schemas, plus per-plan rate limits ``` The second version is longer by roughly forty words and worth every token: it is the difference between an agent knowing three files exist and knowing which one answers the question it was given. The full set of writing conventions lives in the [llms.txt best practices guide](/blog/llms-txt-best-practices/). ## Length Discipline: Descriptions Are a Budget The whole file is read in one fetch, so every extra clause competes with the links that deserve the context window. Practical limits that read well in practice: - Summary: 40–80 words. Past 100 it stops being quotable and starts being ignorable. - Link description: 8–20 words. Long enough to add information, short enough to scan. - No sentence that only fills space — "read more about our great features" is a token spent on nothing. Descriptions are also the fastest way to leak or over-promise. Anything you would not put on a public page does not belong in a description either; the categories are listed in [what not to include in llms.txt](/blog/llms-txt-what-not-to-include/). ## A Twelve-Minute Rewrite Pass 1. Read your blockquote aloud and delete every adjective you cannot measure. 2. Cover the page titles and read only the descriptions. Can you tell what each page contains? If not, rewrite. 3. Check that no two lines describe the same page in different words. 4. Confirm each description is true of the page it points at — copy that describes a retired feature is worse than no copy. 5. Re-run the file through the [llms.txt checker](/checker) to confirm the structure still parses after your edits. If you would rather start from a working skeleton than an empty file, paste your sitemap into the [llms.txt generator](/generator) and spend your time on the three slots that need a human: the H1, the summary, and the descriptions. --- # How AI Crawlers Discover Your llms.txt: Link Relations, Subpaths and Directories (2026) > Source: https://llmstxtgenerator.dev/blog/llms-txt-discovery/ GEO · September 2026 # How AI Crawlers Discover Your llms.txt Most llms.txt advice covers what to write. Almost none covers how the file gets found — which is the part that decides whether writing it was worth an afternoon. Here are the channels that actually exist, what each one is good for, and how to prove yours is reachable. ## Publishing Is Not Discovery A published file is a file at a URL. Discovery is the separate question of how a client that has never visited your site learns that the URL exists. The v2 proposal is explicit about the first and deliberately thin about the second: it fixes the filename, fixes the locations, and recommends one mechanism for pointing at the file from elsewhere. [The v2 changes](/blog/llms-txt-v2/) are worth reading for the spec layer; this article is about the plumbing underneath it. The failure mode is silent: a valid file that no client ever requests looks identical in your editor to one that is working, and only access logs or [crawler measurement](/blog/measure-llms-txt-traffic/) tell them apart. ## Channel 1 — The Path Convention, Root and Subpath The strongest thing your file has going for it is that its location is predictable. The spec places llms.txt at `/llms.txt` or at any path within the site, and states that a file covers the URLs under its path — with the most specific file winning where several apply. That gives a client two cheap moves: ask the root of a domain, and ask the subpath it is already working in. - **Root file** — `/llms.txt`. Covers the whole site by convention. This is the one to ship first, and the one a client is most likely to guess. - **Subpath file** — for example `/docs/llms.txt` for a documentation hub, or `/locations/denver/llms.txt` for a branch. Covers only the pages beneath it, and takes precedence there. A subpath file is not separate from the root file; it is a routing decision inside it. The root file should link to the subpath file the same way it links to pages, so a client that reads the root learns that a narrower map exists. Sites whose documentation dwarfs the rest of the site are the usual case — the root file stays small and points at the dense part of the site. ## Channel 2 — Link Relations: describedby and alternate This is the only discovery mechanism the specification actively recommends, and it is the one most sites skip because it needs a server or template change rather than a text file. Two link relations do the work. `rel="describedby"` points at the llms.txt file that covers the page; `rel="alternate"` with `type="text/markdown"` points at the clean markdown version of that page. They can be expressed as HTML link elements in the head, or as an HTTP `Link` response header. ``` Link: ; rel="describedby", ; rel="alternate"; type="text/markdown" ``` The header form is the interesting one, and the spec calls it out: a header can be added in web server or CDN configuration without touching a single page, and it works for non-HTML responses such as the markdown files themselves. A Cloudflare Worker or an nginx `add_header` directive covering `text/html` and `text/markdown` responses is usually the cheapest implementation. The markdown-twin half of this pattern is covered in [Markdown for Agents vs llms.txt](/blog/markdown-for-agents-vs-llms-txt/). ## Channel 3 — robots.txt and Sitemaps: Hints, and One Way to Break Yourself robots.txt is an exclusion protocol, not a discovery one, and conflating the two is where people get hurt. There is one real discovery signal worth knowing and one real hazard. Move What it is worth Leaving `/llms.txt` crawlable Essential. A broad `Disallow: /` or a pattern like `Disallow: /*.txt` matches the file and kills the fetch A second `Sitemap:` line in robots.txt pointing at `/llms.txt` A non-standard hint. Sitemap directives are defined for XML sitemaps; cheap to add, no verified consumer Listing `/llms.txt` in `sitemap.xml` Mostly redundant — it puts the file in search indexes, not in front of an agent Allow rules for `GPTBot`, `ClaudeBot`, `PerplexityBot` Indirect but real: a blocked crawler never reaches any file, llms.txt included If you have not audited the file in a while, [the 2026 crawler reference](/blog/robots-txt-ai-crawlers/) lists the tokens worth knowing. The delivery failures that look like discovery failures — wrong content type, redirect chains, WAF blocks — are catalogued separately in [llms.txt not working](/blog/llms-txt-not-working/). ## Channel 4 — Directories, Platforms and the Human Layer The proposal itself keeps a short list of directories that index public llms.txt files: **llmstxt.site**, **directory.llmstxt.cloud** and **llmstxthub.com**. None of them routes your file into a model, so treating a listing as an AI-visibility tactic is wishful. What it does give you is a second referencing page and a place to see what comparable sites publish — which is how most llms.txt conventions actually spread. Two platform forces matter more. Hosted documentation products now generate a file for every site they serve, so a growing share of the web has one by default rather than by choice. And Chrome's Lighthouse ships llms.txt as part of its _agentic browsing_ audits, turning the file from a blog-post suggestion into a checkbox in a tool developers already run. Neither guarantees a fetch, but both raise the odds that the file a client looks for will be there. ## The Four-Command Reachability Check Whatever channel someone arrives by, the file has to survive the fetch. Four commands settle it: ``` # 1. status, content type, redirects curl -sI https://example.com/llms.txt # 2. the way a crawler sees it curl -sA "GPTBot/1.2" https://example.com/llms.txt | head -5 # 3. is robots.txt blocking it curl -s https://example.com/robots.txt | grep -i llms # 4. does the header form exist on a real page curl -sI https://example.com/docs/ | grep -i "^link:" ``` Expected results: `200` with `text/plain` or `text/markdown` and no `Location` header; a body that starts with an H1 and the summary blockquote; nothing in robots.txt that matches the path; and an optional `Link` header if you implemented Channel 2. Then stop optimising discovery and look at consumption: a widely cited Ahrefs study published in 2026 found that of roughly 38,000 domains with a valid llms.txt, 97% received no requests for it at all in a month. Discovery is worth getting right precisely because so little of it is happening by itself — but the file has to be aimed at the clients that do fetch it, which is a question about [which engines read llms.txt](/blog/which-ai-engines-read-llms-txt/), not about directories. If you are starting from nothing, generate a first draft with the [llms.txt generator](/generator), publish it at the root, then run the production URL through the [llms.txt checker](/checker) before you spend an afternoon on link headers. --- # How to Add llms.txt to Docusaurus (Docs Sites, 2026) > Source: https://llmstxtgenerator.dev/blog/llms-txt-docusaurus/ Docusaurus tutorial · Static or generated · 6 minutes # How to Add llms.txt to Docusaurus Docusaurus has no official `llms.txt` plugin yet — but it is one of the best-behaved frameworks for the format anyway. Everything in `static/` is copied to your site root at build time, so a spec-compliant file is a 60-second job. For larger docs, a community plugin generates `llms.txt` and `llms-full.txt` from your actual pages. Both routes are covered here. ## Why Docs Sites Are llms.txt's Best Use Case The llms.txt convention exists so AI engines, chatbots and coding agents can find the pages that answer their questions without crawling an entire site. Documentation is the content type where that matters most: docs are factual, densely linked and constantly referenced in AI answers, yet their navigation is often sidebar-heavy and hostile to crawlers. The docs-tooling world noticed early. The llmstxt.org integrations list includes Mintlify, which generates an `llms.txt` and Markdown page versions for every site it hosts, and GitBook, which serves the file for published docs. Real-world adopters include dub.co (via Mintlify) and Cloudflare's developer docs (via a build script). Docusaurus, used by React, Jest and thousands of open-source projects, has no native option yet — an official plugin proposal has sat as an open feature request in the facebook/docusaurus repo since February 2025. In practice you do not need to wait for it. ## Method 1: A Hand-Written File in static/ (60 Seconds) Docusaurus treats `docs/` as content and `static/` as assets: every file in `static/` is copied verbatim to the output root on every build. That is the entire mechanism — create `static/llms.txt` and follow the [v2 spec](/blog/llms-txt-v2/) structure: ``` # Acme Docs > Acme Docs is the official documentation for the Acme platform. > It covers installation, configuration, the REST API and troubleshooting. ## Getting Started - [Quickstart](https://acme.com/docs/quickstart): first project in five minutes - [Authentication](https://acme.com/docs/auth): API keys and access tokens - [Deployment](https://acme.com/docs/deploy): production setup for self-hosting ## API Reference - [REST API](https://acme.com/docs/api): every endpoint, parameter and error - [Webhooks](https://acme.com/docs/webhooks): events your server can subscribe to ``` Build with `npm run build` and the file lands at the root of `build/`, ready for Netlify, Vercel, Cloudflare Pages or any static host. Keep this method when your information architecture changes rarely. The file must point to canonical pages with absolute URLs — never relative paths, which agents resolve inconsistently. ## Method 2: Generate llms.txt and llms-full.txt From Your Docs For a file that stays in sync with a growing sidebar, the community plugin [docusaurus-plugin-llms](https://github.com/rachfop/docusaurus-plugin-llms) (MIT-licensed and listed on llmstxt.org) walks your docs at build time. With no options it writes a curated table of contents to `llms.txt` and combines the full text of every page into `llms-full.txt`: ``` // docusaurus.config.js const config = { // ... your existing config ... plugins: [ 'docusaurus-plugin-llms', ], }; export default config; ``` The plugin is designed for exactly the situations a static file cannot handle: versioned docs (each version can emit its own file, such as `/stable/llms.txt`), custom LLM files for specific audiences, and large references where a full-text dump saves an agent dozens of requests. If your reference section is deep, link the generated dump the way the [llms.txt vs llms-full.txt guide](/blog/llms-txt-vs-llms-full-txt/) describes. ## The Docusaurus Checklist - **One file, one owner** — a hand-written `static/llms.txt` and a plugin that also writes `/llms.txt` will fight each other on rebuild. Pick one approach. - **Curate docs, not marketing pages** — an agent landing on your docs wants the quickstart, the auth guide and the API reference, not your pricing page. See the [best practices guide](/blog/llms-txt-best-practices/) for curation rules. - **Absolute URLs only** — `https://acme.com/docs/quickstart`, never `/docs/quickstart`. - **Watch your baseUrl** — sites deployed under a subpath like `/docs/` should generate the file with the plugin so URLs and placement stay consistent with the spec's subpath-file rules. - **Ship it with every release** — Docusaurus is a static generator: nothing exists on your host until you deploy. Rebuild after sidebar changes so the file never points at removed pages. ## Verify Before Agents Do After deploying, confirm the file serves with a 200 status and a text content type: ``` curl -s -o /dev/null -w "%{http_code} %{content_type}\n" https://your-domain.com/llms.txt # Expect: 200 text/plain ``` Docusaurus serves the file from the same CDN edge as your docs, so check the version you actually deployed — a stale build is the most common reason an llms.txt goes missing after a redesign. Then paste the URL into the free [llms.txt checker](/checker) to validate the H1, the blockquote summary and the section structure. Draft the initial file in seconds by feeding your sitemap into the [free llms.txt generator](/generator), then verify the output with the [checker](/checker) before you wire up the plugin. Documentation sites are the highest-leverage place an `llms.txt` can live — a correct, current file is cheap insurance that your docs are the version AI engines read. --- # llms.txt Examples: Real Files for Every Site Type > Source: https://llmstxtgenerator.dev/blog/llms-txt-examples/ Copy-ready examples · Updated 2026 # llms.txt Examples: Real Files for Every Site Type The fastest way to learn a format is to read a good example — so here are four complete, spec-valid **llms.txt example** files for the site types that publish them most: SaaS documentation, e-commerce, personal blogs, and corporate websites. ## Why Examples Are the Fastest Way to Learn llms.txt The llms.txt specification is deliberately minimal — one required `# H1`, an optional blockquote summary, free-form notes, and `## H2` sections that each contain a list of links. Reading the spec tells you the rules; reading real **llms.txt examples** tells you what the rules look like in practice. The format leaves real decisions to you: which pages deserve a link, how to group them into sections, how many links is too many, and where the optional stuff goes. Every example below follows the llmstxt.org v2 specification exactly: exactly one H1 at the top, a blockquote summary, short notes without headings, H2 file lists with absolute URLs and a `: description` on every link, and a trailing `## Optional` section where it helps. The site names are fictional — swap in your own domain, pages, and descriptions. If you want the rules themselves, the [complete llms.txt format guide](/blog/llms-txt-format/) walks through every element. The second thing examples teach you is that the format scales without changing shape: the same skeleton that fits a ten-link blog file also carries a multilingual corporate index. Once you can read one of these files end to end, you can write one for any site — which is exactly what the generator behind this page automates. ## llms.txt Example 1: SaaS Documentation Site Documentation is the canonical use case for llms.txt — AI coding agents answer questions from docs more than from any other page type. This example shows three organizational habits worth copying: linking straight to `.md` versions of pages, referencing an `llms-full.txt` for agents that want everything, and moving secondary links into `## Optional`. ``` # Pulseboard Analytics > Pulseboard is a product analytics platform for SaaS teams — event tracking, funnels, retention, and dashboards from a single SDK. Pulseboard has SDKs for JavaScript, Python, and mobile. All documentation is published as Markdown, so every link below points directly at a .md file. ## Important links - [Quick start](https://pulseboard.dev/docs/quickstart.md): Install the SDK and send your first event in 10 minutes - [Event tracking API](https://pulseboard.dev/docs/api/events.md): Track, batch, and alias events, with code examples - [Dashboards & reports](https://pulseboard.dev/docs/dashboards.md): Build funnels, retention cohorts, and custom reports - [llms-full.txt](https://pulseboard.dev/llms-full.txt): The complete documentation in a single file ## Guides - [Tracking plan design](https://pulseboard.dev/docs/guides/tracking-plan.md): Structure your events before you ship - [Data privacy & consent](https://pulseboard.dev/docs/guides/privacy.md): GDPR, CCPA, and cookieless tracking - [Migrating from Amplitude](https://pulseboard.dev/docs/guides/migrate.md): Step-by-step migration guide ## Optional - [Changelog](https://pulseboard.dev/changelog.md): Release notes for all past versions - [Pricing](https://pulseboard.dev/pricing.md): Plans, limits, and billing FAQ ``` Notice what makes this file useful to an agent: the blockquote explains what the product is in one breath, the `llms-full.txt` link offers a deep-read option without bloating this file, and `## Optional` keeps pricing and changelog out of the default context. The `.md` URLs matter too — Markdown renders cleanly inside an LLM's context window, with none of the nav, cookie banners, or layout noise of the HTML versions. ## llms.txt Example 2: E-commerce & Product Site For stores, the goal is different: agents should be able to answer product questions — what you sell, what it costs, how shipping works — and recommend pages to shoppers. This **llms.txt example** organizes by product line, adds an HTML comment in the notes area (the only safe place for comments), and keeps time-sensitive pages like sales in `## Optional`. ``` # Terra Outfitters > Terra Outfitters is an online outdoor gear store — tents, backpacks, sleeping bags, and hiking apparel with free shipping over $75. ## Product lines - [Tents](https://terra-outfitters.com/products/tents.md): 2–6 person tents, shelters, and footprints - [Backpacks](https://terra-outfitters.com/products/backpacks.md): Daypacks and multi-day packs from 20L to 75L - [Sleeping bags](https://terra-outfitters.com/products/sleeping-bags.md): Down and synthetic bags rated to -20°F - [Hiking apparel](https://terra-outfitters.com/products/apparel.md): Jackets, base layers, and trail pants ## Help & policies - [Shipping & returns](https://terra-outfitters.com/shipping.md): Delivery times, costs, and 60-day returns - [Sizing guide](https://terra-outfitters.com/sizing.md): Fit guidance for packs, apparel, and footwear - [Product care](https://terra-outfitters.com/care.md): Cleaning and repair instructions ## Optional - [Sale & clearance](https://terra-outfitters.com/sale.md): Current discounts and limited offers ``` The shape is the same as the SaaS example, but the content priorities flipped: category pages and policies carry the weight instead of API references. The HTML comment survives parsing because it sits in the notes area, where the spec allows it. If your store runs seasonal promotions, keep them out of the main sections — a sale page that goes stale in `## Optional` costs far less trust than a dead link at the top of the file. ## llms.txt Example 3: Personal Blog Blogs have the opposite problem from docs sites: too much content, most of it not worth an agent's context. A good blog **llms.txt example** is a curated list — recent or best posts only — with the archive tucked into `## Optional`. Keep the whole file under a couple of hundred words and agents will actually read it. ``` # Field Notes by Maya Chen > Field Notes is a personal blog about distributed systems, Rust, and marathon running, written by Maya Chen, a backend engineer at a fintech startup. Posts are written in Markdown and archived by category. The list below covers the most useful posts; the full archive is under Optional. ## Recent posts - [Why we moved from Postgres to TiDB](https://mayachen.dev/posts/tidb-migration.md): A production postmortem with real query numbers - [Rust async in 15 minutes](https://mayachen.dev/posts/rust-async.md): A practical walkthrough for beginners - [My 2025 reading list](https://mayachen.dev/posts/reading-2025.md): Books on systems, history, and endurance training - [Training for a sub-3 marathon](https://mayachen.dev/posts/marathon-training.md): The exact plan I used, week by week ## Optional - [About](https://mayachen.dev/about.md): Bio, speaking, and contact details - [All posts](https://mayachen.dev/archive.md): Complete chronological archive - [Talks](https://mayachen.dev/talks.md): Slides and recordings from conference talks ``` Ten links, two sections, one blockquote — that's the whole file. If you publish a lot, rotate the featured posts as new ones land, or point `## Optional` at category pages instead of individual posts. "Most useful" beats "most recent": an evergreen tutorial outranks yesterday's link roundup, because agents are usually answering a question, not catching up on news. ## llms.txt Example 4: Corporate Website Corporate sites serve a different audience again: investors, journalists, and job seekers asking factual questions. This **llms.txt example** demonstrates a multilingual layout — a section per language, each with its own absolute URLs — plus company-critical pages up front and legal pages at the end. ``` # Nordhelm Group > Nordhelm Group is a Munich-based industrial automation company — robotics, control software, and factory integration services for manufacturers across Europe. The site is published in English and German. Links are grouped by language, and every page is available as Markdown. ## English - [About Nordhelm](https://nordhelm-group.com/en/about.md): Company history, leadership, and locations - [Products & solutions](https://nordhelm-group.com/en/products.md): Robot arms, vision systems, and control software - [Industries](https://nordhelm-group.com/en/industries.md): Automotive, electronics, and logistics use cases - [Newsroom](https://nordhelm-group.com/en/news.md): Press releases and company news ## Deutsch - [Über Nordhelm](https://nordhelm-group.com/de/ueber-uns.md): Unternehmensgeschichte, Führung und Standorte - [Produkte & Lösungen](https://nordhelm-group.com/de/produkte.md): Roboterarme, Bildverarbeitung und Steuerungssoftware - [Karriere](https://nordhelm-group.com/de/karriere.md): Offene Stellen und Bewerbungsprozess ## Optional - [Investor relations](https://nordhelm-group.com/en/investors.md): Annual reports and financial results - [Imprint & privacy](https://nordhelm-group.com/en/imprint.md): Legal notices (Impressum) and privacy policy ``` The language-split sections are the takeaway: agents can fetch the German pages when the answer should be in German and ignore them otherwise. Note the German descriptions still follow the `[text](url): note` convention — keep descriptions in the page's own language so the link list stays meaningful to every reader. The legal section at the end matters for European sites, where an Impressum is a legal requirement and a common question for agents to answer. ## What to Highlight in Your llms.txt, by Site Type Each site type earns agent traffic for different reasons. Use this comparison to decide what belongs in your file: Site type What agents look for What to prioritize **SaaS documentation** Setup steps, API reference, guides Quick start + API links + an `llms-full.txt` reference **E-commerce / product** What you sell, prices, shipping and returns Product categories, help & policy pages, FAQ **Personal blog** Recent and evergreen posts, author context A curated list of recent/best posts — keep it short **Corporate website** Company facts, products, news, contact info About, products, newsroom — one section per language if multilingual Hybrid sites — a blog with a docs section, a store with a corporate About page — can mix and match: pick the section structure that matches your dominant content type, then add one section for the secondary type. The file stays valid either way, because validity never depends on the topic, only on the order and shape of the elements. ## The Universal llms.txt Template: A Copy-Paste Checklist Strip away the site-specific choices and all four examples share the same skeleton. Work through this checklist in order and you'll have a valid file every time: Step Element What to include 1 `# Site Name` (H1) The only required element — exactly one, at the very top. 2 Blockquote One or two lines of key context about the site. 3 Free-form notes 0–3 short paragraphs; no headings allowed here. 4 `## Section` (H2) Two to five file lists, grouped by topic. 5 Link items `[text](absolute URL): description` — never relative, always described. 6 `## Optional` Secondary links agents can skip when context is short. 7 `llms-full.txt` One link to the expanded file, if you publish one. 8 Validation Run the finished file through the free [llms.txt checker](/checker). When you're done, your file should look like a short README: roughly 40 to 150 lines, one H1, one blockquote, a few notes, and every link absolute and described. If a section is thinner than three links, fold it into another section — the checklist is a guide, not a contract. ## llms.txt Examples FAQ ### Can I copy these llms.txt examples as-is? The structure is copy-safe — that's the point of a template. Replace the site name, links, and descriptions with your own, keep the section order, and run the result through a validator. The fictional domains in these examples will fail link checks if you leave them in. ### How many links should my llms.txt file have? There is no hard limit, but most sites do best with 10–40 links across two to five sections. The file is meant to fit comfortably in a context window — if it reads like a sitemap, cut it down and move the long tail into `## Optional` or `llms-full.txt`. ### Should my llms.txt list every page on my site? No. An llms.txt file is a curated index, not a sitemap — agents trust it because every link is worth reading. List the pages that deserve attention, group them sensibly, and publish the full expansion separately as `llms-full.txt`. ## Keep reading [ ### 📐 The Complete Format Guide Every element of the v2 spec explained, with a valid end-to-end example. ](/blog/llms-txt-format/)[ ### 🛠️ How to Create llms.txt A step-by-step tutorial: build your file in 10 minutes and validate it. ](/blog/how-to-create-llms-txt/) ## Ready to write your own llms.txt? Generate a spec-perfect file from your URLs in under a minute — free, no sign-up. Then read the full format guide for the details. [⚡ Generate your llms.txt](/generator) [Read the full format spec](/blog/llms-txt-format/) --- # llms.txt for API Documentation: What to List and How to Structure It (2026) > Source: https://llmstxtgenerator.dev/blog/llms-txt-for-api-docs/ API Docs · September 2026 # llms.txt for API Documentation An API reference is the one site type where llms.txt has a job that pays for itself: telling a coding agent which page answers which integration question. Here is what to list, in what order, and how to keep the file small enough that an agent actually reads it. ## Why API Docs Are the Exception Most llms.txt files are aimed at a chatbot citation that may never come. Developer documentation is different, because the reader is often a program mid-task: an IDE agent, a CLI assistant or a support bot that is about to write a request against your API. When that reader gets a path wrong, the failure is expensive — a hallucinated endpoint, an SDK version pinned from training data, a webhook signature verified the old way. That is also where adoption is concentrated. The hosted documentation platforms generate a file for every site they serve, so documentation-heavy domains are far more likely to have one than marketing sites. If you maintain an API, you are not early to this — you are behind if the file is missing. ## What Belongs in an API llms.txt: Five Buckets The mistake is treating the file as a sitemap. It is a curriculum: the shortest path from "I need to send a payment" to working code. Five buckets cover almost every API. - **Auth and quickstart** — one page each. Authentication first, then the fastest path to a successful first call. - **Reference, grouped by resource** — charges, customers, payouts. One link per resource concept, not one per method. - **Errors, limits and versioning** — the pages that stop an agent from retrying a 402 forever or calling a deprecated field. - **Operational surfaces** — webhooks and event types, idempotency, pagination, sandbox and test fixtures. - **Machine-readable targets** — the OpenAPI file, Markdown twins, an MCP server, a status page. Leave out the auto-generated permalink for every endpoint, changelog archives, marketing pages and anything behind a login. The [leak rules](/blog/llms-txt-what-not-to-include/) apply with extra force on API docs, where it is easy to paste an internal staging host or an admin route into a section header. ## A Copyable API llms.txt ``` # Acme Payments API > REST API for charges, refunds, customers and payouts. Authenticate with a Bearer secret key. ## Start here - [Authentication](https://docs.example.com/api/authentication): Creating and rotating secret keys, and which key belongs in a browser. - [Quickstart](https://docs.example.com/api/quickstart): First charge in five minutes, with curl and Node examples. - [Test mode](https://docs.example.com/api/testing): Test card numbers, magic amounts and how to simulate failures. ## API reference - [API overview](https://docs.example.com/api/reference): Request format, idempotency, pagination and versioning rules. - [Charges](https://docs.example.com/api/charges): Create, capture, retrieve and refund a charge. - [Customers](https://docs.example.com/api/customers): Create a customer and attach payment methods. - [Payouts](https://docs.example.com/api/payouts): Move balances to a bank account and list payout history. ## Errors and operations - [Error codes](https://docs.example.com/api/errors): Every error type, its HTTP status, and the retry behaviour we recommend. - [Rate limits](https://docs.example.com/api/rate-limits): Limits per key and the headers that tell you when to back off. - [Webhooks](https://docs.example.com/api/webhooks): Event types, signature verification and replaying failed deliveries. ## Machine-readable - [OpenAPI 3.1 specification](https://docs.example.com/openapi.json): The full spec, generated from the same source as the reference. - [llms-full.txt](https://docs.example.com/llms-full.txt): Every page above concatenated as Markdown, for one-shot ingestion. - [MCP server](https://docs.example.com/mcp): Tools an agent can call instead of generating raw HTTP requests. ``` Four sections, thirteen links, roughly 400 tokens. Every line is a decision an agent cannot make from a URL alone — which is exactly what the description slot is for. The [description guide](/blog/llms-txt-descriptions/) covers how to phrase those colons if you want them quoted back accurately. ## Point at Specs, Not Only at Pages A well-run docs site has three surfaces, and the file should name all three because they serve different fetches: Surface Best for Serving note HTML page or its `.md` twin Reading one concept Verify the `.md` returns `text/markdown` OpenAPI 3.1 file Generating request code Serve it at a stable URL, no auth MCP server Calling instead of writing Announce the endpoint; it is not discovery by itself Resend's file is a clean example of this layering: the five sections include a "When to use Resend" block that names the actual endpoints, then links to the OpenAPI spec, an official CLI, an MCP server and discovery manifests. It is under 8 KB and answers the first three questions a coding agent asks. The mechanics of an MCP announcement live in the [llms.txt vs MCP comparison](/blog/llms-txt-vs-mcp/). ## The Hidden Slot: Instructions Before the First H2 Everything between the H1 and the first `##` is free text, and most sites use it for one summary line. Some API teams use it for hard rules. Stripe's file — fetched September 26, 2026, served as `text/markdown`, about 92 KB — opens with plain paragraphs instead of a blockquote, telling agents to check the npm registry rather than pin a memorised SDK version, and which sandbox to use for which workflow. That is a deliberate trade: it spends the file's most-read real estate on the two mistakes that break the most integrations. If your support queue is full of the same agent-generated bug — deprecated parameter, wrong environment, missing idempotency key — that is the paragraph to write. Keep it short: this is a notice board, not documentation. ## Size: Page-Level or Endpoint-Level? Stripe's file runs 453 links across 26 sections, and Mintlify's documentation index shows the other strategy — one section collapses an entire 183-page manual into a single Markdown link. Both work, and the choice comes down to what a fetch costs: - **Under ~40 pages** — one flat list, one link per concept, no nesting. - **40–500 pages** — group by product or resource, one or two levels deep, and let the largest sets collapse into a single index page. - **Every endpoint** — that belongs in `llms-full.txt` or the OpenAPI file, not in the index. Four hundred lines at ~12 tokens each is a 5,000-token tax on every fetch. The split between a curated index and a full dump is covered in [llms.txt vs llms-full.txt](/blog/llms-txt-vs-llms-full-txt/); API docs are the case where shipping both is usually correct. ## Verify the Fetch, Then Verify the Links API docs fail llms.txt in a specific way: the file is fine, the targets are not. A reference page that only renders in the browser returns an empty shell; a versioned docs URL from a previous release 404s after a rewrite. Three commands catch nearly all of it. ``` # 1. how the file itself is served curl -sI https://docs.example.com/llms.txt # 2. every link answered, with its status curl -s https://docs.example.com/llms.txt \ | grep -o 'https://[^)]*' \ | while read u; do echo "$(curl -o /dev/null -s -w "%{http_code}" "$u") $u"; done # 3. the way a Markdown-fetching client sees a target page curl -s -H "Accept: text/markdown" https://docs.example.com/api/errors.md | head -5 ``` Expect `200` and `text/plain` or `text/markdown` on the file, no redirect chain, a status list with nothing but 200s, and a Markdown target whose first line is a heading rather than an empty body. Generate a spec-compliant starting point with the [llms.txt generator](/generator), then run the production URL through the [llms.txt checker](/checker) before you add another link. --- # llms.txt for E-commerce: Product, Category and Policy Pages (2026) > Source: https://llmstxtgenerator.dev/blog/llms-txt-for-ecommerce/ E-commerce · October 2026 # llms.txt for E-commerce: What an Online Store Should List A store is the one site type where llms.txt is easy to get badly wrong: the instinct is to export the catalog. Here is the page set that helps an assistant answer questions about your shop, drawn from the files large retailers publish today. ## Two Kinds of AI Traffic Hit a Store A shopping query can end in two very different places. In the first, a person asks an assistant a question about your business — your return window, shipping destinations, which size fits a 26cm foot — and the answer is read off your public pages. In the second, an agent searches a product feed, compares offers and hands the buyer a link or a checkout. Only the first is llms.txt work; the second is feed work, and mixing them up is why so many store files become a 20,000-line SKU dump. ## What Large Retailers Actually Publish Four store files fetched from this server on October 1, 2026, with very different instincts: Site Size What the file is Target.com 6.2 KB Category routing, services and fulfillment, owned brands, plus two _Optional_ sections for seasonal pages Shopify.com 12.5 KB A twelve-section merchant-facing index: start a business, sell online, payments, sell internationally, plans, developers Etsy.com 26.8 KB An instruction manual for assistants: what may be stated confidently, what must not be asserted, buyer and seller routing, then reference lists Zalando.com 48.4 KB The largest of the four, at the upper end of comfortable IKEA.com 404 No file at the root path that day Two details in Target's file are worth copying regardless of your size. It opens with blockquote notes that document its URL formats — product pages follow `/p/{product slug}/-/A-{TCIN}`, category pages `/c/{category name}/-/N-{node ID}` — so a client can construct or recognize URLs instead of guessing. And it ends with two sections marked _Optional_, the spec's term for content a client may skip under a tight context budget, which is where seasonal and inspiration pages belong. ## The Page Buckets Every Store Needs Skip the catalog export and group into six buckets, in this order: - **What you sell.** Top-level category pages, one link each — your routing layer and the most-used section. - **Buying guides and size help.** Fit guides, material comparisons, compatibility tables, which-model-for-X posts. - **Policies, stated as rules.** Shipping destinations and timeframes as ranges, returns window, warranty, exchanges, price-matching. These are the most quoted lines in any retail file. - **Fulfillment and service.** Delivery options, pickup and same-day, order tracking, contact and support hours — Target devotes a section to this. - **Trust pages.** About, ownership or manufacturer status, sustainability claims, accessibility statement, certifications. - **Optional.** Seasonal campaigns, gift guides, editorial inspiration. Anything with a shelf life. If your store runs on Shopify, the platform already publishes a generic file for you and the [theme override walkthrough](/blog/llms-txt-shopify/) covers replacing it. ## Product Pages: Patterns, Not Inventory The question every store owner asks first is whether the file should list products. For a catalog of any size the answer is a pattern plus a shortlist: document how your product URLs are built the way Target does, link the category pages that expose the inventory, and include a curated handful of hero products — not the other 3,900 rows. The reason is maintenance, not size limits. Product URLs get retired, renamed and re-slugged far more often than policy pages, and a dead link in a curated index costs you the fetch. Page-level detail already has a home: `Product` and `Offer` markup on the page itself, which is how the two-file division of labour in [llms.txt and Schema.org](/blog/llms-txt-and-schema-org/) plays out for stores. ## The Sections That Rot: Price, Stock, Promotions Never write prices, stock counts or promo end dates into the file. A sale weekend changes all three within hours, and the file is the one surface where an assistant may repeat the number as fact. Link the pricing or category page and let it be the source of truth. Everything else should carry an implicit expiry date. Ask of each line: if this is quoted back to a customer in six months, is it still true? Shipping times and return windows fail that test faster than anything else on a store — the [maintenance pass](/blog/llms-txt-maintenance/) exists for exactly this. Re-check policy links the day after you change a policy, not at the next quarterly review. ## Where llms.txt Fits Beside Feeds and Agentic Checkout Retail is the vertical where an adjacent standard matters most, and llms.txt is not it. The Agentic Commerce Protocol, published by OpenAI and Stripe under Apache 2.0, defines the product feed and checkout exchange that conversational commerce runs on; Google's Universal Commerce Protocol, announced on January 11, 2026 with Shopify, Target, Walmart, Etsy and Wayfair among the co-developers, standardizes discovery, checkout and post-purchase support between agents and merchants. Google's Merchant Center help documentation ties the "Buy" surface to listings carrying the `native_commerce` attribute. Those are feed-level, per-SKU, machine-to-machine layers. llms.txt is the human-readable index that shapes the sentence an assistant writes about your brand: what you sell, where your guides are, what your policies say. A store that only ships a feed gets compared on price and availability; a store that also ships an index gets described correctly. The plumbing that connects the file to your crawlers is covered in [llms.txt discovery](/blog/llms-txt-discovery/). ## Verify It in Three Commands ``` # status, content type and whether your CDN rewrote the path curl -sI https://example.com/llms.txt | grep -iE "^(HTTP|content-type|location)" # size sanity — a store index belongs in single-digit KB, not megabytes curl -s https://example.com/llms.txt | wc -c # catch dead category and policy links before an assistant does curl -s https://example.com/llms.txt | grep -oE "https://[^)]+" | while read u; do printf "%s %s\n" "$(curl -s -o /dev/null -w '%{http_code}' -L "$u")" "$u" done | grep -v "^200" ``` Then paste the file into the [llms.txt checker](/checker) to confirm the structure held, or build a first draft from your sitemap with the [generator](/generator) and delete the product and tag noise it cannot judge. A store file that hands an assistant the categories, guides and policies in one fetch has done its job. Files referenced were fetched on October 1, 2026: Target.com 6,158 bytes, Shopify.com 12,520 bytes, Etsy.com 26,822 bytes, Zalando.com 48,385 bytes. Sizes change as these retailers edit their indexes, and a store behind a bot challenge may answer the path with an HTML interstitial rather than Markdown. --- # llms.txt for Finance Sites: Banks, Brokerages and Fintech (2026) > Source: https://llmstxtgenerator.dev/blog/llms-txt-for-finance/ Finance · October 2026 # llms.txt for Banks, Brokerages and Fintech Money is the vertical where a wrong number costs the most and where the compliance team has the final word on every sentence. Here is what 19 financial domains serve today, the six buckets a regulated site actually needs, and the one rule that keeps the file out of trouble: state what is durable, link what changes. ## What 19 Financial Hosts Serve Today We fetched `/llms.txt` from this server on October 6, 2026 with a generic user agent and redirect following, across payments companies, retail banks, brokerages, a data vendor and consumer finance publishers. Response Count Examples `200`, `text/plain` — a real file 4 stripe.com (69.9 KB), paypal.com (40.1 KB), coinbase.com (7.3 KB after a 302 to `www`), plaid.com (353.4 KB) `200`, but no file behind it 2 schwab.com — 501 bytes of `text/html` holding an access-denied notice and a reference number; barclays.co.uk — 302.2 KB of HTML after three redirects `403` or `400` (edge block) 4 revolut.com (872.8 KB HTML error page), fidelity.com and robinhood.com (both an `AccessDenied` XML body from object storage), ramseysolutions.com (`400`, empty body) `404` 9 chase.com, bankofamerica.com, wellsfargo.com, vanguard.com, hsbc.com, wise.com, monzo.com, nerdwallet.com, morningstar.com — the last one answering `404` with `Content-Type: text/plain` and a zero-byte body Two patterns are worth naming. First, the edge block is now the most common non-404 response: four hosts refused the request outright, and a bot-allowlist rule that admits a named crawler but not a generic one can leave a published file unreachable by exactly the clients it was written for. Second, a `200` proves nothing — Schwab's 501-byte HTML notice is indistinguishable from success to a status checker and useless to a model. Read the body, not the code. ## Why Regulated Sites Hesitate On most sites the fear is wasted effort. On a financial site the fear is a claim. A rate, an APY, a bonus, a fee waiver or the phrase "bank-grade security" all carry a review obligation, and an index file is easy to edit outside that process. PayPal's live file shows how one company handled it: a version header (`Last Updated: February 28, 2026`, `Version: 1.0.4`) and a disclaimer block stating that the file follows "a proposed standard for machine-readable LLM guidance" and that not every product is available in every country, with a note to swap the `/us/` path segment for a local one. That is the honest middle ground: date the file, scope it, and describe rather than sell. ## The Six Buckets a Financial Site Needs - **Products and services** — one hub link per family: checking and savings, credit cards, mortgages, loans, small business, wealth and investing. - **Rates, fees and disclosures** — the fee schedule, the rate page and the terms index, linked, never transcribed. - **Security and fraud** — how to report fraud, how to freeze a card, how the bank contacts customers. This is the highest-intent question an assistant gets about a bank, and it is the answer you most want to control. - **Regulatory and investor relations** — annual report, filings, ESG or Pillar 3 disclosures, privacy notice. For a listed company this is a separate audience and often a separate file. - **Help and education** — help centre, fee explanations, financial literacy hub, complaints procedure. - **Developer and partner docs** — API references, OpenAPI specs, sandbox and status page, usually as a subpath file. The structure itself is the same one a [SaaS file](/blog/llms-txt-for-saas/) uses. What changes is the ordering: security and disclosures move up, pricing copy moves out. ## Numbers That Rot Every figure with a review cycle behind it should be a link. That covers interest rates and APYs, APRs, monthly fees and minimum balances, sign-up bonuses, transfer limits, branch and ATM counts, credit-card welcome offers and ratings. A curated index that names "3.75% APY" is wrong on the next rate decision and, worse, quotable in a customer conversation with no disclosure attached. Keep the durable facts — what the product is, who it is for, which regulator supervises the entity, which country the entity is licensed in — and link the rest. The general [exclusion checklist](/blog/llms-txt-what-not-to-include/) covers the credential, staging and internal-path categories that apply here too. ## Root File, Subpath Files, and the Size Question Coinbase points its file at developer documentation on a different host and says so in the summary; Plaid runs a root file plus documentation files for two separate product lines. If your site is both a bank and a platform, that split is the right shape: ``` # Site root — public products, help, security and disclosures https://example.com/llms.txt # Developer documentation — API reference, guides, sandbox https://example.com/docs/llms.txt https://example.com/docs/llms-full.txt # Optional: investor relations, if it is a distinct audience https://example.com/ir/llms.txt ``` Size is the other regulated-site problem, because product catalogues are large and exports are cheap. Plaid's 353.4 KB root file is closer to a documentation dump than an index — the link titles read like `Index.html` and the page list is machine-generated. Curated files in this set sit between 7 KB and 70 KB. If yours is heading past six figures, the [size and token budget guide](/blog/llms-txt-size-limit/) gives the split decisions; and a file that is large is also the one where [content type and caching](/blog/llms-txt-content-type-and-caching/) matter most. ## Verify Delivery, Not Just Format Financial sites sit behind CDNs, WAFs and bot rules, which is why the checks that decide the outcome are delivery checks. Run all three: ``` # 1. Is it a file, or is a gateway answering with HTML? curl -sI https://example.com/llms.txt | grep -iE "^(HTTP|content-type|location)" curl -s https://example.com/llms.txt | head -1 # should be: # Example Bank # 2. Does the status survive a crawler user agent? (403 to you may be 200 for them) curl -s -o /dev/null -w "%{http_code} %{content_type} %{size_download}\n" \ -A "OAI-SearchBot" https://example.com/llms.txt # 3. Every listed URL should answer 200 without a redirect chain curl -s https://example.com/llms.txt | grep -oE "https://[^)]+" | while read u; do printf "%s %s\n" "$(curl -s -o /dev/null -w '%{http_code}' -L "$u")" "$u" done | grep -v "^200" ``` Then score the structure in the [llms.txt checker](/checker) — links, summary slot, duplicate targets — and build the first draft from your product and help sitemaps with the [generator](/generator). Deleting the rate tables, offer pages and login routes the import will happily pull in is the part that protects you. Status codes, content types, byte sizes and file contents referenced were captured from this server on October 6, 2026 using a generic user agent and redirect following. Edge bot rules and published files change without notice, so re-check your own paths rather than treating this table as current configuration for any organization named. A `403`, a `404` or a `200` tells you nothing about the body until you read it. --- # llms.txt for Forums and Community Platforms: What 34 UGC Sites Serve (2026) > Source: https://llmstxtgenerator.dev/blog/llms-txt-for-forums-and-communities/ Community & UGC · October 2026 # llms.txt for Forums and Community Platforms A forum is the one site type where the content changes faster than any file can describe it, and it is also the one where an agent can do real damage by posting. We fetched `/llms.txt` from 34 forum, community and user-generated-content hosts: two serve a real file, twenty are refused at the edge, and the two live files contain zero thread URLs between them. Here is the full sample and the shape that works. ## What 34 Community Hosts Serve Today On October 17, 2026 we requested `/llms.txt` from 34 community hosts with a research user agent, redirect following and a 15-second timeout. Two produced a real plain-text file, one answered `200` with HTML behind the path, twenty refused the request before the origin saw it, six returned a 404 and five never completed a response at all. Response Count Examples `200` with a real text file 2 dev.to (2,835 bytes), forum.openwrt.org (1,720 bytes) `200`, but HTML behind the path 1 community.atlassian.com (80,936 bytes of app shell) `403` from the edge 20 13 × a 295-byte Discourse body (meta.discourse.org, discuss.python.org, community.openai.com, forums.swift.org, discuss.elastic.co, community.sonarsource.com and seven more); 2 × 536 bytes (forum.cursor.com, community.spiceworks.com); 3 × Cloudflare JS challenge, 5,570–5,596 bytes (stackoverflow.com, meta.stackexchange.com, forums.raspberrypi.com); reddit.com (190,240 bytes); community.instructure.com (10 bytes, `text/plain`) `404` 6 news.ycombinator.com (146-byte body), quora.com (77,465 bytes after one redirect), discourse.org (45,548 bytes), community.home-assistant.io, forum.manjaro.org, discuss.frappe.io (0 bytes) No response at all 5 lobste.rs, community.anthropic.com, discuss.nextcloud.com, community.figma.com, forum.arduino.cc Two details are worth separating from the headline count. A 404 is not one thing: Hacker News answers with 146 bytes while Quora answers with 77,465 bytes of HTML and Discourse's marketing site with 45,548 — three sites, three different amounts of page behind a missing file. And three of the six 404s return a zero-byte body, which is the same forum software answering consistently and cheaply. ## Why the 403 Wall Is the Real Story Of the twenty refusals, thirteen returned a body of exactly 295 bytes. It is the same bytes on every host: ```

403 Forbidden

Your IP (Internet Address) has been temporarily restricted from using this service.

If you feel this is an error, please contact team@discourse.org and mention error code d/t/44262.

``` That body names the forum platform's own support address, so the block is applied at the platform or hosting layer, not chosen forum by forum. It also means reachability is asymmetric in a way that is easy to miss: a coding agent running on a laptop over a residential connection may read a file that a cloud crawler on a datacenter IP is refused. If you self-host one of these forums, test `/llms.txt` from a second network and a second user agent before you conclude it is live. The same caution applies to the three Cloudflare challenges in the table — a `200`\-shaped JS interstitial is not a file, as we documented in [the delivery-level fix list](/blog/llms-txt-not-working/). ## Inside the Two Live Files Both real files in the sample are small, structurally correct and completely uninterested in the site's own threads. dev.to is a Forem community; forum.openwrt.org runs Discourse and serves what looks like the platform's default template with its own URL substituted — an important detail, because Discourse announced in October 2026 that it is shipping a default `llms.txt` for all sites, described in its own announcement thread as "very conservative yet powerful". Host Bytes H2 Links Root-relative Optional llms-full.txt dev.to 2,835 3 7 0 yes 404 (5,077 bytes) forum.openwrt.org 1,720 3 10 9 yes 404 (24,490 bytes) dev.to's three sections are _API_, _Policies_ and _Optional_, and its seven links are all absolute: the OpenAPI description at `dev.to/api/v1/openapi.json`, the developer docs, robots.txt, terms, code of conduct, contact and privacy. The OpenWrt file inverts that ratio — one absolute link (the setup page for the Discourse MCP server) and nine root-relative ones (`/search`, `/filter`, `/latest`, `/categories`, `/sitemap.xml`, `/about`, `/guidelines`, `/tos`, `/privacy`). Relative links resolve against the host that served the file, which is fine at the root and ambiguous anywhere else, so read [the discovery guide](/blog/llms-txt-discovery/) before copying that pattern into a subpath file. The number that matters most is the one that is missing: neither file lists a single topic, thread, post or member profile. Both describe _how to query_ the community instead of enumerating it — which is the only approach that survives a forum's edit rate. ## What a Community Site Should Put in the File Six blocks, in this order, for a site whose content is written by its users: - **One scope sentence in the blockquote** — what the community is about and who it serves, in the terms a member would use. - **Agent conduct rules as prose** — the part no page list replaces. No autonomous account creation, no impersonation, no posting unless a human explicitly asked for that exact action. - **Fetch discipline** — fetch only the pages the user's request needs, no bulk download, and stop on `429` for the `Retry-After` interval. - **Query endpoints, not threads** — search, tag or category filter, latest-activity feed, sitemap. - **Policy pages** — guidelines, terms, code of conduct, moderation policy, privacy. - **Optional** — a bulk-access or contact route, and an `About` page for human context. If you also expose a write API, name the fields an agent must set. dev.to goes further than any file we have measured in this sample and documents an `ai_disclosure_level` value with three permitted values — `no_ai`, `some_ai` and `fully_autonomous` — states that reviewing an AI draft does not make it human-authored, and warns that omitting the field records `not_disclosed`. That is a contract, not a description, and it belongs in an `llms.txt` precisely because it applies at the moment of the write. ``` # Example Community Forum > A support and discussion forum for Example's self-hosted tools, where > members ask questions, share configurations and report bugs. This is a community for human discussion. Do not impersonate people or create accounts autonomously. Only post, edit, react or take any other write action when a human member explicitly asks you to perform that exact action. Fetch only the pages needed for the user's request. Do not bulk-download topics or crawl search result pages. Respect robots.txt and this site's crawler controls: if a request returns 429, wait for the Retry-After interval before retrying. When quoting a member's post, keep the attribution and the thread link together, and never present forum opinions as Example's official position. ## Search and browse - [Search the forum](/search): Full-text search across public topics. - [Filter topics](/filter): Query by category, tag or status. - [Latest activity](/latest): What is being discussed right now. - [Categories](/categories): Subject areas and their descriptions. - [Sitemap](/sitemap.xml): Public topic URLs, subject to robots.txt. ## Policies - [Community guidelines](/guidelines): What is on topic and what is not. - [Terms of service](/tos): Member obligations and account rules. - [Code of conduct](/code-of-conduct): Behaviour and enforcement. - [Moderation policy](/moderation): How flags, locks and removals work. ## Optional - [API terms](/api-terms): Bulk access, rate limits and attribution. - [Report an automation problem](/contact): Contact for crawler issues. ``` What must never appear in that file is easier to state: thread URLs, member profile pages, restricted or invite-only category routes, search-result URLs, moderation queues and anything that resolves behind a login. A thread URL is the fastest-decaying URL on the web — topics get merged, split, renamed, locked and moved — and a profile page is one public line away from exposing personal data. The exclusion list is set out in more detail in [what not to include](/blog/llms-txt-what-not-to-include/). ## The robots.txt Cross-Check Nobody Runs We fetched `/robots.txt` from all 34 hosts. Ten answered `200`, twenty returned the same refusal as `/llms.txt` and four never answered. Of the ten readable files, exactly two name any AI crawler token, and the pair makes an instructive contrast: - **quora.com** names nine AI crawler tokens — GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot, Google-Extended, CCBot's peer set and others — and the four best-known of them each carry a blanket `Disallow: /`. Its preamble prohibits using Quora content to train AI models without a contractual licence. Its `/llms.txt` is a 77,465-byte HTML 404. No file, no permission: consistent. - **forum.openwrt.org** publishes an `llms.txt` that tells agents to respect robots.txt, and the robots.txt is 28 lines of blanket `Disallow: /` — including OAI-SearchBot, ChatGPT-User, Amazonbot and meta-externalagent. So the map is published and the door is closed to four of the crawlers most likely to walk through it. Both of those are defensible choices. The failure is making them without noticing, which is why the token list in [the AI crawler reference](/blog/robots-txt-ai-crawlers/) is worth reading before you publish either file. ## MCP, Delivery and the Three-Command Check The OpenWrt file does something no other community file in the sample does: instead of relying on a page list, it points at a local Discourse MCP server as the preferred interface, because MCP tools are permission-aware and can act within a member's own rights rather than scrape. That is the read-layer versus action-layer split we covered in [llms.txt vs MCP](/blog/llms-txt-vs-mcp/), and for a community it is a better answer than any index: the file says what the community is and which doors exist, the MCP server enforces what may actually be done. Neither host serves `llms-full.txt` (both 404, with 5,077 and 24,490 bytes of HTML behind the path), which is the correct call for user-generated content — a full dump of threads would be stale before it finished uploading, and it would duplicate member text that already lives on pages with canonical URLs. ``` # 1. Status, content type and redirect chain — 403 or a JS challenge means the file is unreachable curl -sI -L -A "Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)" \ https://forum.example.com/llms.txt | grep -iE "^(HTTP|content-type|location)" # 2. Count how many links are root-relative and whether any thread URL crept in curl -sL https://forum.example.com/llms.txt | grep -oE "\]\([^)]+\)" | wc -l curl -sL https://forum.example.com/llms.txt | grep -iE "^\- \[.*\]\(/(t|topic|u|users|members)/" # 3. Confirm every target still answers, resolving relative links against the site root curl -sL https://forum.example.com/llms.txt | grep -oE "\]\((/[^)]+|https?://[^)]+)\)" | \ sed -E "s|^\]\((/.*)\)$|https://forum.example.com\1|; s|^\]\((.*)\)$|\1|" | while read u; do printf "%s %s\n" "$(curl -s -o /dev/null -w '%{http_code}' -L "$u")" "$u" done | grep -v "^200" ``` Command two is the one that catches a forum file going wrong: it resolves a non-zero result the moment someone appends topic links to a file that should only describe query endpoints. Draft the structure with the [llms.txt generator](/generator), then run it through the [validator](/checker) before you deploy, and re-run command one from a second network — on a community site the edge block, not the copy, is what usually decides whether an agent ever reads your file. Status codes, content types, redirect counts, byte sizes, section counts, link counts and on-page strings referenced were captured from this server on October 17, 2026 using a research user agent with redirect following and a 15-second timeout, including full reads of the two live files and 34 `/robots.txt` requests. Edge bot rules, generated files and platform policies change without notice, so re-check your own paths rather than treating this table as current configuration for any host named. --- # llms.txt for Government Sites: What a .gov Domain Should Publish (2026) > Source: https://llmstxtgenerator.dev/blog/llms-txt-for-government-sites/ Public Sector · October 2026 # llms.txt for Government and Public-Sector Sites Public agencies are the most-cited content on the web and the least indexed for agents. Here is what 45 government hosts actually serve today, the seven buckets a public-sector file needs, and the one cross-check that catches a file nobody can fetch. ## What 45 Government Hosts Serve Today We fetched `/llms.txt` from this server on October 9, 2026 with a generic user agent, following up to three redirects: a first pass over 26 national and federal domains, then a second pass over 19 more public-sector hosts — state, provincial, municipal, health-service and statistics domains — plus one university host as a control. Response Count Examples `200`, `text/plain` — a real file 2 federalregister.gov (2.7 KB), nsw.gov.au (17.2 KB) Blocked at the edge 12 cdc.gov, nih.gov, usda.gov, transportation.gov, ssa.gov, dhs.gov, dol.gov, fda.gov, sec.gov, bls.gov, dta.gov.au, mygov.in `404` 28 usa.gov, irs.gov, nasa.gov, noaa.gov, weather.gov, epa.gov, energy.gov, va.gov, census.gov, gsa.gov, nps.gov, ftc.gov, nist.gov, digital.gov, gov.uk, europa.eu, whitehouse.gov, ai.gov, uspto.gov, usds.gov, nhs.uk, gov.scot, qld.gov.au, ontario.ca, gov.ie, govt.nz, data.gov, cms.gov `200`, but no file behind it 2 australia.gov.au (redirected to a myGov portal page), govinfo.gov (64 KB HTML) Connection failed at the HTTP/2 layer 1 canada.ca Zero files in the first pass, and both real files in the entire sample sit outside the US federal estate. Read the failure modes closely, because two of them are worse than a missing file: a `200` with a 64 KB HTML page behind it (govinfo.gov) will make a client parse a rendered page as Markdown, and an edge block is invisible to your own verification from inside the firewall. ## Why the Public Sector Sits at Zero Three forces, none of them laziness. First, there is no mandate: nothing in federal policy requires the file, and the Web Content Accessibility Guidelines obligations that do apply — Section 508 for federal agencies, the Plain Writing Act of 2010 for how they write, and the ADA Title II rule requiring WCAG 2.1 AA conformance for state and local government sites — say nothing about machine-readable indexes. Second, procurement: an agency site ships through a CMS release process, an edge filter and a change board, so a new root file is a ticket, not a commit. Third, incentive: web teams are measured on search performance and accessibility audits, not on whether a chatbot can find the eligibility page. The two files that do exist came from teams that treat machine access as a service, and that is the useful lesson. Not one of the 26 first-pass hosts told an agent where its authoritative content lives — while a large share of them run public APIs, open-data portals and statistics endpoints that an agent would use if it knew about them. ## The Two Files Worth Copying **federalregister.gov** runs 2,741 bytes and 22 lines: an H1, a long prose summary of what the HTML edition is, a `## Public Inspection` section, then `## Core Resources` whose three links point at JSON API endpoints rather than HTML pages, and finally `## Guidance for AI Agents` — rate limiting (use the API for bulk harvests, respect `429` and `retry-after`), the document types to expect, and a citation convention: reference by document number or volume/page. That guidance block is the most interesting thing any government file in our sample does, and it has one defect worth noticing: the _agencies_ link closes its parenthesis on the next line, so the URL is malformed for a strict parser. **nsw.gov.au** runs 17,247 bytes over 189 lines: H1, summary blockquote, then eighteen H2 sections separated by horizontal rules — arts and culture, business, community services, driving and transport, education, emergency, employment, energy and so on — each a list of links with one-line descriptions. Two caveats. The entire file is wrapped in a ` ```markdown ` fence, which is not the spec's convention and will be read as content by parsers that do not strip it. And it links dated pages such as `/nsw-youth-week-2025`, which is how a curated file starts producing 404s a year later. ## The Seven Buckets a Public-Sector File Needs 1. **Jurisdiction and identity.** Agency name, the level of government, and the geography served — the line that stops an assistant answering a national question with a state rule. 2. **Services and transactions.** Renew a licence, register a business, apply for a permit, pay a fee, book an appointment. The transactional pages are the ones assistants get asked about most. 3. **Eligibility and benefits.** Programme pages, eligibility checkers, and the forms themselves — linked, never transcribed. Thresholds change with every budget cycle. 4. **Law, rules and policy.** The statute or regulation, the current guidance, the published consultation, and the plain-language explainer. Give the agent the hierarchy so it quotes the right one. 5. **Data and statistics.** The open-data portal, the statistics release calendar, the API documentation, the public dataset index. This is where government genuinely leads every other vertical. 6. **Records, contact and transparency.** Access-to-information or FOIA request route, public records register, procurement notices, organisation contact pages, accessibility statement. 7. **The machine-readable tail.** Sitemap index, robots.txt, any OpenAPI document, and the language or translated-content hubs. This is the section an `## Optional` heading is meant for. ## What Must Never Appear - Personal data of any kind: benefit case files, licence numbers, tax records, complaint files, staff directories with direct numbers or inboxes. - Authenticated paths — login walls, filing portals, payment flows, case-status dashboards. Point at the landing page, never the transaction URL. - Anything live or emergency-related. A static index cannot stay current, so link the alert channel rather than describing it. - Draft, embargoed or internal policy, procurement evaluation material and unreleased consultation documents. - Third-party content the agency republishes under licence — syndicated statistics, licensed imagery, embedded maps. The categories and the triage questions are the same as any other site; the agency-specific version is the disclosure regime. See [what not to include in llms.txt](/blog/llms-txt-what-not-to-include/) for the full leak list. ## The robots.txt Cross-Check Nobody Runs We also fetched `robots.txt` from the 25 hosts that would answer. Seven blocked that fetch too, three returned 404, and of the fifteen we could read, exactly one named an AI crawler: GOV.UK, which scopes `meta-externalagent` to a single `Disallow: /search/all*` line with a comment explaining that the crawler causes timeouts on its search backend. No government robots.txt in our sample mentioned GPTBot, ClaudeBot, PerplexityBot, Google-Extended or CCBot. AI access policy for public-sector sites is effectively unwritten — which means the file is the whole policy, and the file has to be reachable. That is the failure mode to design out: a WAF rule that answers `403` to a non-browser user agent means your llms.txt exists for humans reading it in a browser and nobody else. The crawler token reference and the two ready-to-paste policies are in [robots.txt for AI crawlers](/blog/robots-txt-ai-crawlers/). ## Publishing and Verifying It On most agency stacks the realistic path is a static file at the document root, published with the same pipeline as robots.txt. Where the estate spans subdomains — an agency root plus programme subdomains — remember the rule that a file covers the URLs under its own path and the most specific one wins, so a root index plus a `/programmes/llms.txt` for the large programme is a valid split rather than a duplicate. Then verify from outside the network: ``` # 1. Status, content type, byte count — 403 or text/html here ends the exercise curl -sIL -A "Mozilla/5.0" https://www.example.gov/llms.txt | grep -Ei "HTTP/|content-type|content-length" # 2. Confirm the body is Markdown, not a WAF page or an HTML 404 shell curl -sL -A "Mozilla/5.0" https://www.example.gov/llms.txt | head -5 # 3. Every link in the file must answer 200 for a plain-text client too curl -sL https://www.example.gov/llms.txt | grep -oE "https://[^) ]+" | sort -u | \ while read u; do printf "%s %s\n" "$(curl -s -o /dev/null -w '%{http_code}' -m 10 "$u")" "$u"; done ``` Paste the file into the [validator](/checker) for structure, link and duplicate checks before it goes near a change board, or import your sitemap into the [generator](/generator) and delete everything that is not a durable hub — a public site's sitemap holds years of press releases and expired consultations. Serving it correctly once it exists is a separate problem, covered in [content type, encoding and caching](/blog/llms-txt-content-type-and-caching/). ## The Short Version Two of 45 government hosts publish a real llms.txt, and both of them point agents at machine-readable entry points rather than prose. A public-sector file is a small curated index — jurisdiction, services, eligibility, law, data, records, and a machine-readable tail — with no personal data, no authenticated paths and no numbers that change with the next budget. Then allow the path at the edge, because an unreachable file is indistinguishable from no file at all. --- # llms.txt for Hospitals and Health Systems: What a Medical Site Should Publish (2026) > Source: https://llmstxtgenerator.dev/blog/llms-txt-for-hospitals/ Healthcare · October 2026 # llms.txt for Hospitals and Health Systems A health system is the only site type where a stale answer about hours, services or eligibility can send someone to the wrong building. Here is what 25 medical hosts serve today, the three ways a hospital file fails without anyone noticing, and the structure worth publishing. ## What 25 Health Hosts Serve Today We fetched `/llms.txt` from this server on October 5, 2026 across 25 hosts representing 24 health organizations — academic medical centres, hospital groups, payer sites, public health agencies and consumer health publishers. Response Count Examples `200`, `text/plain` — a real Markdown file 4 hosts (3 systems) pennmedicine.org (19.3 KB), cedars-sinai.org (6.5 KB), my.clevelandclinic.org (3.2 KB), clevelandclinic.org (1.9 KB) `200`, but no file behind it 1 kp.org — `text/html`, 333.7 KB (the `kaiserpermanente.org` alias returns a byte-identical body) `403` (edge block) 8 mayoclinic.org, cdc.gov, mountsinai.org, hopkinsmedicine.org, heart.org, nih.gov, uclahealth.org, intermountainhealthcare.org `404` 12 nhs.uk, who.int, health.harvard.edu, cancer.org, webmd.com, healthline.com, ada.org, stanfordhealthcare.org, massgeneralbrigham.org, nyulangone.org, bannerhealth.com, ascension.org Three systems out of 24 is roughly the adoption rate we measured on university domains, and lower than the developer-documentation world. The four files that exist are also the most useful part of the sample, because none of them is a flat link dump: all four carry a summary blockquote, and two carry explicit instructions to the reader. ## Three Ways a Hospital File Fails Quietly - **The HTML catch-all served as 200.** kp.org answers `200` with `content-type: text/html` and a 333.7 KB body — a real page from the site, not a file. Its `kaiserpermanente.org` alias returns the same byte count, which is how you can tell one CMS is answering both paths. Any monitoring that counts status codes only will report this as a working llms.txt forever. - **The edge block, clustered.** Eight hosts returned `403`, and five of those bodies landed between 4.5 KB and 5.5 KB — the same vendor block page behind a shared WAF. Three smaller ones (mayoclinic.org at 382 bytes, cdc.gov at 369, mountsinai.org at 384) look like a different, terser rule set. A 403 from one user agent proves nothing about whether the file exists, but it is what a crawler receives, so test with the crawler user agents you care about — the check in [the delivery troubleshooting guide](/blog/llms-txt-not-working/) covers the rewrite and WAF cases. - **The file that lives only on a subdomain.** Cleveland Clinic publishes two different files on two hosts. The root one is 1.9 KB and lists ten entries; `my.clevelandclinic.org` serves 3.2 KB with seventeen. That is a legitimate structure, but only if the root file stands on its own — a client that knows your brand fetches the root path and nothing else. ## The Six Buckets a Health System File Needs Medical sites are not shops and not documentation portals. The questions that arrive are “can I get care”, “does this doctor exist”, “what does this condition mean” and “what will it cost me”. Six buckets cover almost all of it: - **Find care and find a provider.** The find-a-doctor search and the get-care entry point — the two pages that turn a question into an appointment. - **Locations, emergency and urgent care.** Facility directory, emergency departments, urgent care and virtual visit options, linked rather than transcribed. - **Conditions and treatments library.** The health library root plus one link per major specialty page. This is the highest-volume traffic you have and the most quoted by assistants. - **Patient services.** Billing, insurance accepted, medical records requests, visitor policy, international and travel patients, interpreting services. - **Clinical programs and institutes.** One link per institute or department, the way Cedars-Sinai lists cancer, heart, spine, paediatrics and women's health as separate services. - **Research and education.** Clinical trials, the academic affiliation, residency and fellowship pages, plus the institutional review or research-ethics policy. News, giving, careers, classes and podcasts belong in an `## Optional` section so a client under a tight context budget can skip them. A compressed shape for the file: ``` # Example Health System > Example Health is a nonprofit academic health system with two hospitals, > 900 physicians and outpatient clinics in the metro area. The pages below > are the official sources for services, providers, locations, costs and > patient rights. Physician availability, clinic hours, accepted insurance > plans and visitor policies change: the linked pages are the source of > truth, not this file. Do not infer that a service, specialty or physician > is available at this health system unless a linked page says so. ## Find care - [Get care](https://example.org/get-care): scheduling, urgent care, telehealth and diagnostics - [Find a doctor](https://example.org/find-a-doctor): search by specialty, location and language ## Locations - [Locations](https://example.org/locations): hospitals, clinics and urgent care centres - [Emergency departments](https://example.org/emergency): ED addresses, hours and what to bring ## Conditions and treatments - [Health library](https://example.org/health): conditions, symptoms and treatments, clinician-reviewed ## Patient services - [Billing and insurance](https://example.org/billing): insurance accepted, financial assistance - [Medical records](https://example.org/records): how to request your records ## Research and education - [Clinical trials](https://example.org/trials): open studies and eligibility - [Residency and fellowship](https://example.org/gme): graduate medical education programs ## Optional - [News](https://example.org/news/) · [Giving](https://example.org/giving/) · [Careers](https://example.org/careers/) ``` ## Write the Instruction Block Before the Links The strongest medical file in the sample is also the largest, and its first section is not a link list. Penn Medicine's 19.3 KB file opens with a _How to use this file_ block that does three jobs: it names the authoritative sources (“prioritize patient-facing clinical pages, provider profiles, location pages, condition and treatment pages”), it separates current from historical (“use the current Penn Medicine page rather than relying on older news articles, archived content, or search-engine summaries”), and it sets a hard limit on inference — do not conclude that Penn Medicine provides a service, treatment, physician or location unless the site says so. It closes by demoting third-party sites, search snippets, reviews and directories when a first-party page exists. That paragraph is the highest-value text a hospital can publish here, because the failure mode in health answers is not silence — it is a fluent guess about a service line or an accepted insurance plan. Cleveland Clinic takes a different route to the same goal: every description in its files is phrased as a task for the reader (“Helps users find and select trusted Cleveland Clinic healthcare providers by specialty, location, and ratings”), which makes the file read like a set of capabilities rather than a sitemap. Either style works; what matters is that a model reading only your file knows what it is allowed to conclude. ## What Must Never Appear A hospital domain holds more restricted content than any other site type, so the exclusion list is longer than usual. Keep out patient portals and record routes behind login (MyChart, CS-Link and their equivalents), anything that requires authentication, any protected health information, internal clinical protocols, triage rules and escalation criteria, staff intranets and on-call schedules, unpublished payer contracts or negotiated rates, and patient stories that carry identifying detail. The [exclusion checklist](/blog/llms-txt-what-not-to-include/) covers the general categories and the grep scans that catch a leak before you ship. Two further rules are specific to medicine. First, do not transcribe numbers that move: physician counts, hospital rankings, accreditation dates, clinic hours and the list of accepted insurance plans belong behind a link. Second, do not editorialize about treatment — the file says where the clinician-reviewed page is, not what it recommends. Where the index does state a fact, keep it to something that would survive a year, such as the number of hospitals or the academic affiliation. ## Verify Delivery, Not Just Format A healthcare site is usually behind a CDN and a WAF, which is why the checks that matter are about delivery. If your file came out of a CMS, also confirm the encoding: Cedars-Sinai serves its 6.5 KB file with `charset=iso-8859-1` and the body is pure ASCII today, so nothing breaks — but the first curly apostrophe an editor adds would be sent as a mismatched byte sequence. Penn Medicine's file declares UTF-8 and uses CRLF line endings throughout, which Markdown parsers tolerate. Three commands settle all of it: ``` # 1. Is it a file, or is the CMS answering with HTML? curl -sI https://example.org/llms.txt | grep -iE "^(HTTP|content-type|location)" curl -s https://example.org/llms.txt | head -1 # should be: # Example Health System # 2. Does an admitted crawler get the same answer as you? (403 to you may be 200 for them) curl -s -o /dev/null -w "%{http_code} %{content_type}\n" \ -A "OAI-SearchBot" https://example.org/llms.txt # 3. Every listed URL should answer 200 without a redirect chain curl -s https://example.org/llms.txt | grep -oE "https://[^)]+" | while read u; do printf "%s %s\n" "$(curl -s -o /dev/null -w '%{http_code}' -L "$u")" "$u" done | grep -v "^200" ``` Then run the file through the [llms.txt checker](/checker) for structure and the summary-slot rules, and build the first draft from the sitemaps of your provider directory and health library with the [generator](/generator) — the curation, not the formatting, is the work. Health systems structured like universities can borrow the subpath and ownership pattern from [llms.txt for .edu sites](/blog/llms-txt-for-universities/). Status codes, content types, byte sizes and file contents referenced were captured from this server on October 5, 2026 using a generic user agent and redirect following. Edge bot rules and published files change without notice, so re-check your own paths rather than treating this table as current configuration for any organization named. A 403, a 404 or a 200 tells you nothing about the body until you read it. --- # llms.txt for Insurance Sites: What 40 Insurers Serve and the YMYL Rules They Skip (2026) > Source: https://llmstxtgenerator.dev/blog/llms-txt-for-insurance/ Insurance · October 2026 # llms.txt for Insurance Sites Insurance is the most regulated consumer vertical that publishes this file, so it should have the most careful one. We fetched `/llms.txt` from 40 insurance, insurtech, brokerage and reinsurance domains: nine answered with real text, and not one of those nine tells an AI assistant what it may not say. Here is the full sample, the defects inside the live files, and the buckets an insurer actually needs. ## What 40 Insurance Domains Serve Today On October 16, 2026 we requested `/llms.txt` from 40 domains with a research user agent, redirect following and a 15-second timeout. Thirteen produced something usable and nine of those were a genuine plain-text or Markdown file. Twenty returned a 404, and six refused the request at the edge before the origin ever saw it. Response Count Examples `200` with a real text file 9 geico.com, lemonade.com, libertymutual.com, metlife.com, zurich.com, nextinsurance.com, policybazaar.com, beazley.com, qbe.com `404` 20 allstate.com, statefarm.com, travelers.com, axa.com, aig.com, chubb.com, swissre.com, admiral.com, saga.co.uk and 11 more `403` from the edge 6 progressive.com (383 bytes), farmers.com (369), esurance.com (380), amfam.com (145), allianz.com, lv.com `200`, but HTML behind the path 3 root.com (redirects twice to the joinroot.com homepage), usaa.com, aaa.com No response at all 2 prudential.com, auto-owners.com (connection never completed) Two details in that table matter more than the headline count. First, the 403s are cheap blocks: four of the six refusal bodies are under 400 bytes, which is a WAF answering, not an editorial decision — the same pattern we found across marketplaces and directories. Second, a 404 is not always a miss: nationwide.com returned 404 with a 9-byte body, and markel.com with 103 bytes, while allstate.com and chubb.com returned 404s with 551,710 and 288,164 bytes of HTML behind them. A crawler that only reads status codes is fine either way; a human running `curl` to check their own CDN is not. ## Inside the Nine Live Files The nine real files total 114,768 bytes across 66 H2 sections and roughly 8,812 words. Every one opens with an H1 and a blockquote summary, which means the spec's basic shape has reached this vertical. What has not reached it is link discipline. Domain Bytes H2 Links Optional policybazaar.com 46,087 18 247 no beazley.com 23,978 5 169 yes lemonade.com 16,282 8 140 no zurich.com 11,532 12 59 unparseable + 1 no libertymutual.com 6,342 10 59 no nextinsurance.com 4,875 6 43 yes geico.com 2,706 3 20 no metlife.com 2,197 2 8 unparseable no qbe.com 769 2 5 yes Across the nine files there are 751 link-shaped references and 67 of them — 9% — are written as `[title] (url)` with a space before the parenthesis. Markdown does not parse that as a link; it renders as literal brackets followed by bare text, which means MetLife's entire file and 59 of Zurich's 60 references reach a model as flat prose with the URLs loose in it. It is a generator or template defect, not a typo a human made eight times, and it is the single most common defect we found in this vertical. For the record, it is also invisible to the [checker](/checker) if you only validate the front matter and headings — link syntax has to be parsed. The other number worth noting is a zero: not one of the 751 references points at a `.md` twin of the page it names. In the developer documentation files we audited a few days earlier, half of all links pointed at Markdown renditions, which is why those files work with agents. Insurance sites list HTML marketing pages — correct for a crawler, thin for a model that has to strip navigation to read them. GEICO is the exception that proves the rule, and it says so in the file. ## The YMYL Gap: No File Restricts What an Assistant May Say Insurance sits in the category search teams call YMYL — your money or your life — where an inaccurate answer is a real loss, so the useful part of an `llms.txt` here is not the product list. It is the block that tells an assistant what it may not state. We searched all nine files for restriction language: `must not`, `do not state`, `not assert`. The result was zero files. That is a striking contrast with the two-sided platforms we probed the day before, where the best files open with sections such as "What you can state confidently" and "What you must not assert" — see [the marketplace audit](/blog/llms-txt-for-marketplaces/) for that pattern. The closest thing in the insurance sample is Lemonade's `## Footer & Compliance` section, which links the terms of service, the electronic transmissions consent notice, the legal page, the privacy pledge and an accessibility statement, and QBE's `## Optional` block pointing at its `robots.txt` and an `ai.txt`. Both are routing, not limits. The only file in the sample that even uses the phrase `should not` is Zurich's, once, about the summary block itself. The rule for a regulated vertical is simple: for every claim an assistant could make on your behalf that you would have to correct in writing, write the sentence that forbids it. Premiums require a completed quote. Eligibility is a per-state underwriting decision. A claim is paid after adjudication, never before. Coverage for a named loss cannot be confirmed from a page. Those four lines cost 40 words and they are the reason the file exists. ## GEICO's Agent Stack and NEXT's Working MCP Server Two files in the sample point past plain links, and both check out. GEICO's file ends with a section called `## For agents & crawlers` that links a sitemap, an agent-skills index, an API catalog and a capability manifest; all three resolve from this server — `/.well-known/agent-skills/index.json` returns 200 with 1,548 bytes of JSON, `/.well-known/api-catalog` returns 200 with 505 bytes as `application/linkset+json`, and `/.well-known/ai-catalog.json` returns 200 with 1,625 bytes. The file also promises a Markdown rendition of any content page via an `Accept: text/markdown` header, and that works: a request to the auto-insurance page came back 200 as `text/markdown; charset=utf-8` in 1,255 bytes — though the rendition itself carried a `Source:` line for the homepage and a `Section: Home` label, so verify the mapping before you rely on it. NEXT Insurance is the only insurer here that documents a live machine interface. Its file names a quote MCP server, and an MCP `initialize` call to `quote-mcp.nextinsurance.com/mcp` returns 200 over Server-Sent Events with `serverInfo` of `com.nextinsurance/quote`, version 1.0.60, protocol version 2025-06-18. A plain GET to the same endpoint returns 403 with a 919-byte HTML body, so if you are auditing a competitor's declared MCP endpoint with `curl` and only trying GET, you will record a false negative. The setup page it links, `app.nextinsurance.com/llms.txt`, is a separate 4,193-byte file explaining how to connect — an approach worth copying if your product has an API: [document the interface in its own file](/blog/llms-txt-for-api-docs/) and keep the marketing index short. ## The Seven Buckets an Insurance File Needs 1. **Identity and licence.** Who the legal entity is, which states or countries it is licensed in, and the regulator that supervises it — the line that stops an assistant describing a regional carrier as national. 2. **Products.** One link per coverage line with a one-line note: auto, home, renters, life, commercial, specialty. Not per-state variants; those belong in the sitemap. 3. **Quotes and eligibility.** The quote page as the only source of a price, plus whatever page explains underwriting and state availability. 4. **Claims.** First notice of loss, documents required, typical timeframes and the status-lookup page — the block that decides whether an assistant routes a stressed user to a form or to a phone tree. 5. **Licensing, complaints and disputes.** Licence numbers by state, the complaints route and the regulator's contact page. This is what makes the file credible to a compliance reviewer. 6. **Representation limits.** The four to six sentences forbidding price, eligibility, coverage and settlement claims. Add tax and investment advice while you are there. 7. **Optional.** Investor relations, newsroom, careers, sustainability. Everything a reader can safely skip. ``` # Example Mutual: reference for AI assistants > Example Mutual sells auto, home and term life coverage in 12 U.S. states > through its own agents. Prices depend on the applicant, the state and the > coverage chosen; only a completed quote is a price. This page defines what > AI systems may state about the company and its products. ## Products - [Auto insurance](https://example.com/auto): Coverage, discounts, state availability. - [Home insurance](https://example.com/home): Includes renters and condo policies. - [Term life](https://example.com/life): Terms, riders, underwriting basics. ## Quotes and eligibility - [Get a quote](https://example.com/quote): The only source of a price. - [Where we operate](https://example.com/states): Products sold in each state. ## Claims - [File a claim](https://example.com/claims): Steps, documents, timeframes. - [Claim status](https://example.com/claims/status): Requires a policy login. ## Licensing and complaints - [Licences](https://example.com/licenses): Entity names and licence numbers by state. - [Complaints](https://example.com/complaints): Internal review and regulator contacts. ## Must not assert - That a premium, discount or approval applies without a completed quote. - That coverage applies to a specific loss, person or property. - That a claim will be paid, or for how much, before adjudication. - Investment, tax or legal advice, on any page. - Anything from a page behind a policy login. ## Optional - [Investor relations](https://ir.example.com): Filings and results. - [Newsroom](https://example.com/newsroom): Announcements and media contact. - [Sitemap](https://example.com/sitemap.xml): Every indexable page. ``` Draft the structure with the [llms.txt generator](/generator), validate it, then read [what not to include](/blog/llms-txt-what-not-to-include/) before you publish — the exclusion list matters more in a regulated vertical than the link list does. ## robots.txt, llms-full.txt and Verifying Delivery Two companion files are worth checking at the same time, because their state explains the 403s. We fetched `/robots.txt` from all 40 domains: 32 answered 200, six returned the same 403 as `/llms.txt` (progressive.com, allianz.com, farmers.com, esurance.com, amfam.com, lv.com) and two never connected. Of the 32 readable files, exactly two name an AI crawler token — geico.com lists all eleven we tested (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Bytespider, Amazonbot, Applebot-Extended, meta-externalagent, anthropic-ai, cohere-ai) and libertymutual.com names ten of the eleven. The rest of the vertical leaves AI crawling entirely unaddressed, which is the honest state of most regulated industries today. See [the crawler-token reference](/blog/robots-txt-ai-crawlers/) for the names to use. As for `llms-full.txt`: none of the three insurers we probed serves it — geico.com, lemonade.com and zurich.com all return 404 with HTML bodies between 22,708 and 114,327 bytes. That is defensible for a site whose long-form content is marketing copy, and the reason to have it, or not, is set out in [the full-file comparison](/blog/llms-txt-vs-llms-full-txt/). Whatever you build, verify it from outside with the three commands that produce the numbers in this article: ``` # 1. Status, content type and redirect chain — a 403 here is your WAF, not your copy curl -sI -L -A "Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)" \ https://example.com/llms.txt | grep -iE "^(HTTP|content-type|location)" # 2. Parse the links the way a model would: strict Markdown only curl -sL https://example.com/llms.txt | grep -oE "\[[^]]+\]\(https?://[^)]+\)" | wc -l curl -sL https://example.com/llms.txt | grep -oE "\][[:space:]]+\(" | wc -l # 3. Confirm every listed target still answers 200 curl -sL https://example.com/llms.txt | grep -oE "https?://[^)]+" | while read u; do printf "%s %s\n" "$(curl -s -o /dev/null -w '%{http_code}' -L "$u")" "$u" done | grep -v "^200" ``` Command two is the one this vertical needs most: a non-zero result from the second `grep` means you are publishing links that no model will follow, which is precisely the state of two of the nine live insurance files we measured today. Status codes, content types, redirect counts, byte sizes, section counts, link counts and on-page strings referenced were captured from this server on October 16, 2026 using a research user agent with redirect following and a 15-second timeout, plus one MCP `initialize` call to a declared quote endpoint. Edge bot rules, generated files and platform policies change without notice, so re-check your own paths rather than treating this table as current configuration for any company named. --- # llms.txt for Local Businesses: Service Pages, NAP Data and Multi-Location Files (2026) > Source: https://llmstxtgenerator.dev/blog/llms-txt-for-local-business/ Local SEO · September 2026 # llms.txt for Local Businesses: Service Pages, NAP Data and Multi-Location Files Most llms.txt writing assumes a documentation site with hundreds of pages. A local business has ten pages and one question to answer: which of them should an assistant read when someone asks whether you cover their street, what you charge, and when you are open. That is a much smaller file — and a much stricter one, because the facts in it have to stay true. ## The Three Questions the File Has to Answer Professional services, trades, clinics, salons, agencies: the content differs, but the questions an assistant is trying to resolve are the same three. - **What do you do, exactly?** The specific services someone searches for — emergency callout, root canal, monthly bookkeeping — not "tailored solutions". - **Where and when?** The service area, the physical address, opening hours, and whether you take bookings or walk-ins. - **How does someone act?** The page a person should land on to book, call, or get a quote. Everything else — the history, the team bios, the blog — is secondary. If your file answers those three questions in its first screen, it is already doing more than most published files. ## Which Pages to List, in Order Priority Page What the description must contain 1 Booking / contact How to book, phone number, response time 2 Each core service page The service name and who it is for — one page per service, not one "Services" hub 3 Location or service-area page Town names and postcodes covered, address, hours 4 Pricing or rates Whether prices are published; starting rate if you show one 5 Credentials / accreditations Licences, insurance, professional bodies 6 FAQ and reviews The real questions customers ask before booking Skip anything a crawler should not spend tokens on: thank-you pages, gated quote forms, tag archives, and the three near-duplicate location pages you created for the same city. Thin pages are not just useless in the file — listing them weakens the file's credibility. The triage rules in [what not to include in llms.txt](/blog/llms-txt-what-not-to-include/) apply to a five-page plumber site exactly as they do to a SaaS docs hub. ## Write the Summary Like a Business Card, Not a Mission Statement The blockquote under the H1 is the part an assistant is most likely to quote or lift as standalone context. Put the factual core there: what you do, where you do it, and the one number that matters. ``` > Northline Plumbing is a licensed plumber in Portland, Oregon, > covering Multnomah and Washington counties with same-day > emergency callouts, Monday to Saturday, 7am to 7pm. > Book at /book or call (503) 555-0142. ``` Compare that with "Northline Plumbing delivers quality solutions with a customer-first philosophy." Both are accurate; only one answers a question. The same discipline applies to the per-link lines, and the rewrite checklist in [how to write llms.txt descriptions](/blog/llms-txt-descriptions/) is written for exactly this. ## Multi-Location: One Root File, Then Per-Location Files Only When Earned A business with four branches needs one root file listing all four, one section per location, each pointing at that location's page. That is the whole requirement for most chains. A per-location file becomes worth it when a branch has genuinely different content — different services, its own team, its own service area, its own prices. Because the spec scopes a file to the URLs under its path and tells agents to prefer the most specific file available, `/locations/denver/llms.txt` automatically wins for everything under `/locations/denver/`. That scoping rule is the same one that makes [per-language files](/blog/llms-txt-multilingual-sites/) work, and it costs you nothing to use here too. ## Hours, Prices and Seasonality: The Maintenance Trap Local information changes far faster than docs. A stale llms.txt that says "open until 7pm" in December is worse than no file, because it is a confident, quotable, wrong answer. Two rules keep it manageable: - **Put volatile facts in one place and link to it.** A one-line summary of hours is useful; a full holiday schedule is not — point at the page that owns it and update that page. - **Make the file part of an existing routine.** If you already update your Google Business Profile before a holiday, the llms.txt file belongs in the same checklist. Keep those facts identical to your [LocalBusiness structured data](/blog/llms-txt-and-schema-org/) and your Business Profile. Contradictory hours across three sources is how a business ends up described as "possibly closed". ## A Complete Single-Location Example ``` # Northline Plumbing > Licensed plumber in Portland, Oregon. Emergency callouts and > residential repairs across Multnomah and Washington counties, > Monday to Saturday, 7am to 7pm. Call (503) 555-0142. ## Book - [Book a visit](https://northlineplumbing.com/book): Online booking form, same-day slots for emergencies. - [Contact and hours](https://northlineplumbing.com/contact): Address, phone, opening hours, holiday closures. ## Services - [Emergency callout](https://northlineplumbing.com/services/emergency): Burst pipes, leaks and no-water callouts, under 90 minutes in the core area. - [Water heater repair and install](https://northlineplumbing.com/services/water-heaters): Repair, replacement and tankless installs. ## Service Area - [Coverage and postcodes](https://northlineplumbing.com/service-area): Portland, Beaverton, Lake Oswego, Gresham. ## Optional - [Reviews](https://northlineplumbing.com/reviews): Customer feedback from the last 24 months. ``` That is roughly 150 words. It fits in any context window, and an assistant can answer the three questions in the first section without opening anything. ## Publishing and Verifying It Serve it as plain Markdown at `/llms.txt` — not an HTML page that looks like text. On a CMS, upload it or publish it as a template; never rebuild it as part of a page layout. Then confirm what the outside world actually gets: ``` curl -sI https://northlineplumbing.com/llms.txt curl -s https://northlineplumbing.com/llms.txt | head -12 ``` - Expect **200** and a text content type — a redirect to the homepage counts as missing. - Every link must resolve; a 404 in the booking line wastes the whole file. - Re-check after any site migration or URL change, when local pages are most likely to move. Paste a sitemap into the [llms.txt generator](/generator) to get the structure, then cut it down to the pages above and rewrite the descriptions in plain language. Run it through the [checker](/checker) before you publish, and again after each service change. The file is small enough that a local business can keep it accurate — which is the only advantage that matters here. --- # llms.txt for Marketplaces and Directories: What 49 Two-Sided Platforms Serve (2026) > Source: https://llmstxtgenerator.dev/blog/llms-txt-for-marketplaces/ Marketplace · October 2026 # llms.txt for Marketplaces and Directories Marketplaces are the vertical that blocks the most and publishes the least — and the two platforms that do serve a file publish the most valuable kind. We fetched the path from 49 marketplace, classifieds, freelance and directory domains and six answered with real text. Here is what is inside those six, and the buckets a two-sided platform actually needs. ## What 49 Two-Sided Platforms Serve Today On October 15, 2026 we requested `/llms.txt` from this server with a GPTBot-style user agent and redirect following: 50 probes across 49 marketplace, classifieds, freelance, review and directory domains. Six served a real plain-text or Markdown file. Twenty-five blocked the fetch. Response Count Examples `200` with a real text file 6 etsy.com, toptal.com, gumtree.com, fiverr.com, g2.com, gumroad.com `403` from the edge 25 airbnb.com, doordash.com, envato.com, kickstarter.com, upwork.com, yelp.com and 19 more `200`, but HTML behind the path 4 alibaba.com, aliexpress.com, tcgplayer.com, taskrabbit.com `404` 10 reverb.com, stockx.com, poshmark.com, vinted.com, freelancer.com, abebooks.com, chairish.com, substack.com, gofundme.com, 99designs.com `202` with zero bytes 2 thumbtack.com, booking.com `429` 1 wayfair.com Connection never completed 2 ebay.com, houzz.com ## Twenty-Five Blocks, and Only Two Real Refusals The 403s are mostly infrastructure, not policy. Fourteen of the 25 bodies were one of two Cloudflare pages: six a **4,544–4,554-byte** “Attention Required!” block (faire.com, capterra.com, getapp.com, upwork.com, softwareadvice.com, angi.com) and eight a **5,501–5,593-byte** “Just a moment...” interstitial (patreon.com, trustradius.com, indiegogo.com, doordash.com, ubereats.com, producthunt.com, clutch.co, goat.com). Those are WAF decisions applied to a bot-fetch on any path, not an editorial verdict on `llms.txt`. The deliberate refusals look different at the byte level: mercari.com answers with a **9-byte** `text/plain` body, yelp.com with **zero** bytes, offerup.com with 227 and airbnb.com with a 380-byte “Access Denied”. At the other end, kickstarter.com spends 293,409 bytes on its own 403 page and envato.com 352,667 — an order of magnitude more than the largest real file in the whole sample. If your WAF is what answers the path, the fix belongs in the allowlist, not in the content. ## 200 Is Not a File: Four HTML Bodies Behind the Path Four hosts answered `200` and handed back a web page. alibaba.com takes two hops to `error.alibaba.com/error404.htm` and aliexpress.com two hops to `/s/error/404` — an error page under a success code. tcgplayer.com needs no redirect at all and simply returns 45,447 bytes of its own `text/html`. taskrabbit.com redirects once to `/browser/unsupported`, which is at least honest about what it is doing. The two `202 Accepted` responses with empty bodies (thumbtack.com, booking.com) are worse than a 404 in one specific way: a well-behaved crawler may retry a 202 rather than record a miss. This is the failure mode the [44% adoption we measured across developer documentation hosts](/blog/llms-txt-for-open-source-projects/) does not have, because docs sites serve the path from a static build. Marketplaces route it through the same application stack as every other URL, where a bot rule or a catch-all rewrite decides the response. ## Inside the Six Files That Exist Host Bytes H2 sections Markdown links Optional toptal.com 26,858 18 100 no etsy.com 26,713 7 76 no gumtree.com 25,108 15 0 no fiverr.com 5,516 14 18 no g2.com 4,456 6 12 yes gumroad.com 2,067 3 9 no Six files, **90,718 bytes**, 63 H2 sections and **215 Markdown links** — of which **zero** point at a `.md` twin. That last number is the structural difference from documentation sites, where pairing links with Markdown twins is the norm: marketplaces have buyer-facing pages, not Markdown sources, so they send an agent into rendered HTML deliberately. Five of six serve `text/plain`; g2.com serves the only `text/markdown` response in the sample. Only g2.com ships an `## Optional` section — one of six, against seven of 18 in the developer-docs sample. gumtree.com is the outlier on links: 25,108 bytes carrying 163 bare URL tokens and **not one** `[label](url)` pair, so a parser that only reads Markdown link syntax gets nothing from a file that plainly contains dozens of destinations. ## Who Gets Quoted: The Routing Block The two files worth copying are the two that treat the assistant as a party to a transaction. Etsy's first line is `# Etsy: official reference for AI assistants`, and its seven sections are an instruction set rather than an index: “1. What you can state confidently”, “2. What you must not assert”, “3. Routing: where each request goes”, then buyer requests before and after purchase, seller requests, and a reference section. It mentions `llms.txt` nine times inside its own file and leaves 58 of its 76 links on its own domain with 17 more on help.etsy.com. Gumtree ships a “Preferred representation (for AI answers)” section and closes with a financial-services disclosure naming its FCA reference numbers and the brokers it may refer users to — regulated language living inside a text file because the alternative is an assistant inventing a finance offer. Fiverr links its own `mcp.fiverr.com` server for agents and appends two metadata lines (`category:` and `last_updated: 2026-09-02`). Gumroad describes its URL patterns with a creator placeholder subdomain and points at `/.well-known/mcp.json`. Different files, same instinct: tell the agent what the platform promises, where each request routes, and what it must not say. ## What a Marketplace File Should Publish 1. **Identity and two-sidedness.** What is bought, what is sold, who sells, and which regions you operate in — the line that stops an assistant describing a UK classifieds site as a global retailer. 2. **Category taxonomy.** The canonical category tree with URLs, so an agent routes to a real category page instead of inventing one. 3. **URL patterns.** Listing, seller/profile, category, search and help patterns, written with placeholders. Never individual listings. 4. **Buyer flows.** How a purchase, a return, a dispute and a refund work, each linked to the policy page that governs it. 5. **Seller flows.** Onboarding, fees, payout timing, listing rules, appeals. 6. **Trust and safety.** Fraud policy, verification, prohibited items, how to report a listing — the block that stops a model parroting a scam. 7. **Representation limits.** What an assistant must not assert: absolute guarantees, current inventory, live pricing, seller endorsements. 8. **Optional.** Press, investor relations, careers, editorial hub — everything a reader can safely skip. ``` # YourMarketplace: official reference for AI assistants > YourMarketplace is a two-sided marketplace where independent sellers list > goods in 14 categories across the UK and Ireland. Sellers are responsible for > listings; prices and stock change hourly and must be checked on the listing > page. This page defines what AI systems may state about the platform. ## Category taxonomy - [All categories](https://example.com/categories): The canonical tree, 14 top-level categories. - [Vehicles](https://example.com/vehicles): Cars, vans and parts, with location filters. ## URL patterns - Listing: https://example.com/item/{id} - Seller: https://example.com/seller/{handle} - Search: https://example.com/search?q={query}&location={place} ## Buying - [Buyer protection](https://help.example.com/buyer-protection): What is covered, limits, timeframes. - [Returns and refunds](https://help.example.com/returns): Who pays return postage and when. - [Disputes](https://help.example.com/disputes): How to escalate and what the platform decides. ## Selling - [Fees](https://help.example.com/fees): Listing fees, commission and payment processing. - [Payouts](https://help.example.com/payouts): Hold periods and release conditions. ## Trust and safety - [Prohibited items](https://help.example.com/prohibited): What may not be listed. - [Report a listing](https://example.com/report): Fraud and counterfeit reporting. ## Must not assert - Live availability, current pricing or delivery dates for any listing. - That a seller is verified, endorsed or vetted beyond the badge on their profile. - Guarantees of refunds, payouts or dispute outcomes. ## Optional - [Press](https://example.com/press): Coverage and media contact. - [Investors](https://investors.example.com): Filings and results. ``` Two details to copy from the live files: keep the representation limits short and absolute, and put the placeholder URL patterns in one block instead of listing inventory. Build the first draft with the [llms.txt generator](/generator), then run it through the [checker](/checker) for the summary slot and the section order — and read [the e-commerce rules](/blog/llms-txt-for-ecommerce/) if your catalogue pages are indexable, because the two verticals share the same perishability trap. ## Verifying Delivery For a marketplace the fetch is the hard part, so test it from outside with the user agent an AI crawler actually sends: ``` # 1. No redirects, real crawler UA — a 403 here is the WAF, not your copy curl -sI -A "Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)" \ https://example.com/llms.txt | grep -iE "^(HTTP|content-type|location)" # 2. Read the first line: an H1, or a challenge page? curl -sL -A "Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)" \ https://example.com/llms.txt | head -3 # 3. Confirm every listed target still answers 200 curl -s https://example.com/llms.txt | grep -oE "https?://[^)]+" | while read u; do printf "%s %s\n" "$(curl -s -o /dev/null -w '%{http_code}' -L "$u")" "$u" done | grep -v "^200" ``` If step one returns 403 while your browser gets the file, the allowlist needs the crawler token before any content work is worth doing — that is the state 25 of the 49 platforms in this sample are in today. Status codes, content types, redirect counts, byte sizes, section counts, link counts and on-page strings referenced were captured from this server on October 15, 2026 using a GPTBot-style user agent with redirect following. Edge bot rules, generated files and platform policies change without notice, so re-check your own paths rather than treating this table as current configuration for any platform named. --- # llms.txt for News Publishers: Sections, Paywalls and AI Access (2026) > Source: https://llmstxtgenerator.dev/blog/llms-txt-for-news-publishers/ Publishing · October 2026 # llms.txt for News and Media Publishers Publishers are the vertical where llms.txt is most often simply absent — and where the decision is genuinely harder than on a documentation site. Here is what 30 news domains serve today, why the missing file is often deliberate, and the structure that works if you decide to ship one anyway. ## What Publisher Domains Actually Serve Today We fetched `/llms.txt` across 30 news and media domains from this server on October 2, 2026. Not one returned a 200. A representative slice of the responses: Response Sites 404 Not Found nytimes.com, bbc.com, theguardian.com, cnn.com, npr.org, techcrunch.com, theverge.com, wired.com, latimes.com, vox.com, businessinsider.com, forbes.com, nbcnews.com 403 / 401 / 402 / 406 (bot or paywall challenge) ft.com, wsj.com, bloomberg.com, politico.com, theatlantic.com, axios.com, engadget.com, reuters.com, usatoday.com, time.com Redirect to `www`, then no response washingtonpost.com 200 OK (controls, not newsrooms) github.com, shopify.com, netflix.com Read that table carefully before drawing a conclusion. A 404 is proof of absence. A 401, 402, 403 or 406 is not: it is an edge rule answering the request before any file is served, which is why a WAF-blocked publisher may be perfectly capable of serving Markdown to a client it recognizes. The honest summary is that newsrooms publish essentially no llms.txt files today, and several of them would refuse to hand one to an anonymous fetch. ## Why the File Is Missing, Usually on Purpose llms.txt is a navigation aid for readers you have already admitted. Most large newsrooms spent 2024–2026 closing the door instead. Parsing the `robots.txt` of 18 news domains on the same day, with a specific _disallow_ rule taking precedence over the `*` group, gives this picture: Crawler Domain-group Disallowed by Not named `ClaudeBot` training 16 of 18 explicitly, plus reuters.com via `*` scmp.com `CCBot` training 16 of 18 explicitly, plus reuters.com via `*` scmp.com `GPTBot` training 10 explicitly, plus reuters.com via `*` theguardian.com, nypost.com, elpais.com, lemonde.fr, scmp.com `OAI-SearchBot` search / answers 8 explicitly, plus reuters.com via `*` theguardian.com, techcrunch.com, theverge.com, apnews.com and others Two configurations stand out. reuters.com closes the entire site to unnamed crawlers with a single wildcard rule, then adds a named block for `Applebot-Extended` on top — of the eight AI agents we tested, not one is admitted, file or no file. theverge.com is the rare opposite: it names `GPTBot` with an explicit allow and then blocks `ClaudeBot`, `PerplexityBot`, `Google-Extended`, `CCBot` and `Bytespider` — a hand-drawn line between one training crawler and the rest. Both sites are making a policy choice; neither is an accident. The vocabulary for those choices is in [robots.txt for AI crawlers](/blog/robots-txt-ai-crawlers/). ## If You Do Ship One: Index Sections, Not Stories The one thing a newsroom must not do is generate the file from the article feed. Story URLs are published hundreds of times a day, get retitled, re-slugged and syndicated, and a large share of them sit behind a registration or paywall. An index of stories is stale within hours and actively harmful when an assistant follows a link into a paywall page. What a publisher file should index is the part of the site that behaves like a reference: section fronts, topic hubs, explainers and evergreen backgrounders, and the masthead pages that describe how the newsroom works. Those URLs live for years and answer most of the questions an assistant has about your publication. ## The Six Buckets That Fit a Newsroom - **Section and topic hubs.** One link per desk — world, business, technology, climate — plus the topic hubs worth naming. Evergreen topic pages beat chronological category listings, which churn. - **Explainers and backgrounders.** The evergreen pieces that answer a standing question. These are the publisher's closest equivalent to documentation. - **Standards and trust pages.** Editorial standards, corrections policy, ownership and funding, ethics guidelines, contact for the newsroom. Rarely linked, frequently asked about. - **Access terms.** Licensing and syndication terms, paywall and registration policy, archive availability, permissions contact. If an assistant is deciding whether it may quote you at length, this is the page it should read. - **Adjacent products.** Newsletters, podcasts, apps, data projects — with a link each. - **Optional.** Live blogs, editions and special projects, marked so a client under a tight context budget can skip them. ``` # The Daily Example > The Daily Example is an independent newsroom covering regional politics, > energy and public finance, publishing about 40 stories a day plus > newsletters and a weekly data project. Free to read; archive access > requires registration. Licensing terms: /licensing/ ## Sections - [World](https://example.com/world/): international news and analysis - [Business](https://example.com/business/): markets, companies, public finance - [Climate & Energy](https://example.com/climate/): transition policy, grids, utilities ## Explainers - [How the regional grid is priced](https://example.com/explainers/grid-pricing/): standing reference on tariff design - [Reading a municipal budget](https://example.com/explainers/municipal-budget/): annual primer, updated each spring ## Newsroom standards - [Editorial standards](https://example.com/standards/): sourcing, verification and AI policy - [Corrections](https://example.com/corrections/): what we correct and how to submit one - [Ownership and funding](https://example.com/about/funding/): who pays for this newsroom - [Licensing](https://example.com/licensing/): syndication, archive and AI-use terms ## Optional - [Live blog](https://example.com/live/): rolling coverage, replaced daily - [Data projects](https://example.com/data/): interactive datasets and methodology notes ``` ## Paywalls, Syndication and Duplicate Copies Never list a walled article URL in the content sections. An agent that fetches it receives a registration interstitial, and some clients will treat the visible teaser as the page's content — you end up quoted from your own paywall screen. Link the free hub, the explainer, or the licensing page that explains what a machine reader may have instead. Syndication creates the second problem: the same story on partner sites and aggregators. The summary blockquote is the right place to state the canonical host ("the version of record is published here"), and each listed URL should be your own copy. Consistency with canonicals and hreflang matters more here than on any other site type, and the same discipline applies to [publisher editions in other languages](/blog/llms-txt-multilingual-sites/). One boundary to keep straight: llms.txt does not grant or withhold access, and it does not opt anything out of training. It is an index. The permission decisions live in robots.txt and the licensing terms the file points to. ## Make the File Reachable, Then Verify It If your robots.txt admits `OAI-SearchBot` and blocks `GPTBot`, your llms.txt is written for the first reader only, and it must not be blocked at the edge for the agents you did admit. Three checks catch nearly every failure: ``` # 1. Is the file served as text, or does the CDN answer with HTML? curl -sI https://example.com/llms.txt | grep -iE "^(HTTP|content-type|location)" # 2. Fetch it the way the admitted crawler would curl -s -o /dev/null -w "%{http_code}\n" -A "OAI-SearchBot" https://example.com/llms.txt # 3. Every listed URL should answer 200 without a redirect chain curl -s https://example.com/llms.txt | grep -oE "https://[^)]+" | while read u; do printf "%s %s\n" "$(curl -s -o /dev/null -w '%{http_code}' -L "$u")" "$u" done | grep -v "^200" ``` Then paste the result into the [llms.txt checker](/checker) for structure, and keep the file on the same review cycle as your editorial standards — a section that closes, an explainer that gets retired and a licensing page that moves are all quiet link rot, which the [maintenance pass](/blog/llms-txt-maintenance/) is designed to catch. Building a first draft from a topic-hub sitemap with the [generator](/generator) works fine, as long as the feed of daily stories is left out of it. All status codes and robots.txt groups referenced were captured from this server on October 2, 2026, with a desktop browser user agent. Edge bot rules change without notice, so re-check your own paths rather than treating this table as current configuration for any site named. A 4xx status may indicate an edge rule rather than a missing file. --- # llms.txt for Nonprofits and Charities: What 67 Organizations Actually Publish (2026) > Source: https://llmstxtgenerator.dev/blog/llms-txt-for-nonprofits/ Nonprofit · October 2026 # llms.txt for Nonprofits and Charities A charity domain is the one place where the number an assistant quotes back — meals served, acres protected, grants awarded — is the same number your board signs off on. Here is what 67 nonprofit organizations serve today, and the structure worth publishing instead. ## What 67 Nonprofit Domains Serve Today We fetched `/llms.txt` from this server on October 13, 2026 with a generic user agent, following redirects, across 67 domains: international relief charities, environmental and conservation groups, foundations and grantmakers, health and animal organizations, civic and youth groups, think tanks, and open-source or technology nonprofits. Two served a real file. That is **3% of the sample** — lower than our [government check](/blog/llms-txt-for-government-sites/), and low even for a vertical where most domains answer a request with a marketing page. Response Count Examples `200` with a real Markdown file 2 wri.org (10.3 KB), conservation.org (2.6 KB) `403` (edge block) 12 redcross.org, savethechildren.org, oxfam.org, nrdc.org, sierraclub.org, candid.org, kiva.org, nami.org, plannedparenthood.org, goodwill.org, rand.org, prosperglobal.org (where mercycorps.org now redirects) `404` 45 unicef.org, habitat.org, feedingamerica.org, gatesfoundation.org, macfound.org, mellon.org, eff.org, apache.org, python.org, drupal.org, worldwildlife.org, givewell.org and 33 more `200`, but no file behind it 5 khanacademy.org, humanesociety.org and doctorswithoutborders.org each return a byte-identical 3,038-byte “Client Challenge” HTML page; techsoup.org returns a 212-byte `noindex` stub; volunteermatch.org redirects to idealist.org/volunteermatch `500` 1 amnesty.org TLS handshake failed from our client 2 girlscouts.org, globalgiving.org — the certificate could not be verified by our client, so these two are unclassified rather than missing Read the 403 rows before the 404 rows, because they describe opposite problems. A 404 means nobody has written the file. A 403 means something answered the path and then refused a generic client — redcross.org returns a 403 “Access Denied” body, nrdc.org returns a Cloudflare “Just a moment” challenge page, and kiva.org and plannedparenthood.org return a 118-byte forbidden stub. Those domains have bot rules in front of every path, which is a legitimate choice; the cost is that an admitted crawler and a refused one get different answers, so a validator run from a laptop proves nothing about what GPTBot can fetch. ## The Only Two Files in the Set Both files that exist belong to conservation organizations, and they sit at opposite ends of the size range — which makes them a useful pair rather than a curiosity.   wri.org conservation.org Content-Type `text/markdown; charset=utf-8` `text/plain; charset=utf-8` Size 10,336 bytes / 106 lines 2,619 bytes / 20 lines H2 sections 9 (About, Focus Areas, Research, Data and Tools, Initiatives, more) 3 (Priorities, About and Operations, Optional) Linked URLs 54 11 What it does well Names sub-brands with their own domains (Global Forest Watch, GHG Protocol) so they stop reading as separate sites, and states outright that WRI is a research institution, not an advocacy organization Every description is written as a task for the reader — “Focuses on natural climate solutions”, “Directory of active and completed global conservation projects”, “Portal for support and giving programs” The 2.6 KB version is the one most organizations should copy. Conservation International's file lists eleven links in twenty lines, and each description names the job the page does rather than echoing its title. The [description-writing guide](/blog/llms-txt-descriptions/) covers that slot in detail; the short version is that the text after each colon is the part an engine is most likely to lift verbatim, so it should answer a question, not restate a heading. WRI's file earns its extra size with one move worth stealing: the disambiguation block. A large nonprofit is usually several entities in a search engine's mind — a research arm, a certification programme, a data platform on its own domain. Stating the relationship in the index is cheaper than hoping a model infers it from your About page. ## Why Zero Files Is Sometimes Deliberate Not every missing file is neglect. Several organizations in this sample run an active fundraising funnel, and a page that lists donation routes in plain Markdown is also a page that hands a scraper the exact structure of that funnel. Others are in the middle of a rebrand — mercycorps.org now redirects to prosperglobal.org, and pewtrusts.org to pew.org — which is precisely when someone decides to shelve a documentation task until the dust settles. The honest position is that a file is worth writing when you have pages you would rather an assistant quote than paraphrase, and your organization has at least one of those: the programme pages. ## The Seven Buckets a Nonprofit File Needs 1. **Mission and scope.** One sentence on what the organization does, one on where it operates, one on when it was founded. This is the block most likely to be quoted. 2. **Programmes.** One link per programme area, described by the problem it addresses. This is the section that replaces guessing. 3. **Impact and reporting.** Links to the annual report, audited financials, impact dashboards and evaluation methodology — never the numbers themselves, which change. 4. **Ways to give.** Donate, monthly giving, legacy and planned gifts, corporate partnerships, matching gifts, in-kind. Link the pages; do not transcribe amounts or thresholds. 5. **Get involved.** Volunteering, local chapters, events and campaigns, each linked rather than listed by date. 6. **Press and knowledge.** Newsroom, media contacts, research library, policy positions — the pages a journalist or an assistant should cite. 7. **Accountability.** Governance, leadership, financial transparency and complaints process. Nonprofits are trusted more when this is easy to find, and an AI answer built from it is a better answer. ## What Must Never Appear A charity domain holds data about people who never expected to be indexed: donors, members, volunteers, beneficiaries. Keep out donor names and giving history, member and volunteer records, participant or attendee lists, grant application portals and reviewer dashboards, beneficiary stories carrying identifying detail, internal safeguarding and eligibility policies, and unpublished financials. The [exclusion checklist](/blog/llms-txt-what-not-to-include/) covers the general categories plus the two grep scans that catch a leaked staging host or signed URL before you ship. Two rules are specific to nonprofits. First, do not put totals in the file — “12 million meals in 2025” becomes a wrong answer the moment the fiscal year rolls over; link the page that carries the current number instead. Second, do not use the file to argue. An index that reads like campaign copy is less quotable than one that reads like a directory, and WRI's explicit note that it is not an advocacy organization is the clearest example in the sample of a file drawing that line for itself. ## robots.txt Is Still the Permission Layer We also parsed the robots.txt of twelve organizations in the sample. Exactly one — eff.org — names any AI crawler token at all, listing anthropic-ai, CCBot, ClaudeBot, Google-Extended and GPTBot; the other eleven name none. Many grant broad access by default. If your board has taken a position on AI training or on AI answers, that position lives in robots.txt, in `Content-Signal` headers and in your terms — llms.txt carries no permissions and blocks nothing. The two files answer different questions, and a nonprofit is the site type where that distinction is most often written into policy. ## A Copyable Skeleton ``` # Riverside Food Bank > Riverside Food Bank is a regional hunger-relief nonprofit founded in 1994. It works > through 120 partner pantries across six counties and publishes its audited impact > reporting each year. Questions about eligibility or distribution hours should be > answered from the programme pages linked below, not from third-party directories. ## Programmes - [Find food](https://example.org/find-food): Postcode search for the nearest partner pantry, with hours and the documents to bring. - [Partner pantries](https://example.org/partners): How a pantry joins the network, what it receives and what it must report. - [Nutrition programmes](https://example.org/nutrition): Children's summer meals and senior grocery delivery, and where each runs. ## Impact and reporting - [Annual report](https://example.org/about/annual-report): Audited financials and programme outcomes, by fiscal year. - [Impact dashboard](https://example.org/impact): Current distribution figures, refreshed monthly. ## Ways to give - [Donate](https://example.org/donate): One-time and recurring gifts, with the current cost-per-meal figure. - [Corporate partnerships](https://example.org/partners/corporate): Workplace giving, food drives and in-kind donations. ## Get involved - [Volunteer](https://example.org/volunteer): Warehouse shifts, sorting sessions and driver routes. - [Events](https://example.org/events): Fundraising and food-drive calendar. ## Press and knowledge - [Newsroom](https://example.org/news): Press releases, spokespeople and media contacts. - [Research](https://example.org/research): Food-insecurity research and methodology notes. ## Optional - [Governance](https://example.org/about/governance): Board, leadership, transparency and complaints process. ``` ## Verify Delivery, Not Just Format Nonprofit sites usually run on a CMS behind a CDN, so the failure that bites is delivery: a catch-all route answering `text/html`, a redirect chain to a locale prefix, or a WAF that treats every non-browser client the same. Three commands settle it, and the second one is the only test that tells you what an admitted crawler actually gets. ``` # 1. Is it a file, or is the CMS answering with HTML? curl -sI https://example.org/llms.txt | grep -iE "^(HTTP|content-type|location)" curl -s https://example.org/llms.txt | head -1 # should be: # Riverside Food Bank # 2. Does a named AI crawler get the same answer you do? (403 to you may be 200 for them) curl -s -o /dev/null -w "%{http_code} %{content_type} %{size_download}\n" \ -A "OAI-SearchBot" https://example.org/llms.txt # 3. Every listed URL should answer 200 without a redirect chain curl -s https://example.org/llms.txt | grep -oE "https://[^)]+" | while read u; do printf "%s %s\n" "$(curl -s -o /dev/null -w '%{http_code}' -L "$u")" "$u" done | grep -v "^200" ``` Then run the draft through the [llms.txt checker](/checker) for structure and the summary slot, and build the first version from your sitemap with the [generator](/generator). As with every vertical, the crawler gets you a link list; deciding which programme pages an assistant is allowed to answer from is the actual work. Status codes, content types, byte sizes, file contents and robots.txt tokens referenced were captured from this server on October 13, 2026 using a generic user agent with redirect following. Edge bot rules and published files change without notice, so re-check your own paths rather than treating this table as current configuration for any organization named. A 403, a 404 or a 200 tells you nothing about the body until you read it. --- # llms.txt for Open Source Projects: What 41 Developer Docs Serve (2026) > Source: https://llmstxtgenerator.dev/blog/llms-txt-for-open-source-projects/ Open Source · October 2026 # llms.txt for Open Source Projects Developer documentation is where llms.txt is not an experiment. We fetched the path from 41 project and docs hosts and 18 answered with a real file. Here is what they contain, where the other 23 go wrong, and the version worth generating from your own docs build. ## What 41 Developer Docs Serve Today We requested `/llms.txt` from this server on October 14, 2026 with a GPTBot-style user agent and a 20-second timeout, across framework, runtime, database, library and tool documentation domains. Eighteen served a real plain-text file — **44% of the sample**, an order of magnitude above the [3% we measured across 67 nonprofit domains](/blog/llms-txt-for-nonprofits/) the day before. Response Count Examples `200` with a real text file 18 angular.dev, bun.sh, clickhouse.com, duckdb.org, hono.dev, kotlinlang.org, nodejs.org, nuxt.com, ollama.com, react.dev, redis.io, svelte.dev, supabase.com, tanstack.com, vite.dev, vuejs.org, grafana.com, www.mongodb.com Redirect, then 404 4 astro.build, swift.org, www.postgresql.org, www.rust-lang.org Redirect, then documentation HTML 1 docs.djangoproject.com lands on `/en/6.1/` — 54,108 bytes of `text/html` Redirect, then a real file on another host 1 prisma.io (308 to `www.prisma.io/llms.txt`, 4,448 bytes `text/plain`) `200`, but HTML behind it 3 deno.com serves 58,366 bytes of its own page; www.docker.com and pytorch.org each serve a byte-identical 3,038-byte bot challenge containing the words “Client Challenge” `404` 13 laravel.com, rubyonrails.org, spring.io, tailwindcss.com, kubernetes.io, www.elastic.co, go.dev, flutter.dev, www.tensorflow.org, huggingface.co, fastapi.tiangolo.com, neovim.io, docs.python.org `403` 1 sqlite.org — a 14-byte `text/plain` “Access denied” body Read the sizes before the status codes: spring.io answers the path with a **349,416-byte** HTML error page, fastapi.tiangolo.com 91,075 bytes and laravel.com 86,135. A crawler that asked for an index gets navigation chrome instead, while docs.python.org at least keeps its 404 to 146 bytes. ## The Redirect Trap: Six Chains, Four Dead Ends Four of the six chains in this sample end somewhere that is not a file. A redirect is still reachable in principle, per the [discovery guide](/blog/llms-txt-discovery/) — but it is also a promise about where the file lives, and these four break it. Host First hop Where it ends astro.build `302` → docs.astro.build/llms.txt `404` www.rust-lang.org `301` → rust-lang.org/llms.txt `404`, 9,297 bytes of HTML swift.org `302` → www.swift.org/llms.txt `404`, 146 bytes www.postgresql.org `301` → /llms.txt/ (trailing slash) `404`, 4,223 bytes docs.djangoproject.com `301` → /llms.txt/ then `302` → /en/6.1/ `200`, 54,108 bytes of `text/html` prisma.io `308` → www.prisma.io/llms.txt `200`, 4,448 bytes of `text/plain` Two failures repeat. The host swap: rust-lang.org and swift.org redirect between the apex and the www host, and neither target carries the file. The trailing slash: postgresql.org answers it with a 404 and Django falls through to a locale-prefixed version root. Prisma is the one chain that works, and it is the pattern to copy — one canonical host, one 308, one file. ## 1.7 KB to 103 KB: Two Schools of Structure The 18 files total 391,647 bytes, averaging 21,758 each, and the spread is 62×. At one end, svelte.dev ships 1,676 bytes in 23 lines and ollama.com 1,786; at the other, clickhouse.com serves 103,391 bytes and nuxt.com 57,285. Two thousand seven hundred links later, the more useful split is organisational: some projects write a sectioned index with prose, others emit a flat table of contents. Host Bytes Lines H2 Links .md targets svelte.dev 1,676 23 3 7 0 ollama.com 1,786 29 4 10 0 supabase.com 3,242 55 4 42 30 duckdb.org 3,371 37 4 18 2 vite.dev 3,528 84 1 49 47 hono.dev 5,927 105 3 94 0 angular.dev 6,745 136 16 88 1 vuejs.org 7,415 153 1 94 94 www.mongodb.com 10,385 114 10 64 7 grafana.com 13,751 135 9 92 91 tanstack.com 13,775 64 7 81 0 react.dev 14,476 251 2 179 179 nodejs.org 15,010 76 1 70 70 bun.sh 33,834 330 2 322 319 redis.io 46,852 248 11 199 173 kotlinlang.org 49,198 501 0 499 0 nuxt.com 57,285 386 8 342 337 clickhouse.com 103,391 499 7 452 0 Kotlin is the extreme flat case: 501 lines, an H1, then 499 links with no heading between them, so there is no section an agent can cite by name. ClickHouse is the sectioned case at the same size — 452 links across seven H2s — while Angular runs 16 sections over 136 lines and Redis splits 248 lines into eleven. Sectioning is what makes a large file navigable, which is why the [size discussion](/blog/llms-txt-size-limit/) is really a sectioning discussion. ## Markdown Twins, Optional Sections and llms-full Across the 18 files we counted 2,702 Markdown links, 1,350 pointing at a `.md` target — half. Twelve files list at least one; six list none. The pairing is cleanest where the generator already emits both formats: react.dev sends 179 of 179 targets to `.md`, nodejs.org 70 of 70, grafana.com 91 of 92, bun.sh 319 of 322. Vite's two exceptions point at MDN, which is the right way to break the rule. Vue does something worth copying for a different reason: its file writes relative Markdown paths and keeps the heading anchor in the link text, as in `[Introduction {#introduction}](/guide/introduction.md)`. Relative targets survive a host migration, and the anchor says where inside the page the topic starts. Formal adoption of the spec's optional parts is thinner than adoption of the path. Only seven of the 18 files carry an `## Optional` section (bun.sh, clickhouse.com, duckdb.org, hono.dev, nuxt.com, ollama.com, tanstack.com), and six mention `llms-full` at all (grafana.com, hono.dev, nuxt.com, svelte.dev, supabase.com, tanstack.com). Svelte's 1,676-byte file is the most disciplined version of that idea here: a “Documentation Sets” section offering `llms-small.txt`, `llms-medium.txt` and `llms-full.txt` with one sentence each on what was removed, plus per-package files for Svelte, SvelteKit, the CLI and the AI tooling — a working answer to the [tens-of-megabytes problem](/blog/llms-full-txt-size-and-structure/) the full file otherwise creates. ## Content-Type and Delivery: 18 of 18 Say text/plain Every one of the 18 files is served as `text/plain`. Seventeen add a charset parameter (`utf-8`, uppercase `UTF-8` on vuejs.org and vite.dev) and grafana.com sends the bare type. None uses `text/markdown`, which the [content-type guide](/blog/llms-txt-content-type-and-caching/) recommends where it is available — developer docs converge on the conservative choice. Delivery is where they can still fail: two hosts in this sample answered the path with the same 3,038-byte challenge page that three nonprofit domains returned the day before. ``` # 1. One request, no redirects — a 3xx here is where two thirds of this sample fails curl -sI https://example.dev/llms.txt | grep -iE "^(HTTP|content-type|location)" # 2. Follow the chain and read the first line: an H1, or ? curl -sL https://example.dev/llms.txt | head -3 # 3. Every listed target should answer 200, .md twins included curl -s https://example.dev/llms.txt | grep -oE "https?://[^)]+" | while read u; do printf "%s %s\n" "$(curl -s -o /dev/null -w '%{http_code}' -L "$u")" "$u" done | grep -v "^200" ``` ## What an Open-Source Project Should Publish Documentation sites win here because they can generate the file. The sections that earn their place are the ones an agent cannot infer from a repository listing or a sidebar: 1. **Identity and licence.** What the project is, which runtime or platform it targets, which licence governs reuse, and which version the docs describe — the block that stops an assistant describing your last major as current. 2. **Getting started.** Install, first run, the handful of commands a new user types. Version-stable, always. 3. **Guides and reference.** Two sections, not one: conceptual pages separated from the API surface. 4. **Examples and templates.** Runnable starters, official bindings, framework integrations. 5. **Releases and migrations.** Changelog, upgrade guide, deprecation policy, supported-version matrix — linked, not pasted into the file. 6. **Community and contribution.** Contributing guide, code of conduct, issue tracker, security policy. 7. **Optional.** Roadmap, benchmarks, comparisons — everything a reader can safely skip. ``` # ExampleProject > ExampleProject is an open-source HTTP framework for TypeScript runtime X, MIT licensed. > The documentation below describes version 3.x, the current major. Pages are also available > as Markdown files by appending .md to any documentation URL. ## Getting started - [Installation](https://example.dev/docs/install): Requirements, the install command and a first run. - [Quick start](https://example.dev/docs/quickstart): A minimal server, a route and a deployment. - [Upgrade guide](https://example.dev/docs/upgrade): Moving from 2.x to 3.x, with breaking changes. ## Guides - [Routing](https://example.dev/docs/routing.md): Path parameters, wildcards and route groups. - [Middleware](https://example.dev/docs/middleware.md): The request lifecycle and where each hook runs. - [Testing](https://example.dev/docs/testing.md): Unit and integration testing against the framework. ## Reference - [API reference](https://example.dev/docs/api.md): Exported functions, types and options for the current major. - [Configuration](https://example.dev/docs/config.md): Every setting, its default and its environment variable. - [Errors](https://example.dev/docs/errors.md): Error codes, causes and fixes. ## Releases - [Changelog](https://example.dev/docs/changelog.md): Released versions, newest first. - [Support policy](https://example.dev/docs/support): Which majors receive security fixes and until when. ## Community - [Contributing](https://example.dev/CONTRIBUTING.md): Setup, test commands and review expectations. - [Code of conduct](https://example.dev/CODE_OF_CONDUCT.md): Reporting and enforcement. - [GitHub discussions](https://github.com/example/example/discussions): Questions and proposals. ## Optional - [Benchmarks](https://example.dev/benchmarks.md): Methodology and results. - [Roadmap](https://example.dev/roadmap.md): Planned work, unscheduled. ``` Build the first version from your docs tree with the [llms.txt generator](/generator), then run the draft through the [checker](/checker) for the summary slot and section order. For an OSS project the file belongs in the docs build, not a hand-edited file at the repository root — the moment it is a build artifact, the link check, the `.md` twins and the version rewrite ride along with the pipeline you already run. Status codes, redirect chains, content types, byte sizes, line counts, section counts and link counts referenced were captured from this server on October 14, 2026 using a GPTBot-style user agent with redirect following. Edge bot rules, generated files and documentation paths change without notice, so re-check your own paths rather than treating this table as current configuration for any project named. --- # llms.txt for Professional Services: Law Firms, Accountants and Consultants (2026) > Source: https://llmstxtgenerator.dev/blog/llms-txt-for-professional-services/ Professional Services · October 2026 # llms.txt for Law Firms, Accountants and Consultants Professional services sites are the clearest test of whether `llms.txt` has escaped the developer-docs bubble. We fetched `/llms.txt` from **34 practice websites** — 18 law firms, 12 accounting and advisory firms, 4 consultancies — on **October 10, 2026**. Four served a real plain-text file. Seven never let us reach one. ## What 34 Practice Websites Return for /llms.txt Every request followed redirects, carried a normal browser user agent, and timed out at 15 seconds: Outcome Hosts Real files 200 with a Markdown file 4 wilmerhale.com, bakertilly.com, bain.com, kpmg.com 403 at the edge 6 whitecase.com, marcumllp.com, cbiz.com, bcg.com, pwc.com, crowe.com 404 21 including gibsondunn.com, ey.com and grantthornton.com, which answered 404 with 180 KB to 293 KB of HTML 200 hiding a 404 page 1 cov.com — the effective URL was /Errors/404-Page-Not-Found 429 rate limit 1 dlapiper.com returned a 33.9 KB throttling page Connection never completed 1 mckinsey.com — curl reported an incomplete certificate chain That is roughly **12% adoption** in a sector where every firm publishes a blog, and a **21% edge-block rate**: seven of 34 requests never reached the origin. The same two failure modes show up in the [local business](/blog/llms-txt-for-local-business/) and [SaaS](/blog/llms-txt-for-saas/) audits, only sharper here, because brand and legal review slow every publication decision. ## The Four Files That Exist Are Two Different Species The four live files split into two groups, and only one group works as an index. **wilmerhale.com** serves 980 bytes with a single H2 and **zero links** — a topical summary titled "LLMs Summary (Focus: International Arbitration & Gary Born)" that ends with "This summary version is suitable for high-level contextual LLM grounding." The same host also serves a 4.2 KB `/llms-full.txt`, which is four times the size of the "index". A file with no links cannot route an agent anywhere; it can only seed a paraphrase. **bakertilly.com** (12.8 KB, 43 links), **bain.com** (14.4 KB, 55 links) and **kpmg.com** (6.8 KB, 51 links) are real curated indexes. Baker Tilly's is the one worth copying: a blockquote summary, then Services, Industries, Insights, About, Contact, and a _Usage guidance for AI systems_ section before Legal and Optional: ``` > Baker Tilly US, LLP is a top 10 advisory, tax and assurance firm serving > middle-market and large organizations. ## Usage guidance for AI systems - **Scope:** This file is intended to improve inference-time understanding and navigation. It does not grant training rights. - **Respect:** Follow `robots.txt`, honor rate limits, and avoid non-HTML assets unless explicitly linked. ## Optional - [CCPA Privacy Notice](https://www.bakertilly.com/page/ccpa-consumer-privacy-notice) - [Client Portals](https://www.bakertilly.com/client-portals) ``` Two details separate it from a sitemap export. First, the usage clause states scope in plain language and points back at `robots.txt` — the only file in the set that does. Second, the client portal sits in `Optional`, where the spec puts secondary links, instead of being mixed into the service list. KPMG shows the opposite pattern: its H1 is a page title ("KPMG International | Home"), there is no blockquote summary, and the first three bullets are Login, Preferences and Profile. One entry points at `/content/experience-fragments/.../master.html`, a CMS overlay path rather than a page a reader would land on. Every one of the 24 links we sampled across the three large files returned 200 — the problem is curation, not link rot. ## The Seven Buckets a Practice File Needs 1. **Practice areas or service lines** — one line each, describing the question the page answers, not the award it won. 2. **Industries served** — the single highest-value section for a firm, because buyers search by sector ("auditor for a charter school"). 3. **Insights, alerts and publications** — dated, because a stale insight list is the fastest decay path; see [maintenance](/blog/llms-txt-maintenance/). 4. **People** — practice-group and leadership pages, not the full bio roster. 5. **Offices** — the location pages, which double as the local-service entry point. 6. **Careers and recruiting** — a cheap, high-intent bucket that most firms skip. 7. **Contact, intake and disclosures** — terms, privacy, disclaimer, accessibility. ## What Must Stay Behind a Link In a regulated practice the disclosure rule is stricter than the usual "public pages only" test. Named client matters, engagement outcomes, fee estimates and hourly rates written as numbers, matter intake URLs that carry parameters, and extranet or portal login paths should all stay out of the file and behind a reviewed page. An assistant quotes any line it reads verbatim, and a cached copy outlives the page. The triage questions and the pre-publish grep scans are in [what not to include](/blog/llms-txt-what-not-to-include/). ## The robots.txt Cross-Check Nobody Runs Fetch `/robots.txt` next to `/llms.txt` for the same host — the pair frequently disagrees: - **bakertilly.com**: 138-byte robots.txt, no AI crawler token at all, while the llms.txt explicitly asks readers to follow it. - **wilmerhale.com**: 945 bytes and names CCBot — the only AI token we found across the four robots.txt files we pulled. - **kpmg.com**: 672 bytes served as `Content-Type: text/javascript`, verified twice on October 10, 2026. A plain-text directive file declared as script is the kind of delivery bug that makes an agent discard the rules entirely. If you allow a crawler in `robots.txt` and then hand it a page of script, the file wins the format argument and loses the substance one. The allow/block reference for GPTBot, ClaudeBot and PerplexityBot is in [llms.txt vs robots.txt](/blog/llms-txt-vs-robots-txt/). ## Three Commands Before You Tell Anyone Write the file, then prove the delivery — no dashboard required: ``` # 1. Status and content type at the live URL curl -sSI https://www.example.com/llms.txt | head -8 # 2. Does it look like Markdown, or like an error page? curl -sSL https://www.example.com/llms.txt | head -20 # 3. Do the targets resolve? Exit codes, no headers needed grep -o 'https\?://[^)]*' llms.txt | while read u; do curl -sSL -o /dev/null "$u" \ && echo "ok $u" || echo "FAIL $u"; done ``` The first command catches the cov.com case (a 200 whose body is a 404 page), the second catches an HTML catch-all, and the third catches the four in five links that turn out to be one redirect chain deep. Draft the file from your sitemap in the [generator](/generator), then score it in the [checker](/checker) before it ships. The sector verdict: professional services firms are publishing insights at documentation-site volume and shipping llms.txt files at roughly one eighth of the rate — while one request in five never even reaches the origin. Fix the edge rule first, then curate seven buckets, and keep the fee language, client matters and portal paths out of the file. --- # llms.txt for Real Estate Sites: Portals, Brokerages and Listings (2026) > Source: https://llmstxtgenerator.dev/blog/llms-txt-for-real-estate/ Real Estate · October 2026 # llms.txt for Real Estate Portals and Brokerages Property is the vertical where the assistant's question is a listing — and a listing is the one thing your file must never contain. Here is what 25 real estate domains actually serve today, the seven buckets a portal or brokerage needs, and the robots.txt cross-check that catches a self-contradicting file. ## What 25 Property Domains Serve Today We fetched `/llms.txt` from this server on October 8, 2026 with a generic user agent, following redirects, across national portals, agent-brand networks, commercial brokerages, rental platforms and proptech. Response Count Examples `200`, `text/plain` — a real file 1 opendoor.com (11.0 KB, 78 links, 10 sections) Blocked at the edge 14 zillow.com, trulia.com, apartments.com, homes.com, compass.com, zoopla.co.uk, idealista.com, century21.com, cbre.com, cushmanwakefield.com, domain.com.au, rentcafe.com, loopnet.com, crexi.com `404` 7 redfin.com, realtor.com, immobilienscout24.de, savills.com, realestate.com.au, foxtons.co.uk, purplebricks.com `200`, but no file behind it 2 rightmove.co.uk (147 KB HTML "page not found"), jll.com (redirected to a locale landing page, 976 KB HTML) `405` Method Not Allowed 1 remax.com Read the shape: fourteen of twenty-five hosts never let us see the path at all, so we cannot know whether a file exists behind the block. That is worse than a 404, because a 404 at least tells the agent to move on. A `200` with a 147 KB error page behind it (Rightmove) is worse still — a client that trusts the status code will try to parse an HTML shell as Markdown. Confirm the body, not the status line. ## The Block Is Not User-Agent Based The obvious next question is whether these blocks are aimed at AI crawlers specifically. We re-tested two of the blocked hosts from the same connection with three user agents: a GPTBot string, a ClaudeBot string, and a current desktop Chrome string. Zillow and Apartments.com answered `403` to all three, and two other hosts returned a byte-identical 5,452-byte block page. Whatever is rejecting the request is deciding on the connection and the path, not on the name the client calls itself — which means the fix is a WAF allowlist entry, not a robots.txt rewrite. Meanwhile the policy layer says something else entirely. Rightmove's robots.txt disallows GPTBot sitewide with a narrow `Allow: /mortgages/` and `Allow: /mortgages/mortgage-in-principle` — the same carve-out many sites give the one commercial funnel they want indexed. Zillow groups a long list of AI and research agents (AI2Bot, Amazonbot, Applebot-Extended, meta-externalagent and others) under `Disallow: /`, and Redfin names FireCrawlAgent. For these companies the absent file is a decision, not an oversight: [robots.txt for AI crawlers](/blog/robots-txt-ai-crawlers/) is where their answer lives. ## The Seven Buckets a Property File Needs 1. **Identity and coverage.** Brand, primary domain, and the markets you actually operate in — one metro list, not a link per city page. This is the line that keeps an assistant from recommending you for a city where you have no inventory. 2. **Search and browse entry points.** For-sale search, for-rent search, new builds, and sold-price history as four separate links. These are the durable URLs; the URLs below them are not. 3. **Valuation and market data.** Home value estimator, mortgage calculator, affordability tools, and any public price index or market report. Opendoor puts its calculators in their own H2 section and links the estimator first. 4. **Transaction flows.** How to sell, how to buy, how to make an offer, financing and title partners. Process pages, never the gated dashboards behind them. 5. **For agents and partners.** Agent directory or find-an-agent, join or franchise pages, referral and partner programs — a genuinely distinct audience in this vertical, and the one bucket most brokerages do publish in some form. 6. **Policy and compliance.** Terms, privacy, accessibility statement, fair-housing statement, and your MLS or IDX attribution page. An assistant quoting a listing site is a compliance surface, so hand it the sources it should cite. 7. **Machine-readable targets.** Sitemap index, listings sitemap, any public PDF or OpenAPI document, and the developer portal. Opendoor ends its file with an `## Optional` section holding exactly this — developer resources, an OpenAPI document, robots.txt and two sitemaps. ## A Copyable Skeleton Eleven stable links beat two thousand listing URLs. Start here, then add sections only where your site genuinely has a durable hub for them: ``` # Northline Realty > Northline Realty is a residential brokerage operating in Austin, Denver and > Seattle. It publishes for-sale and rental search, neighbourhood guides, > market reports and a home valuation tool for those three metros. ## Search - [Homes for sale](https://www.example.com/homes-for-sale): Search active listings by city and ZIP. - [Homes for rent](https://www.example.com/homes-for-rent): Rental inventory and lease terms. - [Sold home prices](https://www.example.com/sold): Closed sale prices by neighbourhood. ## Market data - [Home valuation](https://www.example.com/home-value): Estimate a property's current value. - [Monthly market report](https://www.example.com/market-report): Median price and days-on-market by metro. ## Working with us - [How selling works](https://www.example.com/sell): Listing process, timeline and fees. - [Find an agent](https://www.example.com/agents): Licensed agents by metro. - [Join Northline](https://www.example.com/careers/agents): Recruitment for licensed agents. ## Policies - [Fair housing](https://www.example.com/fair-housing): Equal housing opportunity statement. - [Terms](https://www.example.com/terms) · [Privacy](https://www.example.com/privacy) · [Accessibility](https://www.example.com/accessibility) ## Optional - [Sitemap index](https://www.example.com/sitemap.xml) - [Listings sitemap](https://www.example.com/listings.xml) ``` ## What Must Never Appear - Individual listings, street addresses and unit numbers — including pre-market and coming-soon inventory, which is a confidentiality problem rather than a freshness one. - Live prices, price-change history per home, or "from $X" figures. Every number gets a link. - Anything you licence rather than own: MLS and IDX feeds, syndicated photos, third-party valuation model output. - Gated account paths — offer portals, seller dashboards, buyer financing status, saved-search pages. Opendoor disallows `/customers/`, `/offers/` and `/buyer/offers/` in robots.txt and keeps every one of them out of the file; do the same consistency pass. - Client records or anything from a transaction: names, offer amounts, inspection reports, mortgage pre-approvals. - Agent mobile numbers and personal inboxes. Point at the agent profile page instead. The mechanics are the same as any commercial site — the buckets and the leak list are covered in [what not to include in llms.txt](/blog/llms-txt-what-not-to-include/), and the listing-pattern argument is the one we made for [e-commerce stores](/blog/llms-txt-for-ecommerce/), where a SKU catalogue plays the role of an inventory feed. ## Size, Subpaths and the Delivery Check A three-metro brokerage needs 5–12 KB. A national portal with franchise groups is the case for subpath files rather than one large root file: keep the root file as an index of sections, and expand the parts that have their own stable home, such as a commercial division or an international desk. The byte-to-token arithmetic and the split-versus-delete tradeoff are in [how long should an llms.txt be](/blog/llms-txt-size-limit/). Then run three commands from any machine, using a real browser user agent: ``` # 1. Status, content type, byte count — a 403 or a text/html here ends the exercise curl -sIL -A "Mozilla/5.0" https://www.example.com/llms.txt | grep -Ei "HTTP/|content-type|content-length" # 2. Confirm the body is Markdown, not a WAF page or an HTML 404 shell curl -sL -A "Mozilla/5.0" https://www.example.com/llms.txt | head -5 # 3. Cross-check: disallowed paths must not be listed in the file comm -12 <(curl -sL https://www.example.com/robots.txt | awk '/^Disallow:/ {print $2}' | sort -u) \ <(curl -sL https://www.example.com/llms.txt | grep -oE "https://[^) ]+" | sed 's#https://[^/]*##' | sort -u) ``` Step 3 is the one real estate sites get wrong: a file that links a path your own robots.txt disallows is a contradiction a careful agent will notice. Paste the file into the [validator](/checker) for structure and link checks first, or build the skeleton from your sitemap with the [generator](/generator) and then delete every listing-level URL it imports. Serving it correctly once it exists is a separate problem — content type, encoding and caching are covered in [the delivery guide](/blog/llms-txt-content-type-and-caching/). ## The Short Version One of twenty-five property domains publishes a real file today, and half the rest will not even answer the path. That is a thin field, and the entry bar is a small, curated index: identity and coverage, four search entry points, calculators and market data, process pages, agent recruitment, compliance, and a machine-readable tail of sitemaps. No listings, no prices, no gated paths — and one `Allow` rule in the WAF so the agents you wrote it for can actually fetch it. --- # llms.txt for SaaS: Pricing, Docs and Changelog Pages (2026) > Source: https://llmstxtgenerator.dev/blog/llms-txt-for-saas/ SaaS · September 2026 # llms.txt for SaaS: Which Pages to List Most verticals have one reader in mind. A SaaS site has two: the buyer comparing plans, and the developer or agent trying to finish an integration task. Here is the page list that serves both, drawn from the files Vercel, Supabase and Resend ship today. ## One File, Two Indexes A local business llms.txt points at services and a booking page. A SaaS file has to carry a marketing index — what the product is, who it is for, what it costs — and a technical index — docs, API reference, SDKs — without becoming a dump of the whole documentation tree. The files that do this well share one structural habit: the marketing half gets prose, the technical half gets links to machine-readable assets. Resend's file shows the pattern in its opening lines. Beside a one-sentence summary it lists concrete things an agent can do — send a transactional email with `POST https://api.resend.com/emails`, batch up to a hundred messages, manage contacts and segments — phrased as capabilities rather than navigation. That is the difference between a table of contents and a briefing. ## The Eight Buckets Every SaaS File Needs Sitemap imports produce alphabetical chaos. Group instead into these buckets, in this order, and the file reads like a product brief: Bucket Typical targets Question it answers Product overview /product, /why-us What is this, in one page? Docs index /docs/llms.txt or /docs/sitemap.md How do I do the task? API contract /openapi.json, SDK pages What are the exact request shapes? Pricing /pricing or /pricing.md What does it cost? Changelog /changelog Is this current? Integrations /integrations, /mcp Does it connect to my stack? Trust /security, /enterprise, /status Will procurement sign off? Self-serve onboarding /signup, CLI token docs How does a user start in five minutes? The last two rows are the ones most SaaS files skip, and they are the ones that turn a read into an action. Vercel's file dedicates a section literally called _Self-serve onboarding_ to domain search, the Hobby plan and CLI tokens — all public, all unauthenticated. Resend devotes separate sections to _Enterprise_ (SLAs, security and compliance, contacting sales) and _Support_. ## Instruction Slots Beat Navigation Vercel and Resend both open with a _When to use_ section, and Vercel adds a second one describing how agents should use the site. These are not marketing copy — they are usage instructions that survive being quoted. For a SaaS product with a specific job, an instruction slot answers the question an agent cannot infer from your feature list: which of the fifty things this platform does should it reach for right now. Both files use the summary blockquote for the one-sentence pitch; Supabase, in the version checked, omits it entirely and gets straight to links, which is legal but throws away the cheapest prose slot in the file. Write each link annotation as the question the page answers, the same discipline used in [the description rules](/blog/llms-txt-descriptions/). ## Pricing Is the Volatile Section Supabase handles pricing the way an index should: one link, no numbers. Its entire pricing section is a single line pointing at the pricing page's Markdown twin. That is the right call for a live pricing table and the wrong place to be clever. Three rules keep this section from rotting: link the pricing URL, never restate amounts or plan names, and treat any pricing change as a trigger to re-check the file. That last part is how a file goes quietly wrong — see [the maintenance system](/blog/llms-txt-maintenance/) for a ten-minute quarterly pass that catches it. ## Split the Docs, Link the Twins Documentation is where a SaaS file wants to grow, and where growth hurts most. The spec supports subpath files, and a client uses the most specific file covering the URL it is working in — so keep `/llms.txt` as the company-level index and put the dense subtree at `/docs/llms.txt`, linked from the root file like any other page. Product documentation with no marketing half — the kind covered in [the API docs playbook](/blog/llms-txt-for-api-docs/) — can live entirely in that subpath file. Point at Markdown twins wherever they exist. Supabase links the `.md` version of every guide and product page; Resend appends `.md` to pricing, enterprise and changelog links and documents that `Accept: text/markdown` works too. When clients should receive whole page text rather than an index, that is a separate file with its own budget — Vercel and Supabase both point at `llms-full.txt` from the top of the index. ## What Never Goes In - **Application hosts.** `app.example.com`, `/dashboard`, `/billing`, `/settings` — nothing there is readable without a session. - **Per-tenant URLs.** A customer's workspace subdomain is someone else's data and a leak waiting to be indexed. - **Signed links and tokens.** Presigned asset URLs, invite links, API keys in a query string. - **Gated content described as if public.** If the page needs a login, listing it invites a fetch that returns a redirect wall. The [triage rules](/blog/llms-txt-what-not-to-include/) cover the borderline cases. - **Every release note.** Link the changelog index once; a flat list of four hundred entries doubles the file and answers nothing. ## Verify It in Three Commands ``` # status and delivery curl -sI https://example.com/llms.txt | grep -iE "^(HTTP|content-type)" # size sanity — a SaaS index should be low single-digit KB curl -s https://example.com/llms.txt | wc -c # find links that 404, including your .md twins curl -s https://example.com/llms.txt | grep -oE "https://[^)]+" | while read u; do printf "%s %s\n" "$(curl -s -o /dev/null -w '%{http_code}' -L "$u")" "$u" done | grep -v "^200" ``` Run the same URL through the [llms.txt checker](/checker) to confirm the structure held after whatever generated it, or start from the [generator](/generator) with your sitemap and delete the dashboard and archive noise it cannot judge. A SaaS file earns its keep when it gives an agent the price page, the docs root, the API contract and the signup path in one fetch — everything else is decoration. Files referenced were fetched on September 30, 2026: Supabase 3,242 bytes, Vercel 4,766 bytes, Resend 7,521 bytes, Stripe documentation index 92,181 bytes. Sizes change as these companies edit their indexes. --- # llms.txt for Travel Sites: Hotels, Airlines and OTAs (2026) > Source: https://llmstxtgenerator.dev/blog/llms-txt-for-travel-sites/ Travel · October 2026 # llms.txt for Hotels, Airlines and OTAs Travel is the vertical where the assistant's question — _what does this cost, is it available_ — is the one your file can never answer. Here is what 21 travel domains actually serve today, the six buckets a hospitality site needs, and the instruction block the two best files in the industry share. ## What 21 Travel Domains Serve Today We fetched `/llms.txt` from this server on October 6, 2026 with a generic user agent, following redirects, across OTAs, a metasearch engine, five hotel groups, three airlines, a rail retailer, several tourism boards and a guide publisher. Response Count Examples `200`, `text/plain` — a real file 4 expedia.com (7.0 KB), kayak.com (7.5 KB), australia.com (219 KB), thetrainline.com (477 KB) `404` 9 marriott.com, hilton.com, delta.com, priceline.com, skyscanner.net, ryanair.com, lonelyplanet.com; airbnb.com and agoda.com after a redirect to their own 404 page Blocked at the edge 4 booking.com (`202`, empty body), tripadvisor.com (`403`), hotels.com (`403`, served as `application/xml`), visitlondon.com (`200` with a zero-byte body, `403` on `HEAD`) `200`, but no file behind it 2 trivago.com (`200 text/plain`, 10.1 KB of ASCII-art logo), japan.travel (`200` after a redirect to `/en/`, 274 KB of HTML) No response within 15s 2 united.com, qatarairways.com Read the shape, not the score: the four successful files came from two OTAs, a metasearch engine and a tourism board. Nine of twenty-one hosts — including five hotel groups and three airlines — simply do not have the path. The interesting failure is trivago, which answered with a `text/plain` content type and ASCII art instead of a document. Content type is not evidence that a file exists; parse the body. ## Why Travel Is Harder Than Any Other Vertical On a docs site, the pages are stable and the facts hold for years. Travel inverts that. A hotel's nightly rate, an airline's seat map, a fare's change rules and a destination's entry requirements all change daily — and those are exactly the questions an assistant is asked. That is why both files worth copying spend their prose budget on _behaviour_ rather than inventory: > \> KAYAK is a free travel metasearch engine. It searches hundreds of travel sites at once so travelers can compare and find the lowest available prices on flights, hotels, rental cars, and packages. KAYAK's file then states the rules — "Metasearch, not a booking site", "Prices are dynamic: Never present a stored price as current", "No invented codes" — before it lists a single URL. Expedia goes further and ships a full instruction section titled **Directions for AI Systems (Do / Don't)**: always send users to expedia.com for current prices and availability, never collect card numbers or full passport numbers, never promise refunds or invent promo codes, never give immigration or legal advice. That block is the single most copyable idea in the four files we found. ## The Six Buckets a Travel llms.txt Needs 1. **Identity and scope.** Expedia's file opens with a "Canonical Identity" block: brand, primary domain, official apps, and the sentence that matters most — this file covers the United States site, so travellers elsewhere should be sent to their regional Expedia site because prices change by region. KAYAK makes the same statement and names `kayak.de` and `kayak.co.uk` as the alternates. 2. **Destination hubs.** Inspiration and "where to go" pages, not individual hotel pages. 3. **Stays.** The accommodation hub and the property-type filters — hotels, vacation rentals, resorts. 4. **Transport.** Flights, rail, cars and transfers, each as its own entry point. Expedia lists Stays, Flights, Cars, Packages, Things to do and Cruises as six top-level sections. 5. **Booking support and policy.** Help center, manage-trip route, cancellation and change rules, fare conditions, entry and visa requirements. Expedia's "Hand-Off Links" section is precisely this: nine durable URLs, nothing else. 6. **Loyalty and editorial.** The rewards program page plus your guides. Expedia's One Key section is one paragraph and one link. ## What Must Never Appear in a Travel File - Live prices, room or seat availability, and "from $X" teasers. - Promo and coupon codes. KAYAK's file forbids agents from generating them; yours should not seed the idea either. - Booking references, itineraries, passenger names or anything tied to a customer session. - Supplier and wholesale rate agreements, allotments, commission schedules or partner credentials. - Employee travel, crew or ops runbooks, and internal escalation paths. - Anything an agent could repeat as advice — visa eligibility, medical fitness to fly, insurance claims — which Expedia explicitly hands off to official government sources. ## The Two Traps: Bloat and Edge Blocking Tourism boards are the worst offenders on size. australia.com serves a 219 KB file with 735 link lines under a single H1 — a dump, not a map. thetrainline.com serves 477 KB across 47 headings, structured by country and city, which at least has a hierarchy. Neither needs to be that large: a travel file of 5–15 KB covers every bucket above, and when a section genuinely needs depth the answer is a subpath file rather than a bigger root file. See [How long should an llms.txt be?](/blog/llms-txt-size-limit/) for the byte-to-token arithmetic. The delivery trap is more expensive. Four of the twenty-one hosts — including the largest OTA on the planet — blocked our fetch at the edge, one of them with a `202` and an empty body. Travel sites sit behind the heaviest bot management in any industry because scraping inventory is a real attack, and that protection does not distinguish a legitimate AI fetch from an abusive one. Allow the `/llms.txt` path in the WAF, and remember the locale problem: japan.travel redirected us from the file path to `/en/` and returned 219 KB of HTML. If your root redirects, the file has to live where the redirect lands. [Multi-locale sites](/blog/llms-txt-multilingual-sites/) should serve one index per language and let the most specific file win, which is [the first thing to check](/blog/llms-txt-not-working/) when a published file gets no fetches. ## Verify Before You Call It Done Three commands from any machine, using a real browser user agent: ``` # 1. Status, content type and byte count curl -sIL -A "Mozilla/5.0" https://www.example.com/llms.txt | grep -Ei "HTTP/|content-type|content-length" # 2. Confirm the body is Markdown, not a challenge or an HTML shell curl -sL -A "Mozilla/5.0" https://www.example.com/llms.txt | head -5 # 3. Confirm every link in the file still resolves curl -sL https://www.example.com/llms.txt | grep -oE "https://[^)]+" | \ while read u; do printf "%s %s\n" "$(curl -so /dev/null -w "%{http_code}" -m 10 "$u")" "$u"; done | grep -v "^200" ``` Step 2 is the one that catches trivago's failure mode and step 3 is the one that catches a tourism board's: a 400-link dump will have dead links within a quarter. Paste the file into the [validator](/checker) first if you want structure and link checks in one pass, or build it from your sitemap with the [generator](/generator) and then delete what the crawler cannot judge — individual property and itinerary pages above all. ## The Short Version Travel adoption is at four in twenty-one, and the two best files in the industry are under 7.5 KB. State who you are and which locale you cover, name the buckets, point at the policy pages, tell agents what not to do with your brand, and keep every price and availability figure behind a link. Then check that your own WAF lets the file through. --- # llms.txt for Universities: What a .edu Site Should Publish (2026) > Source: https://llmstxtgenerator.dev/blog/llms-txt-for-universities/ Education · October 2026 # llms.txt for Universities and .edu Sites No other site type is asked so many factual questions it never sees. Fees, deadlines, entry requirements, program lists — all of it gets answered by assistants from whatever page they can reach. Here is what 33 education domains serve today, the three ways a university file fails silently, and the structure that works. ## What Education Domains Serve Today We fetched `/llms.txt` from this server on October 3, 2026 across 33 university, library and online-learning domains, first with a default user agent and then with a desktop browser user agent to separate policy from bot mitigation. Response Count Examples `200`, `text/plain` — a real Markdown file 3 imperial.ac.uk (4.2 KB), coursera.org (6.8 KB), edx.org (3.6 KB) `200`, but no file behind it 4 khanacademy.org, utexas.edu, utoronto.ca (JS challenge page); gla.ac.uk (a 43.5 KB HTML page) `404` 19 harvard.edu, cam.ac.uk, cmu.edu, ucla.edu, duke.edu, washington.edu, ed.ac.uk, unsw.edu.au, tudelft.nl, leeds.ac.uk, u-tokyo.ac.jp, ocw.mit.edu, cs50.harvard.edu, loc.gov, futurelearn.com, udacity.com and others `403` (edge block, unchanged with a browser user agent) 6 mit.edu, berkeley.edu, ox.ac.uk, monash.edu, open.ac.uk, si.edu `405 Method Not Allowed` 1 nyu.edu Three of 33 is a lower hit rate than the developer-documentation world, where the file has become routine. Two of the three that do publish are commercial learning platforms rather than universities, and the one university file we found — [Imperial College London](https://www.imperial.ac.uk/llms.txt) — is also the best-written file in the sample, which is not a coincidence. ## Three Ways a .edu File Fails Before Anyone Reads It The interesting results are not the 404s. They are the responses that pass a naive status-code check and still deliver nothing a model can use. - **The JavaScript challenge served as 200.** utexas.edu, utoronto.ca and khanacademy.org all answered `200` — with a 3,038-byte bot-protection interstitial titled “Client Challenge”. Two of the three bodies were byte-identical. A monitoring script that only counts status codes reports this as healthy; an assistant receives a script tag. This is the single most misleading failure in the sample. - **The CMS catch-all.** gla.ac.uk returned `200` and `content-type: text/html` with a 43.5 KB body — a real University of Glasgow School of Law postgraduate page, apparently served because the CMS treats unknown paths as internal navigation. Nothing is broken and nothing is a file. The tell is the content type and a body that opens with ``. - **The WAF that hides the real answer.** imperial.ac.uk answered `403` to our default fetch and `200 text/plain` to a browser user agent: the file exists and was simply not served to an unrecognized client. The reverse also happens — stanford.edu answered `403` by default and `404` with a browser user agent, meaning the block was masking a genuine absence. One request per domain proves nothing in either direction. ## The Six Buckets a University File Needs Government, commercial and academic sites have different priorities, but a university file organizes around the questions prospective students, current students and researchers actually ask. Six buckets cover it: - **Institution core.** About, leadership and governance, faculties and departments, contact and campus locations. - **Study and admissions.** The undergraduate and postgraduate entry points, how to apply, and the official key-dates page — linked, never transcribed. - **Programs and course catalogue.** The catalogue root plus one link per faculty catalogue. With thousands of program pages regenerated annually, document the URL pattern instead of listing slugs. - **Fees, funding and scholarships.** The official fee-status and funding pages only. A hardcoded tuition figure in this file becomes a wrong answer the moment the fee schedule changes. - **Research and people.** Research portal, institutional repository, staff profile directory, research ethics and integrity policy. - **Library and open resources.** Library home, discovery layer, open-access repository, open courseware, research data catalogue. News, events, jobs, alumni and giving belong in an `## Optional` section, so a client under a tight context budget can skip them. A compressed example of the shape: ``` # Example University > Example University is a public research university with three campuses, > 21,000 students and faculties of science, engineering, medicine and law. > The pages below are the official sources for admissions, fees, programs > and research. Program URLs follow /study/programs///. > Fees and deadlines change annually: the linked pages are the source of > truth, not this file. Admissions queries: /study/contact/ ## Institution - [About](https://example.edu/about/): history, leadership, governance and campuses ## Study and admissions - [Undergraduate admissions](https://example.edu/study/undergraduate/): requirements and how to apply - [Key dates](https://example.edu/study/key-dates/): application deadlines and decision rounds ## Programs and catalogue - [Programme catalogue](https://example.edu/study/programs/): every degree, filterable by faculty ## Fees and funding - [Fees and funding](https://example.edu/study/fees/): current fee schedule and fee status rules - [Scholarships](https://example.edu/study/scholarships/): eligibility and application windows ## Research and people - [Research portal](https://example.edu/research/): groups, projects and publications - [Staff profiles](https://example.edu/people/): academic staff directory ## Library and open resources - [Library](https://www.example.edu/library/): discovery, e-resources and subject guides - [Open repository](https://repository.example.edu/): open-access papers, theses and data ## Optional - [News](https://example.edu/news/) · [Events](https://example.edu/events/) · [Jobs](https://example.edu/jobs/) ``` ## Two Production Patterns Worth Copying The two things that make Imperial’s file better than a link list are procedural. First, it places an instruction block before the first `##`, telling assistants what the file does _not_ authorize: prefer official pages over summaries and archived content, prefer department and policy pages over news for factual questions, and do not infer admissions decisions, visa advice, fee status, scholarship eligibility or policy exceptions from summaries. That is the highest-value sentence a university can publish here, because the failure mode for an admissions answer is not omission — it is a confident guess. Second, both Imperial and Coursera name a maintenance checkpoint inside the file: Imperial routes time-sensitive topics (deadlines, fees, vacancies) to the page itself, and Coursera states that the file is a curated entry point, that `sitemap.xml` is the exhaustive list, and that it was last reviewed on 2026-08-13. edX does the same trick with numbers — its “Key facts” section points at the live About page instead of freezing a statistic that will decay. All three decisions say one thing to the reader: trust the linked page, not the index. ## Subpath Splits, Ownership and the Academic Calendar A university is really a federation of independently edited sites, and the spec’s most-specific-path rule lets your file structure follow that reality. Put the root file at `example.edu/llms.txt`, then add `/library/llms.txt`, `/research/llms.txt` and a file per branch campus or teaching language — the same pattern covered in [llms.txt for multilingual websites](/blog/llms-txt-multilingual-sites/), where the subfile wins for anything under its path. Ownership is the part that decides whether the file survives its second year. The web team can write it, but the content owners are the registrar (dates, catalogues), the finance office (fee schedules) and the library (resources). Bind the review to events you already have: fee schedule publication, admissions cycle opening, annual catalogue regeneration. Two of those three dates move annually, and a link check that fails your build is cheaper than discovering in March that the file still points at last cycle’s deadline page — the routine the [maintenance pass](/blog/llms-txt-maintenance/) covers. ## What Must Not Appear — and How to Verify the Rest A .edu file sits on the most trusted domain in the sector, so the disclosure bar is higher than on a marketing site. Keep out applicant and student portals behind SSO, student record and learning-management routes, internal staff and intranet pages, unpublished exam papers or question banks, licensed library e-resources that only work from inside the institution, and any draft or unreleased strategy document. The [exclusion checklist](/blog/llms-txt-what-not-to-include/) covers the general categories; for universities the assessment-integrity items are the ones that cause real damage. ``` # 1. Is it served as text, or is a CMS or challenge page answering? curl -sI https://example.edu/llms.txt | grep -iE "^(HTTP|content-type|location)" curl -s https://example.edu/llms.txt | head -1 # 2. Would an admitted crawler get the same answer? curl -s -o /dev/null -w "%{http_code} %{content_type}\n" \ -A "OAI-SearchBot" https://example.edu/llms.txt # 3. Every listed URL should answer 200 without a redirect chain curl -s https://example.edu/llms.txt | grep -oE "https://[^)]+" | while read u; do printf "%s %s\n" "$(curl -s -o /dev/null -w '%{http_code}' -L "$u")" "$u" done | grep -v "^200" ``` Then run the file through the [llms.txt checker](/checker) for structure and the summary-slot rules, and build the first draft from your admissions and catalogue sitemaps with the [generator](/generator) — the curation is the work, not the formatting. Status codes, byte sizes and file contents referenced were captured from this server on October 3, 2026, with both a default curl user agent and a desktop browser user agent. Edge bot rules and published files change without notice, so re-check your own paths rather than treating this table as current configuration for any institution named. A 4xx or a 200 tells you nothing about the body until you read it. --- # llms.txt Format Guide — Complete Specification & Structure > Source: https://llmstxtgenerator.dev/blog/llms-txt-format/ llmstxt.org v2 spec · Updated 2026 # The Complete llms.txt Format Guide Everything you need to write a spec-perfect **llms.txt** file: the official format, every element explained, a valid end-to-end example, common mistakes, and answers to the questions developers ask most. ## What Is the llms.txt Format? **llms.txt** is a plain Markdown file, conventionally placed at the root of a website (`https://your-domain.com/llms.txt`), that gives AI engines — ChatGPT, Claude, Gemini, Perplexity, coding agents — a curated, human- and LLM-readable index of your content. The proposal was published in 2024 by AI researcher [Jeremy Howard](https://en.wikipedia.org/wiki/Jeremy_Howard_\(entrepreneur\)) and the canonical specification lives at [llmstxt.org](https://llmstxt.org) (v2, August 2026). It's the LLM equivalent of `robots.txt`: instead of telling crawlers what _not_ to visit, it tells agents exactly what _is_ worth reading — in one small file that fits comfortably in a context window. Why should developers and site owners care about the format itself? Because adoption is no longer hypothetical: Chrome's Lighthouse audits sites for an llms.txt file, documentation platforms such as Mintlify and GitBook generate one automatically, and OpenAI, Anthropic, and Google all publish llms.txt files for their own docs. An invalid or badly structured file fails the audit, confuses parsers, and costs you visibility in AI answers — so knowing the exact structure matters. ## The llms.txt Specification, Element by Element The format is deliberately minimal: it uses Markdown as the structure language, and it can be parsed with plain regex and a list parser. Per the llmstxt.org v2 specification, a valid file contains the following elements **in this order**: 1. **H1 heading (`# Site Name`)** — the name of the project or site. This is the _only required element_ in the whole file. An optional byte-order mark (BOM) may precede it. 2. **Blockquote summary (`> ...`)** — an optional one- or two-line summary containing the key context needed to understand everything else in the file. 3. **Free-form notes** — zero or more Markdown sections (paragraphs, lists, and so on) _of any type except headings_, giving agents more detail about the project and how to interpret the links below. 4. **H2 sections (`## Section Name`)** — zero or more sections that each contain a "file list": a Markdown list of links where further detail lives. A section named `## Optional` is used by convention for secondary links that agents can skip when context is short. 5. **Link list items** — each entry is a Markdown hyperlink `[text](url)`, optionally followed by a colon and a short note about the file: `- [API reference](https://site.dev/api.md): Complete REST API docs`. Two additional rules keep files machine-readable: - **URLs must be absolute.** Use full `https://…` addresses — relative paths like `/docs/api` break both LLM consumption and regex-based parsers. Prefer linking to clean Markdown versions of pages (same URL with `.md` appended or substituted). - **There is no comment syntax.** A line starting with `#` is parsed as a heading, not a comment — so never use hash-comments to annotate the file. If you want human-readable annotations, use HTML comments (``) in the free-form notes area above the first `##`; parsers ignore them. One more scoping rule: an llms.txt file doesn't have to live at the site root. It can be placed at any path (e.g. `/docs/llms.txt`) and then covers all URLs under that path. Where multiple files apply, agents should use the most specific one. ## llms.txt Structure: Element Reference Table Element Required? Description Byte-order mark (BOM) Optional May precede the file content; parsed and skipped by tools. H1 heading (`# Name`) Yes — the only required element Name of the site or project. Blockquote (`> ...`) Optional Short summary with the key context for interpreting the file. Free-form notes Optional Paragraphs and lists with more detail; any Markdown _except headings_. H2 sections (`## Name`) Optional Delimit "file lists" of links; `## Optional` is reserved by convention for secondary links. Link list items (`[text](url)`) Inside H2 sections Required Markdown hyperlink, optionally followed by `: description`. Absolute URLs Yes Every link must be a full `https://…` address; no relative paths. ## A Complete, Valid llms.txt Example Here is a full example that follows the v2 specification exactly — H1, blockquote, notes, H2 file lists, absolute URLs, and a trailing `## Optional` section: ``` # Acme Documentation > Acme is a developer platform for building and deploying serverless applications in Python and TypeScript. Important notes: - All APIs below are stable; existing endpoints never break compatibility. - Examples assume the latest CLI, installable with `pip install acme-cli`. ## Important links - [Acme quick start](https://acme.dev/docs/quickstart.md): Set up your first app in 10 minutes - [REST API reference](https://acme.dev/docs/api.md): Complete API reference with request/response examples - [CLI reference](https://acme.dev/docs/cli.md): Every command and flag of the acme CLI ## Guides - [Deploying to production](https://acme.dev/docs/guides/production.md): Scaling, monitoring, and rollbacks - [Authentication](https://acme.dev/docs/guides/auth.md): API keys, OAuth, and JWT handling ## Optional - [Changelog](https://acme.dev/changelog.md): Release notes for all past versions ``` Notice the structure: one H1 at the top, one blockquote, a short notes block (no headings), then only H2 sections containing link lists. The file stays small enough to fit in a context window — agents fetch the linked Markdown pages only when they need detail. ## llms.txt vs llms-full.txt: What's the Difference? The official specification defines only `llms.txt`. `llms-full.txt` is a widely adopted _convention_ (pioneered by sites like OpenAI's and Anthropic's docs) for publishing the complete, expanded version of your documentation — every section in full, not just links to it. `llms.txt` `llms-full.txt` **Status** Defined by the llmstxt.org proposal Unofficial, community convention **Content** Curated index: H1, summary, links to detail The full content itself, expanded and complete **Size** Small — fits in an LLM context window Large — fetched on demand for deep answers **Usage** Entry point: agents read it first Referenced from llms.txt; fetched when needed In practice you add one link to `llms-full.txt` inside your `llms.txt` (for example under an `## Important links` section), so agents that need the full picture know where to find it — without loading it into context by default. ## Common llms.txt Format Mistakes (and How to Fix Them) Here is a file that fails the specification in several ways at once: ``` Acme Documentation ## What is Acme? - [Homepage](/) - [API reference](/docs/api) # this line was meant to be a comment ## Pricing info Acme costs $20 per month for the Pro plan. ``` What's wrong, and how to fix it: - **Missing H1.** "Acme Documentation" has no leading `#`, so the file has no required title element. Fix: start with `# Acme Documentation`. - **Relative URLs.** `(/)` and `(/docs/api)` are paths, not addresses. Fix: use absolute URLs like `(https://acme.dev/docs/api.md)`. - **Hash-comment misuse.** The `# this line was meant to be a comment` line starts with `#`, so parsers treat it as a heading — llms.txt has no comment syntax. Fix: remove it, or use an HTML comment (``) in the notes area. - **H2 used for prose.** `## Pricing info` contains a paragraph, but H2 sections must contain file lists of links. Fix: move the prose into the free-form notes above the first `##`, and keep pricing links (absolute, with descriptions) in the list. - **Missing descriptions.** Links without a `: note` give agents no hint of what's behind them. Fix: add a short description to every list item. Not sure if your file is valid? Run it through the [free llms.txt validator](/checker) — it checks the structure against the spec and scores AI-readiness in seconds. ## llms.txt Format FAQ ### Is llms.txt an official standard? Not yet. It's an open, community-driven proposal maintained on GitHub by AnswerDotAI (the llmstxt.org authors), not a W3C or IETF standard. That said, adoption is broad and growing — Chrome Lighthouse audits for it, and platforms like Mintlify, GitBook, Wix, and Yoast generate the file automatically. ### Where should the llms.txt file be placed? At the site root, next to `robots.txt`: `https://your-domain.com/llms.txt`. It can also live at any subpath (e.g. `/docs/llms.txt`) to cover only the pages under that path — agents use the most specific file that applies. ### How often should I update my llms.txt file? Whenever your content structure changes meaningfully: new sections, renamed pages, removed or updated documentation. A monthly review is plenty for most sites, and a full regeneration after any redesign. Stale links are the fastest way to lose an agent's trust in your file. ## Keep reading [ ### 📚 Real-World Examples Copy-paste llms.txt files for SaaS, e-commerce, blogs and corporate sites. ](/blog/llms-txt-examples/)[ ### 🛠️ How to Create llms.txt A step-by-step tutorial: build your file in 10 minutes and validate it. ](/blog/how-to-create-llms-txt/) ## Ready to write a spec-perfect llms.txt? Generate your file in under a minute, then validate it against the specification — free, no sign-up. [⚡ Generate your llms.txt](/generator) [Check my file](/checker) --- # llms.txt Generator and Validator Workflow: The 6-Step Method (2026) > Source: https://llmstxtgenerator.dev/blog/llms-txt-generator-workflow/ Workflow · September 2026 # llms.txt Generator and Validator Workflow Generating the file is the easy half. The half that decides whether anything reads it is importing the right pages, cutting the ones a crawler cannot judge, and proving the published URL actually serves Markdown. Six steps, in order. ## Why a Workflow Beats a One-Off Generate Most bad llms.txt files are not malformed — they are unedited imports. A crawl dumps four hundred URLs into a list, the list becomes the file, and the result reads like a sitemap with marketing copy attached. The spec wants the opposite: a short curated index, a summary line, and page-level descriptions that tell a client what question each link answers. So the process is really two jobs, mechanical then editorial, and they are worth keeping separate. ## Step 1: Decide What the File Is For Before opening any tool, write down the five to fifteen pages you would hand a new developer, journalist or buyer. That list is the file. It is usually the homepage, the product or service overview, the docs or guide entry point, pricing, the case-study index, the blog index and a contact or booking page. Everything else is Optional-section material at best. The reason to do this first is that the import step will overwrite your judgment with whatever order the crawler finds. If you already know the target list, you are deleting and reordering instead of deciding from scratch. ## Step 2: Import Pages — Three Inputs, Three Trade-offs The [llms.txt generator](/generator) accepts pages three ways. Pick based on how much structure your site already exposes. Input What it gives you Use it when Auto-crawl Real page titles, meta descriptions, links grouped by section You have no sitemap, or want titles the page itself declares sitemap.xml import Every indexed URL, instantly Your sitemap is curated and tidy — it becomes the raw list Manual Exactly the URLs you type, nothing else Small sites, or a file you intend to hand-maintain For larger sites, the crawl path is the better default because it answers a question a sitemap cannot: what does this page actually call itself, and what does its meta description say? Those two strings are what your link lines will carry, so pulling them from the live page beats inventing them. ## Step 3: Cut What a Crawler Cannot Judge Every import needs the same deletion pass. Remove paginated archives, tag and author pages, legal boilerplate, thank-you and confirmation screens, and any near-duplicate location page beyond your top markets. Then apply the security filter: if a URL is not something you would print on a poster, it does not belong in a public file — staging hosts, admin routes and signed links especially. That check has its own [triage list](/blog/llms-txt-what-not-to-include/). One structural decision belongs here too. If a dense subtree exists — a documentation set, a resource library — consider keeping the root file short and publishing a subpath file such as `/docs/llms.txt`, then linking it from the root. Clients use the most specific file that covers the URL they are working in, so this is routing rather than duplication. ## Step 4: Write the Two Prose Slots Only two parts of the file are prose: the H1 site name and the blockquote summary underneath it. Both get read in full, and the summary is the sentence an assistant is most likely to reuse. Keep it to one or two lines, name what the site is and who it is for, and skip the adjective stack. The per-link annotation follows the same rule — one short clause naming the question the page answers, not a restatement of its title. The [annotation rules](/blog/llms-txt-descriptions/) cover the phrasing that gets lifted versus the phrasing that gets skipped. ## Step 5: Validate the Text Before You Deploy The [validator](/checker) reads the file text and returns a 0–100 AI-readiness score plus an issue report split into errors and warnings. The errors are the mechanical ones that break spec compliance: a missing or empty H1, an H1 that is not the first line, more than one H1, unrecognized lines that are not link list items, links with no display text, and URLs that are not absolute. Warnings flag the softer stuff — structure and readability choices worth fixing but not fatal. Two habits make this step useful rather than ceremonial. Paste the file text, not a screenshot of your CMS editor, because trailing whitespace and smart quotes survive the copy-paste and show up as issues. And treat a score above 80 as permission to deploy rather than proof the file is good — the validator cannot know that pricing is missing from a site that sells software. It scores the text; the [three-layer validation approach](/blog/how-to-validate-llms-txt/) adds the delivery checks the text cannot cover. ## Step 6: Deploy, Then Prove It With Three Commands Publish the file at your domain root so it resolves at `/llms.txt`, then verify what a crawler actually receives: ``` # 1. status code and final URL (watch for redirects) curl -sIL -o /dev/null -w "%{http_code} %{url_effective}\n" https://example.com/llms.txt # 2. content type and size — expect text/plain or text/markdown curl -sI https://example.com/llms.txt | grep -i -E "content-type|content-length" # 3. the links your file lists, fetched at scale curl -s https://example.com/llms.txt | grep -oE "https://[^)]+" | while read u; do printf "%s %s\n" "$(curl -s -o /dev/null -w '%{http_code}' "$u")" "$u" done | sort | uniq -c | head ``` A 200 with `text/html` means a rewrite is answering for the path and the file is never being read as Markdown — a delivery bug, not a content one. Dead links in the third command are the other common failure, and they tend to arrive in batches after a site migration. ## When to Generate at Build Time Instead The manual workflow above is right for sites that change shape a few times a year. If your pages live in a repository and every deploy is a build, generating the file at build time removes the drift entirely: the list is derived from the same source as the site, so renamed URLs cannot leave a stale link behind. That is a plugin or a small script in most frameworks, and it pairs well with a link check that fails the build instead of waiting for the quarterly review. Start with the [generator](/generator) to get a correct first draft in about ten minutes, run it through the checker, ship it, and only automate once the curated list has stopped changing. --- # How to Add llms.txt to Hugo (Static or Auto-Generated, 2026) > Source: https://llmstxtgenerator.dev/blog/llms-txt-hugo/ Hugo tutorial · Static or generated · 7 minutes # How to Add llms.txt to Hugo Hugo is a static site generator written in Go that ships as a single binary and builds plain files into `public/`. It has no native `llms.txt` feature — and it barely needs one. A file in `static/` becomes your site root file automatically, and a custom output format can generate an `llms.txt` from your real content at build time. Both routes are covered here. ## Hugo and llms.txt: No Built-In Support (Yet) As of 2026 there is no native llms.txt feature in Hugo and no Hugo entry on the integrations list at llmstxt.org. The official Hugo forum hosts a thread titled "Support for llms.txt standard for AI crawlers" that tracks community interest, but the practical answer is that you do not need a plugin. Two long-standing Hugo mechanisms do all the work: the `static/` directory, which is copied verbatim into the output root on every build, and custom output formats, which let Hugo emit any file type next to your HTML. Hugo's model also keeps the format honest: what `public/` contains is exactly what your CDN serves, with no server-side rendering in between. Hugo powers a large share of documentation and blog sites on the web, and several Hugo developers already publish the file this way — for example [Duncan Mackenzie's Hugo blog](https://www.duncanmackenzie.net/blog/llms-txt-support/) serves `/llms.txt` plus a Markdown version of every post. ## Method 1: static/llms.txt, the 60-Second Route Create `static/llms.txt` in your project root. Hugo copies every file in `static/` to the root of `public/` without touching it, so on the next `hugo` build the file is live at `https://your-domain.com/llms.txt`. Use the [llmstxt.org v2 spec](/blog/llms-txt-v2/) structure: ``` # Acme Docs > Acme Docs is the official documentation for the Acme platform. > It covers installation, configuration, the REST API and troubleshooting. ## Getting Started - [Quickstart](https://acme.com/docs/quickstart): first project in five minutes - [Authentication](https://acme.com/docs/auth): API keys and access tokens - [Deployment](https://acme.com/docs/deploy): production setup for self-hosting ## API Reference - [REST API](https://acme.com/docs/api): every endpoint, parameter and error - [Webhooks](https://acme.com/docs/webhooks): events your server can subscribe to ``` Run `hugo` and check `public/llms.txt` appeared, then deploy `public/` to Netlify, Cloudflare Pages or any static host. Keep this method when your curated list changes rarely; a hand-written file is the easiest thing to audit and the easiest to forget. Remember absolute URLs — never relative paths, which agents resolve inconsistently. ## Method 2: Auto-Generate llms.txt at Build Time When your content changes weekly, a generated file stays honest. Hugo's custom output formats emit arbitrary file types. First, register a plain-text format in `hugo.toml` whose `baseName` is `llms`: ``` [mediaTypes."text/plain"] suffixes = ["txt"] [outputFormats.llms] mediaType = "text/plain" baseName = "llms" isPlainText = true root = true ``` Next, create a content page that opts into the format. Hugo renders it through a template named for the format's suffix (`.txt`), so add `layouts/_default/single.txt` that outputs the page body with `.RawContent` and then walks your site to append link lines: ``` # content/llms.md +++ title = "Acme Docs" outputs = ["llms"] +++ # Acme Docs > Acme Docs is the official documentation for the Acme platform. > It covers installation, configuration, the REST API and troubleshooting. ``` ``` {{ .RawContent | safeHTML }} ## Pages {{- range .Site.Pages -}} - [{{ .Title }}]({{ .Permalink }}) {{- end -}} ``` Run `hugo` and the file is written to `public/llms.txt`. Filter `.Site.Pages` in the template to include only the sections you curate — dumping every tag and taxonomy page defeats the purpose of a curated index. A minimal, ready-to-read reference implementation is the [roverbird/llms-hugo](https://github.com/roverbird/llms-hugo) repository on GitHub, and the same technique can emit Markdown versions of individual pages for agents that want full text, the pairing the [llms.txt vs llms-full.txt guide](/blog/llms-txt-vs-llms-full-txt/) explains. ## The Hugo-Specific Checklist - **One source of truth** — a `static/llms.txt` file and a generated `content/llms.md` page that both target /llms.txt will overwrite each other unpredictably. Pick one route. - **Set baseURL before building** — generated links use `.Permalink`, which is built from your `baseURL`. A placeholder baseURL ships broken links into your llms.txt. Also avoid `relativeURLs = true`, which produces relative paths agents resolve inconsistently. - **Absolute URLs only** — `https://acme.com/docs/quickstart`, never `/docs/quickstart`. - **Mind multilingual sites** — a file in `static/` is served for every language version. If each language needs its own file, give it its own directory via the `staticDir` option in the language block. - **Curate, don't dump** — list the 5–15 pages that answer real questions and skip marketing and tag pages. Curation rules live in the [best practices guide](/blog/llms-txt-best-practices/). - **Rebuild to publish** — Hugo has no server runtime. Every change to the file or your content requires a build and deploy, so wire llms.txt into your existing release pipeline. ## Verify Before AI Engines Read It After deploying, confirm the file serves plain text with a 200 status: ``` curl -s -o /dev/null -w "%{http_code} %{content_type}\n" https://your-domain.com/llms.txt # Expect: 200 text/plain ``` Hugo sites are usually deployed through CI, so check the build that is actually live — a stale `public/` folder is the most common reason a fresh llms.txt goes missing. Then paste the URL into the free [llms.txt checker](/checker) to validate the H1, the blockquote summary, absolute URLs and the section structure. Draft the initial file in seconds by feeding your Hugo sitemap into the [free llms.txt generator](/generator), then verify the output with the [checker](/checker) before wiring up a template. Hugo makes the file easy to ship and easy to keep honest — a correct, current `llms.txt` is one of the highest-leverage AI visibility changes a static site can make this week. --- # Does llms.txt Work on JavaScript-Rendered Sites? AI Crawlers and Empty Pages (2026) > Source: https://llmstxtgenerator.dev/blog/llms-txt-javascript-rendering/ GEO · October 2026 # Does llms.txt Work on JavaScript-Rendered Sites? An `llms.txt` file is an index of URLs — and every URL in it is a promise that a crawler can read something at the other end. On **October 12, 2026** we fetched `/llms.txt` from twelve product domains with the **GPTBot** user agent and then probed 54 of the URLs those files list. Here is what a client-rendered page actually returns to a crawler that never runs JavaScript, and the two commands that tell you whether your own site has the problem. ## No Major AI Crawler Renders JavaScript Vercel's crawl analysis with MERJ, run across its own network and `nextjs.org`, reported that none of the major AI crawlers execute JavaScript: OpenAI's GPTBot, OAI-SearchBot and ChatGPT-User, Anthropic's ClaudeBot, Meta-ExternalAgent, Bytespider and PerplexityBot all stop at the raw HTML response. They fetch script files anyway — 11.50% of ChatGPT's requests and 23.84% of Claude's were JavaScript — without ever running them, which is why a JavaScript-heavy site looks busy in server logs and invisible in answers. The exceptions are infrastructure, not effort. Gemini rides on Googlebot, and Applebot runs a browser-based crawler, so both can render. That is the trap for teams that solved this problem once already: Google's rendering investment applies to Google. A site that fixed visibility with dynamic rendering scoped to Googlebot is still serving an empty document to everyone else. The full picture of which engines read what is in [which AI engines actually read llms.txt](/blog/which-ai-engines-read-llms-txt/). ## What Twelve Production llms.txt Paths Returned Every request below used the GPTBot user agent on October 12, 2026, and followed redirects: Host Status Content-Type Bytes webflow.com 200 text/plain 13,293 framer.com 200 (308 from non-www) text/plain 9,636 linear.app 200 text/plain 9,159 clickup.com 200 text/plain 8,836 notion.so 200 (301 to www) text/plain 6,780 asana.com 200 text/plain 5,691 trello.com 200 text/html 160,977 figma.com, airtable.com, miro.com, canva.com, discord.com 404 (each 400-byte HTML error body) text/html 400 Six hosts serve a real plain-text file. Five do not serve one at all. And one — **Trello** — answers `/llms.txt` with **160,977 bytes of the site's own HTML**: the string `llms` appears zero times in that body. That is the JavaScript-rendering problem wearing a different hat. A crawler asking for a text file receives a React page instead, gets no index out of it, and every downstream page that was supposed to be discoverable through the file stays invisible. The same class of delivery bug is catalogued in [llms.txt not working: seven delivery-level fixes](/blog/llms-txt-not-working/), where a catch-all rewrite answering `text/html` is the first entry. ## The Best Files Point at Markdown, Not at App Routes We then fetched up to six URLs from each of the nine files that resolved, 54 in total. **53 returned readable text**, and the pattern behind that hit rate is worth copying: - **linear.app** — entries resolve to `.md` twins served as `text/markdown` (208 to 1,024 words each). No rendering required. - **react.dev** — index links carry the `.md` suffix and arrive as `text/plain`, from 318 to 10,020 words. - **svelte.dev** — the index is six lines pointing at `llms-small.txt`, `llms-medium.txt`, `llms-full.txt` and per-section files. - **squareup.com** — product entries point at Markdown documents on its asset CDN rather than at site routes. - **notion.com, asana.com, clickup.com, webflow.com, framer.com** — HTML, but server-rendered: 437 to 6,491 words of text in the first response. The single failure is a reminder that a curated file still needs its links checked: `squareup.com/us/en/licenses`, listed as _Licenses_, returned **404** to GPTBot, ClaudeBot, PerplexityBot and a desktop Chrome user agent alike. Markdown twins and how they are connected to the index are covered in [Markdown for Agents vs llms.txt](/blog/markdown-for-agents-vs-llms-txt/). ## Where the Empty Shells Actually Appear We measured fourteen app, login and dashboard URLs with the same user agent, counting words of text in the raw response: URL Status Words in raw HTML app.shopify.com 200 1,577 app.slack.com 200 914 github.com/lukuime 200 586 app.asana.com 200 48 vercel.com/dashboard 200 47 dashboard.stripe.com 200 34 tiktok.com 200 5 instagram.com, notion.so/login 200 1 each x.com (402), app.webflow.com (404), app.linear.app and app.framer.com (no response) failure — Six of the fourteen answered **200 OK with fewer than 60 words** of extractable text. Most of those are login walls that are not meant to be indexed, and that is fine. The measurement matters because it is the same architecture: whatever is rendered on the client arrives at a non-rendering crawler as an almost empty document. The reassuring half of the dataset is the twelve public marketing homepages we measured in the same pass — every one returned an `

` and between **616 and 2,849 words** in the first response. Marketing sites are usually static or server-rendered by now. Public content routes inside an app shell are where the gap survives. ## Two Commands That Reveal the Problem Do this for the five pages you most want an AI engine to quote: ``` # 1. Fetch exactly what an AI crawler receives, with no JavaScript executed curl -sS -o page.html \ -A 'Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)' \ -w '%{http_code} %{content_type} %{size_download}\n' \ https://www.example.com/your-page/ # 2. Count the words a parser would find, and look for one sentence you know is on the page sed -e 's/]*>.*<\/script>//g' -e 's/<[^>]*>/ /g' page.html | wc -w grep -c 'the exact sentence your page is supposed to answer with' page.html # 3. If you publish a Markdown twin, prove it resolves curl -sS -o /dev/null -w '%{http_code} %{content_type}\n' https://www.example.com/your-page.md ``` A word count in the hundreds means the crawler got your content. A count under about sixty, or a `grep` that returns `0` for a sentence you can see in the browser, means the page exists only after JavaScript runs. The `.md` check decides your fix: if the twin answers `200` with `text/markdown`, the cheapest repair is to point the `llms.txt` entry at the twin. ## The Four Fixes, in Order of Effort 1. **Link the twin.** Serve Markdown at the page URL plus `.md`, declare it with `rel="alternate" type="text/markdown"`, and list the twin rather than the route. This is a content-pipeline change, not an architecture change, which is why Linear, React and Square already ship it. 2. **Server-render the pages you list.** Pre-render at build time where the content is stable; server-render at request time where it is not. What matters is that headings, body text, tables and internal links are present in the first response. 3. **Stop scoping fixes to Googlebot.** Dynamic rendering configured for one user agent leaves the AI crawlers on the shell, and they do not fall back to anything. 4. **Then verify both layers.** A file that passes a structural validation can still list dead or unreadable targets. Build the index in the [generator](/generator), run the link and structure pass in the [checker](/checker), then fetch the pages it points at the way a crawler would — which is the part of [llms.txt validation](/blog/how-to-validate-llms-txt/) that involves no file at all. **The verdict:** llms.txt works perfectly on a JavaScript-rendered site — as long as the URLs inside it serve readable HTML or Markdown to a client that never runs a script. Six of fourteen app routes we measured did not, one major domain answers `/llms.txt` itself with an HTML shell, and a curated file still managed to list a 404. Rendering is the invisible half of AI visibility: fix the response before you tune the file. --- # How to Add llms.txt to Jekyll (GitHub Pages, 2026) > Source: https://llmstxtgenerator.dev/blog/llms-txt-jekyll/ Jekyll tutorial · GitHub Pages · 7 minutes # How to Add llms.txt to Jekyll Most GitHub Pages blogs run on Jekyll, which has no native `llms.txt` feature. Its file rules make it one of the easiest SSGs to fix, though: a static file in 60 seconds, auto-generation from your posts with a few lines of Liquid (no plugin, so GitHub Pages accepts it), or the `jekyll-llms-output` plugin for `llms.txt` plus `llms-full.txt`. ## The Jekyll File Rule That Makes This Easy Every root-level file in Jekyll meets one of two fates: with YAML front matter it is rendered through Liquid and written to the same path in `_site`; without front matter it is copied verbatim. A root file named `llms.txt` with a `layout: null` front matter block gets its Liquid processed and published at `/llms.txt` as plain text — no custom output formats needed. ## Real-World Case: jtemporal.com Developer advocate Jessica Temporal (Jekyll + GitHub Pages) [published her exact setup](https://jtemporal.com/how-to-create-llms-txt-in-jekyll). Her live file, curl-verified on 2026-09-07: ``` $ curl -sI https://jtemporal.com/llms.txt HTTP/2 200 content-type: text/plain; charset=UTF-8 content-length: 32551 ``` The `_Generated` timestamp proves Liquid wrote the file at build time. Her setup: `layout: null` front matter, a `{% for post in site.posts %}` loop enumerating every post, and `llms.txt` in the `_config.yml` `include` list (custom include lists omit unlisted files). ## Method 1: A Static File in 60 Seconds If your site is small and stable, create `llms.txt` at the Jekyll root _without_ front matter — it is copied into `_site` unchanged: ``` # Acme Blog > Acme Blog publishes practical tutorials on web performance and SEO. ## Essentials - [About](https://acme.com/about/): who writes here and why - [Archive](https://acme.com/archive/): every post, newest first ## Popular Posts - [Core Web Vitals in 2026](https://acme.com/cwv-2026/): what changed and what to fix - [Heading Hierarchy](https://acme.com/headings/): H1 to H6 done right ``` Caveat: if your `_config.yml` defines a custom `include` list, add `llms.txt` to it — otherwise the file never reaches the build output. The cost of this route is maintenance: every new post is a hand edit, and a file that drifts from your content is one agents follow into stale territory. ## Method 2: Auto-Generate from site.posts with Liquid For a blog, the no-plugin template is the sweet spot: no gem, so it runs inside the stock GitHub Pages build, and it regenerates on every `bundle exec jekyll build`: ``` --- layout: null --- # Acme Blog > Acme Blog publishes practical tutorials on web performance and SEO. > Generated: {{ site.time | date: "%Y-%m-%d" }} ## Posts {% for post in site.posts %} - [{{ post.title }}](https://acme.com{{ post.url }}): {{ post.excerpt | strip_html }} {% endfor %} ``` `layout: null` emits plain text instead of your theme's HTML wrapper. URLs are hard-coded because `site.url` stays empty until you set `url:` in `_config.yml`. A one-line description per post becomes the link description agents read. Variant: link to raw Markdown sources using `post.path`: `raw.githubusercontent.com/USER/REPO/main/{{ post.path }}`. Clean Markdown is the most LLM-friendly form — like the per-page `.md` files the [llms.txt v2 spec](/blog/llms-txt-v2/) defines. ## Method 3: jekyll-llms-output for llms.txt + llms-full.txt For spec-grade output without writing Liquid, [jekyll-llms-output](https://github.com/abhinavs/jekyll-llms-output) generates `/llms.txt` and `/llms-full.txt`. In curated mode, drop a `_data/llms.yml` mapping sections to links; without it, auto mode emits one `## Section` per collection, one bullet per document. Add the gem to your Gemfile: ``` # Gemfile gem "jekyll-llms-output", group: :jekyll_plugins # terminal bundle install ``` Paired with [jekyll-markdown-output](https://github.com/abhinavs/jekyll-markdown-output), it writes a clean `.md` sibling for every page, and `llms-full.txt` concatenates every document's full body under `# Title` headers. See the [llms.txt vs llms-full.txt guide](/blog/llms-txt-vs-llms-full-txt/). **The GitHub Pages catch:** Pages only allows a whitelist of Jekyll plugins, and this gem is not on it — building from a `gh-pages` branch means it silently never runs. Build in CI and deploy the resulting `_site` directory instead. ## Static, Liquid or Plugin? Approach Setup GitHub Pages Auto-updates llms-full.txt Static root file 1 minute Yes No No Liquid template ~5 minutes Yes, no plugin needed Every build No jekyll-llms-output ~10 minutes Via CI only Every build Yes For a typical blog the Liquid route offers the best return: it stays inside GitHub Pages' constraints and your post list can never go stale. Choose the plugin when you also want `llms-full.txt` or per-page Markdown and already build in CI. Either way, follow [llms.txt best practices](/blog/llms-txt-best-practices/) — a summary per link beats a raw title. ## Verify and Maintain - **Confirm the include list** — with a custom `include`, add `llms.txt` or it never publishes. - **Set `url:` in `_config.yml`** — or hard-code absolute URLs. - **Use `layout: null`** for the Liquid route — without it Jekyll serves HTML as text. - **Verify the deployed file** — confirm it serves as plain text: ``` curl -s -o /dev/null -w "%{http_code} %{content_type}\n" https://yourblog.com/llms.txt # Expect: 200 text/plain ``` Paste the URL into the free [llms.txt checker](/checker), or feed your blog's sitemap into the [free llms.txt generator](/generator), then let Liquid keep it current — a zero-maintenance byproduct of the build that already publishes your blog. --- # llms.txt Maintenance: How to Keep Your File From Going Stale (2026) > Source: https://llmstxtgenerator.dev/blog/llms-txt-maintenance/ Maintenance · September 2026 # llms.txt Maintenance: Keeping the Map Honest Publishing the file takes ten minutes. The interesting question is the one nobody writes about: what happens to it over the next two years, while your URLs, your product names and your documentation structure keep moving. ## Staleness Is a Content Bug, Not a Delivery Bug Delivery problems — the wrong content type, a rewrite rule, a blocked crawler — are loud and fixable in an afternoon; they are covered in [the delivery-level troubleshooting guide](/blog/llms-txt-not-working/). Staleness is quiet. The file still returns `200`, the format is still valid, and every link is still syntactically perfect — but three of them now 404, and the H2 section called _Pricing_ points at a page you retired last spring. An agent that follows your map and hits 404s learns that your curated file is less reliable than your sitemap, which is the opposite of the file's purpose. There is a second, slower cost: summaries drift. If the blockquote under your H1 still describes a product you repositioned six months ago, that sentence travels with every retrieval that touches your domain. Curation is a claim about your site, and claims expire. ## What Actually Changes (and What Each Change Forces) Most site changes need no edit at all. These six do: Change Edit required A new top-level section or product line Add an H2 and its 3–6 links; update the summary if the positioning moved URL restructuring or a rename Replace the URL — do not keep the old one alongside the redirect A page in the file was deleted Remove the line; leave the section if it still has three or more links A page is now your best entry point Move it into the first section — order is editorial information A new language or regional site Decide root file versus per-language files before adding URLs Content moved to a subdomain you do not advertise Point at the canonical host, never the alias an agent cannot index Nothing on that list requires a new format, a version bump or a migration. It is editing a markdown file you already own, which is why the expensive part is remembering to do it. ## Layer 1: Make Dead Links Fail Your Build The single highest-value habit is a link check that runs on every deploy and turns a 404 into a failed pipeline. A file that lists twenty URLs is small enough for a plain shell script — no dependency, no framework: ``` #!/usr/bin/env bash # scripts/check-llms-links.sh — exits 1 if any URL in llms.txt is not 200 fail=0 while read -r url; do code=$(curl -s -o /dev/null -w '%{http_code}' -L -A "GPTBot/1.5" "$url") if [ "$code" != "200" ]; then echo "DEAD LINK $code $url" fail=1 fi done < <(grep -oE 'https?://[^ )]+' public/llms.txt | sort -u) exit $fail ``` Run it locally before you push, and once after every deploy against the production host. The `-L` flag matters: it follows redirects, so a link that only works through a hop shows up as a pass. If you would rather not allow redirects at all — several engines will not follow them for this file — drop `-L` and treat a `301` as a failure too. Add the same script to your CI workflow and to a weekly scheduled run. Weekly is enough: a link that breaks on Monday and is reported on Friday costs you four days of failed fetches, not four months. ## Layer 2: Generate What Can Be Generated, Curate the Rest Split the file into the part a machine can keep current and the part only you can write. Sitemap derived lists of every URL belong to the automated side; the `/generator` builds exactly that from a `sitemap.xml` if you want a starting point. The header summary, the section names and the ordering are the editorial side, and they should be reviewed by a human — automation that refreshes the whole file nightly tends to produce an alphabetical dump that loses the curation that made the file useful in the first place. The [nine best-practice rules](/blog/llms-txt-best-practices/) are mostly about that editorial layer. ## Layer 3: A Quarterly Review You Can Actually Finish Ten minutes, once a quarter, in this order: - Run the link check and delete or fix every failure. - Read the summary line aloud. Would a stranger understand what the site is today? - Check the first section: is your single best page still in it? - Compare against the sitemap — did a whole section appear that the file never learned about? - Confirm the file still returns `200` with `text/plain` from the canonical host. - Check the cache headers on that response, so you know how long a wrong version persists. - If you measure fetches at all, compare the last quarter's numbers against the previous one — see [how to measure llms.txt traffic](/blog/measure-llms-txt-traffic/). ## When Not to Touch It Not every week is a version. Adding a blog post that is not one of your best pages is not a reason to edit the file — that is what `sitemap.xml` is for, and the two files have different jobs; [the comparison](/blog/llms-txt-vs-sitemap-xml/) covers the split. Constant churn also fights caching: agents and proxies that hold a copy of your file have to reconcile changes, and a file that reorders itself weekly is noise. Keep the URL fixed, keep absolute links, and let the file improve by accretion. Once the automation is in place, staleness stops being a memory problem. The build fails, you fix the link, and the map stays honest. If you have not published a file yet, start with the [generator](/generator), then push the production URL through the [checker](/checker) before you automate anything. --- # 6 llms.txt Mistakes That Make AI Engines Ignore Your File > Source: https://llmstxtgenerator.dev/blog/llms-txt-mistakes/ Common errors · August 2026 · AI readiness # 6 llms.txt Mistakes That Make AI Engines Ignore Your File You published `/llms.txt`, so why do AI engines seem not to care? In most cases the file is technically "there" but fails in one of six ways: wrong placement, format errors, dead links, a site-wide dump, marketing language, or the wrong success metric. Here is how to find and fix each one. ## Mistake 1: The file is where agents can't find it AI agents look for llms.txt at the root of your domain over HTTPS — the same place as robots.txt. If your file only works on `www` while the crawler checks the apex domain, or a redirect chain swallows the request, the agent gets a 404 and moves on. The [llmstxt.org](https://llmstxt.org) spec also allows subpath files such as `/docs/llms.txt`, which cover the pages under their path — useful when you only control a directory, but a root file is what most agents check first. - Serve the file at `https://yourdomain.com/llms.txt` — lowercase, exact name, root path. - Make sure HTTPS redirects work before that URL, with no infinite loops. - Confirm the status code, not just that the file exists on disk: ``` curl -I https://example.com/llms.txt # expect: HTTP/2 200 curl -s https://example.com/llms.txt | head -5 ``` ## Mistake 2: The format fails validation The spec is deliberately small, and the most common failures are the simplest. The H1 with your site name must be the _very first line_ — the only required element. A byte-order mark, an HTML comment, or a blank line pushed in front of it by your CMS is enough to break the file for strict validators. - First line: `# Your Site Name`, nothing before it. - Blockquote summary directly after the H1. - Absolute URLs only: `https://...` — never `/about`. - One H1; markdown link list items under H2 headings such as `## Important links`. Run every revision through a validator and fix everything reported as an error, not just the warnings. The [free validator](/checker) checks exactly these rules — missing H1, H1 not on the first line, relative URLs, unrecognized lines — and scores the result 0–100. ## Mistake 3: The links lead nowhere useful llms.txt is a list of links. If those links 404, redirect to a login wall, or point at HTML pages buried in navigation and cookie banners, an agent that follows them gets no clean context. Links rot fastest after a redesign — a fact every site with an old llms.txt learns the hard way. - Link to markdown versions of pages when you can; both `.md` and `.html.md` URL forms are valid in the v2 spec. - Verify every listed URL returns 200, and re-check after each deploy. - Add a short description after each link so an agent knows what the page is about before fetching it. ## Mistake 4: You dumped the whole site in sitemap.xml lists _every_ URL for search crawlers; llms.txt should curate the handful of pages that best represent your site for an AI agent that reads the file inside its context window. Hundreds of links dilute the signal — an agent skims, and your five most important pages disappear into the noise. See [llms.txt vs sitemap.xml](/blog/llms-txt-vs-sitemap-xml/) for the full comparison. llms.txt sitemap.xml Purpose Curated index for AI agents Complete URL list for search crawlers Typical size 5–15 links Thousands of URLs Readers AI engines and agents Google and other search engines Detail One-line description per link Metadata and priorities Keep the file under a few dozen links. Secondary material can live in an `Optional` section, which agents may skip — put your core value in the first sections. ## Mistake 5: You wrote marketing copy, not context Agents summarize from what the file actually says. "Industry-leading," "next-generation," and "empowering businesses" carry zero information — an AI model has no way to verify them, so it ignores them. Write like you are explaining the site to a smart colleague: ``` # Acme Inc > Acme is the industry-leading provider of next-generation > solutions that empower businesses worldwide. ``` Compare that with a summary an agent can actually use: ``` # Acme Inc > Acme builds payroll software for US small businesses, > with 40,000 customers and a 12-person support team. ``` The same rule applies to link descriptions: "About us" is weaker than "About: founding story, team, and 12 years of payroll history." Specificity is what makes an agent confident enough to cite you. ## Mistake 6: You expect rankings — and never re-check In June 2026, Google updated its AI optimization guide to state plainly that llms.txt has no effect — positive or negative — on Search rankings or AI Overviews. Independent data points the same way: ALLMO pulled 94,614 cited URLs from 11,867 AI answers across five AI platforms and found exactly one /llms.txt page among them (0.001%), and Limy's analysis of 500M+ LLM bot traffic events shows GPTBot, ClaudeBot, PerplexityBot and Google-Extended overwhelmingly crawl HTML directly and skip the file. We covered the evidence in detail in [does llms.txt affect SEO?](/blog/does-llms-txt-affect-seo/). None of that makes llms.txt useless — it is a control layer. Adoption is real and rising (about 5.6% of the top 10,000 sites had a valid file in June 2026, and Shopify pushed one to every store by default in 2026), and Chrome Lighthouse audits for the file by default. The mistake is judging it by ranking metrics instead of by whether it stays correct. Every quarter, and after every site change, run this checklist: - `curl -I` returns 200 at the exact root URL. - First line is still the H1 with your site name. - All links are absolute and return 200. - The file is under a few dozen links and still curated. - Summary and descriptions are factual, not hype. - Re-validated after every deploy. Fix the six mistakes above and your llms.txt does its real job: it gives agents that do check it a fast, clean, accurate path to your best content. Generate a compliant file from your sitemap with the [generator](/generator), then confirm every point above with the [validator](/checker). --- # How to Add llms.txt to MkDocs (Material for MkDocs, 2026) > Source: https://llmstxtgenerator.dev/blog/llms-txt-mkdocs/ MkDocs tutorial · Plugin or static · 6 minutes # How to Add llms.txt to MkDocs MkDocs is the Python-based documentation generator behind a large share of open-source docs sites, usually dressed in the Material for MkDocs theme. It has no native `llms.txt` feature — but it barely needs one. A plain file in `docs/` is copied straight to your site root on every build, and the community `mkdocs-llmstxt` plugin generates a spec-compliant file from your own config. Both routes are covered here. ## MkDocs and llms.txt: No Native Support (Yet) Software documentation is where llms.txt gets used most heavily — the llmstxt.org spec notes that coding agents follow the file to find API references and tutorials — and MkDocs is one of the most common engines behind that documentation. Yet the integrations list at llmstxt.org still has no MkDocs entry: it names hosted platforms like Mintlify and GitBook and CMS plugins like Yoast, but nothing for self-hosted MkDocs. The gap is visible in the ecosystem. A community discussion on the Material for MkDocs repository ([squidfunk/mkdocs-material #8384](https://github.com/squidfunk/mkdocs-material/discussions/8384)) asks for native llms.txt generation, and the answer so far is a plugin. Timothée Mazzucotelli's `mkdocs-llmstxt` is the reference implementation, and real projects use it in production — the [Instructor documentation](https://python.useinstructor.com/blog/2025/08/29/mkdocs-llmstxt-plugin-integration) pipeline regenerates its llms.txt automatically on every deploy. Note that MkDocs pages can sit at any subpath, so the file works for versioned docs too — the [llms.txt v2 spec](/blog/llms-txt-v2/) defines how subpath files and Markdown page versions behave. ## Method 1: Ship a Static docs/llms.txt MkDocs copies every non-Markdown file in `docs/` verbatim into the output directory, preserving its path — the same mechanism that makes `docs/robots.txt` work. Create `docs/llms.txt` and your next `mkdocs build` serves it at `https://docs.example.com/llms.txt`: ``` # Acme Docs > Acme Docs is the official documentation for the Acme platform. > It covers installation, configuration and the REST API. ## Getting Started - [Quickstart](https://docs.example.com/getting-started/): first project in five minutes - [Authentication](https://docs.example.com/authentication/): API keys and scopes ## API Reference - [REST API](https://docs.example.com/api/): every endpoint, parameter and error - [Webhooks](https://docs.example.com/webhooks/): events your server can subscribe to ``` This is the right route for small sites whose structure changes rarely. The cost is manual maintenance: every new page is a hand edit, and when your docs drift from the file, agents follow a stale index. Keep URLs absolute — never relative paths — and remember the file only describes pages under its own path, so docs served at `/en/latest/` want a file in that same tree. ## Method 2: Auto-Generate with the mkdocs-llmstxt Plugin Install the plugin and add it to `mkdocs.yml`. Three keys matter: `site_url` is required (all links are built from it), `site_description` becomes the blockquote, and your `sections` define the curated heading groups, with optional per-file descriptions and glob support: ``` # terminal pip install mkdocs-llmstxt ``` ``` # mkdocs.yml site_name: Acme Docs site_url: https://docs.example.com/ plugins: - search - llmstxt: markdown_description: Long-form context an agent needs before navigating. sections: User Guide: - index.md: What Acme Docs covers - getting-started.md - deployment.md API Reference: - api/*.md ``` Every `mkdocs build` now writes `site/llms.txt`. The plugin parses the rendered HTML (BeautifulSoup + Markdownify) and converts it back to clean Markdown, so executed code blocks, Jinja-generated snippets and API documentation survive in the output — and each page you list is published as its own Markdown file, with llms.txt linking those `.md` URLs. Set `full_output: llms-full.txt` to also emit the full-text dump, the pattern the [llms.txt vs llms-full.txt guide](/blog/llms-txt-vs-llms-full-txt/) compares. For versioned builds on Read the Docs, the `base_url` option rewrites every generated link to a subdirectory such as `https://docs.example.com/en/0.1.34`. One honesty note: the plugin's repository is in maintenance mode — the author moved on to another project and is looking for a maintainer. It is widely used and stable, but pin your version and keep the static-file fallback in mind. An alternative, `mkdocs-llmstxt-md`, takes a raw-Markdown approach (enabled as `llmstxt-md`) and generates both llms.txt and llms-full.txt from your source files by default, deriving sections from your `nav` when you configure nothing. ## Which Route Fits Your Docs? Approach Setup Stays current Extra outputs Static `docs/llms.txt` 1 minute Manual edits only None `mkdocs-llmstxt` ~5 minutes Every build .md page versions, optional llms-full.txt `mkdocs-llmstxt-md` ~5 minutes Every build llms.txt + llms-full.txt by default A static file is fine while your sitemap is small and stable. The moment docs change weekly — new guides, moved pages, added API endpoints — a generated file stays honest, because the llms.txt an agent reads is only as trustworthy as its last update. ## The MkDocs-Specific Checklist - **Set a canonical site\_url** — plugin links are derived from it. A placeholder `site_url` ships broken or relative links into your llms.txt. - **Curate, don't dump** — list the pages that answer real questions: getting started, auth, deployment, the API. Skip the 404 page, the changelog and search indexes. Curation rules live in the [best practices guide](/blog/llms-txt-best-practices/). - **Write useful descriptions** — "what problem this page solves" beats "documentation for module X" when an agent decides which link to fetch. - **Handle versioned docs per tree** — each language or version wants its own subpath file; use `base_url` in per-version builds rather than one file claiming to cover all of them. - **Rebuild to publish** — MkDocs is static: llms.txt changes only when CI runs `mkdocs build`, so wire it into your existing deploy pipeline. ## Verify Before Agents Read It After deploying, confirm the file serves as plain text with a 200 status: ``` curl -s -o /dev/null -w "%{http_code} %{content_type}\n" https://docs.example.com/llms.txt # Expect: 200 text/plain ``` Then paste the URL into the free [llms.txt checker](/checker) to validate the H1, blockquote summary, absolute URLs and section structure against the v2 spec. Draft the initial file in seconds by feeding your MkDocs sitemap into the [free llms.txt generator](/generator), then switch to the plugin once your sections are settled. MkDocs makes the file trivial to ship and easy to keep current — for a documentation site, that is the highest-leverage AI visibility change you can make this week. --- # llms.txt for Multilingual Websites: One File or One Per Language? (2026) > Source: https://llmstxtgenerator.dev/blog/llms-txt-multilingual-sites/ Global SEO · September 2026 # llms.txt for Multilingual Websites: One File or One Per Language? Multilingual sites are the awkward case for `llms.txt`. English-only sites write one file and move on; global sites have to decide whether a single curated index can represent five languages, or whether each language version deserves its own. The spec does not answer the question — but it does give you the one rule that settles it. ## The Rule That Settles It: Scope The [specification](https://llmstxt.org/) is explicit about what a file covers: an `llms.txt` file describes the URLs _under its own path_, and where more than one file applies to a given page, agents should use the **most specific** one. The file may sit at the root or at any subpath. That is the whole ballgame. A root `/llms.txt` covers your entire domain, every language included. A file at `/de/llms.txt` covers everything under `/de/` and **wins** for those URLs. The v2 spec revision (modified August 10, 2026) kept this wording unchanged — see [what changed in v2](/blog/llms-txt-v2/) for the rest of the update, including the `rel="describedby"` link relation that lets a page point at the file covering it. Note also that the format is language-neutral: nothing in it requires English. Descriptions should be written in the language of the content they describe, because the consumer is often a localised assistant answering a question in that language. ## Three Structures That Work Structure Best for Maintenance cost **A.** Single root file, one H2 per language 2–3 languages, shorter sites Lowest — one file to edit **B.** Root index + `/en/llms.txt`, `/de/llms.txt` per locale 4+ languages, separate locale paths Higher — one file per locale, but each stays short **C.** Root file + `Optional` section for secondary languages One dominant market, one or two minor ones Low, but secondary languages are second-class ## Structure B in Practice The pattern that keeps showing up in practitioner write-ups is the sitemap-index shape: a small root file at the domain root whose job is to route agents to the right language file, plus one real file per language root. The root file stays stable for years because it changes only when you add or retire a language: ``` # Example Store > Global retailer selling household goods in 6 languages. ## Language editions - [English](https://example.com/en/llms.txt): full product and support content, primary market - [German](https://example.com/de/llms.txt): full product and support content, DACH market - [French](https://example.com/fr/llms.txt): product catalogue and shipping policy - [Japanese](https://example.com/ja/llms.txt): product catalogue and support centre ``` Each locale file then does the real work under its own path — the top-level sections an assistant needs, with one-line descriptions, exactly as the [best-practice rules](/blog/llms-txt-best-practices/) describe. Keep the count of curated links per locale in the 20–40 range. If you are generating a first draft from a sitemap, [the generator](/generator) can start you off, but split the output by locale before you publish it — dumping six languages into one file is the failure mode this whole structure exists to avoid. One caveat specific to subpaths: your locale directories must be real paths (`/de/`, `/ja/`), not cookie- or `Accept-Language`\-based switching. If the language is decided at request time, there is no stable path for a file to cover, and there is no URL for an agent to cite. ## Keep It Aligned With hreflang and Canonicals - **List canonical URLs only.** If `/de/produkte/` is the German canonical, do not also list the parameterised or tracking variants. - **One entry per language version, not per region.** Your `hreflang` annotations already tell engines that `de-DE` and `de-AT` share a page. Duplicating it in `llms.txt` just adds noise. - **Never list a page that redirects.** A locale link that 301s to another language is worse than no entry at all — the agent burns a request and may cite the wrong version. - **Self-reference with `rel="describedby"`.** Adding the link relation on localised pages (in the HTML `head` or as an HTTP `Link:` header) tells agents which file covers that page, which is how a deep `/ja/...` URL gets routed to `/ja/llms.txt`. ## What Big Multilingual Sites Actually Ship We checked a sample of global sites with `curl` on September 14, 2026. The picture is consistent: the root file is becoming normal, per-language files are not there yet — and the most multilingual site of them all publishes nothing at all. URL checked Status `docs.stripe.com/llms.txt` 200, `text/markdown`, ~90 KB `docs.stripe.com/de/llms.txt` 404 `developers.cloudflare.com/llms.txt` 200, `text/plain` `developers.cloudflare.com/zh-cn/llms.txt` 404 `www.shopify.com/llms.txt` 200, `text/plain` `www.shopify.com/de/llms.txt`, `/fr/llms.txt` 404 (both) `about.gitlab.com/llms.txt` 200, `text/plain`, ~10 KB `en.wikipedia.org/llms.txt`, `/es/`, `/ja/` 404 (all three) `www.ikea.com/us/en/llms.txt` 404 Two lessons. First, publishing any valid file puts you ahead of most of the web. Second, if you already run locale paths, per-language files are a differentiator rather than a catch-up move: Stripe's single 90 KB file is genuinely useful for English-speaking developers, and useless to a German-speaking one asking about payment methods in German. ## A Five-Minute Verification Pass Check every path you claim to support, from one shell, and confirm the file is served as text and not as an HTML error page: ``` curl -sI https://example.com/llms.txt curl -sI https://example.com/en/llms.txt curl -sI https://example.com/de/llms.txt # content type and first lines curl -s https://example.com/de/llms.txt | head -20 ``` - Every URL returns **200** with `text/plain` or `text/markdown` — a 302 to your homepage counts as missing. - Every link inside each file resolves, and every link under `/de/` really is German content. - The H1 and summary in each locale file are written in that locale, not machine-translated boilerplate. Run the files through the [llms.txt checker](/checker) after each language launch, and re-run it whenever you add a locale. Structure follows content: add the language version first, then give it a file. The short answer to the question in the title: one file if you have two or three languages, one file per language once locale paths are a real part of your site. Either way, make sure the file that applies to a page is the most specific one available — that is the only rule the spec asks you to respect. --- # How to Add llms.txt to Next.js (App Router, 2026) > Source: https://llmstxtgenerator.dev/blog/llms-txt-nextjs/ Next.js tutorial · App Router · 6 minutes # How to Add llms.txt to Next.js Next.js still has no built-in llms.txt file convention — but shipping one takes minutes. Here are the three ways to do it, from a 30-second static file to a fully dynamic, ISR-cached route handler. ## Why Next.js Sites Are the #1 llms.txt Use Case The llms.txt convention exists mainly for software documentation, where coding agents follow the file to find API references and tutorials. That is exactly the content Next.js is most often used to build. The early adopters read like a who's who of the Next.js ecosystem: Vercel publishes [vercel.com/docs/llms.txt](https://vercel.com/docs/llms.txt), the Next.js docs themselves serve [nextjs.org/docs/llms.txt](https://nextjs.org/docs/llms.txt), and Strapi, Vultr, Langbase, Checkly and CircleCI all publish one at their docs roots. If your Next.js project is a SaaS site, a docs portal, or a dev-tool marketing page, your audience is literally the people asking ChatGPT and Claude how your product works. An `llms.txt` is the cheapest way to make sure the answer they get links back to your canonical pages — this is core [GEO (generative engine optimization)](/blog/llms-txt-v2/) for framework-built sites. ## Method 1: Static File in public/ (30 Seconds) The fastest option works on both App Router and Pages Router projects. Create a file at `public/llms.txt` (or `src/../public/llms.txt` with the standard `src/` layout — Next.js copies the whole `public/` folder to the root of the build output): ``` # Example SaaS Docs > Example SaaS is a hosted API platform. This file maps the pages > coding agents should read first: quickstart, API reference, guides. ## Quickstart - [Quickstart](https://example.com/docs/quickstart): 5-minute setup with your first API call - [Authentication](https://example.com/docs/auth): API keys and token scopes ## API Reference - [REST API Reference](https://example.com/api-reference): all endpoints, params and errors - [Webhooks](https://example.com/docs/webhooks): events your server can subscribe to ``` Deploy and the file is live at `https://your-domain.com/llms.txt`. One caveat: a static `public/` file always wins over a route handler with the same path — delete the static file if you later switch to Method 2. ## Method 2: Dynamic Route Handler (App Router) For a file that should reflect live data — your CMS, your database, your sitemap — use a route handler. A folder named `llms.txt` works fine in the App Router (no rewrites needed): ``` // app/llms.txt/route.ts import { siteName, siteDescription, sections } from '@/lib/llms-config'; export const dynamic = 'force-static'; // optional: pre-render at build time export async function GET() { const lines = [ '# ' + siteName, '', '> ' + siteDescription, '', ]; for (const section of sections) { lines.push('## ' + section.title, ''); for (const link of section.links) { lines.push('- ' + link.name + ': ' + link.description + ' (' + link.url + ')'); } lines.push(''); } return new Response(lines.join('\n'), { headers: { 'Content-Type': 'text/plain; charset=utf-8' }, }); } ``` Two details matter. First, the `Content-Type` must be `text/plain` — agents and validators expect plain text, not HTML. Second, with `export const dynamic = 'force-static'` the file is generated once at build time and served from the edge, so it costs nothing at request time. ## Method 3: ISR — Keep It Fresh Without Rebuilding Static is fast but goes stale. If your docs change hourly (release notes, changelogs, pricing), swap `force-static` for an ISR interval: ``` // app/llms.txt/route.ts export const revalidate = 3600; // regenerate at most once per hour ``` The route handler then rebuilds lazily on the next request after the interval expires, so your `llms.txt` never describes pages that no longer exist. You can even fetch your `sitemap.xml` inside the handler and filter it down to the genuinely important pages — a good way to avoid hand-maintaining the file. If you prefer a ready-made solution, the community [next-llms-txt](https://github.com/bke-daniel/next-llms-txt) plugin adds this as a build-time step. ## What Next.js Doesn't Ship (Yet) In July 2025 a feature request ([vercel/next.js discussion #81182](https://github.com/vercel/next.js/discussions/81182)) asked for an `llms.txt` file convention in the app directory, mirroring `sitemap.ts` and `robots.ts`. A community PR (#90580) is in progress, but native support has not shipped. The good news: the file convention only matters for ergonomics. The `public/` file and route-handler approaches above produce exactly the same result at `/llms.txt`, so there is no reason to wait. ## Verify Your File After deploying, check that the route serves plain text with a 200 status: ``` curl -s -o /dev/null -w "%{http_code} %{content_type}\n" https://your-domain.com/llms.txt # Expect: 200 text/plain ``` Then paste the URL into our free [llms.txt checker](/checker) to validate the H1, the blockquote summary, absolute URLs, and section structure before AI engines start reading it. ## Best Practices Checklist for Next.js - **Absolute URLs only** — `https://example.com/docs/start`, never `/docs/start`. Agents resolve links from the file's location. - **Curate, don't dump** — 10–50 links beats 500. Point agents at canonical docs pages, not marketing landing pages. - **Watch for interceptors** — make sure no middleware, redirect, or `next.config` rewrite catches `/llms.txt` before your route does. - **Treat it like a README** — update the file whenever the underlying page changes; stale files erode agent trust over time. - **Add llms-full.txt for deep docs** — for large reference sections, link a generated [llms-full.txt](/blog/llms-txt-vs-llms-full-txt/) companion so agents can fetch the full content on demand. Once it's live, run your sitemap through the [free llms.txt generator](/generator) to build the initial curated file in seconds, then validate it with the [checker](/checker) before you deploy. A correct, current `llms.txt` is the single highest-leverage AI visibility change most Next.js sites can make this week. --- # llms.txt Not Working? 7 Delivery-Level Fixes (Content-Type, Redirects, WAF, 2026) > Source: https://llmstxtgenerator.dev/blog/llms-txt-not-working/ Troubleshooting · September 2026 # llms.txt Not Working? Seven Delivery-Level Fixes The file is in the repository, it passed validation, and no AI engine ever asks for it. Usually the problem is not the format but delivery: what your server actually returns when something requests `/llms.txt`. ## The 30-Second Test Ask the server, not your editor. Two commands answer the status code, the content type, whether a redirect sits in front of the file, and whether the body is markdown or an HTML shell: ``` # headers only: status, Content-Type, Location, cache status curl -sI https://example.com/llms.txt # fetch the way an AI crawler would, and look at the first lines curl -sA "GPTBot/1.2 (+https://openai.com/gptbot)" https://example.com/llms.txt | head -20 ``` A healthy response is `200`, `content-type: text/plain; charset=utf-8` (or `text/markdown`), no `Location` header, and a body starting with a markdown H1 and the summary line. Anything else maps onto the table below. Symptom What it usually means 200, but `text/html` Catch-all rewrite served your app shell — fix 1 `text/x-robots` or the wrong body A rule mapped `/llms.txt` onto `/robots.txt` — fix 1 301 or 302 chain, or a redirect to the homepage Canonical host mismatch — fixes 2 and 3 404 for that path only Path case, extension not whitelisted, or wrong subpath — fix 3 403, 429 or an empty reply WAF, bot rule or a `robots.txt` block — fix 4 200 locally, 404 in production The file never shipped in the build — fix 5 Old content, or a 404 that comes and goes Edge cache holding a previous response — fixes 6 and 7 ## Fix 1: The Response Is HTML, Not Markdown This failure is invisible in a browser, because the browser renders the app shell anyway. A single-page app or a rewrite-everything rule answers _every_ unknown path with `index.html` — including `/llms.txt`. A Common Crawl survey of published llms.txt files found the two signatures: sites that serve their SPA for any unknown path answer `text/html`, and sites whose rewrite sends `/llms.txt` to `/robots.txt` answer `text/x-robots`. A static file in `public/` (or `static/`, or your host's root directory) is served as `text/plain` automatically — add an exception so the SPA fallback never sees that path. If a route handler generates the file, set the header yourself. The v2 spec, modified August 10, 2026, names no required MIME type, but it does require a markdown file at that path, and a response labelled `text/html` is not one. ## Fixes 2 and 3: Redirect Chains and Path Case Redirects turn a working file into an unreliable one. Some agents follow them, some do not, and a chain across hosts doubles the failure surface. Publish the file on the exact hostname you advertise. A redirect whose destination is your homepage is worse than a 404 — a missing file that looks like a success at the HTTP level. Path case is the quieter version of the same problem: object storage and Linux hosts are case-sensitive, so `/LLMS.TXT` and `/Llms.txt` are 404s even though `/llms.txt` exists, and NGINX rules that whitelist extensions can 404 a newly added `.txt` file. Subpath files follow the spec's scope rule — a file describes the URLs _under its path_, most specific wins — so `/docs/llms.txt` must be reachable at that path, not only at the root. ## Fix 4: Something Is Blocking the Request Check the two layers in front of your origin. First, `robots.txt`: the major AI crawlers honour it, so a broad `Disallow: /` or a pattern such as `Disallow: /*.txt` blocks the llms.txt fetch itself — the file never gets read. Second, the firewall: bot protection, WAF rules and aggressive rate limiting return 403 or 429 to clients that do not look like browsers. Test with a crawler user agent and compare: ``` curl -s -o /dev/null -w '%{http_code}\n' -A "GPTBot/1.2" https://example.com/llms.txt curl -s -o /dev/null -w '%{http_code}\n' -A "ClaudeBot/1.0" https://example.com/llms.txt ``` A 403 for the crawler while your browser sees 200 puts the block in the CDN or WAF, not your site. Staging has the same trap in another form: basic authentication on the whole hostname fails every fetch, which is harmless on staging and fatal if that rule is ever applied to production. ## Fix 5: The File Never Reached Production Local success is not deployment success. The file may sit in a source folder the build does not copy into the output directory, the deploy may have run from a commit that predates it, or a dynamic route may catch `/llms.txt` and return something else. The only reliable check is against the production URL after every deploy — and if you deploy manually, confirm the file made it into that build, not the previous one. ## Fixes 6 and 7: Caches, Copies and the Verification Pass An edge cache can keep serving a 404 your origin stopped returning, or a copy from before your last edit. Purge the exact path, and read the cache headers while you are there: an `age` measured in days on a file you changed an hour ago tells you which copy agents get. Watch for a second hostname on the same site, too — a staging or preview alias that answers `/llms.txt` is a file agents can find and you will forget. Then run the pass once, and again after every deploy that touches the file: - Canonical host, `curl -sI`: **200**, `text/plain` or `text/markdown`, no `Location`. - Crawler user agent: same status as your browser, not a 403. - Body: a markdown H1 and summary line, not an HTML document. - Every link inside the file returns 200 — dead links waste the curation even when delivery is perfect. - No `robots.txt` or firewall rule matches the path. If the content is the suspect rather than delivery, the [six mistakes that get a file skipped](/blog/llms-txt-mistakes/) cover that side, the [crawler reference](/blog/robots-txt-ai-crawlers/) shows which tokens matter per engine, and the [measurement walkthrough](/blog/measure-llms-txt-traffic/) answers whether anyone reads the file at all. Run the URL through the [llms.txt checker](/checker) the way an agent would, and build the file itself with the [generator](/generator) if it is still a draft. --- # How to Add llms.txt to Nuxt (public/, Server Routes, nuxt-llms, 2026) > Source: https://llmstxtgenerator.dev/blog/llms-txt-nuxt/ Nuxt tutorial · Nitro · nuxt-llms · 7 minutes # How to Add llms.txt to Nuxt Nuxt ships no `llms.txt` output of its own — but its `public/` directory makes Nuxt apps one of the easiest to fix: a static file in 60 seconds, a `server/routes/` handler for a live-generated file, or the official `nuxt-llms` module for build-time `llms.txt` and `llms-full.txt`. Nuxt's own documentation site, curl-verified for this guide, shows what a good file looks like. ## The Nuxt Rule That Makes This Easy Nuxt serves every file inside `public/` at the site root, untouched by the build — the same mechanism that publishes `robots.txt` and `favicon.ico` (the directory was called `static/` in Nuxt 2). That one rule means an `llms.txt` file on disk becomes your live `/llms.txt` with zero routes, zero config and zero plugins. There is no native Nuxt convention beyond it, so your choice is really about how much of the file you want to maintain by hand. ## Method 1: A Static public/llms.txt in 60 Seconds Create the file and publish — the whole setup is one command and one edit. A spec-compliant file needs only an H1; add a summary and grouped link lists for the context AI agents need: ``` # Acme Docs > Acme Docs covers the Acme API: REST reference, SDKs and step-by-step tutorials. ## Essentials - [Quickstart](https://acme.com/docs/quickstart): first API call in five minutes - [API Reference](https://acme.com/api): every endpoint, error and rate limit - [Guides](https://acme.com/docs/guides): auth, webhooks and data models ## Optional - [Changelog](https://acme.com/changelog): what changed in each release ``` Links must be absolute URLs, and every entry benefits from a short description — the same rules as the full [llms.txt format guide](/blog/llms-txt-format/). A second file named `public/llms-full.txt` gives agents the full-text sibling; see [llms.txt vs llms-full.txt](/blog/llms-txt-vs-llms-full-txt/) for when that pays off. The cost of this route is maintenance: every new page is a hand edit, and a file that drifts from your content is one agents follow into stale territory. ## Real-World Case: nuxt.com/llms.txt The Nuxt project dogfoods the format. On 2026-09-09 the official docs site served a 58,277-byte file at `/llms.txt`: ``` $ curl -s -o /dev/null -w "HTTP %{http_code}, %{size_download} bytes\n" https://nuxt.com/llms.txt HTTP 200, 58277 bytes ``` The file itself is a good model for a framework site. It opens with the `# Nuxt Docs` H1 and a blockquote that states exactly when the docs should be used. Then it links `nuxt.com/llms-full.txt` (the whole documentation in one file), advertises per-page Markdown — append `.md` to any URL or send `Accept: text/markdown` — and points agents at a public MCP server under `/mcp`. Clean Markdown and `llms-full.txt` matter enough to both [the v2 spec](/blog/llms-txt-v2/) and Nuxt's own file. ## Method 2: Generate It on the Fly with server/routes/ For content that changes faster than your deploy cadence, serve the file from a Nitro route. Nuxt maps `server/routes/llms.txt.get.ts` to `/llms.txt` — server routes outside `server/api/` get no `/api` prefix — and auto-imports `defineEventHandler`. Query Nuxt Content or your CMS and return plain text: ``` // server/routes/llms.txt.get.ts const header = '# Acme Docs\n\nDocumentation for the Acme API and SDKs.\n\n## Documentation\n\n' export default defineEventHandler(async () => header + (await getPages()).join('\n')) ``` Return a string and Nitro serves it as text; return a `Response` to set the `Content-Type` header explicitly. Every request rebuilds the file, so a new CMS entry is reflected instantly with no redeploy — at the cost of a query per request, which is why build-time generation is the better default for most sites. ## Method 3: nuxt-llms — Build-Time llms.txt and llms-full.txt [nuxt-llms](https://github.com/nuxtlabs/nuxt-llms), maintained by NuxtLabs and published on npm since February 2025 (v0.2.0, January 2026), generates and prerenders `/llms.txt` at build time from a small `nuxt.config.ts` block: ``` # terminal npm i nuxt-llms // nuxt.config.ts export default defineNuxtConfig({ modules: ['nuxt-llms'], llms: { domain: 'https://acme.com', title: 'Acme Docs', description: 'API reference and guides for the Acme platform.', sections: [{ title: 'Guides', links: [{ title: 'Quickstart', href: '/docs/quickstart' }] }], }, }) ``` What you get for that config: - **llms.txt prerendered at build** — set `prerender: false` to serve it dynamically instead (for example behind ISR). - **llms-full.txt on demand** — add a `full.title` and `full.description` and the module emits the second file too. - **Runtime hooks** — `llms:generate` lets modules and server plugins push extra sections, so CMS content can extend the file. - **Nuxt Content integration** — with `@nuxt/content` 3.2.0 or newer, every file in your `content/` directory is added to `llms.txt` and `llms-full.txt` automatically. ## Which One Should You Pick? Approach Setup Auto-updates llms-full.txt Best for Static public/ file 1 minute No Second file Small, stable sites server/routes handler ~10 minutes Every request Your code Content that changes between deploys nuxt-llms module ~5 minutes Every build Yes, via full option Docs and Nuxt Content sites For a documentation site on Nuxt Content, the module is the clear winner: the file cannot drift, because the build writes it from the same content your pages render. Static suits a brochure site that rarely changes; a server route only earns its per-request cost when freshness beats the deploy interval. ## Verify and Maintain - **Confirm the file is at the root** — `public/llms.txt`, never `public/llms/llms.txt`. - **Use absolute URLs** and one descriptive line per link — see the [llms.txt best practices guide](/blog/llms-txt-best-practices/). - **Check the deployed file** after every deploy: ``` curl -s -o /dev/null -w "HTTP %{http_code}, %{content_type}\n" https://yoursite.com/llms.txt # Expect: HTTP 200, text/plain ``` Paste the URL into the free [llms.txt checker](/checker), or feed your sitemap into the [llms.txt generator](/generator) for a first draft, then hand the maintenance to nuxt-llms — a zero-maintenance byproduct of the build that already publishes your Nuxt site. --- # How to Add llms.txt to Shopify: Replace the Default File (2026) > Source: https://llmstxtgenerator.dev/blog/llms-txt-shopify/ Shopify tutorial · E-commerce · 6 minutes # How to Add llms.txt to Shopify: Replace the Default File Good news: Shopify now publishes an `llms.txt` for every store automatically. Bad news: the default is Shopify's pitch, not yours. This guide shows how to swap in a curated file with a single theme template — no app, no Cloudflare Worker — and verify it is live. ## Shopify Now Serves llms.txt Natively For most of 2025, adding `llms.txt` to a Shopify store meant a workaround: intercept `/llms.txt` in a Cloudflare Worker fronting the store, or upload the file to Shopify Files and create a URL redirect from `yourstore.com/llms.txt` to the CDN link. Both approaches are dead now. Shopify serves the route at the platform level, and every store gets a generated file — open `yourstore.com/llms.txt` and you will get a real response, no setup required. That sounds like the problem is solved. Then you read the file and find the catch: it was generated by Shopify, _for_ Shopify — and it shows. ## Why the Default Shopify llms.txt Undersells Your Store - **It leads with Shopify, not you.** The default recommends the Shop app, links Shopify's developer tools, and points to pages for starting a new store — in some cases before a single product appears. An AI engine that reads the file first learns about the platform, not about your brand. - **It skips what matters.** Your top collections, best sellers, FAQ and about page are not prioritized — and may be missing entirely. - **It has no descriptions.** Bare URLs force an AI to fetch each page and guess what it covers, which is exactly what `llms.txt` exists to prevent. - **It wastes space on developer links.** Lines aimed at people building on Shopify carry zero value for a shopper asking ChatGPT or Perplexity which store sells what they want. Aspect Shopify default Your custom file Leads with Shopify ecosystem Your brand and catalog Catalog links Generic, unprioritized Top collections and best sellers Link descriptions Rare One line per link Update control Platform decides You decide The fix is a curated file — the same principle as the [e-commerce examples in our llms.txt sample library](/blog/llms-txt-examples/), but written around your store. ## Step 1: Write Content an AI Can Act On Start from your sitemap: Shopify generates one at `yourstore.com/sitemap.xml`. Paste it into the free [llms.txt generator](/generator) to draft the file, then curate — delete out-of-stock lines, thank-you pages and thin pages until only the links you want AI engines to reach remain. A proven shape for a store: ``` # Meadow & Stone Tea Co. > Meadow & Stone sells loose-leaf tea and brewing kits, ships worldwide from Portland, OR. ## Collections - [Best Sellers](https://meadow.example.com/collections/best-sellers): the 20 teas customers reorder most - [Gift Sets](https://meadow.example.com/collections/gift-sets): curated tea and kit bundles under $60 ## Products - [Jasmine Silver Needle](https://meadow.example.com/products/jasmine-silver-needle): single-origin jasmine green tea, 50 g pouch - [Cold Brew Kit](https://meadow.example.com/products/cold-brew-kit): carafe, filter and two sampler tins ## Useful Pages - [About](https://meadow.example.com/pages/about): sourcing, story and the team - [Shipping & Returns](https://meadow.example.com/pages/shipping): delivery times and the return policy ``` Curated beats exhaustive: a dozen strong links with clear descriptions outperform a dump of hundreds of SKUs, and AI engines reward files that follow the [llms.txt best practices](/blog/llms-txt-best-practices/) — short summaries per link, grouped sections, no dead URLs. ## Step 2: Add a templates/llms.txt.liquid Theme File Shopify treats `llms.txt` the way it treats `robots.txt`: a special template overrides the generated file. Confirmed on the Shopify developer community and in 2026 guides, the method is: 1. In your Shopify admin go to **Online Store → Themes**. 2. Open the three-dot menu next to your live theme and choose **Edit code**. 3. Inside the `templates` folder, click **Add a new template**. 4. Pick a plain **template** (not JSON), and name it exactly `llms.txt.liquid` — the `.liquid` extension is mandatory; a file named `llms.txt` will not be served. 5. Paste your curated Markdown content into the template and click **Save**. Your content is now served at `yourstore.com/llms.txt`, replacing the Shopify default. Theme code edits do not affect your store front end: the template only answers that one route. ## Step 3: Verify, Then Keep It Fresh Confirm the file is live and served correctly: ``` curl -s -o /dev/null -w "%{http_code}" https://your-store.com/llms.txt # Expect: 200 — then view the URL and check it is your content ``` Paste the URL into the free [llms.txt checker](/checker) to validate structure and links against the spec. Then set a maintenance cadence: refresh the file after major catalog or pricing changes, seasonal collection swaps, and product retirements. Each subdomain or regional storefront needs its own file at its own root, since URLs, currency and inventory differ between them. Once the default is replaced, Shopify stops speaking for you and your catalog speaks first. Draft the file from your sitemap with the [free llms.txt generator](/generator), publish it through the template, and validate with the [checker](/checker) — a five-minute job that puts your products ahead of the platform in every AI answer. --- # How Long Should an llms.txt Be? Size Limits and Token Budgets (2026) > Source: https://llmstxtgenerator.dev/blog/llms-txt-size-limit/ GEO · September 2026 # How Long Should an llms.txt Be? Every guide quotes a different ceiling — 50KB, 100KB, five thousand tokens — and none of them comes from the specification. Here is the rule the spec actually gives, where the practical ceiling really comes from, and the cut list for a file that grew past it. ## The Spec Gives You a Rule, Not a Number The proposal at llmstxt.org is deliberately quiet about bytes. Its only size statement is functional: the file stays small enough to fit in context, the detail lives behind the links, and is fetched only when a client needs it. That is a design constraint dressed up as a sentence, and it is more useful than a number, because the number would be wrong within a year. [The v2 revision](/blog/llms-txt-v2/) kept that framing. The figures people repeat come from three other places. A third-party AI discovery file specification recommends no more than 50KB. Platform tutorials aimed at marketers say keep it under 100KB and UTF-8 encoded. Agency guidance usually lands nearer five thousand tokens, with the observation that most marketing sites sit comfortably under that. All three are budgets rather than rules, and the useful move is to know which one your consumer enforces — an indexing tool, a context window, or nothing at all. ## Where the Real Ceiling Comes From The binding constraint is whatever parses the file. Two public examples bracket it. Mintlify cites Turborepo's llms.txt at roughly 116,000 tokens as the cautionary tale — a file so large it defeats the purpose of being cheap to read. On the tooling side, developers in the Cursor community report that files above roughly 50,000 to 60,000 tokens get unstable during indexing, while a 4MB file (somewhere between 800,000 and 1.2M tokens at about five characters per token) worked in one test and is well outside anything else quoted. For a normal site the practical answer is much lower than any of that. An index whose links carry one-line descriptions is a few kilobytes. Once your file is bigger than your homepage, someone has started concatenating content instead of curating it. File size Roughly Verdict Under 5 KB 800 – 1,500 tokens Healthy curated index. Typical for a marketing site or blog 5 – 20 KB 1,500 – 5,000 tokens Fine, but audit the descriptions first — they are the usual growth 20 – 50 KB 5,000 – 12,000 tokens Over budget. Split the dense area into a subpath file 50 – 100 KB 12,000 – 25,000 tokens Above every published ceiling. Expect truncation or a skipped read Over 100 KB 25,000+ tokens Stop calling it an index. This is llms-full.txt territory ## Measure It in Two Commands Never estimate size from your editor — CDN compression, encoding and duplicated links all change the answer. Fetch what a crawler fetches: ``` # bytes and word count of the live file curl -s https://example.com/llms.txt | wc -c curl -s https://example.com/llms.txt | wc -w # structure, broken links and missing elements curl -sI https://example.com/llms.txt | grep -i "content-type" ``` An index-style file should come back in the low kilobytes. If the byte count is in the tens of thousands, compare it against the table above and then run the same URL through the [llms.txt checker](/checker) to confirm the structure survived whatever generated it. ## The Five Things That Bloat an llms.txt Size problems are nearly always one of five decisions made once and never revisited: - **Enumerating every post.** List the pillar pages, the category hubs and the feed, not four years of articles. The index points; it does not archive. - **Pagination and tag archives.** `/page/2/` through `/page/40/` and every tag landing page add lines that resolve to the same content with no new value. - **Descriptions that became paragraphs.** One line, eight to twelve words. The [annotation rules](/blog/llms-txt-descriptions/) matter more than usual on a large file, because every extra clause is multiplied by every link. - **Near-duplicate pages.** Thirty city pages with the city name swapped is fifty kilobytes of noise. List the hub and the top markets. - **Concatenated content.** If full page text is in the index, it belongs in [llms-full.txt](/blog/llms-txt-vs-llms-full-txt/) instead. ## Split Before You Delete The spec supports subpath files, and the rule is that a client uses the most specific file covering the URL it is working in. That turns size into a routing decision: keep `/llms.txt` as a short front door for the whole site, and put the dense subtree at `/docs/llms.txt`, linked from the root file like any other page. The discovery mechanics for those files — [paths, link relations and reachability](/blog/llms-txt-discovery/) — decide whether they are found. Two things worth remembering about the split. The root file must link to the subpath file, or nothing connects them. And the subpath file inherits the same budget pressure, so the same rules apply one level down. ## What Size Discipline Does Not Buy A lean file is a cost decision, not a growth lever. SE Ranking's study of roughly 300,000 domains found a 10.13% adoption rate and no correlation between having llms.txt and being cited in AI answers — removing the variable actually made their prediction model slightly more accurate. Ahrefs reached a similar place from the log side in May 2026: across 137,210 domains with a valid file, 97% received no request for it in a month. Which is the argument for keeping the file small rather than for skipping it. The clients that do fetch it — developer tools, coding agents, a handful of crawlers — pay for every token, and a file that fits in context is the one that gets read. Start from the [llms.txt generator](/generator) if you want a lean first draft, keep it under a few kilobytes, and re-run the byte count whenever you add a section. --- # llms.txt v2: What Changed in the 2026 Spec Update > Source: https://llmstxtgenerator.dev/blog/llms-txt-v2/ Spec update · August 2026 · AI readiness # llms.txt v2: What Changed in the 2026 Spec Update Two years after the original proposal, the llmstxt.org specification has its first major revision. v2 (August 2026) adds discoverability, defines subpath files, and changes how agents are expected to use the file. Here is exactly what changed — and what you should update on your site. ## Why a v2 at All? The original llms.txt proposal was published by Jeremy Howard on September 3, 2024, when the idea that language models would routinely read websites was still speculative. By 2026 that is routine: thousands of sites publish an llms.txt file, documentation platforms such as Mintlify generate one automatically, Chrome's Lighthouse audits sites for one as part of its agentic browsing checks, and OpenAI, Anthropic, and Gemini all publish llms.txt files for their own developer docs. The revision (updated August 10, 2026) reflects what two years of adoption taught the community — and the most common request was _discoverability_. ## Change 1: Two URL Forms for Markdown Versions v1 specified one way to expose a machine-readable version of a page: append `.md` to the full URL, so `page.html` becomes `page.html.md`. In practice, many publishing tools replace the extension instead. v2 blesses both forms: - Appended: `https://example.com/docs/page.html.md` - Replaced: `https://example.com/docs/page.md` For URLs without a filename, append `index.html.md` or `index.md`. Agents that follow a link from your llms.txt to a markdown page should now accept either form. ## Change 2: Subpath Files Are Officially Defined v1 allowed llms.txt in subpaths without saying what that meant. v2 defines it: a file covers the pages under its path, and where more than one file applies, agents should use the most specific one. A root `/llms.txt` covers the whole site; `/docs/llms.txt` covers only the documentation. This matters for sites that control just a directory — a GitHub Pages project site, for example, can publish `llms.txt` in its own path even though it can never touch the host root. The FastHTML project does exactly this, placing its file at `/docs/llms.txt` to cover only its documentation pages. ## Change 3: Standard Link Relations for Discoverability Given a page, how does an agent find its markdown version — or the llms.txt file that covers it — without guessing? v2 answers with two standard link relations: - `rel="alternate" type="text/markdown"` — points to the page's markdown version - `rel="describedby"` — points to the llms.txt file that covers the page You can provide them as HTML link elements in the page head: ``` ``` Or as an HTTP `Link:` response header, which also works for non-HTML resources and can be added in web server or CDN configuration without touching any pages: ``` Link: ; rel="alternate"; type="text/markdown", ; rel="describedby" ``` ## Change 4: Agents View, Then Follow Links v1 described a tool, `llms_txt2ctx`, that expanded a file into an LLM context, and the `Optional` section carried special meaning for it. v2 drops the context-expansion tooling from the proposal and states the expectation directly: agents view or search the llms.txt to find what they need, then follow the relevant links — which should point to LLM-friendly content such as the markdown versions above. The `Optional` section is still allowed and remains a useful convention for secondary links an agent can skip, but it no longer has mechanical semantics. The practical takeaway: keep the file small enough to fit in context, and put the detail behind the links. ## What Did Not Change The core format is untouched: an optional byte-order mark, the H1 with your site name (the only required element), a blockquote summary, optional context paragraphs, and H2 sections with markdown file lists. Root placement still works. Everything you learned from the [format guide](/blog/llms-txt-format/) and [best practices](/blog/llms-txt-best-practices/) remains valid — existing files don't break. ## Your v2 Update Checklist - Publish markdown versions of your key pages (either `.md` or `.html.md` form). - Point the links in your llms.txt at the markdown versions, not the HTML pages. - Add `rel="alternate"` and `rel="describedby"` via HTML link elements or an HTTP `Link:` header — see the [Cloudflare Pages guide](/blog/llms-txt-cloudflare-pages/) for CDN-level headers. - Use a subpath file such as `/docs/llms.txt` when you control only a directory. - Keep secondary links in an `Optional` section; keep the file under a few dozen links. - Re-validate after every change with the [free validator](/checker). The v2 update is small but meaningful: it makes llms.txt easier for agents to find, easier to consume, and usable by sites that only control a path. Generate a spec-compliant file with the [generator](/generator), check it with the [validator](/checker), and you're ready for the agentic web. --- # How to Add llms.txt to VitePress (Static or Auto-Generated, 2026) > Source: https://llmstxtgenerator.dev/blog/llms-txt-vitepress/ VitePress tutorial · Static or plugin · 7 minutes # How to Add llms.txt to VitePress VitePress is the Vue-powered static site generator behind the documentation of Vite, Vue.js, Vitest and many other flagship open-source projects. It has no native `llms.txt` feature — yet its own documentation ships one, and so do the docs of every major project in its ecosystem. Two routes get you there: a static file in `public/`, or build-time generation with `vitepress-plugin-llms`. Both are covered here, with real config and a curl check. ## VitePress Docs Sites That Already Publish llms.txt Documentation is where llms.txt earns its keep — the llmstxt.org spec notes that coding agents follow the file to reach API references and tutorials — and the VitePress ecosystem was among the first to dogfood it. These are not hypothetical examples. Verified with curl on 2026-09-06: Site Built with VitePress for /llms.txt on 2026-09-06 vuejs.org Vue.js HTTP 200 vite.dev Vite HTTP 200 vitest.dev Vitest HTTP 200 vitepress.dev VitePress itself HTTP 200 The shape of these files is worth studying. The one on vitepress.dev starts with a single H1, a one-line blockquote summary, and a Table of Contents: ``` # VitePress > Vite & Vue Powered Static Site Generator Markdown to beautiful docs in minutes ## Table of Contents - [Getting Started](/guide/getting-started.md): Get up and running with VitePress. Learn how to install, scaffold, and start developing your documentation site. - [Asset Handling](/guide/asset-handling.md): Learn how to reference and handle static assets such as images, media, and fonts in VitePress. ``` Notice the links: every page is listed as a `.md` URL. That is the markdown page form the [llms.txt v2 spec](/blog/llms-txt-v2/) defines — a VitePress page rendered as clean Markdown is exactly the format an LLM wants to ingest, and it is what `vitepress-plugin-llms` outputs. ## Method 1: Drop a Static File in the public Directory VitePress gives you a shortcut that most SSGs do: everything in the `public` directory — `docs/public` by default, directly under your source directory — is copied verbatim to the root of the build output. The official docs name `robots.txt` and favicons as the typical use case; `llms.txt` works the same way. Create `docs/public/llms.txt`: ``` # Acme Docs > Acme Docs is the official documentation for the Acme platform. > It covers installation, configuration and the REST API. ## Getting Started - [Quickstart](https://docs.example.com/getting-started/): first project in five minutes - [Authentication](https://docs.example.com/authentication/): API keys and scopes ## API Reference - [REST API](https://docs.example.com/api/): every endpoint, parameter and error ``` The next `npm run docs:build` serves it at `https://docs.example.com/llms.txt`. Two notes: if you set a custom `srcDir`, the public directory follows it, and if the site deploys under a subpath via the `base` option, the file still lands at the output root while your listed URLs stay absolute. This route is fine while your docs are small and stable — the cost is that every new page means a hand edit, and a file that drifts from your content is a file agents follow into stale territory. ## Method 2: Auto-Generate with vitepress-plugin-llms The ecosystem workhorse is [okineadev/vitepress-plugin-llms](https://github.com/okineadev/vitepress-plugin-llms) — roughly 400 stars as of September 2026, and adopted across the VoidZero ecosystem with the blessing of Vue.js. The flagship sites above use it: the file served on vitepress.dev is its output. Install and register it: ``` # terminal npm install vitepress-plugin-llms --save-dev ``` ``` // .vitepress/config.ts import { defineConfig } from 'vitepress' import llmstxt from 'vitepress-plugin-llms' export default defineConfig({ vite: { plugins: [llmstxt()], }, }) ``` Zero configuration required. Every build now writes three things into `.vitepress/dist`: `llms.txt` (the curated index with section links), `llms-full.txt` (all documentation merged into one file — see how the two relate in the [llms.txt vs llms-full.txt guide](/blog/llms-txt-vs-llms-full-txt/)), and a clean Markdown version of every page, which llms.txt links to. The plugin authors call out one practice that visibly improves the output: add a `description` to each page's frontmatter and it becomes the link description in llms.txt — which is exactly what an agent reads when deciding which link to fetch. Two finer points from the project's documentation. First, you can mark sections of a Markdown source file as `llm-only` or `llm-exclude` to control what reaches the LLM versions — useful for instructions aimed at agents, or human-only notes. Second, for repositories with documentation in multiple languages the author recommends enabling the plugin for the English docs only; that is all an LLM needs. ## Alternative: vitepress-plugin-llmstxt If you need finer control than zero-config gives you, [vitepress-plugin-llmstxt](https://www.npmjs.com/package/vitepress-plugin-llmstxt) (npm 0.5.x, zero dependencies) also generates `llms.txt`, `llms-full.txt` and per-page Markdown files, then adds: glob-based `ignore` patterns to exclude routes, a `transform` callback that rewrites generated content, dynamic-route and i18n support, and experimental compatibility with VitePress 2.0 alpha. A configured example: ``` // .vitepress/config.ts import { defineConfig } from 'vitepress' import llmstxt from 'vitepress-plugin-llmstxt' export default defineConfig({ vite: { plugins: [ llmstxt({ hostname: 'https://docs.example.com', ignore: ['**/api/**/*'], }), ], }, }) ``` The `hostname` option pins the absolute URLs in your file (useful when VitePress infers the wrong origin), and `ignore` keeps internal or generated pages out of the index. ## Static File or Plugin? Approach Setup Per-page .md llms-full.txt Stays current Static `docs/public/llms.txt` 1 minute No No Manual edits only `vitepress-plugin-llms` ~5 minutes Yes Yes Every build `vitepress-plugin-llmstxt` ~5 minutes Yes Yes Every build A hand-written file is fine while your sitemap is small and your structure rarely moves. The moment docs change weekly — new guides, migrated pages, added API endpoints — generation wins, because the llms.txt an agent reads is only as trustworthy as its last update, and a plugin cannot forget to update itself. ## The VitePress Checklist - **Confirm the public directory path** — it is `docs/public` by default but moves with `srcDir`; a static file in the wrong folder quietly never gets published. - **Register the plugin under `vite.plugins`**, not at the top level of the VitePress config — these are Vite plugins and VitePress only loads them there. - **Write a `description` in every page's frontmatter** — it becomes the link description agents use to choose pages. "What problem this page solves" beats a module name. - **Keep it English-only for LLM purposes** — for multi-language docs, generate the file from the English tree only, per the plugin's own guidance. - **Verify the deployed file, not just the build** — redirect rules, domain moves and docs restructures break llms.txt in production after the build goes green. Confirm it serves as plain text: ``` curl -s -o /dev/null -w "%{http_code} %{content_type}\n" https://docs.example.com/llms.txt # Expect: 200 text/plain ``` Then paste the URL into the free [llms.txt checker](/checker) to validate the H1, blockquote summary, absolute URLs and section structure against the spec. To draft the initial file in seconds, feed your docs sitemap into the [free llms.txt generator](/generator), then let the plugin keep it current from there. VitePress made llms.txt a zero-maintenance default for the biggest projects in the Vue ecosystem — your docs can be next. --- # llms.txt vs llms-full.txt: What's the Difference and Which Do You Need? > Source: https://llmstxtgenerator.dev/blog/llms-txt-vs-llms-full-txt/ llms.txt spec · AI content strategy · Updated 2026 # llms.txt vs llms-full.txt: What's the Difference and Which Do You Need? **llms.txt** is a curated index of your best pages. **llms-full.txt** is the entire site content in one Markdown file. Different jobs, different sizes — here's how to choose. ## The Short Answer `llms.txt` gives AI agents a _map_ of your site: a small, curated Markdown list of the pages that matter most. `llms-full.txt` gives them the _territory_: the full text of your content, formatted as Markdown, in a single file. The spec at [llmstxt.org](https://llmstxt.org) treats llms.txt as the primary file — llms-full.txt is an optional companion for sites where the map alone isn't enough. ## What llms.txt Is (and Isn't) An llms.txt file is a short Markdown document at your site root. It opens with an H1 (your site name) and a one-line `>` blockquote summary, followed by a curated list of links — usually your 5 to 15 most important pages, grouped under `##` headings. ``` # Example Site > A concise description of what the site is and who it's for. ## Documentation - [Getting Started](https://example.com/docs/getting-started) - [API Reference](https://example.com/docs/api) ## Blog - [How We Built X](https://example.com/blog/how-we-built-x) ``` The job of llms.txt is _prioritization_. An agent that lands on your site can read this file in milliseconds and know exactly which pages are worth fetching — instead of guessing from a crawl or landing on your marketing pages. ## What llms-full.txt Is llms-full.txt is the same idea scaled up: your site's full content, converted to Markdown and merged into one file, served from `https://your-domain.com/llms-full.txt`. It's meant for situations where the curated index isn't enough material for the agent: - **Documentation sites** — an agent needs the actual API docs, not just a link to them. - **Long-form content** — tutorials and guides that span thousands of words. - **JS-rendered pages** — content that doesn't exist in the raw HTML an agent fetches. - **Research-heavy sites** — papers, data pages, reference material agents should read in full. You link to it from your llms.txt with a relative Markdown link: ``` # Example Site > A concise description of what the site is and who it's for. ## Documentation - [Getting Started](https://example.com/docs/getting-started) - [API Reference](https://example.com/docs/api) [llms-full.txt](llms-full.txt) ``` ## Key Differences at a Glance llms.txt llms-full.txt **Content** Curated link list Full site content as Markdown **Typical size** 1–5 KB 10 KB – several MB **Purpose** Prioritize what agents read Give agents everything at once **Required?** Yes (the primary file) Optional **Best for** Every site Docs, long-form, JS-heavy sites ## Which One Do You Need? Start with **llms.txt alone**. It's the file every AI engine checks first, it's trivial to maintain, and for most sites the curated list is exactly what an agent needs. Add **llms-full.txt** when you observe that agents need more material — typically once your site has substantial documentation or long-form content, or when you want AI engines to be able to answer from your content without fetching every page. There's no penalty for having both, and no penalty for having just llms.txt. The only real mistake is treating llms-full.txt as a replacement for the curated index — an agent expects the index first, and llms-full.txt is reached _through_ it. ## Validate Both Files After publishing either file, run it through our free [llms.txt validator](/checker) — it checks the H1, absolute URLs, Markdown formatting, and structure, and gives you an AI Readiness Score so you know the file will parse cleanly for agents. Then use the [generator](/generator) to produce a fresh llms.txt from your sitemap in seconds. --- # llms.txt vs MCP: The Read Layer vs the Action Layer for AI Agents (2026) > Source: https://llmstxtgenerator.dev/blog/llms-txt-vs-mcp/ AI Agents & Discovery · September 2026 # llms.txt vs MCP: The Read Layer vs the Action Layer Two protocols get mentioned in the same breath and solve opposite problems. `llms.txt` answers _"what is on this site?"_ — a static map an agent reads once. MCP answers _"what can I do here?"_ — a live interface an agent calls. Confusing them leads to the wrong investment: a file you never needed to make dynamic, or a server nobody can find. ## Two Questions, Two Protocols Ask a chat assistant to "summarise how this company handles refunds" and it needs **content**: which page, in what form. Ask an agent to "find the cheapest plan for eight seats" and it needs **capabilities**: a query it can run against your data or your UI. `llms.txt` is the accepted answer to the first, MCP to the second, and a third, browser-level answer — WebMCP — is emerging underneath both. ## llms.txt Is the Read Layer `/llms.txt` is a Markdown file at your site root, proposed by Jeremy Howard in September 2024 and maintained by the community as a convention rather than a formal standard — the current shape of it is documented at [llmstxt.org](https://llmstxt.org/). It carries an H1 name, a summary blockquote and curated sections of links with one-line descriptions. Nothing executes; nothing authenticates. Its value is proportional to how bad your website is to parse. An agent landing on an HTML marketing page burns context on navigation, cookie banners and markup before it reaches the answer; a curated file replaces that with a handful of links. That is why documentation-heavy sites adopted first — and why independent crawl studies put overall adoption at roughly **10% of domains**, far below robots.txt or XML sitemaps. Treat it as the cheap, static, cacheable layer. [The format guide](/blog/llms-txt-format/) covers every element and the [generator](/generator) turns a sitemap into a first draft. Google has said repeatedly that the file has no effect on Search rankings — see [what Google actually says](/blog/does-llms-txt-affect-seo/). That makes it an agent-facing asset, not a ranking asset. ## MCP Is the Action Layer The Model Context Protocol, launched by Anthropic in late 2024 and now maintained as an open specification at [modelcontextprotocol.io](https://modelcontextprotocol.io/), defines how a client (an assistant or IDE) connects to a server that exposes tools, resources and prompts. Instead of reading your documentation, the agent calls it: a search tool, a pricing lookup, a status endpoint. The **2026-07-28 revision** is the biggest change since remote MCP shipped. The `initialize` handshake and the `Mcp-Session-Id` header are retired: requests are stateless, each carrying its protocol version, client identity and capabilities in request metadata. A capability inventory is now an optional `server/discover` call, and the new `Mcp-Method` and `Mcp-Name` headers let a gateway route on the operation without parsing the body: ``` POST /mcp HTTP/1.1 MCP-Protocol-Version: 2026-07-28 Mcp-Method: tools/call Mcp-Name: search Authorization: Bearer Content-Type: application/json ``` The practical consequence: with no session store to share, a server can run behind a plain round-robin load balancer — or, as Cloudflare describes in [its MCP v2 write-up](https://blog.cloudflare.com/mcp-v2/), entirely inside a single Worker. Remote servers are expected to use OAuth 2.0 with PKCE, and updated TypeScript, Python, Go and C# SDKs shipped with the spec. MCP has one weakness llms.txt does not: **the protocol has no discovery**. Agents connect only to servers they were configured with, so yours can be excellent and unknown. This is where the read layer earns its keep: a curated file naming your MCP endpoint, what it exposes and who it is for is the most reliable "here is my server" signal you control. ## WebMCP: The Third, In-Browser Layer WebMCP moves the same idea into the page. Rather than a server the agent connects to, the browser exposes tools the site declares: _declaratively_ by annotating existing HTML forms, or _imperatively_ with `navigator.modelContext.registerTool`. The agent then operates the interface the user is already looking at, inside the visitor's own session — no separate endpoint, no separate auth. WebMCP is still experimental. Chrome's documentation for [Lighthouse's Agentic Browsing category](https://developer.chrome.com/docs/lighthouse/agentic-browsing/scoring) — which verifies tool registration, checks the accessibility tree, measures layout stability and inspects `llms.txt` discoverability — requires Chrome 150 or later, and its WebMCP audits require the origin trial. Not a ranking factor, but a clear signal of where browser vendors are heading. ## Side by Side Layer What it is Who consumes it Cost to ship `llms.txt` Static Markdown index at the site root Reading crawlers and assistants An afternoon, then occasional edits MCP server Live tools and resources over a stateless protocol Configured clients: IDEs, assistants, agents Engineering work, auth, ongoing maintenance WebMCP Browser-exposed tools from an existing page Agents acting in a live session Form annotations today, experimental API ## The Adoption Order That Makes Sense Layer these in sequence, and only move down when the layer above is measurably working: - **1\. Publish and validate the file.** Run it through the [llms.txt checker](/checker): every link must resolve, every entry must carry a description, and the header block must follow the spec. - **2\. Measure whether anything reads it.** Start with access logs. If fetches are still zero after two months, the [crawler data](/blog/which-ai-engines-read-llms-txt/) and the [measurement framework](/blog/measure-llms-txt-traffic/) tell you whether that is normal for your niche. - **3\. Make the file point at real capabilities.** Documentation subsections, a machine-readable pricing page, a status endpoint, your MCP server if you have one. This is where the two layers compound. - **4\. Build a server only for repeated, high-value questions.** Support deflections, catalogue lookups, quote generation. If nobody asks twice a week, a smaller file beats a bigger server. - **5\. Watch the browser layer.** If your value sits in a form, configurator or checkout flow, WebMCP is the place to prototype — but not on the roadmap until the audits leave origin-trial status. The short version: `llms.txt` is how an agent learns what you have, MCP is how it acts on it, and neither replaces the other. Ship the file, instrument it, and let real request data — not vendor enthusiasm — decide whether a server comes next. --- # llms.txt vs robots.txt vs sitemap.xml: What's the Difference? > Source: https://llmstxtgenerator.dev/blog/llms-txt-vs-robots-txt/ Site root files · Search & AI discovery · Updated 2026 # llms.txt vs robots.txt vs sitemap.xml: What's the Difference? Three tiny files, three very different jobs. **robots.txt** controls which crawlers may access your site, **sitemap.xml** tells search engines which URLs exist, and **llms.txt** tells AI engines what's worth reading. Open the root directory of almost any serious website and you will find a small set of plain files: `robots.txt`, `sitemap.xml`, and — increasingly — `llms.txt`. They look similar — tiny, text-based files in the same folder, served to automated visitors — so it's tempting to assume they do the same job. They don't: each is read by a different consumer and answers a different question. Below we compare all three and show how they work together. ## What Is robots.txt? — Crawler Permission Control **robots.txt** is the oldest of the three, dating back to the Robots Exclusion Protocol of 1994. It is a plain-text file placed at the root of a domain (`https://example.com/robots.txt`) that tells compliant crawlers which parts of the site they may or may not crawl. **Who reads it:** search-engine crawlers — Googlebot, Bingbot, YandexBot, DuckDuckBot — and increasingly AI crawlers such as GPTBot, ClaudeBot, and PerplexityBot. Any compliant crawler checks `robots.txt` before it fetches anything else on your domain. **What it does:** `robots.txt` does not rank, index, or describe anything — it is a permission system. Rules like `Disallow: /admin/` stop compliant crawlers from requesting those paths, keeping private sections out of the crawl. It can also point crawlers at your sitemap via a `Sitemap:` directive. ``` User-agent: * Disallow: /admin/ Disallow: /private/ User-agent: GPTBot Disallow: / Sitemap: https://example.com/sitemap.xml ``` One crucial caveat: `robots.txt` only governs well-behaved crawlers, and blocking a URL does _not_ remove it from Google — it can even leave a bare URL-only result with no snippet. For true removal, use a `noindex` meta tag or HTTP header instead. ## What Is sitemap.xml? — The Complete URL Inventory **sitemap.xml** is an XML file that follows the Sitemap Protocol from sitemaps.org. It is an inventory of your site: a machine-readable list of every URL you want search engines to know about, optionally annotated with ``, ``, and `` hints. **Who reads it:** search-engine indexers. Google, Bing, and others check the sitemap (typically referenced from `robots.txt` or submitted via Search Console) and use it to discover and prioritize new or rarely linked URLs. **What it does:** a sitemap has no effect on rankings — it improves _discovery_. Pages that would otherwise wait weeks to be found through internal links get crawled quickly, and a fresh `` helps crawlers notice updates. Large sites split sitemaps into multiple files via a sitemap index. ``` https://example.com/ 2026-08-01 https://example.com/docs/getting-started 2026-08-10 ``` Keep in mind a sitemap is a machine inventory, not a recommendation: it lists every URL with no notion of what is most important for a human or an AI to read first. ## What Is llms.txt? — A Content Index Built for AI Engines **llms.txt** is the newest of the three. Proposed by AI researcher [Jeremy Howard](https://en.wikipedia.org/wiki/Jeremy_Howard_\(entrepreneur\)) in 2024 and specified at [llmstxt.org](https://llmstxt.org), it is a small Markdown file at the root (`https://example.com/llms.txt`) that gives AI engines a curated, human- and LLM-readable index of your content. The file opens with an H1 title and an optional blockquote summary, followed by sections of Markdown links, each with a short description. It is designed to fit in a model's context window, so an agent can understand your whole site from one small file. **Who reads it:** AI engines and agents — ChatGPT, Claude, Gemini, Perplexity, coding assistants, and retrieval tools. Adoption is accelerating: Chrome Lighthouse audits for it, platforms like Mintlify and GitBook generate it automatically, and OpenAI, Anthropic, and Google publish llms.txt files for their own docs. **What it does:** where `robots.txt` says what not to crawl and `sitemap.xml` lists every URL, `llms.txt` tells an AI engine what the site is about and which pages to read first. It is curation, not inventory: you order links by importance, add descriptions, and point agents at clean Markdown versions of your pages. There is no ranking or crawling behavior attached — it is simply a content index. ``` # Example Docs > Example is a developer platform for building serverless applications in Python and TypeScript. ## Important links - [Quick start](https://example.com/docs/quickstart.md): Set up your first app in 10 minutes - [API reference](https://example.com/docs/api.md): Complete REST API reference ## Optional - [Changelog](https://example.com/changelog.md): Release notes for all past versions ``` ## llms.txt vs robots.txt vs sitemap.xml: Side-by-Side Comparison Here is the full comparison. The short version: robots.txt manages crawler access, sitemap.xml feeds indexing, and llms.txt serves AI engines — three different consumers, three different jobs. Dimension `robots.txt` `sitemap.xml` `llms.txt` **Intended reader** Search-engine and AI crawlers (Googlebot, Bingbot, GPTBot) Search-engine indexers and crawlers AI engines and agents (ChatGPT, Claude, Gemini, Perplexity) **File format** Plain text (Robots Exclusion Protocol) XML (Sitemap Protocol) Markdown (llmstxt.org spec) **Location** Root only: `/robots.txt` Any URL; typically referenced from robots.txt Root by convention: `/llms.txt` (subpaths allowed) **Primary purpose** Control which crawlers may access which paths List every URL you want discovered and indexed Curate an index of what AI engines should read **Impact on Google rankings** None None None — built for AI engines, not ranking **How it is discovered** Checked automatically by compliant crawlers Via `Sitemap:` directive or Search Console submission Fetched by convention at `/llms.txt` or found via links **Typical size** A few lines to a few kilobytes Up to 50,000 URLs per file (usually split) Small enough to fit a context window **Typical example** `Disallow: /admin/` `` with `` entries `# Site name` + sections of Markdown links **If it is wrong or missing** Crawlers may access things you wanted private New pages are discovered slowly Agents misread your site or miss your best content ## Can You Use robots.txt, sitemap.xml, and llms.txt Together? Absolutely. None of these files substitutes for another — they work best as a pipeline answering three different questions: robots.txt Read by crawlers (Googlebot, GPTBot) Answers: “May I crawl this?” sitemap.xml Read by search-engine indexers Answers: “Which URLs exist?” llms.txt Read by AI engines (ChatGPT, Claude, Perplexity) Answers: “What should AI read?” Crawler flow: robots.txt → sitemap.xml → pages. AI flow: llms.txt → linked Markdown pages. `robots.txt` points crawlers at the sitemap via a `Sitemap:` directive; `llms.txt` is served directly to AI engines. A practical setup for a modern site: - Point crawlers at your sitemap with a `Sitemap:` directive in `robots.txt`, so discovery is automatic. - Allow GPTBot, ClaudeBot, and PerplexityBot instead of blanket-blocking them, then control what they consume with a well-curated `llms.txt`. - Link to `llms.txt` from your homepage or footer, and keep it in sync with your sitemap — add new guides to both files. - Use `robots.txt` to protect genuinely private paths, and `noindex` — not robots.txt — to remove indexed pages from Google. ## Common Misconceptions About These Three Files - **“llms.txt will boost my Google rankings.”** — No. Google does not read `llms.txt` for ranking; the file targets AI engines such as ChatGPT, Claude, and Perplexity. For better Google rankings, invest in content quality, internal links, and Core Web Vitals — and treat `llms.txt` as a separate channel for AI visibility. - **“robots.txt removes pages from Google.”** — No. `robots.txt` blocks crawling, not indexing. A blocked URL can still appear in results. Use `noindex` for removal and robots.txt for crawl control. - **“llms.txt is just a sitemap in Markdown.”** — No. A sitemap is an exhaustive machine inventory; `llms.txt` is a curated, ordered recommendation with a summary and descriptions. Sitemaps are read by indexers; llms.txt by agents that reason about your content. - **“A complete sitemap.xml is enough for AI engines.”** — Increasingly not. AI engines can crawl sitemaps, but increasingly look for `llms.txt` by convention because it tells them what matters in one small file — different consumers, complementary files. ## FAQ: llms.txt vs robots.txt vs sitemap.xml ### What's the difference between robots.txt and sitemap.xml? `robots.txt` is a permission file for crawlers (“may I crawl, and where?”); `sitemap.xml` is a URL inventory for indexing (“these URLs exist”). They interact via the `Sitemap:` directive in robots.txt, but permission control and discovery are two different jobs. ### Does llms.txt replace my sitemap.xml? No. `sitemap.xml` feeds search-engine indexing with every URL you own; `llms.txt` feeds AI engines with the curated subset that matters most — different consumers, formats, and goals. Most sites should publish both, plus `robots.txt`. ### Where should llms.txt live, and how do AI engines find it? At the site root by convention — `https://your-domain.com/llms.txt`, next to `robots.txt`. Agents that support the convention check that URL automatically; linking from your homepage or footer helps discovery. Subpaths (e.g. `/docs/llms.txt`) work too, covering just that section. ## Keep reading [ ### 🧭 What is llms.txt? The beginner-friendly overview: what it is and why it matters now. ](/guide/)[ ### 📐 The Complete Format Guide Every element of the v2 spec explained, with a valid end-to-end example. ](/blog/llms-txt-format/) ## Ready to add llms.txt to your site? Generate a spec-compliant llms.txt in under a minute, then validate it for AI-readiness — no sign-up needed. [⚡ Generate your llms.txt](/generator) [Check my file](/checker) --- # llms.txt vs sitemap.xml: Why AI Engines Need a Different File > Source: https://llmstxtgenerator.dev/blog/llms-txt-vs-sitemap-xml/ Site root files · AI discovery · Updated 2026 # llms.txt vs sitemap.xml: Why AI Engines Need a Different File Your `sitemap.xml` is a complete inventory for search engines. Your `llms.txt` is a curated recommendation for AI agents. Here's why one can't do the other's job. ## What sitemap.xml Does A sitemap is an XML file that lists _every URL_ you want search engines to index — product pages, articles, category pages, sometimes thousands of entries — with metadata like last-modified dates and change frequency. Googlebot and Bingbot read it to discover pages they might otherwise miss. ``` https://example.com/docs/getting-started 2026-08-01 ``` Its strengths: complete, machine-readable, and the industry standard for indexing. Its weakness for AI: it's a _flat list_. Nothing says which pages matter, what each one is about, or what order an agent should read them in. ## What llms.txt Does An llms.txt file is a short Markdown document — typically 5 to 15 curated links — that tells an AI agent exactly what your site is and which pages are worth reading: ``` # Example Site > A concise description of what the site is and who it's for. ## Documentation - [Getting Started](https://example.com/docs/getting-started) - [API Reference](https://example.com/docs/api) ## Blog - [How We Built X](https://example.com/blog/how-we-built-x) ``` It's a _curation_ file. The agent gets your H1, a one-line summary, and a prioritized list in seconds — so it fetches your best pages instead of crawling blindly or landing on a tag archive. ## The Comparison sitemap.xml llms.txt **Format** XML Markdown **Audience** Search crawlers (Googlebot, Bingbot) AI engines (ChatGPT, Claude, Perplexity, Gemini) **Scope** Every URL 5–15 curated pages **Priorities** No Yes (order + H2 groups) **Human-readable?** No Yes **Effect on Google rankings** Yes (indexing) None (Google doesn't read it) ## Why You Need Both Different consumers, different jobs. Your sitemap keeps Google's index complete; your llms.txt makes AI answers about your site accurate and grounded in your best content. The two also support each other operationally: many teams [generate llms.txt from the sitemap](/blog/llms-txt-cloudflare-pages/) automatically, and our [generator](/generator) does exactly that — paste your sitemap URL and get a curated, spec-compliant llms.txt in seconds. The realistic caveat: AI crawler adoption of llms.txt is still ramping up, and Google has explicitly said the file does not affect its rankings. Treat llms.txt as an [AI-visibility signal](/blog/does-llms-txt-affect-seo/), not a ranking factor — and keep your sitemap healthy for the search side of the equation. ## Quick Checklist - Submit your sitemap in Google Search Console and Bing Webmaster Tools. - Serve llms.txt at the site root, spec-compliant (H1 + blockquote + absolute URLs). - Validate it with our [free checker](/checker) after every content change. - Regenerate when you add or remove important pages — a stale index is worse than none. --- # What Not to Include in llms.txt: Confidential Pages, Staging URLs and Leaks (2026) > Source: https://llmstxtgenerator.dev/blog/llms-txt-what-not-to-include/ Content security · September 2026 # What Not to Include in llms.txt Most llms.txt advice is about what to add. This one is about what to leave out — because `/llms.txt` is a public file at a predictable path, and every entry in it is a disclosure decision you make once and cannot quietly reverse. ## A Public File, Served Without Asking An llms.txt file sits at the site root or at any subpath and covers the URLs under that path. It is fetched with a plain GET request — no login, no session, no referrer. That is the point of the convention: an agent working for a user should be able to pull a concise map of the site in one request. The same property makes it a disclosure surface. Anything written there is readable by crawlers that respect nothing in particular, by SEO tools, by directories that index llms.txt files, and by anyone who fetches the path on a hunch. The spec is explicit that this is not a permission system: robots.txt declares what access is acceptable, while llms.txt is consulted on demand to help an agent understand content. It grants nothing and withholds nothing. So the test for a line in llms.txt is not "is this link useful to a model?" It is "am I comfortable with this URL being public, quotable and indexed for good?" ## Five Categories That Should Never Appear 1. **Preview, staging and branch URLs.** Deploy previews, `*.staging` hosts, branch aliases and temporary campaign domains. They are usually unlisted but unprotected, they drift, and they often carry work-in-progress claims that contradict your live site. 2. **Private application and admin routes.** Dashboards, client portals, internal search endpoints, ticket queues, `/admin` and `/internal` paths. Even without credentials in the URL, naming these paths in a curated file is a roadmap for automated probing. 3. **Credentials, tokens and signed URLs.** API keys, `?token=` query strings, unlisted share links with embedded secrets, webhook URLs, private bucket paths. Time-limited links are worse: the file outlives the signature. 4. **Customer, employee or partner data.** Gated case studies, named-account pricing, private reports, anything under NDA. This is where OWASP's risk register maps directly — sensitive information disclosure is a first-class entry in the LLM Top 10, and a public index is one of the simplest ways to publish something you meant to keep internal. 5. **Unreleased roadmap and embargoed pricing.** A page about a product not yet announced, next quarter's pricing, a migration guide for a plan you have not deprecated yet. Agents will read it, and assistants will summarise it to customers before your launch email goes out. If a page fits any of those five, it stays out of the file — however useful it would be as context. The broader curation problem is covered in [llms.txt mistakes](/blog/llms-txt-mistakes/). ## Why "It's Only a URL" Underestimates the Damage Three things happen to a URL once it appears in a public llms.txt file: - **It gets syndicated.** Community directories collect and re-publish llms.txt files, so your entry is copied off your domain within days. Removing it later does not remove the copies. - **It gets fetched by more than agents.** Link checkers, monitors and scrapers treat the file as a link inventory and walk every entry, including the ones you hoped nobody would notice. - **It becomes quotable context.** A note like "internal pricing model, subject to change" travels with the link. Assistants repeat what the file says. There is a second-order effect too: a file that names your internal structure — campaign naming, roadmap shape, which accounts get special pricing — is competitive intelligence even when each URL alone is harmless. ## A Three-Question Triage for Borderline Pages Most decisions are obvious at the extremes. For the ones in the middle — a partner case study, a comparison page, a docs section behind a login — run three questions in order: Question If the answer is no Can it be fetched with no login? Leave it out; llms.txt will not unlock it. Would a competitor reading it be fine? Leave it out; the file is public by design. Would you want it quoted in an AI answer? Leave it out; anything listed is context an assistant may repeat. One related trap: a page that is disallowed in robots.txt should not be listed either. Cross-checking the two files against each other is covered in [llms.txt vs robots.txt](/blog/llms-txt-vs-robots-txt/), and the short version is that a URL blocked for crawlers but listed for agents produces contradictory signals. ## Pre-Publish Checklist Run these two scans against the file before it ships. They catch the patterns that leak in through generators and copy-paste from sitemaps: ``` # Flag previews, staging hosts, admin paths and anything with a query string grep -inE "staging|preview|localhost|127\.0\.0\.1|\.internal|admin|portal|token=|api_key|secret" llms.txt # Print every distinct URL so you can eyeball domains and paths grep -oE "https?://[^) ]+" llms.txt | sort -u ``` Then confirm three things by hand: every link points at a host you control, no entry carries a query string, and nothing in the notes would be awkward in a press release. If a generator produced the file, this pass matters more, not less — auto-generated files inherit whatever the source inventory contained. ## If Something Already Leaked Do these in order: 1. **Rotate first, edit second.** Any credential, token or signed link that appeared in the file is disclosed. Reset it before you touch the markup. 2. **Remove the line and redeploy** so the live file stops advertising the path. Then purge the CDN cache — a stale copy at the edge keeps serving the old file. 3. **Assume the URL is known.** Watch logs for traffic against it in the following weeks. If it was only obscure rather than protected, either add real access control or retire it with a 410. 4. **Add the pattern to your build check** so the same class of path cannot come back on the next deploy. When you are done, re-check the file structurally — the [llms.txt checker](/checker) confirms every link resolves and every entry follows the spec. If you are rebuilding rather than editing, the [generator](/generator) drafts one from your sitemap; this article is then your review pass, because the generator knows nothing about which pages are meant to stay quiet. --- # How to Add llms.txt to WordPress (2026 Guide) > Source: https://llmstxtgenerator.dev/blog/llms-txt-wordpress/ WordPress tutorial · Updated 2026 # How to Add llms.txt to WordPress (2026 Guide) WordPress powers more than 40% of the web — but adding **llms.txt** to WordPress is different from a static site. This guide covers three proven ways to add llms.txt to WordPress in 2026: uploading a generated file, using a WordPress llms.txt plugin such as Yoast SEO, or adding a few lines of code. No matter which method you pick, you'll be done in under ten minutes. ## Why llms.txt Is Different on WordPress On a static site, adding llms.txt is trivial: drop a file into the build folder and redeploy. On WordPress it's trickier, for three reasons. First, WordPress sites are **dynamic** — posts, pages, categories, tags, archives and pagination are generated from the database, so a site can easily expose hundreds or thousands of URLs. Second, the **plugin ecosystem** is where WordPress SEO actually happens: the tools that already manage your `robots.txt` and sitemap are the natural home for llms.txt too. And third, WordPress adds infrastructure concerns that static sites don't have — caching plugins, multisite networks and subdirectory installs can all silently break an llms.txt file. The good news: because llms.txt is just a plain Markdown file served at `https://your-domain.com/llms.txt`, every method below works with any WordPress host — shared hosting, managed WordPress, or a VPS. ## Method 1 (Recommended): Upload a Generated llms.txt to Your Site Root The fastest, most reliable way to add llms.txt to WordPress requires no plugins and no code: generate the file, then upload it to the root of your WordPress installation. You keep full control over what goes into the file, and there is nothing to break later. 1. **Generate your file.** Use the free [llms.txt generator](/generator) — enter your URL, pick the pages you want AI engines to read, and download the resulting `llms.txt` file. 2. **Find your WordPress root.** The root is the folder that contains `wp-config.php` — usually `/public_html` (cPanel and most shared hosts), `/var/www/html`, or whichever folder your hosting file manager opens first. If WordPress is installed in a subdirectory (for example `example.com/blog/`), the root is _that_ subdirectory, not the document root above it. 3. **Upload the file there.** Three ways, pick one: - **Hosting file manager:** open `public_html` → _Upload_ → select `llms.txt`. - **cPanel File Manager:** navigate to `public_html` → _Upload_, then move the file if needed. - **FTP/SFTP:** connect with FileZilla or any FTP client, open `/public_html/`, and drag the file in. 4. **Verify.** Open `https://your-domain.com/llms.txt` in a browser — you should see your file rendered as plain text, not a 404 or the homepage. Two notes. Don't upload the file inside `wp-content/themes/…`: theme updates can delete it. And if your site serves through Cloudflare or another CDN, purge the cache after uploading so visitors and AI crawlers get the fresh copy. ## Method 2: Use a WordPress llms.txt Plugin If you prefer automation — especially on a busy site where content changes daily — a WordPress llms.txt plugin is the way to go. Plugin-generated files stay in sync with your content, respect your `noindex` rules, and need zero maintenance: - **Yoast SEO (free and premium).** Yoast shipped built-in llms.txt support in 2025 and it's fully part of the plugin in 2026. To enable it: go to **Yoast SEO → Settings → Site Features**, scroll to the **AI tools** section, switch the **LLMS.txt** toggle on, and save. Yoast automatically picks your most important and recently updated pages, respects `noindex` settings, and regenerates the file weekly — no code, no manual uploads. - **Website LLMs.txt (WordPress.org).** A dedicated llms.txt plugin that generates the file from your published content, pulls titles and meta descriptions from Yoast, Rank Math, SEOPress or AIOSEO, excludes anything marked `noindex`, and serves the file from your site root. Good choice if you don't use Yoast. - **Rank Math.** Also offers llms.txt support in its SEO suite, useful if Rank Math is already your SEO plugin of choice. Method Difficulty Best for Upload a generated file (Method 1) Easy — no plugins Full control, one-time setup, any host WordPress llms.txt plugin (Method 2) Easiest — automatic Frequently updated sites, non-technical owners Code in a child theme (Method 3) Intermediate — PHP Developers who want zero extra plugins ## Method 3: Add llms.txt to WordPress with Code If you'd rather not install another plugin, a few lines of PHP serve your llms.txt file directly from your theme. Two rules first: **never edit WordPress core files** (`wp-includes`, `wp-admin`, or the bundled default themes), and **always use a child theme** so a parent theme update can't wipe your changes. Step 1 — create the file `llms.txt` inside your child theme folder (for example `wp-content/themes/my-theme-child/llms.txt`) and paste in your generated content. Step 2 — add this snippet to your child theme's `functions.php`: `` Step 3 — flush the rewrite rules once: go to **Settings → Permalinks** and click _Save Changes_ (no other change needed). The snippet registers `/llms.txt` as a route, then serves your theme's file with the correct `text/plain` content type. ## What to Put in a WordPress llms.txt File Here is the temptation to resist: your WordPress site can produce hundreds of URLs — posts, pages, categories, tags, author archives, pagination, media attachments, WooCommerce products. **Do not list all of them.** An llms.txt file is a _curated index_: AI agents read it first and treat it as "these pages are worth reading." A giant auto-generated list dilutes your most important content and wastes the agent's context window. Include your homepage, about page, key landing pages, best articles, documentation and top products. Exclude tag and category archives, paginated listings, admin pages, and anything marked `noindex`. A healthy WordPress llms.txt is usually 10–30 links: ``` # Example Coffee Co. > A specialty coffee roaster in Portland, Oregon — beans, brewing guides, and our story. ## Important links - [Home](https://example.com/): Storefront, featured products, and seasonal offers - [Shop](https://example.com/shop/): All coffees and brewing equipment - [Brewing guides](https://example.com/guides/): Step-by-step recipes for every method - [About us](https://example.com/about/): Our story and sourcing philosophy ``` ## Common llms.txt WordPress Pitfalls - **Caching plugins serve stale files.** WP Rocket, W3 Total Cache, LiteSpeed Cache and Cloudflare can all cache `/llms.txt` and keep serving an old version after you update it — which is exactly how an AI engine ends up citing a deleted page. After every change, purge the page cache and the CDN cache, and consider excluding `/llms.txt` from caching entirely. - **WordPress in a subdirectory.** If your site lives at `example.com/blog/`, the file goes in the `blog/` folder, not `public_html` — otherwise agents fetch `example.com/llms.txt` and get a 404. - **Multisite networks.** Each site in a WordPress multisite should have its own llms.txt at its own root: a subdomain site's file goes in that subdomain's root, a subdirectory site's file in that subdirectory. - **Redirects and plugins hijacking the path.** Some hosts or security plugins redirect unknown paths to the homepage. After deploying, confirm `/llms.txt` returns a `200` with a `text/plain` content type — not a redirect to `/`. ## WordPress llms.txt FAQ ### Does Yoast SEO support llms.txt in 2026? Yes. llms.txt generation is built into both the free and premium versions of Yoast SEO. Enable it under Yoast SEO → Settings → Site Features → AI tools → LLMS.txt. Yoast picks your most important and recently updated pages, respects noindex rules, and regenerates the file weekly. ### Will a caching plugin break my llms.txt file? It can — a page cache may keep serving an old copy after you update the file. Purge your page cache and CDN cache after every change, and if you can, exclude /llms.txt from caching so crawlers always get the current version. ### Do I need llms.txt if I already have a sitemap? Keep both — they serve different purposes. An XML sitemap lists every URL for search-engine crawlers; llms.txt is a short, curated, Markdown index that AI agents read directly. A WordPress site should ship both: Yoast handles the sitemap, and any of the three methods above handles llms.txt. ## Keep reading [ ### 🛠️ How to Create llms.txt A step-by-step tutorial: build your file in 10 minutes and validate it. ](/blog/how-to-create-llms-txt/)[ ### 📚 Real-World Examples Copy-paste llms.txt files for SaaS, e-commerce, blogs and corporate sites. ](/blog/llms-txt-examples/) ## Ready to add llms.txt to WordPress? Generate your file in under a minute, then validate it against the spec — free, no sign-up, works with any of the three methods above. [⚡ Generate your llms.txt](/generator) [Check my file](/checker) `` --- # Markdown for Agents vs llms.txt: Content Negotiation or Curated Index? (2026) > Source: https://llmstxtgenerator.dev/blog/markdown-for-agents-vs-llms-txt/ Comparison · Content negotiation · Updated 2026 # Markdown for Agents vs llms.txt: Content Negotiation or Curated Index? Two 2026 conventions are fighting for the same phrase: "serve Markdown to AI." One is a file that describes your site. The other is a request header that changes how a single page arrives. They are not competitors, and treating them as such is why a lot of agent-readiness work misses half the surface. ## The Short Answer llms.txt Markdown for Agents Question it answers What exists on this site, and what is worth an agent's context window? Can I have this one page as clean Markdown, please? Unit of work The whole site, or every page under a path such as `/docs/llms.txt` A single URL per request Mechanism A static Markdown file you curate HTTP content negotiation on the `Accept` header Discovery The `/llms.txt` convention and the `rel="describedby"` link relation None — the client has to know to ask Where you implement it Your repo, CMS or static assets Your CDN, edge worker, or framework routes The one-line version: llms.txt handles **discovery**, Markdown for Agents handles **delivery**. An agent that never finds your page does not benefit from your perfect Markdown endpoint, and an agent that finds your page but receives 180,000 tokens of div soup pays for the privilege. ## llms.txt Is a Map, Not a Pipe The proposal by Jeremy Howard was updated to v2 in August 2026 and still describes a plain Markdown file: one H1, a blockquote summary, then H2 sections of linked resources with one-line descriptions. The v2 text formalises what practitioners had already started doing — the file can sit at the site root or at any path, covering the pages beneath it. That is why Cloudflare's developer documentation ships a root `developers.cloudflare.com/llms.txt` that is mostly a list of per-product files (Workers, R2, D1, AI Gateway and so on), each with its own full index: ``` > Each product below links to its own llms.txt, which contains a full > index of that product's documentation pages and is the recommended > way to explore a specific product's content. ## Developer platform - [Workers](https://developers.cloudflare.com/workers/llms.txt): Build, deploy, and scale serverless applications - [D1](https://developers.cloudflare.com/d1/llms.txt): Create managed, serverless databases with SQL semantics ``` What the file cannot do is hand over content. It points. Whether the agent then fetches HTML or Markdown is a separate decision — which is exactly the gap content negotiation fills. For the spec details, see the [llms.txt v2 breakdown](/blog/llms-txt-v2/). ## Markdown for Agents Is a Delivery Format Cloudflare shipped "Markdown for Agents" on 12 February 2026: for zones with the feature enabled, a request carrying `Accept: text/markdown` makes the network convert the origin's HTML to Markdown in real time and return it from the same URL. Vercel published a similar pattern on 3 February 2026, implemented as a rewrite in `next.config.ts` that routes Markdown-preferring requests to a route handler, plus Markdown sitemaps at URLs such as `vercel.com/docs/sitemap.md`. ``` curl https://developers.cloudflare.com/workers/ \ -H "Accept: text/markdown" HTTP/2 200 content-type: text/markdown; charset=utf-8 vary: accept x-markdown-tokens: 725 content-signal: ai-train=yes, search=yes, ai-input=yes ``` Three headers on that response are worth more than the Markdown itself. `Vary: accept` keeps caches from serving the Markdown version to browsers. The `content-type` tells the client it got what it asked for, so it skips its own conversion step. And `x-markdown-tokens` is a hint about context cost that an agent can use for chunking decisions. ## Which Agents Actually Ask for Markdown Content negotiation only pays off when the client sends the header. Checkly tested seven widely used coding agents in February 2026 by pointing their built-in web-fetch tools at a header echo endpoint. The result was a near-even split: Agent Sends `Accept: text/markdown`? Claude Code (Fetch, v2.1.38) Yes — `text/markdown, text/html, */*` Cursor (WebFetch, 2.4.28) Yes — `text/markdown` with q-values OpenCode (webfetch, 1.2.5) Yes — `text/markdown;q=1.0` OpenAI Codex (Feb 2026 build) No — browser-style HTML header Gemini CLI (0.28.2) No — `*/*` GitHub Copilot (fetch\_webpage) No Windsurf (read\_url\_content) No Treat the table as a snapshot, not a law: tool versions move monthly, and crawlers such as GPTBot or OAI-SearchBot are a different population from IDE agents. The underlying mechanics are also unchanged — a server that ignores the header loses nothing, because the client will fall back to HTML and convert it itself. ## Where the Two Meet: Markdown Twins The interesting design work happens when the map points at the pipe. llms.txt v2 recommends publishing a Markdown twin for each listed page — either by appending the extension (`page.html.md`) or replacing it (`page.md`) — and it defines two link relations to make the pairing machine-readable: `rel="alternate" type="text/markdown"` for the Markdown version of a page, and `rel="describedby"` for the llms.txt file that covers it. Checkly's live file, fetched in September 2026, is a clean example of both layers in one place: it documents `Accept: text/markdown` support for every page URL, lists a set of hand-authored twins (`/pricing.md`, `/cli.md`), and ships scoped indexes at `/docs/llms.txt` and `/product/llms.txt` — so an agent can arrive through the index and then read each page in its cheapest form. Vercel's post describes the fallback for agents that never send the header: a `link rel="alternate" type="text/markdown"` tag in the HTML head. ## What Neither One Buys You - **Rankings.** Google has consistently said llms.txt is not a Search signal, and changing the representation served to a requesting agent is not a ranking tactic either. Both belong to agent readiness, not to ranking. - **Citation share.** Getting cited in an AI answer still depends on the content, not the container. Markdown reduces cost and latency; it does not make a thin page authoritative. - **Coverage.** Markdown for Agents helps the agents that already browse your site. It does nothing for the ones that never discovered it — that is the index's job, and most published files still see very few fetches. ## How to Ship Both in an Afternoon 1. **Publish the index first.** Draft `/llms.txt` from your sitemap with the [generator](/generator), keep it to the 10–50 pages that answer real questions, and validate it with the [checker](/checker) before you touch the CDN. 2. **Add the delivery layer.** On Cloudflare, enable Markdown for Agents in the zone's Quick Actions. Off Cloudflare, read the `Accept` header and return Markdown with `Content-Type: text/markdown`, or pre-generate twins at build time if your content is already authoring in Markdown. 3. **Connect them.** Point the links inside llms.txt at the Markdown twin where one exists, and add `rel="alternate" type="text/markdown"` to the pages you care about most. 4. **Verify both paths.** One request per page: a plain fetch that must still return HTML, and an `Accept: text/markdown` request that must return Markdown plus `Vary: accept`. If the second returns HTML, the conversion never triggered — that is the failure to watch for. Related reading: the same "two layers" split in the action domain, covered in [llms.txt vs MCP](/blog/llms-txt-vs-mcp/), and the measured reality of who fetches these files in [which AI engines read llms.txt](/blog/which-ai-engines-read-llms-txt/). --- # How to Measure Whether AI Engines Read Your llms.txt (Logs, GA4 and Cloudflare, 2026) > Source: https://llmstxtgenerator.dev/blog/measure-llms-txt-traffic/ Measurement & analytics · September 2026 # How to Measure Whether AI Engines Read Your llms.txt Publishing `/llms.txt` is the easy half. Proving anyone read it is the half that decides whether you keep maintaining the file. Here is a three-layer measurement stack — raw logs, crawler behaviour, and AI referral traffic — that separates what actually happened from what your analytics dashboard implies happened. ## Why llms.txt Measurement Is Different Most web measurement assumes a browser: JavaScript loads, a session starts, a referrer gets recorded. AI crawlers do none of that. When `OAI-SearchBot` fetches your `/llms.txt`, no GA4 session exists, no cookie is set, and no channel group fires. The only evidence is a line in your access log. That single fact reorganises the whole reporting problem into three independent questions: - **Was the file read?** Direct fetches of `/llms.txt` — the only signal that proves the file works. - **Is your content being read?** AI crawler hits on your pages, which happen with or without an llms.txt file. - **Is it producing anything?** Humans arriving from AI assistants in GA4. These signals move on different timelines and disagree for months — heavy crawler traffic with zero AI referrals is normal. Report them separately and you will never confuse a crawl with a citation. ## Layer 1 — Count Direct llms.txt Fetches On your own server, the file is one path. Filter for it and group by user agent: ``` # Did any AI agent request the file? grep -i "llms.txt" /var/log/nginx/access.log # Group the hits by crawler identity grep -i "llms.txt" access.log | awk '{print $1, $12}' | sort | uniq -c | sort -rn # Include rotated logs from the last month zgrep -i "llms.txt" /var/log/nginx/access.log.*.gz | grep -iE "GPTBot|OAI-SearchBot|ClaudeBot|PerplexityBot" ``` What you are looking for is a small number of non-zero hits, not volume. The baseline across the wider web is grim: our breakdown in [Which AI Engines Actually Read llms.txt?](/blog/which-ai-engines-read-llms-txt/) covers the crawler audits showing that most published files are never requested even once. Two hits a month is a working file; zero after eight weeks means the file is [being skipped for a reason](/blog/llms-txt-mistakes/) — usually placement, an unparsable format, or links the agent cannot resolve. User-agent strings are trivially spoofed, so confirm identity before reporting: resolve the requesting IP to a hostname, resolve that hostname back to an IP, and compare against the operator's published ranges. If they do not match, it was not that bot. ## Layer 2 — Track AI Crawler Hits on Content Pages llms.txt is a map. Whether agents follow it shows up as requests to the pages it lists. Pull the top paths per crawler token: ``` # Top pages fetched by AI agents this month grep -iE "GPTBot|OAI-SearchBot|ClaudeBot|Claude-SearchBot|PerplexityBot" access.log \ | awk '{print $12}' | sort | uniq -c | sort -rn | head -20 ``` This is the most useful diagnostic in the stack, because it tests curation directly. If the paths agents fetch match the pages you listed in `/llms.txt`, the file is working as a prioritisation layer. If they keep fetching pages you deliberately left out — category archives, paginated tag pages, thin landing pages — the file is not steering anything yet, and the fix is structural, not a matter of publishing it again. ## Layer 3 — Read AI Referral Traffic in GA4 On May 13, 2026, Google added an **AI Assistant** entry to GA4's Default Channel Group. Sessions whose referrer matches a recognised AI assistant are tagged automatically: Field Value assigned Default channel group `AI Assistant` Session medium `ai-assistant` Campaign `(ai-assistant)` Find it in _Reports → Acquisition → Traffic acquisition_ with Session default channel group as the primary dimension. Two caveats matter before you put the number in a report. First, Google's live Default Channel Group documentation lists ChatGPT, Gemini, Deepseek, Copilot and Grok, and states the channel excludes AI Overviews and AI Mode — so if you want Claude, Perplexity or AI Mode measurement, you still need a custom channel group with a regex on session source above the generic Referral channel. Second, this channel counts humans who clicked through. A busy AI crawler fleet produces exactly zero of these sessions. ## Layer 4 — Use Platform-Side Tooling Where You Have It If your site sits behind Cloudflare, AI Crawl Control exposes a crawler-level view you cannot get from GA4: which AI services are accessing the domain, request patterns over time, whether a crawler respects your robots.txt, and per-crawler enforcement rules. Its Metrics tab also reports a Content Format breakdown — what content types AI systems request versus what your origin serves — and the changelog documents an option that redirects verified AI training crawlers to a page's canonical URL while humans and search crawlers see the original. Cloudflare's older "Block AI bots" toggle is documented as deprecating on September 15, 2026, so plan a migration if your enforcement depends on it. And these dashboards report what reached the edge — not what an assistant chose to cite. Useful governance, not a truth machine. ## A 30-Day Benchmark Table Put the three signals in one table and review monthly. This is the whole reporting system: Signal Where to read it What good looks like llms.txt fetches Access logs, filtered on `llms.txt` Any non-zero count from a verified AI agent Crawler hits on listed pages Access logs, grouped by user agent and path Top paths overlap the sections in your file AI referral sessions GA4 AI Assistant channel, plus a custom channel group Any stable, non-seasonal trickle worth segmenting Before you spend a quarter tracking it, spend five minutes validating the file itself. Run it through the [llms.txt checker](/checker) to confirm each link resolves, every entry carries a description, and the header block follows the spec. If you are rebuilding from scratch, the [generator](/generator) turns a sitemap into a first draft. A file that fails validation will never produce a log line worth reporting — and no dashboard will tell you why. --- # robots.txt for AI Crawlers in 2026: Allow, Block or Feed GPTBot, ClaudeBot and PerplexityBot > Source: https://llmstxtgenerator.dev/blog/robots-txt-ai-crawlers/ AI crawlers & robots.txt · September 2026 # robots.txt for AI Crawlers: Who to Allow and Who to Block robots.txt was designed to referee search engines. In 2026 it is refereeing a dozen AI agents from six different companies — and getting the policy right depends on one thing most guides skip: which class of crawler you are talking to. Here is the token list, two pastable policies, and how your llms.txt file picks up where robots.txt stops. ## The Three Classes of AI Crawler Every AI user agent worth configuring falls into one of three job descriptions. The class decides what blocking actually costs you: Class What it does Cost of blocking **Training** Collects pages for model pre-training corpora None for today's citations — it affects future model weights **Search index** Indexes pages so the assistant can retrieve and cite them Your pages stop being discoverable in AI answers **User-triggered** Fetches one page live when a user pastes a URL or asks about it Often unavoidable — these behave like a browser on a user's behalf That split is the whole game. Publishing teams that block "AI" wholesale by blocking one or two famous tokens usually block the harmless training crawler while leaving the retrieval agents untouched — or the reverse, and quietly remove themselves from AI answers. ## The 2026 User-Agent Token Reference These are the tokens site owners ask about most. "Honours robots.txt" reflects each operator's published documentation, not a guarantee — verify against your own logs. Token Operator Class `GPTBot` OpenAI Training `OAI-SearchBot` OpenAI Search index `ChatGPT-User` OpenAI User-triggered `ClaudeBot` Anthropic Training `Claude-SearchBot` Anthropic Search index `Claude-User` Anthropic User-triggered `PerplexityBot` Perplexity Search index `Perplexity-User` Perplexity User-triggered `Google-Extended` Google Training opt-out token `CCBot` Common Crawl Training corpus `meta-externalagent` Meta Training Two traps hide in that table. First, `Google-Extended` is not a crawler at all — it is a robot token that opts you out of Gemini and Vertex training while leaving Googlebot, Search and AI Overviews completely unaffected. You cannot opt out of AI Overviews without blocking Google Search itself. Second, Anthropic formalised the split in February 2026, documenting ClaudeBot, Claude-SearchBot and Claude-User separately and warning that blocking Claude-SearchBot reduces your site's visibility in its search results. Meanwhile OpenAI's user-triggered `ChatGPT-User` is documented as fetching on a user's behalf, which makes it effectively unmanageable from robots.txt. ## Two robots.txt Policies You Can Paste Today ### Policy A — Get cited, do not get trained The split most publishers want: retrieval allowed, training denied. ``` # Retrieval agents: allow User-agent: OAI-SearchBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: PerplexityBot Allow: / # Training crawlers: deny User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: CCBot Disallow: / User-agent: meta-externalagent Disallow: / # Google/Gemini training opt-out token User-agent: Google-Extended Disallow: / ``` ### Policy B — Maximum visibility If your goal is AI citations above all — documentation, tools, service pages — allow the agents that can recommend you, and keep them out of private paths only: ``` User-agent: GPTBot User-agent: OAI-SearchBot User-agent: ClaudeBot User-agent: Claude-SearchBot User-agent: PerplexityBot Allow: / Disallow: /admin/ Disallow: /checkout/ ``` Rules for a specific token always override the wildcard group for that bot, so add a bot's group deliberately rather than relying on `User-agent: *`. And always test the published file, not the draft — `curl https://yourdomain.com/robots.txt` is the only version agents see. ## Cloudflare Content Signals: Usage Terms Inside robots.txt Since Cloudflare introduced its Content Signals Policy, robots.txt has grown a second layer that describes _how_ content may be used, not just whether it may be fetched. Three signals are defined — `search` (building a search index), `ai-input` (feeding content into models for live answers) and `ai-train` (training or fine-tuning) — and sites can declare preferences with comma-delimited yes/no values: ``` # Content Signals Policy User-agent: * Content-Signal: search=yes, ai-train=no Allow: / ``` Cloudflare's documentation states that domains without a robots.txt of their own may be served a default signal set of search permitted and AI training declined, and that content signals are a stated reservation of rights rather than a technical enforcement layer — enforcement still happens at the edge, through bot management rules. Treat signals as documentation of intent that strengthens your position; treat the allow/disallow rules above as the part that actually changes behaviour. ## Where llms.txt Fits: Permission vs Curation robots.txt answers "may you read this?" llms.txt answers "what is worth reading?" Once you have allowed the retrieval agents, an `/llms.txt` file gives them a curated map: your best pages, one-line descriptions, and Markdown links that skip the HTML scaffolding. For large sites, llms.txt v2 formalises subpath files such as `/docs/llms.txt` and an `llms-full.txt` dump for full-context reads — Cloudflare's own developer docs run exactly this hub-and-spoke pattern. The practical sequence is: audit your robots.txt groups, allow the retrieval tokens you want, then publish the map. Generate a first draft from your sitemap.xml with the [llms.txt generator](/generator), validate it with the [checker](/checker), and compare the two files properly in [llms.txt vs robots.txt](/blog/llms-txt-vs-robots-txt/). Remember that permission plus a map still leaves you at square one if agents never fetch: our crawl-data breakdown in [Which AI Engines Actually Read llms.txt?](/blog/which-ai-engines-read-llms-txt/) shows how uneven that coverage still is in 2026. ## Verify the Bots With Your Own Logs Every claim above is testable on your own server. The monthly log check: ``` # Which AI agents actually reached you? grep -iE "GPTBot|OAI-SearchBot|ClaudeBot|Claude-SearchBot|PerplexityBot" access.log | awk {print $1, $12} | sort | uniq -c | sort -rn | head -20 ``` Then confirm identity — user-agent strings are trivially spoofed. The standard check is a double reverse-DNS lookup: resolve the requesting IP to a hostname, resolve the hostname back to an IP, and compare against the operator's published ranges. If the values do not match, it was not that bot. Run this audit once a month, and you will know within a single cycle which of your robots.txt rules are reshaping real traffic — and whether the llms.txt file you published is being read at all. --- # Which AI Engines Actually Read llms.txt? (2026 Crawler Data) > Source: https://llmstxtgenerator.dev/blog/which-ai-engines-read-llms-txt/ AI visibility · Crawler research · August 2026 # Which AI Engines Actually Read llms.txt? llms.txt is pitched as the file that helps ChatGPT, Claude, Perplexity and Gemini understand your site. Two years of independent crawler audits tell a more honest story — here is the evidence, straight from server logs, and what it means for your site. ## The Short Answer: Almost No One Fetches It If you publish an llms.txt today, the realistic outcome is that nothing automated will ever request it. In May 2026, Ahrefs analyzed 137,000 domains that publish the file: **97% of those files received zero requests** during the entire month — no crawler, no agent, no human. The 3% that were fetched were hit almost entirely (96%) by bots, and only about a fifth of those fetches came from named AI tools. This is not one outlier study. Independent audits keep arriving at the same number: zero. ## What 500 Million Bot Requests Reveal Two large log-based studies are worth knowing about because they measure what bots _actually do_, not what press releases say: - **Weekerp (2025):** across two production sites, 68,759 AI bot requests in a month — and `/llms.txt` received **0 requests**. In the same period, `robots.txt` was fetched 1,938 times and sitemaps 1,224 times, proving AI bots do fetch supporting files when they care about them. - **Limy (2026):** analysis of 515 million LLM bot traffic events found that GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot and Google-Extended "overwhelmingly skip the file and crawl HTML directly." Crawler Owner Fetches llms.txt? Evidence `GPTBot` OpenAI Rarely — the most common named fetcher Ahrefs 2026: top named AI fetcher `ClaudeBot` / `Claude-Code` Anthropic Training crawler: not observed; Claude-Code: second most common Ahrefs 2026; Weekerp: 0 requests `PerplexityBot` Perplexity Not observed Weekerp, Limy audits `OAI-SearchBot` OpenAI Not observed Weekerp, Limy audits `Google-Extended` / Googlebot Google No — officially unsupported Illyes, July 2025 ## Google's Position Is On the Record Google is the only major player that has answered the question publicly and unambiguously. In July 2025, Search Relations' Gary Illyes said Google has **"no plans to support llms.txt"**, and John Mueller compared the file to the keywords meta tag — a self-declared signal that was abandoned because it was trivially gameable. Google's 2026 guidance for generative AI visibility lists llms.txt by name as a tactic that does not help. Yes, some Google properties (like `ai.google.dev`) still serve the file — Mueller confirmed that is an internal content system artifact, not an endorsement. We covered the full nuance in [Does llms.txt Affect SEO?](/blog/does-llms-txt-affect-seo/) ## Adoption Is Growing Anyway Despite the crawl data, adoption keeps climbing. Casey Burridge's crawl of the top 10,000 sites found **5.61% with a valid llms.txt in June 2026**, up from 1.04% in July 2025 — roughly 5.4x growth in a year, or about 39,000 sites across the top million. SE Ranking measured 10.13% across 300,000 domains, and Rankability found 8.7% of the top 1,000 (15.8% of the 549 it could actually reach). Documentation platforms like Mintlify generate the file automatically, which is why the [v2 spec](/blog/llms-txt-v2/) and so many dev-tool sites adopted it early. So who does read your file, realistically? Three groups: - **Occasional fetchers.** GPTBot and Claude-Code are the only named AI tools with meaningful request records (Ahrefs). - **Humans and agents on demand.** Users and AI assistants that are asked to summarize your site will often read llms.txt directly — this is the file's most reliable audience. - **Future adopters.** The ecosystem is young; the v2 spec, subpath files and markdown link relations were added specifically to make adoption easier. ## Why Bots Skip Your llms.txt - **Training runs from datasets, not live fetches.** Most model training uses pre-built corpora like Common Crawl, so there is no crawl step where llms.txt would matter. - **It costs crawl budget.** Probing /llms.txt on every domain adds a request per host for little confirmed benefit, so most crawlers don't bother. - **It is still unofficial.** No LLM lab has committed to honoring the file, which makes it a speculative investment for crawler teams. - **robots.txt already works.** Access control is handled by a 30-year-old standard every crawler respects — llms.txt controls nothing and blocks nothing. ## What This Means for Your Site The honest playbook in 2026: publish a spec-compliant llms.txt because it is cheap and because the file's on-demand audience is real — then spend your remaining SEO time on things with proven returns, like crawlable HTML content, a clean robots.txt, and site speed. There is no evidence that llms.txt moves rankings, traffic or citations today. - Generate a valid file with the [llms.txt generator](/generator) and validate it with the [checker](/checker). - Keep it small (5–15 links) and point links at markdown versions per the v2 spec. - Check your own logs once a month to see if anything requests it: ``` grep -c "llms.txt" /var/log/nginx/access.log grep -i "GPTBot" /var/log/nginx/access.log | head -20 ``` If the second command shows hits, your file is being discovered — but don't expect it. Ship the file, keep it valid, and let the ecosystem catch up. You're ready for the agentic web either way. ---