llms.txt is a simple idea with a lot of hype around it. Here is what llms.txt actually is, what it is not, and the honest answer on when to bother.
llms.txt is an emerging convention proposed in late 2024 for serving LLMs a curated index of your most important content at /llms.txt. It is not officially honored by OpenAI, Anthropic, or Google as of 2026. It costs nothing to ship, signals AI-readiness, and pairs with strong GEO scores in benchmarks.
llms.txt is an emerging convention proposed in late 2024 for serving LLMs a curated index of your most important content at /llms.txt. It is not officially honored by OpenAI, Anthropic, or Google as of 2026. It costs nothing to ship, signals AI-readiness, and pairs with strong GEO scores in benchmarks.
What llms.txt actually is
llms.txt is a plain markdown file served at the root of your domain (yoursite.com/llms.txt) that gives LLMs a curated index of your most important pages, in a format optimized for context window consumption. It was proposed by Jeremy Howard in late 2024.
The format
A typical llms.txt looks like this:
# Site Name
> One-paragraph site description optimized for LLM context.
## Key Pages
- [Page name](url) - description
## Documentation
- [Doc name](url) - descriptionWhat llms.txt is NOT
- Not officially honored by OpenAI, Anthropic, Google, or Perplexity as of April 2026
- Not a replacement for robots.txt, sitemap.xml, or proper schema
- Not a guaranteed ranking lever in any AI search engine
- Not a substitute for content quality, entity clarity, or extractable structure
When it's worth shipping
Always. The cost is near zero (a single markdown file plus optional /llms-full.txt) and the upside is that any LLM that does eventually honor it gets a clean, optimized read of your site. In benchmarks, sites that publish llms.txt consistently pair with stronger overall GEO scores, even if the file itself is not the causal lever.
How to write a good file
- Start with a one-paragraph site summary that defines who you are, what you do, and who you serve
- List your most important pages first (about, key services, flagship content)
- Keep descriptions short (one sentence per link)
- Link to /llms-full.txt for a deeper, longer-form version
- Update whenever you publish a major new page
Frequently asked questions
Does Google honor llms.txt?
No. Google has explicitly stated that no special file or schema is required for AI Overviews. llms.txt has no documented effect on Google AI features as of 2026.
Does ChatGPT honor llms.txt?
Not officially. OpenAI has not published documentation indicating that GPTBot, OAI-SearchBot, or ChatGPT-User read or honor llms.txt.
Should I worry about my llms.txt being scraped or misused?
No. Anything in llms.txt is content you publish publicly anyway. The file is meant to make your public content easier to consume, not to expose private content.
Where can I see a real llms.txt example?
yanivgoldenberg.com/llms.txt is a working example. Reference deployments include Anthropic's documentation site and many open-source library docs.
How to write a good file
A good llms.txt file reads like a thoughtful table of contents for a machine. Start with a short paragraph naming the site and what it is for, then group your best links under clear headings, each with a one-line description of what the page covers and who it helps. Point only to pages you would genuinely want quoted: your strongest docs, canonical explainers, and reference material, not every URL you have ever published. A bloated file is worse than none, because it buries the pages that matter.
Keep it honest and current. If your file promises a pricing page or an API reference, that link had better resolve and match what the page actually says, or you train models to distrust your map. Update it when your key pages change, and treat it as a living index rather than a set-and-forget file. Above all, remember the order of operations: crawlability and content quality first, llms.txt second. The file is a small, cheap lever that helps well-built sites, not a shortcut that rescues a site the engines cannot read or do not trust. Ship it when you have earned the fundamentals, and it becomes a genuine convenience for the models you want quoting you.
Common llms.txt mistakes to avoid
The most common mistake is treating it as a ranking hack. It is not: no major engine promises to honor the file, and none will rank you higher because you shipped one. The second mistake is bloat, dumping every URL you own into the file so the genuinely important pages drown in noise. A good one is short and opinionated, pointing only to the pages you would most want an AI to quote. The third mistake is letting it rot: a file that that links to pages you have since changed or removed teaches models to distrust your map entirely.
The fourth mistake is shipping it before the fundamentals. If your site is not crawlable, your content is thin, or your entity is unclear, a perfect file changes nothing, because the file is a convenience layer on top of quality, not a substitute for it. The fifth is forgetting the human test: read your llms.txt as if you were a busy analyst trying to understand the site in thirty seconds. If the descriptions are vague or the structure is confusing to you, it will be no clearer to a model. Fix those five and llms.txt becomes a small, honest lever that helps a well-built site get quoted, exactly what it was designed to be and nothing more.
The honest summary: llms.txt is worth the twenty minutes it takes to write when you have a large or documentation-heavy site and you want to hand models a clean index of what matters. It will not rescue a weak site, and a good llms.txt on a strong site is a small edge, not a silver bullet. Ship it once the fundamentals are in place, keep it current, and treat the file as one honest signal among many rather than the thing that wins AI visibility on its own.
"An llms.txt file is a map, not a magic spell. I would not trade $1 of real crawlability work for a perfect llms.txt on a site the bots still cannot reach."
At its core, llms.txt is a curated Markdown file at your root that points AI systems to your most useful, cleanest pages, so a model does not have to guess which of your thousand URLs actually matter. It is a convenience layer, not a directive: no major engine is contractually bound to honor it, and it does not override crawlability, quality, or your existing SEO. Treating the file as a ranking hack is the fastest way to be disappointed.
So when is llms.txt worth shipping? When you have a large or complex site, especially docs, and you want to hand models a clean index of the pages you would most want quoted. Write it like a good table of contents: a short description of the site, then grouped links with one line each on what the page covers. Keep it current, keep it honest, and treat it as one small lever on top of the fundamentals that actually move AI visibility, not a substitute for them.
Further reading: Text file and Web crawler. Related: GEO (AI search), AI visibility audit, More on AI search.
