llms.txt: what it is, what it isn’t, and when to bother
Led growth at cnvrg.io before its acquisition by Intel.
llms.txt is a simple idea with a lot of hype around it. Here is what llms.txt actually is, what it is not, and the honest answer on when to bother.
llms.txt is a proposed curated Markdown index for readers and tools, first published by Jeremy Howard on September 3, 2024 and revised as v2 on August 10, 2026. Some documentation sites publish one. That does not establish a search-ranking or citation benefit; maintaining it has a cost.
What llms.txt actually is
llms.txt is a plain markdown file served at the root of your domain (yoursite.com/llms.txt) that gives LLMs a curated index of your most important pages, in a format optimized for context window consumption. It was proposed by Jeremy Howard in September 2024.
The format
A typical llms.txt looks like this:
# Site Name
> One-paragraph site description optimized for LLM context.
## Key Pages
- [Page name](url) - description
## Documentation
- [Doc name](url) - descriptionWhat llms.txt is NOT
- A documentation index can be published or used by a specific tool without establishing a universal search-bot or ranking role.
- Not a replacement for robots.txt, sitemap.xml, or proper schema
- Not a guaranteed ranking lever in any AI search engine
- Not a substitute for content quality, entity clarity, or extractable structure
When it's worth shipping
Consider llms.txt when a maintained, curated index would help readers or tools navigate a large or complex site. Prioritize crawlability and useful content first. The file is optional; its maintenance has a cost, and correlation with a readiness score does not demonstrate a citation benefit.
How to write a good file
- Start with a one-paragraph site summary that defines who you are, what you do, and who you serve
- List your most important pages first (about, key services, flagship content)
- Keep descriptions short (one sentence per link)
- Link to /llms-full.txt for a deeper, longer-form version
- Update whenever you publish a major new page
Frequently asked questions
Does Google honor llms.txt?
Google says no special AI text file or schema is required for its AI Search features. That establishes that llms.txt is not an eligibility requirement; it does not establish universal non-use by every engine or a ranking or citation benefit.
Does ChatGPT honor llms.txt?
OpenAI's developer documentation publishes an llms index. That is different from documentation that its search or training bots use your site's file as a control or ranking signal. Verify the intended consumer and purpose rather than infer citation support.
Should I worry about my llms.txt being scraped or misused?
Only include links and descriptions approved for public release. Anyone may read or copy a public file, and it is not access control. Do not place private content, credentials or sensitive metadata in it.
Where can I see a real llms.txt example?
yanivgoldenberg.com/llms.txt is a working example. Reference deployments include Anthropic's documentation site and many open-source library docs.
What is llms.txt in plain terms?
llms.txt is a plain markdown file served at the root of your domain that gives LLMs a curated index of your most important pages. Jeremy Howard proposed it in September 2024. A typical file opens with a site name and a one-paragraph site description, then lists grouped links, each followed by a short description of the page.
Is llms.txt worth shipping for my site?
It is worth it when a maintained, curated index would help readers or tools navigate a large or complex site, such as a documentation-heavy one. The file is optional and its maintenance has a cost. Ship it once the fundamentals are in place, because if your site is not crawlable, your content is thin, or your entity is unclear, a perfect file changes nothing.
What should I put in my llms.txt file?
Start with a one-paragraph site summary that defines who you are, what you do, and who you serve. Then list your most important pages first, such as about, key services, and flagship content, with one sentence per link. Point only to pages you would want quoted, because a bloated file is worse than none. You can also link to a longer-form version.
How often should I update my llms.txt?
Update it whenever you publish a major new page or when key pages change, and treat it as a maintained index. A stale map sends readers and tools to the wrong place, so links and summaries should resolve and match the current page content.
Published: 2026-04-27 · By Yaniv Goldenberg · Get audited
How to write a good file
A good llms.txt file reads like a thoughtful table of contents for a machine. Start with a short paragraph naming the site and what it is for, then group your best links under clear headings, each with a one-line description of what the page covers and who it helps. Point only to pages you would genuinely want quoted: your strongest docs, canonical explainers, and reference material, not every URL you have ever published. A bloated file is worse than none, because it buries the pages that matter.
Keep the index honest and current. Links to pricing pages or API references should resolve and match the page content, so readers and tools reach the right source. Update it when key pages change, and treat it as a maintained index. Crawlability and useful, verifiable content come first. llms.txt is an optional navigation aid, not evidence of a ranking or citation benefit.
Common llms.txt mistakes to avoid
Treat llms.txt as an optional navigation aid, not a ranking or citation guarantee. Keep it short enough to highlight the sources you most want an agent to understand, without hiding other useful URLs. Update links and summaries when pages change; a stale map sends readers and tools to the wrong place.
Another mistake is shipping it before the fundamentals. If your site is not crawlable, your content is thin, or your entity is unclear, a perfect file changes nothing, because the file is a convenience layer on top of quality, not a substitute for it. A further mistake is forgetting the human test: read your llms.txt as if you were a busy analyst trying to understand the site in thirty seconds.
Make the descriptions understandable and keep links current for the intended reader or tool. Those checks establish a maintained navigation aid, not an observed citation advantage.
Consider the file when a maintained index solves a real navigation need. Estimate the writing and maintenance effort for your site and verify the intended consumer. A useful index does not establish a ranking or citation edge.
"An llms.txt file is a map, not a magic spell. I would not trade $1 of real crawlability work for a perfect llms.txt on a site the bots still cannot reach."
At its core, llms.txt is a curated Markdown file at your root that points AI systems to your most useful, cleanest pages, so a model does not have to guess which of your thousand URLs actually matter. It is a convenience layer, not a directive: no major engine is contractually bound to honor it, and it does not override crawlability, quality, or your existing SEO. Treating the file as a ranking hack is the fastest way to be disappointed.
Use llms.txt where a curated, maintained index helps the intended reader or tool find public content. Group useful links and keep descriptions accurate. Evaluate its navigation purpose separately from search visibility and citation outcomes.
Further reading: Text file and Web crawler. Related: GEO (AI search), AI visibility audit, More on AI search.
The Growth Blocker
Get the next playbook in your inbox
Work with Yaniv
Fractional Head of Growth for B2B SaaS, B2C, and e-commerce. Revenue that compounds, not spikes. Any channel, any motion, any stage.
About the author
Yaniv GoldenbergFractional CMO & Head of GrowthFractional CMO & Head of Growth — I lead strategy and personally execute across the revenue funnel. He scales post-product-market-fit companies to the revenue milestone that unlocks their next funding round. He scaled Elementor from $200K to $20M in ARR (a 100x increase), grew Riverside.fm’s Net MRR by 337%, and led demand generation at cnvrg.io, which Intel acquired in 2020. With 10+ years operating across every channel, motion, and stage, he hands the growth engine back once a team can run it without him.
- Elementor $200K→$20M ARR
- Riverside +337% Net MRR
- cnvrg.io → Intel
- 10+ yrs operating