llms.txt explained: a robots.txt for AI answer engines
llms.txt is an emerging convention that gives LLMs a clean, curated map of your most important pages. Here is what it is, how to write one, and whether it works.
As models increasingly read the web to answer questions, a gap appeared: HTML is noisy. Navigation, cookie banners, scripts and ads bury the substance. The llms.txt convention, proposed at llmstxt.org, addresses this by offering a clean, Markdown map of your most valuable pages — the way you would brief a researcher before they read your site.
What goes in the file
An llms.txt file lives at your root and follows a light structure: an H1 with your name, a short blockquote summary, then grouped links with one-line descriptions. Order matters — put what you most want models to summarize first.
# Acme
> Acme builds observability tools for data teams.
## Product
- [Overview](https://acme.com/): what Acme does, in one page.
- [Pricing](https://acme.com/pricing): plans and limits.
## Editorial
- [Guide to X](https://acme.com/blog/x): the definitive primer on X.llms.txt vs llms-full.txt
The short llms.txt is a table of contents. Some sites also publish llms-full.txt: the entire editorial corpus as a single Markdown document, optimized for retrieval-augmented systems that would rather ingest one clean file than crawl hundreds of pages. Publish both when your content is worth ingesting wholesale.
Does it actually work?
Honestly, adoption is early and no major engine has committed to honoring llms.txt as a ranking or retrieval guarantee. But the file is cheap to generate from data you already have, it makes your priorities explicit, and it signals that you take machine readability seriously. The asymmetry is favorable: negligible cost, plausible upside, zero risk.
How to generate it
- Drive it from structured content so it never drifts from your real pages.
- Keep descriptions to a single, factual sentence — this is what models quote.
- Group by how a buyer navigates: product, proof, editorial, legal.
- Serve it as text/plain and link machine-readable feeds like your sitemap and RSS.
Frequently asked questions
- Is llms.txt a standard?
- It is a community convention, not an official standard ratified by a search engine. It is gaining traction but remains advisory.
- Does it replace robots.txt?
- No. robots.txt controls what may be crawled; llms.txt suggests what is important. They are complementary.
- Where do I host it?
- At your site root, served at /llms.txt as UTF-8 text. Generate it dynamically so it stays in sync with your content.