llms.txt: What It Is and How to Create One

What Is llms.txt?

llms.txt is a Markdown file that gives AI agents a short, curated map of a site's most useful content, so they can find what they need without parsing full HTML pages. It was proposed by Jeremy Howard (co-founder of Answer.AI and fast.ai) on 3 September 2024, and substantially revised as v2 on 10 August 2026.

The problem it solves is not a marketing problem. It is a token-economics problem:

"But web pages are built for people. An HTML page wraps its information in navigation, ads, and JavaScript, and converting it back into clean text is difficult and imprecise. Context windows, while larger than they were, are still too small for most websites in their entirety, and every wasted token costs time and money."

— Jeremy Howard, The /llms.txt file, v2, published 3 September 2024, revised 10 August 2026

That origin matters, because it explains what llms.txt is good at — and what it was never designed to do.

Does llms.txt Help You Rank in Google? No.

This is the single most common misconception about the file, and it is worth being blunt about it before you spend any time on implementation. Google's own documentation addresses it directly:

"You don't need to create new machine readable files, AI text files, or markup to appear in these features. There's also no special schema.org structured data that you need to add."

— Google Search Central, AI features and your website — on appearing in AI Overviews and AI Mode

Google has been consistent on this in public too. At Google's Search Central Deep Dive event in the Asia Pacific region in July 2025, Gary Illyes stated that Google does not support llms.txt and is not planning to, and that normal SEO practice is what gets content into AI Overviews — as reported by Search Engine Land. John Mueller has separately compared the format to the old keywords meta tag: a self-declared signal a search engine would have to verify against the real page anyway.

Who Actually Reads llms.txt

The file has real, documented adoption — just not where most SEO articles claim. Its heaviest use is software documentation consumed by coding agents:

• The AI labs publish their own. Per the v2 spec, OpenAI, Anthropic, and Gemini all publish llms.txt files for their developer documentation.
• Chrome's Lighthouse checks for one. The v2 spec notes that Lighthouse audits sites for an llms.txt file as part of its agentic browsing checks.
• Documentation platforms generate one automatically, which is why thousands of sites now publish one without anyone hand-authoring it.

The spec is explicit that the intended mode of use is inference, not training — an agent reads it on demand, while helping a user with a specific task, rather than a crawler sweeping it up into a training corpus.

So the honest case for adding one: if people use coding agents, documentation assistants, or research agents against your content, an llms.txt gives those agents a cleaner path in. If your site is a marketing site with no documentation and no API, the realistic benefit today is close to zero.

The v2 Format, Precisely

Most guides describe the v1 format and get two things wrong. First, the file does not have to live at your domain root. Second, only one section is actually required.

Per the v2 spec, a conforming file contains these sections as Markdown, in this order:

1. An optional byte-order mark (BOM).
2. An H1 with the name of the project or site. This is the only required section.
3. A blockquote with a short summary of the project, containing key information necessary for understanding the rest of the file.
4. Zero or more Markdown sections of any type except headings — paragraphs, lists — giving more detail about the project and how to interpret the files.
5. Zero or more sections delimited by H2 headers, containing "file lists" of URLs where further detail is available. Each list item is a required Markdown hyperlink [name](url), then optionally a : and notes about the file.

The spec's own mock example:

# Title

> Optional description goes here

Optional details go here

## Section name

- [Link title](https://link_url): Optional link details

## Optional

- [Link title](https://link_url)

The ## Optional section is a reserved convention, not just a label — it marks secondary links an agent may skip when it needs a shorter context. Putting genuinely important pages under ## Optional is a self-inflicted wound.

Placement: root or any subpath

A file at /llms.txt covers the whole site. A file at /docs/llms.txt covers everything under /docs/. Where more than one applies, agents should use the most specific one. For a large site, a focused /docs/llms.txt is often more useful than one root file trying to describe everything.

v2's Second Half: Markdown Versions of Pages

This is the part almost every llms.txt article omits, and it is arguably more useful than the index file itself. v2 proposes that pages agents might need also serve a clean Markdown version at the same URL — either with .md appended (page.html.md) or with the extension replaced (page.md). URLs without file names append index.html.md or index.md.

To let clients discover them, the spec recommends standard link relations: rel="alternate" type="text/markdown" points to the Markdown version of a page, and rel="describedby" points to the llms.txt file covering it. These work as HTML <link> elements or as an HTTP Link: response header:

Link: </docs/page.html.md>; rel="alternate"; type="text/markdown",
      </docs/llms.txt>; rel="describedby"

The header form is worth noting for anyone who does not want to touch page templates: it can be added in web server or CDN configuration, and it also works for non-HTML resources such as the Markdown files themselves.

Five Mistakes Worth Avoiding

1. Expecting Google results. Covered above. Google says it does not use the file; publishing one will not move your rankings or your AI Overview citations.
2. Treating it as a robots.txt substitute. They solve different problems. robots.txt governs what automated tools may access; llms.txt describes what your content is, and is read on demand. Neither replaces the other.
3. Dumping your sitemap into it. A sitemap lists every indexable page for search engines. llms.txt is a curated overview — its value is in what you leave out. If it is long enough to blow a context window, it has failed at its one job.
4. Linking to HTML when Markdown exists. The links should point to LLM-friendly content. Pointing an agent at a JavaScript-heavy HTML page recreates the exact problem the file was invented to solve.
5. Writing it once and letting it rot. A file describing pages that have since moved is worse than no file — it sends agents to 404s while claiming authority over your structure.

So Should You Add One?

A defensible rule of thumb, given what is actually known today:

• Yes, if you publish documentation, an API, or a developer-facing product. This is the use case the file was designed for and where it demonstrably gets read.
• Probably, if it is generated automatically by your docs platform. The cost is zero and it is kept current for you.
• It is a cheap hedge for a pure marketing site. It costs a few minutes, will not hurt anything, and no evidence shows it affecting Google visibility — but Google is not the only engine that matters, and the others have published nothing about what they use.

That last point is the whole argument, and it is worth stating plainly. Google has told you its minimum; ChatGPT, Claude and Perplexity have told you nothing at all. The asymmetry favours doing a little more than Google asks rather than exactly as much: the extra work here is measured in minutes, it cannot hurt your Google performance, and it is the only hedge available against engines that publish no rulebook. Treat llms.txt as belonging to that category — not as a ranking tactic, and not as something to skip because Google shrugged at it.

What does determine whether AI answer engines can reach and cite you is far more mundane: whether their retrieval crawlers are allowed in at all. That is a robots.txt question, and it is the subject of our guide to getting cited by AI answer engines — where the distinction between training crawlers and search crawlers turns out to matter a great deal.

Frequently Asked Questions