What Is llms.txt? The Emerging AI Content Standard

What is llms.txt?

llms.txt is a proposed text file, written in markdown, that sits at the root of a website and gives AI systems a curated summary of the site's most important pages. Instead of forcing a large language model to crawl and parse full HTML — with its navigation menus, scripts, ads, and layout markup — llms.txt hands over a clean, structured list of links and short descriptions that point straight to the content that matters.

The key thing to understand up front: llms.txt is not an official standard. It has no backing from the IETF, W3C, or any other standards body. It was proposed by developer Jeremy Howard in 2024 as a convention, and its adoption depends entirely on whether AI companies choose to look for it and use it. That distinction matters for how much weight you should put on implementing it.

What actually goes inside an llms.txt file

An llms.txt file is a markdown document, typically containing an H1 with the site or project name, a short blockquote summarizing what the site does, and then a series of H2-headed sections listing links with one-line descriptions. A minimal example might have a "Docs" section linking to key documentation pages, a "Guides" section linking to tutorials, and an "Optional" section for lower-priority material.

This format matters because markdown is cheap for a model to parse and low on ambiguity compared to HTML. There's no need to strip out <nav> elements, ad scripts, or CSS classes — the file is already reduced to the signal: page title, URL, and a sentence of context.

llms.txt vs. llms-full.txt

Some sites publish two versions: llms.txt as a lightweight index of links and descriptions, and llms-full.txt as a more complete document that inlines the actual content of key pages rather than just linking to them. The idea is that llms.txt works like a table of contents for a model doing light-touch reasoning about a site, while llms-full.txt gives a model everything it needs in one file if it can't or won't follow the links. Not every implementation includes both — many sites just publish the shorter llms.txt.

How llms.txt differs from robots.txt and sitemap.xml

llms.txt does a different job than either of the two files it's most often confused with, and it doesn't replace them.

robots.txt controls permission. It tells crawlers which paths they're allowed or disallowed from accessing. It's an actual, decades-old convention with broad, consistent support across search engines and crawlers, and it governs access, not content quality or context.

sitemap.xml supports discovery. It's a flat, machine-readable list of URLs (often with metadata like last-modified dates) designed to help crawlers find all the pages on a site, especially ones that are hard to reach through internal links alone. It doesn't describe what's on those pages beyond basic technical metadata.

llms.txt is about comprehension. It doesn't grant or deny access, and it doesn't try to list every URL on a site. Its job is to say, in plain language, "these are the pages that matter, and here's what each one covers." A crawler could ignore it entirely and still be permitted to crawl everything under robots.txt; conversely, a page can be blocked in robots.txt regardless of whether it's mentioned in llms.txt. The three files operate on separate layers — permission, discovery, and comprehension — and a well-configured site can use all three without conflict.

The honest state of adoption

Here's the part worth being direct about: no major AI vendor has publicly confirmed that it treats llms.txt as an authoritative input to how it retrieves, ranks, or cites content. Some AI-related crawlers and tools have reportedly checked for or used llms.txt files, and a growing number of documentation platforms, SaaS products, and developer-tool sites now publish one. But "some tools reportedly use it" is a meaningfully different claim than "the major labs have committed to supporting it," and right now the evidence sits closer to the former.

That doesn't make llms.txt worthless — it makes it a low-cost bet rather than a guaranteed lever. Writing one takes a few hours at most: you're organizing links you likely already have into a markdown outline. If some subset of AI crawlers does read it now or starts reading it later, you've already got clean, structured context sitting at your root domain. If none of them ever standardize on it, you've lost very little. That risk-reward profile is why it's reasonable to publish an llms.txt file without treating it as a core part of your AI-visibility strategy or expecting it to move the needle on its own.

Should you create an llms.txt file for your site?

Yes, if you can produce one accurately and keep it current — but don't build your AI-visibility roadmap around it. The highest-value use case right now is for documentation-heavy sites, developer tools, and API references, where the content is naturally structured as a list of discrete pages with clear purposes. For content-marketing sites, blogs, and ecommerce catalogs, the payoff is less clear because the "curated list of key pages" format fits less naturally, and the pages you'd list are also the ones most likely to already be well-linked and crawlable through normal means.

If you do create one, keep it accurate. A stale llms.txt that links to deprecated pages or missing content is worse than no file at all, because it actively feeds a model bad context instead of none. Treat it the way you'd treat a sitemap: something to regenerate or review whenever your site's structure changes materially, not a one-time artifact you publish and forget.

The broader point is that llms.txt is one small, cheap piece of a much larger AI-search-visibility picture that also includes how clearly your HTML pages communicate their own content, whether your site is technically crawlable at all, and whether your content is the kind of clear, well-structured material that gets cited when a model does read it directly.

Frequently asked questions

Is llms.txt an official web standard?

No. llms.txt is a community-proposed convention, not a ratified standard from a body like the IETF or W3C. Its rules exist because a developer proposed a format in 2024, not because any organization enforces or governs it, which means support is voluntary and inconsistent across AI systems.

Where should I put my llms.txt file?

At the root of your domain, the same location convention used by robots.txt — for example, yoursite.com/llms.txt. This is where any tool that checks for the file would look first, following the pattern established by other root-level convention files.

Does llms.txt replace robots.txt or sitemap.xml?

No. robots.txt handles crawler permissions, sitemap.xml handles URL discovery, and llms.txt handles content summarization for AI comprehension. They serve different functions and are meant to coexist rather than substitute for one another.

Will llms.txt guarantee my content gets cited by AI models?

No. There's no public confirmation from major AI vendors that llms.txt is treated as an authoritative input to retrieval or citation decisions. It's best understood as a low-cost, low-risk addition to your site, not a lever with proven, guaranteed impact.

What's the difference between llms.txt and llms-full.txt?

llms.txt is typically a short index of links with brief descriptions, while llms-full.txt inlines the fuller content of key pages directly into the file. Sites choose to publish one or both depending on how much context they want to make available without requiring a model to follow links.