What Is llms.txt? The Emerging AI Content Standard

The key thing to understand up front: llms.txt is not an official standard. It has no backing from the IETF, W3C, or any other standards body. It was proposed by developer Jeremy Howard in 2024 as a convention, and its adoption depends entirely on whether AI companies choose to look for it and use it. That distinction matters for how much weight you should put on implementing it, and it's the reason this article treats it as a low-cost bet rather than a settled best practice.

What an llms.txt file actually is and what it contains

An llms.txt file is a markdown document, typically containing an H1 with the site or project name, a short blockquote summarizing what the site does, and then a series of H2-headed sections listing links with one-line descriptions. A minimal example might have a "Docs" section linking to key documentation pages, a "Guides" section linking to tutorials, and an "Optional" section for lower-priority material. There's no required schema beyond this loose convention, so two sites' files can look quite different while still following the same basic idea.

This format matters because markdown is cheap for a model to parse and low on ambiguity compared to HTML. There's no need to strip out <nav> elements, ad scripts, or CSS classes — the file is already reduced to the signal: page title, URL, and a sentence of context. You can generate this structure yourself by hand, or use a tool like KinetixSEO's llms.txt generator, which builds the file from a site's existing page list rather than requiring you to write the markdown outline from scratch.

llms.txt vs. llms-full.txt

Some sites publish two versions: llms.txt as a lightweight index of links and descriptions, and llms-full.txt as a more complete document that inlines the actual content of key pages rather than just linking to them. The idea is that llms.txt works like a table of contents for a model doing light-touch reasoning about a site, while llms-full.txt gives a model everything it needs in one file if it can't or won't follow the links. Not every implementation includes both — many sites just publish the shorter llms.txt, reserving the fuller file for cases where the content set is small enough to inline without becoming unwieldy.

What an llms.txt file is used for

An llms.txt file is used to give a language model a shortcut to understanding what a site is and which pages on it matter, without the model having to crawl every URL and infer structure from HTML. For a documentation site, that might mean pointing a model straight at the API reference and the getting-started guide instead of leaving it to find those pages by following internal links. For a model that does read it, the practical effect is faster, cleaner context: it gets a page title, a URL, and a sentence of description instead of having to parse a full page template to extract the same information. It does not request or grant crawl access, and it does not guarantee a page gets read, summarized, or cited — it only describes the content if and when a system chooses to consult the file at all.

What is llms.txt in SEO: how it fits alongside existing AI-visibility work

In SEO terms, llms.txt is a content-comprehension aid, not a ranking or indexing mechanism — it sits alongside, not inside, the practices covered in GEO best practices and the broader GEO guide. Traditional SEO signals like structured data, clean HTML, and internal linking still do the heavy lifting of making a page understandable to any crawler, AI or otherwise. llms.txt doesn't substitute for those signals; at best it adds a shortcut on top of them for the subset of AI crawlers that check for it. Because no major AI vendor has confirmed using it as a ranking or citation input, teams running AI-visibility programs should treat it as a supplementary, low-effort addition rather than a core tactic, and keep their attention on things like answer-block structure and technical crawlability, covered in how to write an answer block AI engines can cite.

How llms.txt differs from robots.txt and sitemap.xml

llms.txt vs. robots.txt vs. sitemap.xml

	File	Primary job
robots.txt	File: robots.txt	Primary job: Controls crawler permission (allow/disallow paths)
sitemap.xml	File: sitemap.xml	Primary job: Supports URL discovery for crawlers
llms.txt	File: llms.txt	Primary job: Gives a curated, markdown summary for AI comprehension

llms.txt does a different job than either of the two files it's most often confused with, and it doesn't replace them. The three operate on separate layers of how a crawler or model interacts with a site:

• robots.txt controls permission. It tells crawlers which paths they're allowed or disallowed from accessing. It's an actual, decades-old convention with broad, consistent support across search engines and crawlers, and it governs access, not content quality or context.
• sitemap.xml supports discovery. It's a flat, machine-readable list of URLs (often with metadata like last-modified dates) designed to help crawlers find all the pages on a site, especially ones that are hard to reach through internal links alone. It doesn't describe what's on those pages beyond basic technical metadata.
• llms.txt is about comprehension. It doesn't grant or deny access, and it doesn't try to list every URL on a site. Its job is to say, in plain language, "these are the pages that matter, and here's what each one covers."

A crawler could ignore llms.txt entirely and still be permitted to crawl everything under robots.txt; conversely, a page can be blocked in robots.txt regardless of whether it's mentioned in llms.txt. Permissioning still matters as a prerequisite, separate from llms.txt: KinetixSEO's own crawl check against its robots.txt on October 23, 2026 confirmed it allows OAI-SearchBot, PerplexityBot, and Claude-SearchBot from the site root, which is the kind of baseline check worth running before worrying about an llms.txt file at all, since a blocked crawler won't read either file. A well-configured site can use all three files without conflict — they answer different questions, not competing versions of the same one.

The honest state of llms.txt adoption among AI crawlers

No major AI vendor has publicly confirmed that it treats llms.txt as an authoritative input to how it retrieves, ranks, or cites content, and that gap between reported use and confirmed use is the core thing to be honest about. Some AI-related crawlers and tools have reportedly checked for or used llms.txt files, and a growing number of documentation platforms, SaaS products, and developer-tool sites now publish one. But "some tools reportedly use it" is a meaningfully different claim than "the major labs have committed to supporting it," and right now the evidence sits closer to the former.

One concrete illustration: on October 19, 2026, a tracked prompt asking Gemini (web search) "Can ChatGPT crawl my website? How do I check whether it can?" returned 9 cited sources — xseek.io, openai.com, unveilrai.com, crawlercheck.com, thats.be, and four others — with no mention of an llms.txt file as a factor in any of them. That single check doesn't prove llms.txt has no effect anywhere, but it's a reminder that even on a query directly about crawler access, citation behavior isn't visibly tied to whether a site publishes the file.

That doesn't make llms.txt worthless — it makes it a low-cost bet rather than a guaranteed lever. Writing one takes a few hours at most: you're organizing links you likely already have into a markdown outline. If some subset of AI crawlers does read it now or starts reading it later, you've already got clean, structured context sitting at your root domain. If none of them ever standardize on it, you've lost very little. That risk-reward profile is why it's reasonable to publish an llms.txt file without treating it as a core part of your AI-visibility strategy or expecting it to move the needle on its own, a theme also covered in how to measure GEO ROI when clicks don't count.

What an llms.txt file is good for versus what it can't do

llms.txt is good for giving a model a fast, structured map of a documentation-heavy site; it can't guarantee citation, replace good HTML, or substitute for the kind of content AI answer engines actually pull from when they do cite sources. KinetixSEO's own citation tracking across ChatGPT (web search), Claude (web search), Gemini (web search), Google AI Mode, Google AI Overviews, Grok, Meta AI, and Perplexity found that just 1% of the 891 source links cited across 111 tracked-prompt checks over the last 90 days (measured October 23, 2026) were community forum pages, based on KinetixSEO's GEO citation tracking methodology, which classified each cited domain as a forum, reference wiki, product documentation, or unclassified. That figure says nothing directly about llms.txt, but it underscores the larger point: what gets cited is overwhelmingly driven by content type and structure, not by whether a site published an auxiliary index file. If a site's pages aren't the kind of clear, well-structured material that earns citations once a model reads them, an llms.txt file won't change that outcome, a pattern explored further in AI crawls my site but doesn't cite it: a 3-stage fix.

Should you create an llms.txt file for your site?

Yes, if you can produce one accurately and keep it current — but don't build your AI-visibility roadmap around it. The highest-value use case right now is for documentation-heavy sites, developer tools, and API references, where the content is naturally structured as a list of discrete pages with clear purposes. For content-marketing sites, blogs, and ecommerce catalogs, the payoff is less clear because the "curated list of key pages" format fits less naturally, and the pages you'd list are also the ones most likely to already be well-linked and crawlable through normal means.

If you do create one, keep it accurate. A stale llms.txt that links to deprecated pages or missing content is worse than no file at all, because it actively feeds a model bad context instead of none. Treat it the way you'd treat a sitemap: something to regenerate or review whenever your site's structure changes materially, not a one-time artifact you publish and forget. The broader point is that llms.txt is one small, cheap piece of a much larger AI-search-visibility picture, one worth checking off alongside the other items in the GEO checklist, which also covers whether your site is technically crawlable and whether your content is structured in a way that gets cited when a model reads it directly.

Frequently asked questions

Is llms.txt an official web standard?

No. llms.txt is a community-proposed convention, not a ratified standard from a body like the IETF or W3C. Its rules exist because a developer proposed a format in 2024, not because any organization enforces or governs it, which means support is voluntary and inconsistent across AI systems.

Where should I put my llms.txt file?

At the root of your domain, the same location convention used by robots.txt — for example, yoursite.com/llms.txt. This is where any tool that checks for the file would look first, following the pattern established by other root-level convention files.

Does llms.txt replace robots.txt or sitemap.xml?

No. robots.txt handles crawler permissions, sitemap.xml handles URL discovery, and llms.txt handles content summarization for AI comprehension. They serve different functions and are meant to coexist rather than substitute for one another.

Will llms.txt guarantee my content gets cited by AI models?

No. There's no public confirmation from major AI vendors that llms.txt is treated as an authoritative input to retrieval or citation decisions. It's best understood as a low-cost, low-risk addition to your site, not a lever with proven, guaranteed impact.

What's the difference between llms.txt and llms-full.txt?

llms.txt is typically a short index of links with brief descriptions, while llms-full.txt inlines the fuller content of key pages directly into the file. Sites choose to publish one or both depending on how much context they want to make available without requiring a model to follow links.