llms.txt: What It Is, Do You Need It, and How to Set It Up

What is llms.txt

llms.txt is a proposed specification for a Markdown file that you publish on your website. It contains a short description of the project, important context and links to key materials.

Jeremy Howard published the proposal in September 2024. The current version of the specification is v2, updated on August 10, 2026. Keep in mind that this is a proposal for standardisation, not a mandatory web standard.

Put simply, llms.txt is a short guide for an AI agent:

here is what this site is about, which materials matter most and where to find the details.

The file does not have to contain everything on the site. Quite the opposite: its job is to give concise context and links to materials the agent can fetch when it needs them.

What problem llms.txt solves

Regular HTML pages are built for people and are cluttered with navigation, code and ads. llms.txt gives an AI agent a compact entry point: a short description of the site and a structured list of important resources. The agent follows the links for details only when it actually needs them.

That said, it is not a guarantee: having the file does not mean every AI service will read or use it.

How llms.txt is structured

Only one element is required: an H1 heading with the name of the site or project. The other elements are recommended and come in this order:

• # H1 — the project name (required).
• > blockquote — a short summary and key context.
• Free-form paragraphs or lists (no headings) — extra notes on how to interpret the materials.
• ## H2 — topic sections with lists of links in the format - [Name](URL): short description of the resource.
• ## Optional — a section for secondary materials that an AI agent can skip when it needs to save context.

Where llms.txt lives

The most common option is the domain root:

https://example.com/llms.txt

But specification v2 also allows you to place the file in a separate section of the site:

https://example.com/docs/llms.txt

In that case, the file covers the URLs under its path. If more than one file applies to a URL, the agent should use the most specific one. This is especially handy for large sites with independent sections such as documentation, a knowledge base or a separate product.

How llms.txt relates to LLMs and AI search

llms.txt is designed primarily for inference: the moment an AI agent, while handling a request, needs to find its way around a site and locate relevant materials. It works as a site guide, not as a replacement for robots.txt to control crawler access.

Do ChatGPT, Claude, Gemini and other systems use llms.txt

The format is still a proposal, not a mandatory standard. None of the major AI services has confirmed that it reads llms.txt when answering users.

An Ahrefs study (June 2026) covered 137,210 domains, and 28% of them had an llms.txt file. Among the roughly 38,000 sites with the file, 97% received no requests for it at all in May 2026. Of the requests that did come in, 96% were made by bots. And 77% of those bots did not belong to AI services at all: they were SEO auditors, technology profilers and other crawlers. AI retrieval bots, which fetch pages to answer queries in AI search (OAI-SearchBot, PerplexityBot), accounted for just 1.1% of bot requests.

The study does not prove the file is never used, but it shows that, for now, very few systems actually read it.

So llms.txt is an optional navigation format for compatible agents, not a guaranteed channel to every AI system.

llms.txt vs robots.txt vs sitemap.xml

These three files do different jobs.

Does llms.txt replace the other technical files? No. llms.txt does not set access rules and does not replace other files:

• robots.txt sets crawling rules for automated crawlers.
• sitemap.xml lists the URLs the site owner wants search engines to find and crawl.
• llms.txt only provides a curated structure and context for AI.

All three files usually sit in the site root: /robots.txt, /sitemap.xml, /llms.txt. But that is not required: the sitemap can live at another address (you point to it in robots.txt), and llms.txt can sit in a subdirectory.

Does llms.txt affect SEO and AI visibility

llms.txt is not a ranking factor. Google states clearly that you don’t need such files to appear in Google Search, including its generative AI features, because Google Search ignores them. So llms.txt neither helps nor hurts your site’s visibility or rankings in Google.

Creating the file on its own:

• does not guarantee you will appear in ChatGPT, Perplexity or Google;
• does not replace quality content, technical SEO and foundational optimisation practices.

Who should create llms.txt

The file is most useful for projects with a large amount of structured and technical information:

• Documentation and APIs. The main use case for the specification: coding agents quickly find the tutorials and references they need.
• SaaS and knowledge bases. It helps AI find its way around features, pricing, guides and integrations.
• Large corporate and research websites. It serves as a concise index across hundreds of pages of expert content.

For a small site with a handful of pages, the practical benefit of a separate navigation file is limited. Creating llms.txt purely for SEO makes no sense, because Google does not treat it as a ranking signal.

How to create llms.txt properly

The file does not need complex markup. The main thing is to pick the materials that really matter and present them compactly.

1. Identify your key pages

You can include:

• main product or service pages;
• documentation;
• API reference;
• guides and instructions;
• a knowledge base;
• important research;
• expert guides;
• policy pages or other information needed to understand the product.

You don’t need to add every URL on the site.

2. Add a name and short context

A basic file can look like this:

# Site name
> A short description of the company, product or project, plus the context needed to understand the materials below.
## Documentation
- [Getting started](https://example.com/docs/start): a guide for new users.
- [API Reference](https://example.com/docs/api): a full description of the API and its parameters.
## Products
- [Product 1](https://example.com/product-1): features and use cases.
- [Product 2](https://example.com/product-2): functionality, limitations and terms of use.
## Guides
- [Setup guide](https://example.com/guide): step-by-step instructions.

Under the specification, only the H1 is required. The blockquote and link sections are recommended parts of the structure.

3. Add short descriptions to links

It is better not to leave a list of URLs without context. Compare:

- [Documentation](https://example.com/docs)

and:

- [Documentation](https://example.com/docs): full description of setup, the API and the main use cases.

The second version tells the agent much more about what it will find behind the link.

4. Don’t turn llms.txt into a copy of your site

The file should be a compact index, not a long article. Detailed explanations, product specs, instructions and other materials are better kept on separate pages that llms.txt links to.

5. Create Markdown versions of pages where you can

Specification v2 suggests going beyond llms.txt itself and giving AI agents clean Markdown versions of pages at the same address with a .md extension. For example, for this page:

https://example.com/docs/api.html

the Markdown version can live at:

https://example.com/docs/api.html.md or https://example.com/docs/api.md

To help agents find these versions, the specification recommends standard link relations:

<link rel="alternate" type="text/markdown" href="/docs/api.md">
<link rel="describedby" href="/docs/llms.txt">

The first points to the Markdown version of the page, the second to the llms.txt file that covers it. You can provide these relations as HTML <link> tags or as an HTTP Link header, which is handy if you’d rather not change page templates.

6. Publish the file on your site

For the whole site:

https://example.com/llms.txt

For a single section:

https://example.com/docs/llms.txt

After publishing, check that the file opens without authentication, returns a 200 status code and that every link inside it works.

7. Keep the file up to date

When products, documentation, URLs or the site structure change, update llms.txt as well. An outdated list of links is less useful than a current, compact index.

How to set up llms.txt in WordPress and other CMSs

In WordPress, you can create llms.txt by hand and upload it to the site root, or generate it with a plugin: Yoast SEO and AIOSEO, for example, both include an llms.txt generator. Wix creates llms.txt automatically for every site, and documentation platforms do the same for published docs.

After generating the file automatically, always review the result. Generators often add every page, post and category in a row, which is exactly the mistake the specification warns against. Remove utility and secondary URLs and keep the key ones. For small sites, creating the file by hand is usually enough.

Example of a good llms.txt

For a hypothetical SaaS product, the file might look like this:

# Example SaaS
> Example SaaS is a platform for automating marketing workflows.
## Product
- [Platform overview](https://example.com/product): key features and use cases.
- [Pricing](https://example.com/pricing): plans and limits.
## Documentation
- [Getting started](https://example.com/docs/getting-started): first steps.
- [API Reference](https://example.com/docs/api): API documentation.
## Optional
- [Blog](https://example.com/blog): product news.

Here the Optional section holds secondary content. This is a convention of the specification, not a command that forces an AI system to process the file in a particular way.

What is llms-full.txt

llms-full.txt is often mentioned alongside llms.txt, but they are different things.

The llms.txt specification does not describe llms-full.txt. This format is used by some documentation platforms, which put the full text of their documentation into a single file.

So an llms-full.txt file on a particular site is that platform’s own choice, not a requirement for all websites.

Common mistakes when creating llms.txt

Adding every URL on the site

llms.txt should not turn into a second sitemap.xml. It is better to choose materials that genuinely help explain the product, services or the site’s subject.

Duplicating sitemap.xml

The goal of the file is not to list every page but to provide structured, curated context.

Using llms.txt instead of robots.txt

llms.txt does not control crawler access. Crawling rules belong in robots.txt.

Leaving broken or outdated links

If a URL changes, update it in llms.txt too. Checking the links periodically is part of normal file maintenance.

Expecting automatic growth in traffic or AI mentions

Having llms.txt does not guarantee that an AI system will cite your site, mention your brand or send extra traffic. For Google Search this is confirmed directly: Google Search ignores llms.txt.

Where llms.txt fits into GEO

GEO (Generative Engine Optimization) covers approaches to improving content visibility in generative search and AI systems. We explained how GEO differs from classic SEO in a separate article. llms.txt can be seen as one of the optional technical pieces of this work, but it does not replace foundational SEO, quality content and a technically accessible site. For generative AI features in Search, Google recommends sticking with foundational SEO practices rather than looking for separate “AI hacks”.

Why llms.txt alone is not enough

You can compare the file to a book’s table of contents: it helps you understand the structure but does not replace the content itself. If the site lacks quality documentation, useful pages or up-to-date information, a list of links will not make the content more valuable for users or AI systems.

What matters more than the file itself

For a site that wants to be visible in search and AI services, the basics remain:

• useful, up-to-date content;
• a clear site structure;
• internal links;
• important content that is accessible for crawling;
• correct indexing;
• clear text content;
• external sources and mentions, where they arise naturally in your field.

llms.txt can complement this system, but on its own it does not decide whether a particular AI system will cite a page or mention a brand. We covered what actually influences appearing in ChatGPT, Perplexity and other AI answers in a separate article.