llms.txt: what it is, who uses it and whether it helps

What is llms.txt?

At a website's root, the file is a plain Markdown document with a short description of the site and links to pages that may help AI systems understand it. Think of it as a reading list for systems that choose to look for it.

The proposal at llmstxt.org sets out a standard format for sharing information with agents. A typical file starts with the site name as an H1, then a brief summary and useful links grouped under H2 headings. An Optional section can hold less essential material.

The proposal does not require every AI system to fetch or follow it. A compatible reader may find selected pages more easily, but the file's presence does not show that a crawler has read it. For the wider question of AI search visibility, see our GEO vs SEO guide.

What the file looks like

A useful file has a clear name, a short description and links to your best explanatory pages. The format is simple enough to edit without a special tool.

Some sites also publish llms-full.txt, a larger document that collects full page content in one place. A tool that supports it may find that useful, though a large file takes more care to keep accurate. In our sample, valid llms.txt files ranged from under 1 KB to over 1.6 MB. A few had no Markdown links, which defeats the purpose of a curated list.

Who publishes llms.txt?

We found a valid file on 34% of the marketing and tech domains we checked, compared with 18% of everyday service domains. Across both groups, that is 163 valid files among 692 domains. Five per cent of each sample returned a soft 404: an HTML page with a successful HTTP status.

That pattern suggests publication is more common in marketing and software circles than among service businesses. It does not tell us whether a system fetches those files or whether they affect search results. A soft 404 matters because a browser can show a page even when no valid file exists.

Lighthouse now checks for llms.txt in its Agentic Browsing category, as Chrome's documentation explains. That makes the file visible in a technical audit. It does not show that a search system uses it to choose sources.

Does llms.txt help you get cited by AI?

Our data shows no positive association with Google AI Overview citations. Of the domains cited for the same everyday-service searches, 14% had a valid llms.txt. Among uncited top-10 domains, 20% had one.

This was a cross-tab, not a controlled test. It cannot show whether the file helped, had no effect or was more common on sites with other advantages. We also have no log evidence that AI crawlers read these files at scale. These percentages describe this sample on one day.

Google's current guidance says there are no additional requirements to appear in AI Overviews or AI Mode. Its AI features guidance says the usual SEO best practices still apply. Our AI visibility guide starts with pages people and search systems can reach and understand.

llms.txt vs robots.txt vs sitemap.xml

llms.txt suggests useful pages, robots.txt gives crawler directives, and a sitemap lists URLs for discovery.

Google explains that robots.txt controls crawler access, while sitemaps help it discover URLs (robots.txt documentation, sitemap documentation). Neither file replaces access controls or guarantees that a page will be indexed. llms.txt is intended to be a selective list.

Most domains did not name an AI crawler in robots.txt: 21% of marketing and tech domains did, alongside 17% of everyday service domains. We found a site-wide GPTBot block on 3% and 5% respectively.

What our robots.txt data says about AI crawlers

GPTBot and Google's systems do different jobs. OpenAI says GPTBot relates to model training, while OAI-SearchBot is used for ChatGPT search. Google uses its own crawlers. Eight domains cited by Google AI Overviews in our sample blocked GPTBot, which fits with those systems working separately.

Anthropic documents its ClaudeBot controls through robots.txt. Before blocking a bot, check what it does and whether you want that use of your pages. A rule for one vendor does not automatically apply to another.

How to create an llms.txt file

You can write llms.txt by hand or use a generator for a first draft. Review every link and description before publishing. The useful part is choosing the right pages.

1. Choose a small set of pages that explain your services, policies and expertise. Start with the pages you would send to a new customer.
2. Write the site name as an H1, followed by one short sentence about the site.
3. Group links under H2 headings such as Services, Guides or About. Use a clear page title and full URL for each Markdown link.
4. Save the document as plain Markdown and publish it at /llms.txt. Check that it returns a successful response and contains text, not an HTML error page.
5. Review links when you update your main pages or sitemap. Add llms-full.txt only if you can keep a longer version current.

A generator can save typing, but it cannot choose which pages deserve attention or spot a stale URL without review. Check the live response in a browser or with a command-line request. A server may return an HTML fallback with status 200, which looks successful to a basic status check.

Common mistakes

The clearest problems in our scan were URLs that did not serve a real file, link lists without useful direction, and content that would be hard to keep current.

• A soft 404: the URL returns status 200 but displays an HTML page. We recorded this on 5% of domains in each sample.
• A large export without useful selection. Valid files ranged above 1.6 MB in our scan, so check whether the extra material is worth maintaining.
• A file with no links, or links that point to pages that have moved.
• An HTML fallback served at /llms.txt instead of Markdown text.
• A crawler rule that blocks the bot you meant to let read the file. Check the specific user-agent groups in robots.txt.
• A document that drifts away from the current site because nobody owns its links.

We found soft 404s in both search samples. Check the response body and content type, as well as whether the URL opens. Keep links to pages that help a reader understand your organisation.

Should your business add llms.txt?

Add the file if your useful pages are in good shape and you can publish and maintain the file quickly. Leave it for later if core pages are missing, outdated, hard to crawl or unclear to customers. Our scan gives no reason to expect more AI citations from adding it.

Start with a technically sound site and service information that answers customer questions. Then decide whether a curated file is worth maintaining. Our agentic SEO guide covers the wider work of making pages usable by automated systems.

Aristral maintains a hand-written llms.txt and permits major AI crawlers in robots.txt. In September 2026, we also shipped agent-readiness work and moved from Level 1 to Level 5 on Cloudflare's isitagentready.com checker. That score describes the checks in that tool. We have no evidence that the file caused a ranking or citation change. Our Google Maps ranking research covers a different kind of visibility.

How Aristral approaches agentic SEO

We check a site's crawl rules, page structure, Markdown delivery and discovery files. Then we fix problems that affect access before adding optional formats.

Our work spans SEO and automation from our base in Clifton, Bristol. For a broader review, see our agentic SEO service. A file may fit when someone owns it and its links help readers.

When a curated file makes sense, we agree who will maintain it and review its links alongside the main pages. A Lighthouse check can confirm that the file exists; it cannot show that it changed a search result. For a review of your site, contact us.