robots.txt, sitemap.xml and llms.txt Explained

Three small text files sit at the root of most websites, quietly telling crawlers, search engines, and increasingly AI systems, what they’re allowed to do and where to find things. Two of them, robots.txt and sitemap.xml, are long-established standards. The third, llms.txt, is new, unofficial, and its actual usefulness is genuinely unclear as of 2026. This post explains what each one does and why they shouldn’t be treated the same way.

robots.txt: a standard, but an advisory one

robots.txt is a plain text file placed at the root of a domain, such as example.co.uk/robots.txt, that tells crawlers which parts of a site they may or may not access. It’s been in informal use since 1994, but it was only formally standardised in 2022 as RFC 9309, the Robots Exclusion Protocol, published by the Internet Engineering Task Force. Standardisation cleared up a lot of inconsistency between how different crawlers had been interpreting the file, defining precisely how rules should be parsed, how conflicts between rules should be resolved, and how the file should be cached.

What RFC 9309 does not do, and this is worth being clear about, is enforce anything. Compliance with robots.txt is entirely voluntary. Reputable crawlers, Googlebot among them, respect it because ignoring it would damage their reputation and their relationship with site owners, but nothing in the protocol itself stops a badly behaved bot from ignoring the rules entirely. robots.txt is a request, not a lock. It should never be relied on to keep sensitive content genuinely private or secure.

A basic robots.txt file might look like this:

User-agent: *
Disallow: /admin/
Disallow: /cart/
Allow: /

Sitemap: https://www.example.co.uk/sitemap.xml

sitemap.xml: helping crawlers find what matters

Where robots.txt is about restricting access, sitemap.xml does the opposite job: it’s a discovery aid, listing the URLs on a site that you’d like crawlers to know about. It’s especially useful for large sites, new sites with few external links pointing in yet, or sites with pages that are hard to discover just by following internal links.

A sitemap doesn’t guarantee a page will be crawled or indexed, and it certainly doesn’t guarantee ranking, but it does make the site’s structure explicit rather than leaving a crawler to stumble across everything by chance. Most content management systems, including WordPress with a standard SEO plugin, generate and maintain one automatically.

llms.txt: an emerging idea, not an established standard

llms.txt is a newer proposal, put forward in 2024, suggesting that sites offer a simple Markdown file summarising their content in a format intended to be easy for large language models to digest, as an alternative to having an AI system try to parse a full HTML page.

The honest state of play in 2026 is that it remains experimental. Adoption has grown, with estimates suggesting it’s now used on a small percentage of websites, but as things stand, no major AI provider, not OpenAI, not Anthropic, not Google, has publicly confirmed that its production crawlers or answer engines actually read and act on llms.txt files. Google’s own search guidance has explicitly stated that llms.txt isn’t needed for its AI-powered search features. Server logs from sites that have implemented the file generally show it being fetched rarely, if at all, by the major AI crawlers. A website monitoring service that logs crawler activity is a reasonable way to check for yourself rather than taking adoption estimates on faith.

None of this means llms.txt is pointless to add. It costs little to create and does no harm. But it shouldn’t be treated as essential technical SEO the way robots.txt and sitemap.xml are. It’s a bet on where things might go, not a confirmed lever that currently moves anything.

AI crawlers now come in training and search varieties

One genuinely important change for anyone writing robots.txt rules deliberately around AI crawlers is that the major AI companies have split their crawlers into separate user agents for different purposes, rather than running one bot that does everything.

• OpenAI runs GPTBot for gathering training data, and a separate OAI-SearchBot for building the index used by ChatGPT’s search features.
• Anthropic runs ClaudeBot for training, Claude-SearchBot for search indexing, and Claude-User for real-time fetches triggered directly by a user’s request within a conversation.

This split matters because blocking the wrong one has very different consequences. Disallowing a training-only crawler stops your content being used to train future models, without necessarily affecting whether your pages can still be cited or found through that company’s search or assistant products. Disallowing the search-specific crawler, on the other hand, can mean your content disappears from AI-generated answers entirely. If you want to allow citations while opting out of training, or vice versa, you need to name the specific user agents individually in robots.txt rather than relying on a single blanket rule, and check each provider’s current documentation, since bot names and behaviour have changed before and are likely to again.

Summary

robots.txt, standardised as RFC 9309, tells crawlers what they may access, on a voluntary, advisory basis with no real enforcement behind it. sitemap.xml helps crawlers discover the URLs on your site that matter. llms.txt is a newer, unofficial idea for guiding AI systems that remains experimental in 2026, with no major AI provider confirmed to be acting on it yet, so it’s worth adding cheaply but not prioritising over the fundamentals. Finally, remember that AI crawlers now split into separate training and search user agents, so blanket rules aimed at “blocking AI” may do something quite different from what you intended unless you name the specific bots.