llms.txt is a single, human-readable Markdown file that asks a simple question: which pages on your site should AI assistants read first? For marketers and product teams wrestling with documentation, APIs or sprawling content archives, llms.txt offers a low-cost way to point large language models (LLMs) and AI agents to the resources you value most — but it remains experimental, not a guaranteed path to AI citations or search ranking gains.

What llms.txt is, and where it came from

llms.txt is a voluntary specification for a Markdown-formatted guide to a website’s priority resources. AI researcher Jeremy Howard proposed the format in September 2024; the specification continued to evolve, with a version 2 published in August 2026. The recommended location is the site root (for example, /llms.txt), though the file can also live under a specific site path when you want to scope the guidance to a subsite.

Typical contents include an H1 with the site name, a short summary, grouped H2 headings and bulleted links with concise descriptions. The goal is curation: surface a compact set of product pages, documentation, API references, FAQs or other high-value materials rather than listing every URL (that’s what your sitemap is for).

The proposal also defines llms-full.txt for documentation-heavy sites. An llms-full.txt can bundle larger collections of Markdown content so a compatible agent can fetch more material from a single location instead of visiting many individual HTML pages.

How llms.txt differs from robots.txt and sitemap.xml

These three files have distinct roles:

  • robots.txt expresses crawler access rules — which bots may crawl which parts of a site.
  • sitemap.xml supplies search engines with a comprehensive list of site URLs for discovery and crawling prioritization.
  • llms.txt offers a curated table of contents directed at AI agents, with human-written context for a prioritized subset of resources.

llms.txt complements, rather than replaces, robots.txt or sitemap.xml: it’s intended to be a cleaner, curated route into the content you want AI systems to consult first.

Evidence, limits and current platform support

The idea behind llms.txt is straightforward and useful in concept: reduce noise and point AI systems at authoritative content. But early tests have not established a causal link between publishing an llms.txt and better visibility in AI-generated answers.

For example, testing by Semrush on its sister publication found no correlation between implementing llms.txt and improved performance in AI results, and server-log analysis recorded no visits to the llms.txt file from the major AI crawlers monitored during the test window. There is likewise no evidence that llms.txt is a search-engine ranking signal.

Adoption is increasing and the specification is still evolving. That combination — rising publisher interest plus a living spec — makes llms.txt worth monitoring. Treat it as an experimental technical convention, not a proven marketing lever.

When to experiment with an llms.txt, and how to implement it

Because the implementation is straightforward, llms.txt is best treated as a low-cost experiment for sites where prioritizing content for AI agents could reduce friction. It’s particularly relevant for developer docs, product knowledge bases, API references and other technical repositories.

Practical steps:

  • Choose priority content. Pick the pages and documents that most clearly explain your product, services or technical surface — product overviews, setup guides, API references, FAQs and troubleshooting pages are common choices. Resist the urge to list everything; the point is curation.
  • Create the file in Markdown. Include an H1 title, a short summary, grouped H2 headings and links with one- or two-line descriptions. Use llms-full.txt when you need to surface a much larger corpus in one place.
  • Publish at the recommended path. Place the file where the specification expects it (commonly /llms.txt), or under a subpath if you want scope control.
  • Maintain it. Review links and descriptions periodically so the file reflects current priorities. If you generate it automatically, audit outputs to ensure the content remains curated and accurate.

Keep expectations realistic: an llms.txt file won’t replace good content strategy, technical SEO, accessible documentation or broader GEO/AEO efforts. It’s a complementary asset that can make compatible AI agents’ job easier.

Also be deliberate about exposure: only surface public, intentional resources; do not list private or sensitive endpoints in llms.txt.

What to watch next

For practical teams, the next steps are straightforward: if you publish an llms.txt, monitor server logs to detect crawler activity against the file, and fold the file into an AI-readiness checklist alongside clear documentation and accessible content formats (Markdown is useful where appropriate). Track two signals closely: whether major AI providers begin to fetch and honor the file, and how wider publisher adoption influences norms and best practices.

llms.txt is worth testing for documentation-heavy properties because the cost is low and the potential payoff — cleaner discovery by compatible agents — is real. But for now, it’s an operational experiment, not a substitute for fundamentals: accurate docs, strong content structure and explicit discovery signals remain the primary drivers of being found and cited by AI systems.