What Is llms.txt? Format, Uses, and Limits

What the llms.txt proposal actually describes

llms.txt is a proposed Markdown file placed at the root of a site, usually at /llms.txt. Its job is to give an LLM-friendly directory of the pages a publisher wants an assistant or agent to read first. The proposal uses ordinary Markdown: an H1 for the site or project name, a short blockquote for the summary, optional prose, then H2 groups containing links with brief descriptions.

The format is closer to a compact documentation index than an access-control file. It does not replace robots.txt, a sitemap, canonical tags, or page-level permissions. It has no standard User-agent, Allow, Disallow, or Crawl-delay rules. A site that needs to limit crawling must keep using controls that crawlers publish and honor.

The proposal also mentions an optional llms-full.txt file containing a fuller text bundle. That second file may suit small documentation sets, but it can become unwieldy on a large site. A short, selective llms.txt is easier to audit and less likely to point assistants toward duplicate, stale, or low-priority pages.

What adoption does and does not prove

Publishing the file proves only that the URL is available. It does not prove that ChatGPT, Claude, Gemini, Perplexity, Google AI, or any crawler read it. It also does not prove that a cited page, brand mention, or referral came from the file. Vendor behavior can differ by product, model, retrieval mode, and date.

Treat the file as a low-cost documentation experiment. Save the exact file, note the publish date, and track server requests to the URL. Run a fixed set of prompts before and after publication. Record citations, answer wording, model name, market, and run time. Even then, use careful language: a movement after publication is an observation, not proof of cause.

A sound file anatomy

A useful file begins with one H1, followed by a one-sentence summary in a blockquote. Add a short explanation of the site, then group selected links under descriptive H2 headings. Each link can carry a short phrase stating what the page contains. Keep URLs absolute, use canonical destinations, and avoid session parameters or tracking codes.

Pick pages that answer stable questions well: official product docs, pricing, policies, research methods, terminology, and trusted guides. Do not place private or unpublished URLs in the file. Do not list every page simply because it exists. A concise directory is easier for both people and software to read.

A small example

# Acme Analytics
> Reporting software for subscription businesses.

Acme Analytics tracks revenue, retention, and account activity.

## Product
- [Product overview](https://example.com/product): Main features and intended users.
- [Pricing](https://example.com/pricing): Current plans and billing terms.

## Documentation
- [Getting started](https://example.com/docs/start): Setup from account creation to the first report.

This example is intentionally short. The description states what the site does, while the link text and trailing phrases tell a visitor what sits behind each URL. The file remains readable in a browser, terminal, or agent workflow.

Where it fits beside existing files

robots.txt states crawler rules for agents that choose to honor them. XML sitemaps list URLs for discovery. llms.txt presents a selected reading list in Markdown. Those jobs are different, so one file should not be used as a substitute for another.