llms.txt Validator: Checks to Run Before Publishing

What the llms.txt proposal actually describes

llms.txt is a proposed Markdown file placed at the root of a site, usually at /llms.txt. Its job is to give an LLM-friendly directory of the pages a publisher wants an assistant or agent to read first. The proposal uses ordinary Markdown: an H1 for the site or project name, a short blockquote for the summary, optional prose, then H2 groups containing links with brief descriptions.

The format is closer to a compact documentation index than an access-control file. It does not replace robots.txt, a sitemap, canonical tags, or page-level permissions. It has no standard User-agent, Allow, Disallow, or Crawl-delay rules. A site that needs to limit crawling must keep using controls that crawlers publish and honor.

The proposal also mentions an optional llms-full.txt file containing a fuller text bundle. That second file may suit small documentation sets, but it can become unwieldy on a large site. A short, selective llms.txt is easier to audit and less likely to point assistants toward duplicate, stale, or low-priority pages.

What adoption does and does not prove

Publishing the file proves only that the URL is available. It does not prove that ChatGPT, Claude, Gemini, Perplexity, Google AI, or any crawler read it. It also does not prove that a cited page, brand mention, or referral came from the file. Vendor behavior can differ by product, model, retrieval mode, and date.

Treat the file as a low-cost documentation experiment. Save the exact file, note the publish date, and track server requests to the URL. Run a fixed set of prompts before and after publication. Record citations, answer wording, model name, market, and run time. Even then, use careful language: a movement after publication is an observation, not proof of cause.

A sound file anatomy

A useful file begins with one H1, followed by a one-sentence summary in a blockquote. Add a short explanation of the site, then group selected links under descriptive H2 headings. Each link can carry a short phrase stating what the page contains. Keep URLs absolute, use canonical destinations, and avoid session parameters or tracking codes.

Pick pages that answer stable questions well: official product docs, pricing, policies, research methods, terminology, and trusted guides. Do not place private or unpublished URLs in the file. Do not list every page simply because it exists. A concise directory is easier for both people and software to read.

Validation checks that catch real errors

Start with the file response. The root URL should return HTTP 200 as plain text or Markdown, without a login wall. Check that there is one H1, that every section heading uses H2, and that each list item contains a valid absolute HTTPS URL.

Open every destination. Flag redirects, error pages, canonical mismatches, duplicate links, tracking parameters, and private paths. Compare the descriptions with the pages themselves. A technically valid file can still be misleading when its wording is stale.

Check scope next. The file should read like a selected directory, not a sitemap export. Pricing, product, policies, documentation, methods, and trusted guides usually deserve a place. Thin archives, tag pages, search results, and account screens rarely do.

A compact validator report