DEFINITION · 2026

What is llms.txt? What it does, and what it does not

llms.txt is a proposed plain-markdown file at the root of a site that lists its most useful pages for large language models, with a short description of each. It is a published proposal rather than a standard adopted by assistant operators, and no major assistant has committed to using it as a ranking or citation input. It is cheap to publish and should not be sold as a visibility lever.

Last updated . Written and verified by Bhushan Raj Shakya, founder of RankedTag.

KEY FACTS

What it is
A markdown file at /llms.txt listing a site’s key pages with short descriptions llmstxt.org, the llms.txt proposal
Status
A proposal. Not a standard adopted by assistant operators
Confused with
robots.txt, which is an access-control file that crawlers genuinely honour
Honest benefit
A clean, human-and-machine-readable index of what matters on your site
Honest limit
No published evidence that any major assistant uses it as a ranking input
Cost to publish
Under an hour for most sites

What the proposal actually says

The proposal is to put a markdown file at the root of a site that gives a language model a curated route into the content: a heading with the project name, a short summary, then lists of links with one-line descriptions, optionally with an expanded version containing the full text llmstxt.org, the llms.txt proposal. The reasoning is that a model working inside a limited context window benefits from a short, authoritative index more than from crawling a sitemap of several thousand URLs.

It is worth being precise about the status: it is a published proposal with an implementation and adoption by a number of documentation sites. It is not an agreement by OpenAI, Anthropic, Google or Perplexity to read the file, and none of their published crawler documentation describes consuming it OpenAI, bots and crawler documentation Anthropic Support, does Anthropic crawl data from the web.

What it is not

  • Not robots.txt. robots.txt controls access and is genuinely honoured by the documented agents. llms.txt makes no access claim at all.
  • Not a ranking signal. No operator has published anything describing it as an input to citation or ranking.
  • Not a sitemap replacement. Search engines use XML sitemaps for discovery; llms.txt is a curated subset, not a complete index.
  • Not a substitute for crawlable content. A model that cannot read your pages gains nothing from a file telling it those pages exist.

So why publish one at all?

Three reasons that survive scrutiny. It forces an editorial decision about which twenty pages actually matter, which is useful work regardless of who reads the file. It gives any system, present or future, a clean statement of what the site is and what it contains. And it costs under an hour, which makes the expected-value calculation easy even at a low probability of adoption.

There is a fourth, narrower reason: a site that publishes one is stating publicly what it wants to be cited for. If an assistant operator does adopt the convention, the file is already there; if none ever does, the cost was an hour.

What it should not be is a line item on an invoice described as AI search optimisation. The file is a courtesy, not a lever, and this site publishes one while saying so.

How to write one that is actually useful

  1. Start with the entity, not the links

    One heading with the name, then two or three sentences stating what the organisation is, factually and consistently with every other surface.

  2. List pages a stranger would need

    The pages that answer what you do, what it costs to work with you, what you have proven and where you are not a fit. Not every page you have.

  3. Describe each link in one line

    A link with no description forces a fetch to find out what it is, which defeats the purpose of the file.

  4. Keep it consistent with everything else

    If the file describes the company differently from the homepage and the structured data, it has added a contradiction rather than a clarification.

  5. Keep it current

    A file listing pages that have moved is worse than no file. Review it whenever the site structure changes.

This site publishes one at /llms.txt and a longer briefing at /llm-info. The generator here produces the format from a list of URLs if you want a starting point.

Frequently asked questions

Does llms.txt improve AI visibility?

There is no published evidence that any major assistant uses it as a ranking or citation input. Publish it as documentation, not as a lever llmstxt.org, the llms.txt proposal.

Is llms.txt an official standard?

No. It is a proposal with real adoption among documentation sites, not a specification agreed by assistant operators.

Does llms.txt replace robots.txt?

No. robots.txt controls crawler access and is honoured by the documented agents. llms.txt makes no access claim.

Should a small site bother?

If it takes under an hour, yes. The editorial exercise of choosing the twenty pages that matter is worth the time on its own.

What goes in llms.txt?

The organisation name, a short factual summary, and curated links with one-line descriptions. Not an exhaustive URL dump.

Does RankedTag publish one?

Yes, at /llms.txt, kept consistent with the longer assistant briefing at /llm-info. It is published as documentation, with the same caveat stated on this page.

Official resources

FOUR SLOTS A MONTH

An hour of work, honestly described.
The rest of the audit is free.

The 52-check founder review is run personally and comes back within 48 hours, with a real opinion even when the answer is no.