llms.txt: What It Is, an Example and vs robots.txt

What it means

The format was proposed at llmstxt.org by Jeremy Howard in 2024. The only required element is an H1 with the site or project name; a blockquote summary and H2 sections listing links with short notes are recommended. Its stated aim is to help language models use a site at inference time, when answering a question, rather than for training.

llms.txt is a convention, not a standard that search engines or answer engines have committed to. Google says you do not need AI text files to appear in Google Search or its generative AI features, because Google Search does not use them.

How to create an llms.txt file

List the 10 to 30 pages an AI tool should read first: product, pricing, documentation and comparisons. Leave out tag pages and blog archives.
Start the file with an H1 holding your site or product name. It is the only required element.
Add a one-paragraph summary in a blockquote, a line starting with >, that says what you sell and to whom.
Group the links under H2 headings, one per line: - [Page name](https://example.com/page): a short note on what the page answers.
Put links an agent can skip under a final ## Optional heading.
Save it as plain UTF-8 text at /llms.txt on your domain, then check it against the format with a validator.

llms.txt example

This is a shortened copy of the file RecoProof publishes at recoproof.com/llms.txt. It has the required H1, a summary in a blockquote, H2 lists of links with a note after each colon, and an Optional section that an agent can skip when it needs a shorter context.

# RecoProof

> RecoProof sells AI visibility audits for B2B SaaS companies.

## Product

- [AI visibility audit](https://recoproof.com/ai-visibility-audit): What the expert-reviewed audit measures, what it delivers and who it is not for
- [Pricing](https://recoproof.com/pricing): The free check, the one-time Recommendation Audit and monthly monitoring

## Free tools

- [llms.txt generator and validator](https://recoproof.com/tools/llms-txt-generator): Draft an llms.txt from a homepage and sitemap, or check a file against the llmstxt.org format

## Optional

- [All guides](https://recoproof.com/guides): Every published RecoProof guide

llms.txt vs robots.txt

robots.txt tells crawlers which URLs they may fetch. llms.txt suggests which pages are worth reading once they can. They do different jobs, so a site can use both; listing a page in llms.txt does not override a Disallow rule for a crawler that obeys robots.txt.

What differs	robots.txt	llms.txt
What it does	Tells crawlers which URLs they may or may not fetch.	Points AI tools to the pages worth reading, with a short note on each.
Status	An IETF standard, RFC 9309, published in September 2022.	A proposal published at llmstxt.org in September 2024. No search or answer engine has committed to it.
Format	Plain text: User-agent, Allow and Disallow lines.	Markdown: an H1 name, a blockquote summary and H2 lists of links.
Where it lives	Only at the root of the host: /robots.txt.	Usually /llms.txt. The current version of the proposal also allows a subpath, such as /docs/llms.txt.
Can it block AI crawlers?	Yes, for crawlers that obey it, such as GPTBot. It is a request, not access control.	No. It does not control crawling or training.
Does Google Search use it?	Yes. Googlebot follows robots.txt.	No. Google says Search does not use AI text files.

Why it matters

For a B2B SaaS site, a clean llms.txt is a cheap way to point agents and developer tools that read the file to accurate product, pricing and documentation pages. Judge it on that modest benefit, not as a lever for AI recommendations.

Example

A developer platform publishes /llms.txt with its name, a one-sentence summary and three sections: documentation, API reference and pricing, each linking to the canonical page with a short note. Coding assistants that read the file can find the current API docs instead of an outdated blog post.

Common misunderstandings

There is no evidence that publishing llms.txt makes answer engines recommend a brand; treat it as a map for tools that read it.
llms.txt does not control crawling or training; robots.txt does.
A missing llms.txt is not an SEO or AI visibility defect.

Definition