- Treat llms.txt as a curated index for documentation, not a ranking or citation tactic, because Google's current guidance says Search doesn't use it.
- Keep access rules in robots.txt, where they're honored by convention, and never put allow or disallow lines in llms.txt, since nothing reads it as a permission system.
- Before building one, grep your server logs for requests to /llms.txt grouped by user agent; if no tool you care about ever fetches it, spend the hour on crawlable, citable pages instead.
Every few months someone sells an llms.txt file as the cheap shortcut to getting cited by ChatGPT. It takes about twenty minutes to write, and that’s part of the pitch. The problem is that the people selling it rarely say which system reads it, what that system does with the contents, or whether anything measurable changes. So here’s the direct answer. llms.txt is a real file with a narrow purpose. Google says its search doesn’t use it. For most businesses it isn’t worth building for rankings or citations, though an afternoon can be well spent on the right kind of site.
What llms.txt is, and where it came from
llms.txt is a Markdown file served from the root of your domain, at /llms.txt. Jeremy Howard, a co-founder of fast.ai, proposed the format in September 2024 in a short spec published at llmstxt.org. The reasoning was simple. A language model reading a web page has to swallow everything on it: navigation menus, cookie banners, script tags, footer links, and the actual content somewhere in the middle. A model with a limited context window gets a poor deal from that. The proposal was a curated, plain-text map: here is who we are, here are the pages that matter, read these.
The spec is short enough to hold in your head. Here is a stripped-down example, using a made-up company:
# Example Analytics
> Example Analytics is a product analytics platform for B2B SaaS teams.
> The docs cover the JavaScript SDK and the REST API.
## Documentation
- [Quickstart](https://example.com/docs/quickstart.md): Install the SDK and send the first event
- [API reference](https://example.com/docs/api.md): Every endpoint, with request and response shapes
- [Changelog](https://example.com/changelog.md): Release notes by month
The only required element is the H1 with the name of the project or site. The blockquote is the summary. Each H2 groups a set of links, and each link can carry a short note after a colon. The section titled Optional has a specific meaning in the spec: a tool working under a tight context budget can skip it.
The spec has two companion ideas. The first is llms-full.txt, which concatenates the full text of the linked content into a single file so a tool can load everything in one request. The second is per-page Markdown versions, where /pricing has a sibling /pricing.md. Documentation platforms have made both easy to generate. Most of the argument, though, is about the first file.
One caveat before going further. llms.txt is a convention proposed by one person and adopted by a community. It is not an IETF or W3C standard. That distinction matters more than it sounds, and it’s the first thing to keep in mind when someone tells you it’s “required” or “the new robots.txt.”
llms.txt vs robots.txt: access versus description
People mix these two up because both are plain-text files at the root of a domain, and both are addressed to bots. They do different jobs.
robots.txt is an access control file. It tells crawlers which paths they may fetch. Its rules were formalized in RFC 9309 in 2022, and reputable crawlers honor them by convention. It is advisory, not enforced. A badly behaved scraper can ignore it. But every major search and AI crawler documents how it reads robots.txt, which makes it the file that decides whether a bot may read a page at all.
llms.txt is a content map. It says what’s worth reading and blocks nothing. It has no allow or disallow rules, and a page you list there can still be disallowed in robots.txt. That combination is a contradiction you should fix, and I’ll come back to it.
| Question | robots.txt | llms.txt |
|---|---|---|
| What it does | Controls which URLs crawlers may fetch | Lists the pages worth reading, with short notes |
| Standard status | Formal standard (RFC 9309) | Community proposal (llmstxt.org) |
| Blocks access? | No, but reputable crawlers honor it | No, it has no access rules at all |
| Who reads it | Search and AI crawlers, before they fetch | Tools a person points at your site, and some crawlers |
| Format | Directives such as User-agent, Allow, Disallow | Markdown with an H1, a summary, and link lists |
| Effect on Google Search | Governs crawling, so it matters a great deal | None, according to Google’s current guidance |
The distinction you can act on is that robots.txt is where AI-specific decisions belong. Google-Extended is a robots.txt token that controls whether Google can use your content to train its Gemini models, and Google documents that it has no effect on inclusion in Search. OpenAI separates GPTBot, which gathers data for training, from OAI-SearchBot, which powers search inside ChatGPT. You can disallow one and allow the other. Those are robots.txt decisions. A “do not train” line inside llms.txt does nothing, because no one treats that file as a permission system.
Which tools actually read llms.txt today
This is the section most llms.txt content skips. Here’s what you can verify, and where the evidence runs thin.
Google’s position is the clearest, and it’s on the record. In June 2026, Google updated the AI optimization guidance on Search Central with a section headed “Clarifying guidance on llms.txt files.” According to the trade press coverage of that update, the guidance says Google Search doesn’t use these files. Keeping one neither helps nor harms your visibility in Search. You don’t need one to appear in AI features such as AI Overviews and AI Mode. Site owners may still publish them for other services that use them. Read the wording on Search Central yourself rather than trusting a paraphrase, this one included.
Lighthouse, the auditing tool built into Chrome DevTools, added an llms.txt check in a release in May 2026. It sits in an “Agentic Browsing” category and checks two things: whether the file is present, and whether fetching it returns a server error. If the file is missing, the audit reports Not Applicable, because the file is optional. The category doesn’t produce a score, so it can’t move a number you track. Treat it as a line in a diagnostic report, not a ranking signal.
The AI answer engines
As far as I can find, none of the major answer engines has published a commitment to use llms.txt for ranking or citation. That’s the sentence the sales pitches leave out.
Server logs can show AI bot requests to /llms.txt on some sites, and that is worth knowing. But a request proves only that the file was fetched. It doesn’t show that the file changed an answer, changed a citation, or sent anyone to your site. Treat any claim that goes beyond “a bot requested the file” as unproven until someone shows you the citation data.
Developer tools and documentation
This is where llms.txt has real, demonstrable use. A developer working in an AI coding assistant often wants the current documentation for a library loaded into context. Some of those tools let you add a documentation site by URL, then use its llms.txt to find the right pages, or use llms-full.txt to load the whole thing at once.
The consumer here is different from a crawler. A person chose the site and asked for it. The tool fetches on request, and it isn’t weighing your page against ten competitors for a ranked answer. In that setting the curated map earns its keep. The agent gets the quickstart and the API reference, not a 40-page marketing site with the docs buried in the footer.
That’s the honest case for llms.txt: an index for documentation that tools read on demand.
Do you need one? A verdict by site type
The answer depends almost entirely on what kind of site you run.
Probably worth it: a developer product with a large documentation or API reference, where your users work in AI coding tools. The file costs an hour to write, most documentation platforms can generate it, and it stays useful as long as you keep it in sync.
Probably not, as a visibility play: marketing sites, B2B landing pages, service businesses, local businesses, and most blogs. The file won’t make an answer engine cite you, and Google says it won’t help in Search. That hour is worth more on the pages themselves.
Never as a substitute for content: a thin site with a tidy llms.txt is a thin site with a tidy map to its thin pages.
Don't pitch it as a ranking tactic. If a vendor promises more citations for an llms.txt file, ask for two things: the log data behind the claim, and the line from the answer engine's own documentation that says it uses the file. In most cases neither exists.
Before you build anything, check whether anything is asking for the file. On an nginx server using the standard combined log format, this one-liner counts requests to /llms.txt by user agent:
grep '"GET /llms.txt' /var/log/nginx/access.log | awk -F'"' '{print $6}' | sort | uniq -c | sort -rn
An empty result means nothing has requested the file, which is the most common outcome for a marketing site.
Test it first: run the grep, then ask whether the requests come from tools your buyers or developers actually use. If the only hits are unidentified scrapers, you have your answer.
How to write one, if you decide to
If your docs justify the file, the build is quick. These are the steps in order.
- Choose the pages worth indexing. Aim for the 15 to 30 that answer what developers actually ask. A list of 2,000 URLs isn’t a map, it’s a sitemap with extra steps.
- Use the H1 for the product or site name, written exactly as people know it.
- Write the blockquote as one or two factual sentences. Plain claims, no slogans. A tool may quote it directly, so write it the way you’d want it quoted.
- Group links under H2 headings that match how people think, such as Getting started, API reference, Guides, and Changelog.
- Give each link a note that says what the page covers. “Our best product!” tells a tool nothing it can use.
- Move lower-priority pages under an Optional heading, since the spec explicitly lets tools skip that section when their budget is tight.
- Serve it at
/llms.txtwith a 200 status code, without a redirect chain, and update it whenever URLs change. - Check that every linked page is readable without a browser. This is the step most people skip. If a linked page is an empty shell until JavaScript runs, the tool gets nothing from it. Fetch each URL with curl and read the output. If the content isn’t in the HTML, serve a Markdown version or fix the rendering.
Failure modes that make llms.txt worse than nothing
A file that’s out of date is a worse signal than no file. The common failures are predictable.
- Stale links. A map with 404s tells a tool your documentation is unmaintained. Validate every link on each release, or generate the file from your docs source so it can’t drift.
- Contradictions with robots.txt. If robots.txt disallows
/docs/and llms.txt lists those pages, you’ve handed a tool a map to the pages you asked crawlers to skip. Decide which you actually mean, and make the two files agree. - Marketing copy as the summary. The blockquote gets read as a description of what you are. “The world’s most powerful platform for growth” is noise. A factual sentence about what the product does is more useful to a reader and to a tool.
- Listing everything. An llms.txt that includes every URL teaches the reader nothing about which pages matter. Curation is the entire point of the format.
- Expecting the file to rescue weak pages. The quality bar still applies page by page. A map doesn’t make a thin page worth reading.
What actually drives AI citations
The file isn’t where citation happens. Answer engines pull from pages that are crawlable, specific, and answer a question directly. For that, the work is the same as it was before anyone proposed a text file: headings that match real questions, claims with numbers and named sources, and HTML that renders without tricks. The selection side is covered in how Google AI Overviews choose their sources, and the content side in the work of getting cited by AI answer engines. Start there before you spend time on a text file.
The honest wrap-up
llms.txt is a small, sensible idea. It’s a curated index of your documentation for tools that a person points at your site, and it’s worth an hour if you run developer docs. It is not a lever for ranking in Google, and as of this writing no major answer engine has committed to using it for citations. Treat any pitch that says otherwise as a claim to check, not a plan to follow.
The discipline underneath is the one that matters for every page you publish: know which question a page answers, make it crawlable, and make it specific enough to be worth quoting. That’s the work AutoSEO Stack is built to automate on your site. It researches the questions your buyers actually search, writes pages that answer them in a human voice, and publishes on a schedule with the quality bar built in. Its AI visibility side covers how your brand shows up in answer engines. The fastest way to judge it is to see it running on your own domain.