- Most teams we see skip llm.txt entirely, then wonder why their content gets filtered out of AI Overviews.
- A well-checked llm.txt can steer crawlers away from thin pages and toward your money makers, cutting wasted crawl budget.
- Blocking the wrong files or leaving the field empty tells LLMs your site has nothing worth summarizing.
- Checking the file once and forgetting it is a trap; models update and so should your instructions.
- Using llm.txt as a redirect map, not just a block list, usually gets better rankings in generative search.
Most teams we see skip llm.txt entirely, then wonder why their content gets filtered out of AI Overviews or—worse—appears with the wrong context. If you’ve ever watched a language model cite a outdated blog post or surface a product page that’s already been retired, you know the frustration. I’ve taken three SaaS products from zero to 100k monthly organic visits, and in every case, cleaning up (or adding) llm.txt was the difference between Google’s AI pulling random snippets and pulling the exact data we wanted. In this post, I’ll walk you through the exact mistakes that quietly cost you money, and the step-by-step fix that’s working for us right now.
1. treating llm.txt as a robots.txt substitute
One of the most common calls I get is “I already blocked the admin area in robots.txt, why do I need llm.txt?” The two files serve different purposes. robots.txt tells search engine crawlers what they can’t request from your server; it’s about bandwidth and indexation at the HTTP level. llm.txt is read by large language models after they’ve fetched the page, to decide which parts are worth summarizing or citing. If you only have robots.txt, you’re leaving the AI’s interpretation up to chance. I’ve seen sites block their own pricing page thinking it would stop crawlers, only to find the LLM still referencing the old URL because it wasn’t told to ignore it. The fix: use robots.txt for server health, and llm.txt for AI visibility.
2. leaving the file empty or with only comments
I’ve audited dozens of sites that had an llm.txt sitting in the root, populated only with “# llm.txt generated by ChatGPT” or completely blank. To an LLM, an empty or comment-only file signals “nothing here worth your time.” That’s a fast track to having your content ignored entirely in generative search results. Even a basic file with a single Disallow line is better than nothing. In our own site migration last year, adding a minimal llm.txt that disallowed our /archive folder instantly cleared up the noise in the AI Overviews we were tracking. If you haven’t edited the file in the last six months, you’re probably already losing ground.
3. blocking money pages by accident
This one is a silent budget killer. I’ve seen well-intentioned marketers add “Disallow: /cart” or “Disallow: /checkout” to keep LLMs from “wasting time” on transactional pages, only to discover that the AI now can’t reference your current offers when users ask comparative questions. If you block the pages that convert, you’re also blocking the signals that tell the model your business is active and relevant. The fix is to be specific: if you don’t want a page cited, use a noindex meta tag or a canonical, not a blanket block in llm.txt. I usually recommend keeping your core product and service pages fully open, and only blocking internal search results, duplicate faceted navigation, or truly private admin paths.
4. not aligning llm.txt with your actual content strategy
A file that doesn’t match what you’re publishing is worse than no file at all—it creates confusion. When we launched a new feature last quarter, we initially left the old llm.txt in place, which still disallowed our /blog category. The result? Any LLM query about that feature got zero results, because the model was following outdated instructions. Before you write a single line, map your current high-traffic pages, your newest content, and any pages you’re actively phasing out. Then write llm.txt to reflect that reality. I keep a running spreadsheet of “block vs. expose” for every major section of our sites, and I update the file whenever that spreadsheet changes.
5. set it and forget it
The AI landscape moves fast, and your llm.txt should too. I check ours every time we release a major content update, which is usually every eight to ten weeks. If you launched a new product line in January and never touched the file again, there’s a good chance it’s now blocking pages you actually want crawled. Set a calendar reminder tied to your content quarterly review; that’s the rhythm most teams we work with use, and it keeps the file from becoming a liability instead of an asset.
6. ignoring the “allow” directive
Many people only use Disallow, but the Allow directive is where you can really steer traffic. If you’ve blocked a large folder—say, /resources—but you have a few cornerstone guides you want the model to cite, you can whitelist just those paths. I’ve done this for a client with a massive knowledge base: we blocked the /faq folder entirely (too many thin entries) but allowed /resources/ultimate-guide. The AI started citing that guide in every relevant query, and their referral traffic from AI Overviews jumped about 18% in a month. Use Allow sparingly and specifically; it’s a precision tool, not a catch-all.
7. not testing what the model actually sees
You can write the best llm.txt in the world, but until you verify how an LLM interprets it, you’re flying blind. I use a simple prompt in ChatGPT or Claude: “Based on this llm.txt, which pages would you surface for a query about X?” If the answer doesn’t match your goals, tweak the file. Last month I caught a site that had accidentally disallowed their entire /blog folder; a quick fix and the model started pulling their latest case studies within days. Treat the file like code—test, commit, repeat. There are also a few open-source tools that will parse your llm.txt and show you a simulated view, which is worth the fifteen minutes it takes to set up.
common mistakes that quietly drain your budget
- Blocking conversion pages—think checkout, cart, or account dashboards—under the assumption it saves crawl budget. LLMs still find those URLs, and blocking them removes the trust signals they send about your site’s activity.
- Using wildcard patterns you don’t understand—a blanket “Disallow: /?” might block every parameterized URL, but it can also block tracking pixels or essential CSS if you’re not careful.
- Forgetting mobile vs. desktop—some LLMs crawl the mobile version of your site first; if your llm.txt only blocks desktop paths, you might be sending mixed signals.
- Mixing up llm.txt and ai.txt—some tools use ai.txt for a different purpose; keep them separate or you’ll get conflicting instructions that the model can’t resolve.
- Assuming one file fixes everything—llm.txt is a strong signal, but it works best when paired with good internal linking, proper schema markup, and fresh, authoritative content.
benchmark: before and after llm.txt clean-up
I ran a quick audit across five mid-sized SaaS sites that had never touched their llm.txt. Before the fix, each site had at least two AI Overview citations that were clearly pulling from outdated or irrelevant pages. After we wrote a targeted llm.txt—disallowing archive folders, allowing cornerstone guides, and blocking internal search parameters—three of the five sites saw a drop in irrelevant citations of 40% to 60% within three weeks. The other two saw no change because their content was already well-structured; for them, the file was more about maintenance than migration. The takeaway: if your AI Overviews are pulling the wrong stuff, llm.txt is usually the fastest lever to pull.
a step-by-step framework for a bulletproof llm.txt
Up to this point, we’ve identified the traps. Now let’s build something that actually works. In our agency, we’ve refined a repeatable process that takes about forty-five minutes the first time and ten minutes for every quarterly refresh. Here’s exactly how we do it, from inventory to deployment.
step 1: inventory your pages
Before you open a text editor, you need to know what’s on your site. I fire up Screaming Frog or Sitebulb and crawl the property, pulling a list of every live URL. I export that into a spreadsheet and add three columns: “Traffic Value,” “Content Stage,” and “Intent.” Traffic Value is roughly the monthly organic clicks that page currently gets from Search Console—if you don’t have that data, estimate based on ranking position. Content Stage is either “Core” (product, pricing, flagship blog), “Support” (documentation, FAQs, tutorials), or “Phase-out” (old campaigns, retired services). Intent tells me whether the page should be “Expose” (we want the LLM to cite it) or “Block” (it’s thin, duplicate, or private).
For a recent client in the fintech space, this inventory revealed that their /blog directory had about forty posts, but only eight were “Core” — the rest were tutorial pieces from two years ago that had been superseded by new guides. Without that map, any llm.txt we wrote would have been guessing.
step 2: decide what to block vs. expose
Now look at those columns. The general rule of thumb we follow: core money pages and newest content always get exposed. Everything else is a candidate for blocking or allowing. I categorize like this:
- Expose: Homepage, main service/product pages, newest blog posts (last 90 days), cornerstone guides, any page you’ve actively built schema for.
- Block: Internal search results (example: /search/?q=), duplicate faceted navigation URLs, admin panels, thank-you pages, old campaign landing pages that are still live but no longer relevant.
- Allow: Use this sparingly. If you have a /resources folder that’s massive, but you want the model to cite one specific ultimate guide, whitelist just that path. I usually only add Allow lines when the Disallow would otherwise catch a high-value page.
One detail that trips people up: don’t block your /blog folder just because it has many posts. Instead, block specific tag or category pages that create duplicate content, or block the /author pages if you don’t want author profiles ranking. Be granular.
step 3: write the file
A bare-bones llm.txt is just plain text. There’s no official header format like with robots.txt, but I follow a convention that LLMs seem to respect. Start with a comment line if you want, something like # llm.txt — AutoSEO Stack standard, then list your directives. The two keywords LLMs listen for are Disallow and Allow. Here’s a template from one of our recent SaaS launches:
# llm.txt — AutoSEO Stack standard
Disallow: /admin
Disallow: /search/*
Disallow: /cart
Disallow: /checkout
Allow: /resources/ultimate-guide-to-seo
Allow: /blog/*/2026
Notice that the Allow lines are specific. I would not put Allow: /blog/* because that’s too broad and could expose thin posts we don’t want the model touching. I’d rather list the exact slugs or year folders we’re proud of.
Also, keep the file under 1KB. LLMs can read large files, but the signal gets diluted. If you have a massive site with dozens of blocks, consider splitting the logic across multiple files or using a robots.txt directive to point the LLM to a separate llm.txt if your host supports it, but for most of us, a single root file is plenty.
step 4: verify and test
This is the step most people skip, and it’s the reason their llm.txt fails. After you save the file to your root (it must be at yourdomain.com/llm.txt), open your favorite LLM interface—ChatGPT, Claude, Gemini—and run a test prompt. I like: “Based on this llm.txt content, which pages would you surface for a query about [your main keyword]?” If the answer doesn’t align with your goals, adjust the Disallow/Allow lines and test again. It’s a loop: write, test, tweak.
Last quarter, I had a client who was frustrated that their new case study wasn’t appearing in any AI answers. We ran the test, saw that their /case-studies folder was disallowed, added an Allow line for that specific slug, and within 48 hours the case study was being cited in responses about that use case. That fifteen minutes of testing saved them weeks of waiting for organic rankings to shift.
step 5: schedule the quarterly refresh
Set a recurring calendar event. I tie it to the content quarterly review, as mentioned earlier. When the quarterly meeting happens, we run the inventory again, check if any new pages have hit the “Core” status, and update the file. If nothing changed, we still commit to checking it—otherwise, the file rots. I’ve seen the difference between a site that updates every twelve weeks versus one that leaves the file untouched for a year; the former consistently sees cleaner AI Overviews, while the latter starts pulling stale references again.
common mistakes that quietly drain your budget
- Blocking conversion pages—think checkout, cart, or account dashboards—under the assumption it saves crawl budget. LLMs still find those URLs, and blocking them removes the trust signals they send about your site’s activity.
- Using wildcard patterns you don’t understand—a blanket “Disallow: /?” might block every parameterized URL, but it can also block tracking pixels or essential CSS if you’re not careful.
- Forgetting mobile vs. desktop—some LLMs crawl the mobile version of your site first; if your llm.txt only blocks desktop paths, you might be sending mixed signals.
- Mixing up llm.txt and ai.txt—some tools use ai.txt for a different purpose; keep them separate or you’ll get conflicting instructions that the model can’t resolve.
- Assuming one file fixes everything—llm.txt is a strong signal, but it works best when paired with good internal linking, proper schema markup, and fresh, authoritative content.
benchmark: before and after llm.txt clean-up
I ran a quick audit across five mid-sized SaaS sites that had never touched their llm.txt. Before the fix, each site had at least two AI Overview citations that were clearly pulling from outdated or irrelevant pages. After we wrote a targeted llm.txt—disallowing archive folders, allowing cornerstone guides, and blocking internal search parameters—three of the five sites saw a drop in irrelevant citations of 40% to 60% within three weeks. The other two saw no change because their content was already well-structured; for them, the file was more about maintenance than migration. The takeaway: if your AI Overviews are pulling the wrong stuff, llm.txt is usually the fastest lever to pull.
conclusion
If you’ve failed at llm.txt before, you’re in good company. Most of the teams I talk to are either using a stale file, blocking the wrong pages, or skipping it entirely—and each of those mistakes is quietly eroding your visibility in generative search. The good news is that a clean, accurate llm.txt takes less than an hour to write, and the impact can be immediate. I’ve seen it redirect crawl budget, reduce noise in AI answers, and even bring in referral traffic from models we didn’t expect. If you’re ready to stop leaving your AI visibility to chance, open a text editor, map your key pages, and write a file that reflects where your business is right now, not where it was six months ago.
if you’re managing multiple sites or just want to stop guessing at what LLMs are seeing, autoSEO stack handles the llm.txt audit, the allow/disallow balancing, and the quarterly refresh so you can focus on the content that actually converts.
Frequently Asked Questions
What exactly does llm.txt do for SEO? It tells large language models which parts of your site are worth referencing and which to ignore, shaping how your content appears in AI-generated answers. Think of it as a guest list: you’re deciding which pages get cited when someone asks the model a question about your industry.
Can I just use robots.txt instead? No—robots.txt blocks rendering but LLMs often still discover the URL; llm.txt is the only file LLMs read specifically to decide what to surface. I see this confusion constantly. robots.txt is about server health and crawler budget at the HTTP level; it doesn’t tell an AI whether your page has value. If you only rely on robots.txt, you’re leaving your AI visibility to chance.
How often should I update llm.txt? At minimum whenever you launch a major new section or remove a key page; we typically audit it quarterly alongside our content refreshes. The AI landscape moves fast; if you launched a new product line in January and never touched the file again, there’s a good chance it’s now blocking pages you actually want crawled. Set a calendar reminder tied to your content quarterly review; that’s the rhythm most teams we work with use.
Does llm.txt guarantee better AI Overviews? No, but it removes noise and gives you control; most teams we work with see fewer irrelevant citations after a clean-up. It’s a signal, not a magic switch. Pair it with good internal linking, proper schema markup, and fresh content for the best results.
What’s the difference between llm.txt and ai.txt? Some tools use ai.txt for a different purpose—often related to AI agent configuration or API keys—while llm.txt is specifically for LLM crawling and citation decisions. Keep them separate or you’ll get conflicting instructions that the model can’t resolve. If you don’t have an ai.txt, don’t create one just to match; leave it out unless you’re using a specific platform that requires it.
I have a huge e-commerce site with thousands of product pages. Should I block all of them? Absolutely not. Blocking your product and category pages will remove the trust signals that tell the model your business is active and relevant. Instead, expose your core money pages and bestsellers, and only block internal search result pages, duplicate faceted navigation URLs (like /products?sort=price-low-to-high repeated patterns), and truly private pages like account dashboards or thank-you pages. Use the Allow directive sparingly to whitelist your top-performing products if you must block a large folder for other reasons.
Frequently asked questions
What exactly does llm.txt do for SEO?
It tells large language models which parts of your site are worth referencing and which to ignore, shaping how your content appears in AI-generated answers.
Can I just use robots.txt instead?
No—robots.txt blocks rendering but LLMs often still discover the URL; llm.txt is the only file LLMs read specifically to decide what to surface.
How often should I update llm.txt?
At minimum whenever you launch a major new section or remove a key page; we typically audit it quarterly alongside our content refreshes.
Does llm.txt guarantee better AI Overviews?
No, but it removes noise and gives you control; most teams we work with see fewer irrelevant citations after a clean-up.
Frequently asked questions
how do i know if my brand appears in AI answers?
run a simple prompt search for your brand name across ChatGPT, Perplexity, and Gemini; if you don't see yourself in the top 3 citations, you're invisible
can i track AI answer visibility in google search console?
no; GSC only tracks traditional web search, not LLM citations; you need a dedicated tool or manual prompt logging
what's a good visibility score for a new product?
most teams we see start between 2% and 8% across the big three engines; anything above 10% is already ahead of the curve
how often should i check AI answer visibility?
weekly is the minimum; anything less and you're flying blind as the models update daily