
- The llms.txt specification mandates serving the file directly from the domain root, not a subdirectory or an unmapped SPA route.
- Nginx does not recognize markdown MIME types by default and will often force a binary download unless explicitly configured with text/markdown.
- Single-page application try_files directives will swallow /llms.txt requests and return your index.html with a 200 OK status unless matched specifically.
- Reverse proxy setups (like Cloudflare or AWS ALB in front of Nginx) require cache-control overrides and exact URI passing to avoid serving stale upstream content.
Place your llms.txt file directly in your website document root, such as /var/www/html/llms.txt or /usr/share/nginx/html/llms.txt, so it resolves publicly at yourdomain.com/llms.txt. AI user agents expect it at the root alongside robots.txt. If your site uses a frontend framework where assets live inside a public or dist subfolder, put it there so the build pipeline copies it to the production web root. Never hide it inside an /assets/ or /docs/ directory unless you write an explicit Nginx rewrite rule to map it back to the root.
Where should llms.txt be placed in the web server directory?
Most engineering teams we talk to lose between two hours and two days debugging why automated scrapers refuse to read their site documentation. The engineer tests the URL in Chrome, sees lines of clean markdown text, assumes the job is done, and closes the ticket. Three weeks later, an inspection of access logs reveals that every major AI crawler hitting the endpoint either bailed after reading the response headers or choked on a rendered single-page application shell.
Figuring out exactly where to place your llms.txt file on Nginx sounds trivial until you remember how Nginx handles static file resolution, fallback routing, and MIME types. If your web server serves a modern web app, sits behind a reverse proxy, or handles strict rewrite rules, dropping a flat text file into a directory rarely works out of the box.
Request: GET /llms.txt
│
▼
┌──────────────────────────────────────────────┐
│ Nginx Server Block Resolution │
├──────────────────────────────────────────────┤
│ 1. Match: location = /llms.txt │
│ └─► Found? Serve static file directly │
│ 2. Match: try_files $uri $uri/ /index.html │
│ └─► Missing file? Serves HTML shell (BUG) │
│ 3. Header check: Content-Type │
│ └─► Missing type? application/octet-stream│
└──────────────────────────────────────────────┘
The standard for /llms.txt demands that the file live at the absolute root of your domain. Not in a /docs/ subfolder, not hidden behind an API gateway, and not routed through an unhandled dynamic controller that forces chunked transfer encoding errors. Let us walk through the physical server paths, configuration blocks, and edge cases required to serve this file cleanly.
The Physical Directory: Where to Place Your llms.txt File on Nginx
When deploying a flat file to an Nginx server, the physical location on disk depends entirely on the root directive defined inside your active server block. Unlike traditional shared hosting environments with fixed public directories, modern Linux servers distribute web roots based on your distribution, container setup, or deployment pipeline.
In a clean Debian or Ubuntu installation, standard static files reside in /var/www/html or /var/www/yourdomain/public. On Red Hat Enterprise Linux, AlmaLinux, or Alpine-based Docker containers, default roots often point to /usr/share/nginx/html.
/var/www/yourdomain.com/
├── public/ <-- Document root (nginx `root` directive)
│ ├── index.html
│ ├── robots.txt
│ ├── sitemap.xml
│ ├── llms.txt <-- Place primary summary file here
│ └── llms-full.txt <-- Place complete context file here
├── src/
├── package.json
└── nginx.conf
To verify where Nginx looks for files on your production machine, search your active site configuration:
grep -R "root" /etc/nginx/sites-enabled/
If the output points to /var/www/site/dist, that compiled directory is where the file belongs. Putting the file into /var/www/site/ will cause an immediate 404 Not Found error because Nginx constructs the file path by appending the request URI directly to the root path:
$$\text{File Path} = \text{root path} + \text{request URI}$$
If root is set to /var/www/site/dist, a request for /llms.txt resolves to /var/www/site/dist/llms.txt. If your build process (like Vite, Webpack, or Next.js static exports) wipes the destination directory during deployments, you must store llms.txt in your source directory’s /public folder so your build pipeline copies it across on every run. If you miss this step, you will fall into the trap discussed in our guide on why your llms.txt is failing (and the fix).

Configuring the Server Block for llms.txt
Relying on generic static file matching is a mistake. Most Nginx sites running modern applications use fallback directives, aggressive asset caching, or security headers that break text endpoints. You need an explicit, isolated location block for /llms.txt and its companion file, /llms-full.txt.
Here is an ideal configuration snippet for your site’s server block:
server {
listen 443 ssl http2;
server_name example.com;
root /var/www/example.com/public;
# Primary entry for llms.txt and llms-full.txt
location ~* ^/llms(-full)?\.txt$ {
# Ensure correct MIME type delivery
types { } default_type "text/markdown; charset=utf-8";
# Explicit file handling
try_files $uri =404;
# Access rules
allow all;
# Headers for automated agents and proxies
add_header Cache-Control "public, max-age=3600, stale-while-revalidate=86400";
add_header Access-Control-Allow-Origin "*";
add_header X-Content-Type-Options "nosniff";
# Turn off access logging to avoid log bloating from heavy scrapers
access_log off;
log_not_found off;
}
# Your standard application fallback remains untouched below
location / {
try_files $uri $uri/ /index.html;
}
}
This configuration achieves four essential goals:
1. It matches requests for both /llms.txt and /llms-full.txt using a case-insensitive regular expression.
2. It strips inherited MIME type configurations and explicitly forces a text/markdown; charset=utf-8 header.
3. It uses try_files $uri =404; to prevent missing files from silently falling through to an index routing script.
4. It enables Cross-Origin Resource Sharing (Access-Control-Allow-Origin: *), which is mandatory for browser-based AI extensions and web workers trying to ingest your file directly.
The MIME Type and Encoding Pitfall
The single most common silent error we see during audits is an incorrect Content-Type header.
By default, Nginx reads its file extension associations from /etc/nginx/mime.types. In almost every default package distribution, .txt files map to text/plain. While text/plain is acceptable to some parsers, the current proposed draft standard for /llms.txt specifies markdown as the preferred content format. Worse, if your Nginx installation does not have a mapping for the file at all, it drops back to the default fallback: application/octet-stream.
When an HTTP client receives application/octet-stream, it assumes the resource is arbitrary binary data. Web browsers will trigger a “Save As” download dialogue, and automated AI scrapers will abort parsing immediately to prevent loading unvetted binaries into their ingestion pipelines.
| Header Combination | Crawler Behavior | Browser Behavior | Production Status |
|---|---|---|---|
text/markdown; charset=utf-8 |
Parses markdown structures cleanly | Renders as raw plain text | Target standard |
text/plain; charset=utf-8 |
Parses text, may miss markdown nuance | Renders as plain text | Acceptable fallback |
application/octet-stream |
Fails silently / Aborts request | Downloads file to disk | Broken |
text/html; charset=utf-8 |
Rejects file as malformed | Renders HTML shell | Broken (SPA leak) |
To guarantee your server never returns the wrong header, wipe the local MIME table inside the location block using empty brackets types { } and set default_type "text/markdown; charset=utf-8";.
Never forget the character encoding attribute. If your markdown file contains em-dashes, non-ASCII bullet characters, or international translations, omitting charset=utf-8 causes byte misinterpretations when tokenizers ingest your text. For a deeper understanding of standard specifications, read our breakdown on llms.txt explained: what it is and whether you need one.

Handling SPAs and Reverse Proxies
If your Nginx instance operates purely as an origin server behind Cloudflare, AWS CloudFront, or Fastly, or if it acts as a reverse proxy passing traffic to a Node.js, Go, or Python backend, routing rules become complicated.
External Scraper
│
▼
┌──────────────┐
│ CDN / Proxy │ (Caches based on edge headers)
└──────┬───────┘
│
▼
┌──────────────┐
│ Nginx Server │ ──► Does /llms.txt exist physically?
└──────┬───────┘ ├─► YES: Return static file
│ └─► NO: Forward to upstream Node/Next.js app
▼
┌──────────────┐
│ Upstream App │ ──► Catches unmatched route, returns 200 OK with HTML
└──────────────┘
The Single-Page Application (SPA) Trap
In a React, Vue, or Angular deployment, the standard Nginx configuration looks like this:
location / {
try_files $uri $uri/ /index.html;
}
If you deploy your site and forget to upload llms.txt, or place it in the wrong physical directory, Nginx checks if the file exists on disk. When it finds nothing, it proceeds to the next parameter: /index.html.
Your server returns an HTTP status code 200 OK, but the body payload is your full HTML shell containing client-side JavaScript tags. An automated crawler requesting /llms.txt receives this HTML response, marks the markdown parsing operation as failed, and abandons your site. Because the server returns a 200 status code instead of a 404, monitoring tools will not alert you to the failure.
To prevent this, ensure your location block specifies an explicit =404 termination:
location = /llms.txt {
root /var/www/site/dist;
try_files $uri =404;
}
Reverse Proxying to a Headless Backend
If your Nginx server forwards all traffic to an upstream application server (such as Next.js running on localhost:3000), you should intercept the request at the Nginx layer. Do not waste application server worker threads rendering flat markdown files.
# Handle static AI index at the edge Nginx layer
location = /llms.txt {
alias /var/www/static/llms.txt;
types { } default_type "text/markdown; charset=utf-8";
add_header Cache-Control "public, max-age=3600";
}
# Pass all other traffic upstream to application
location / {
proxy_pass http://127.0.0.1:3000;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
Testing and Verification via Log Files
Do not trust a web browser to verify your Nginx setup. Browsers hide missing headers, automatically repair broken character encodings, and mask incorrect content types. Instead, test your live endpoint using curl directly against your server.
curl -I -X GET "https://example.com/llms.txt" -H "User-Agent: Mozilla/5.0"
Look at the raw HTTP response headers. You are looking for a few specific fields:
HTTP/2 200
server: nginx
date: Wed, 21 Oct 2026 14:12:00 GMT
content-type: text/markdown; charset=utf-8
content-length: 1420
last-modified: Wed, 21 Oct 2026 12:00:00 GMT
cache-control: public, max-age=3600, stale-while-revalidate=86400
access-control-allow-origin: *
x-content-type-options: nosniff
If content-type reads text/html, your SPA rewrite engine is still swallowing the request. If it reads application/octet-stream, your MIME types configuration is missing.
Next, monitor your access logs to observe how AI crawlers interact with the file. Run this command on your Nginx host:
tail -f /var/log/nginx/access.log | grep -E "llms(-full)?\.txt"
You can simulate a crawler request by sending a custom User-Agent string:
curl -s -A "GPTBot/1.2" https://example.com/llms.txt -o /dev/null -w "%{http_code}\n"
If this returns anything other than 200, examine /var/log/nginx/error.log immediately. The most common errors and their exact root causes are detailed below.
Error Message in error.log |
Root Cause | Direct Fix |
|---|---|---|
open() "/var/www/..." failed (2: No such file or directory) |
Physical file is missing or root directive path is wrong |
Check file path; move file into the configured web root |
directory index of "/var/www/..." is forbidden |
Path matches a directory instead of the file | Ensure your location pattern does not point to an empty subfolder |
*1 open() failed (13: Permission denied) |
Nginx worker process cannot read the file | Run chmod 644 llms.txt and verify folder ownership |
client intended to send too large body |
Not applicable to GET; misconfigured proxy buffer | Check upstream buffer sizes in your core nginx.conf |
Implementation Checklist
To avoid deployment failures, follow this sequence when placing and configuring your file on an Nginx environment.
- Verify your physical document root: Check your active Nginx server configuration (
nginx -T) to locate the exact path specified by therootdirective for your primary domain. - Generate clean markdown files: Create both
llms.txtandllms-full.txtlocally, verifying they contain standard Markdown formatting without invalid unescaped control characters. - Move files to production root: Copy the files to your active web root using secure copy or your automated CI/CD pipeline (for example,
/var/www/site/public/). - Set correct Linux file permissions: Execute
chmod 644 llms.txtandchown www-data:www-data llms.txt(or the equivalent user for your OS, such asnginx:nginx) so the Nginx worker process can read the file. - Add the dedicated location block: Insert the
location ~* ^/llms(-full)?\.txt$directive inside the main HTTPS server block in your site configuration file. - Set markdown MIME headers: Ensure
types { } default_type "text/markdown; charset=utf-8";is declared within the block to override global server MIME limits. - Bypass single-page application fallback rules: Explicitly include
try_files $uri =404;to stop unmatched static requests from loading your JavaScript bundle. - Configure CORS permissions: Add the
Access-Control-Allow-Origin "*"response header to allow client-side AI scrapers and extensions to ingest the file directly. - Validate Nginx syntax: Run
nginx -ton your server to verify there are no syntax errors or duplicate block declarations before reloading the service. - Gracefully reload the web server: Run
systemctl reload nginx(ornginx -s reload) to apply the configuration changes without dropping active TCP connections. - Perform an external HTTP header check: Query the endpoint using
curl -I https://example.com/llms.txtto confirm that the server returns HTTP 200 with thetext/markdownheader intact.
For a wider overview of site architecture requirements beyond server configuration, check out llms.txt: the complete checklist and avoid the revenue hits covered in llms.txt mistakes that quietly cost you money.
Maintenance and Updates
Your llms.txt file is not a static asset you set once and ignore for two years. As your technical documentation, products, and landing pages evolve, your summary markdown file must mirror those updates. Failing to maintain this file leads to stale LLM responses, as explained in our analysis of why most llms.txt advice is outdated.
Cache Expiration Strategy
Because AI scrapers ingest these files to build runtime context, your caching policy should balance origin server load with content freshness.
Do not set aggressive multi-week browser caching on llms.txt. An ideal caching policy uses a short max-age with a generous stale-while-revalidate window:
add_header Cache-Control "public, max-age=3600, stale-while-revalidate=86400";
This instruction allows edge proxies and CDN nodes to serve the cached file for up to one hour (max-age=3600). If a request arrives after that hour, the edge server returns the stale version instantly while fetching a fresh copy from Nginx in the background. If you need a comprehensive reference on file structure and ongoing maintenance patterns, refer to llms.txt: a practitioner’s field guide.
File Size and Token Consumption
Keep your root /llms.txt file compact. Its purpose is to serve as a curated, high-density table of contents for AI models, providing links to deeper resources. If your root file grows beyond 50 KB, split your extended technical documentation into /llms-full.txt or separate topic-specific markdown files.
Request: GET /llms.txt (Lightweight Index: ~10KB)
│
├──► Link: /docs/authentication.md
├──► Link: /docs/billing-api.md
└──► Link: /llms-full.txt (Comprehensive File: ~200KB)
Keep your primary index clean and maintainable. This preserves crawl budget, keeps token usage low for small-context agents, and ensures your server processes requests with minimal resource overhead.
What to do this week
- Run
curl -I https://yourdomain.com/llms.txtright now to check your live HTTP status and confirm yourContent-Typeheader is not returningapplication/octet-streamortext/html. - Inspect your primary Nginx server configuration for any
try_filesdirectives that could swallow missing text requests and convert them into SPA fallback routes. - Check file permissions across your document root to verify that your active Nginx worker user has read access (
chmod 644) to bothllms.txtandllms-full.txt. - Add the dedicated regex location block provided above, reload Nginx using
nginx -t && systemctl reload nginx, and test with a simulated AI user-agent string.
If managing server blocks, build pipeline exports, and custom MIME headers across dynamic documentation sounds like unnecessary operational overhead, AutoSEO Stack automates and validates these deployment workflows across your infrastructure.
Frequently asked questions
Where should llms.txt be placed in the web server directory?
Place your llms.txt file directly in your website document root, such as /var/www/html/llms.txt or /usr/share/nginx/html/llms.txt, so it resolves publicly at yourdomain.com/llms.txt. AI user agents expect it at the root alongside robots.txt. If your site uses a frontend framework where assets live inside a public or dist subfolder, put it there so the build pipeline copies it to the production web root. Never hide it inside an /assets/ or /docs/ directory unless you write an explicit Nginx rewrite rule to map it back to the root.
Why does Nginx download my llms.txt file instead of displaying it?
Nginx triggers a file download when it serves an unknown MIME type, sending Content-Type: application/octet-stream instead of a text format. Nginx ships with a mime.types file that historically lacks a mapping for .txt or markdown variants like .md. When an LLM crawler or browser requests /llms.txt, the browser does not render it as plain text. You fix this by explicitly setting 'types { text/markdown txt; }' or adding 'default_type text/plain;' inside the specific location block for /llms.txt in your nginx.conf file.
Can I place llms.txt inside the .well-known folder on Nginx?
While the original /llms.txt standard specifies the absolute root of the origin, some draft extensions propose fallback support inside /.well-known/llms.txt. However, you should always serve it directly at /llms.txt first. If you want to support both standards without maintaining duplicate files, place the actual physical file at your document root and configure an Nginx rewrite or alias directive that routes requests for /.well-known/llms.txt directly to the root file. Serving it exclusively inside /.well-known/ will cause major LLM crawlers to miss it entirely.
Why does my React or Vue app on Nginx return an HTML page for llms.txt?
This occurs because modern SPA routing relies on a catch-all 'try_files $uri $uri/ /index.html;' directive in your Nginx configuration. If the physical llms.txt file is missing from the build output directory, Nginx fails to find the static asset and falls back to serving your single-page app shell with a 200 OK status code. The AI crawler attempts to parse your HTML and JavaScript bundle as markdown and fails. You must place an explicit 'location = /llms.txt' block above your SPA fallback directive.
How do I test if my Nginx llms.txt configuration is working?
Test the endpoint using curl in your terminal: 'curl -I https://yourdomain.com/llms.txt'. Verify three critical elements in the response headers: an HTTP status of 200 OK, a Content-Type header showing 'text/markdown; charset=utf-8' or 'text/plain; charset=utf-8', and proper cache headers. Avoid testing solely in a desktop browser because desktop browsers automatically render raw text even if the Content-Type header is completely misconfigured, hiding server-side configuration mistakes that cause programmatic AI scrapers to reject your file.
Should llms.txt and llms-full.txt be in the same Nginx directory?
Yes, both files should reside side by side in your primary document root directory. The llms-full.txt file acts as the comprehensive, unabridged companion to your summary index file. Both must be served with identical text MIME types and cache headers. In your Nginx configuration, you can use a single regex location block like 'location ~* ^/llms(-full)?\\.txt$' to apply the exact same headers, character sets, and access rules to both files simultaneously without repeating configuration directives.
Frequently asked questions
how to track ai answer visibility?
Track impressions, citations, and ranking shifts for your AI-generated content.
how to monitor ai answer visibility?
Use tools like AutoSEO Stack to log citations and visibility trends automatically.
can ai detect website traffic?
No, AI models don’t detect traffic; they use public signals and citations to generate answers.
how to improve ai visibility?
Create clear, citation-friendly content that directly answers user questions.
how to track ai answers?
Monitor how often your content is referenced in AI-generated responses across platforms.
Frequently asked questions
How do I track AI answer visibility for my brand?
Set up a weekly scan using a dedicated visibility tool, then compare appearance rates month-over-month rather than chasing absolute numbers.
What's a good AI visibility benchmark?
Most B2B SaaS brands sit between 8-15% appearance rate in their primary categories; e-commerce typically sees 3-8% depending on product specificity.
Can AI visibility tracking predict traffic changes?
Yes—watch for appearance rate shifts of 3% or more combined with ranking position changes; that combination usually precedes a traffic shift within 30 days.
Should I optimize content specifically for AI answers?
Focus on answering user questions directly in the first 100 words of each page; AI models prioritize clear, concise answers over keyword-stuffed content.
Frequently asked questions
how do i know if my brand appears in AI answers?
run a simple prompt search for your brand name across ChatGPT, Perplexity, and Gemini; if you don't see yourself in the top 3 citations, you're invisible
can i track AI answer visibility in google search console?
no; GSC only tracks traditional web search, not LLM citations; you need a dedicated tool or manual prompt logging
what's a good visibility score for a new product?
most teams we see start between 2% and 8% across the big three engines; anything above 10% is already ahead of the curve
how often should i check AI answer visibility?
weekly is the minimum; anything less and you're flying blind as the models update daily