- AI companies run different crawlers for different purposes, training, live retrieval, and indexing, each with its own user-agent, and robots.txt lets you allow or disallow them individually.
- The crucial distinction is between crawlers that gather training data and crawlers that fetch your page to answer a live question; blocking the wrong one can quietly cost you citations.
- robots.txt is a public, honor-system instruction file: well-behaved crawlers respect it, so use it deliberately to permit the crawlers whose reading you want and restrict the ones you do not.
Every bot that visits your site introduces itself. Along with each request, a crawler sends a user-agent, a short string that names who it is, and that small piece of identification is the key to controlling what the AI crawlers reading your site are allowed to do. If you want to decide which AI bots may read your content and which may not, the mechanism starts with knowing their names and ends with a file called robots.txt, where you write the instructions they read.
This sounds simple, and in its basics it is, but there is a subtlety that catches many people out: AI companies do not run a single crawler each. They run several, for different purposes, and those crawlers deserve different treatment. Getting this right means allowing the crawlers whose reading you want while restricting the ones you do not, and the single most expensive mistake is blocking the crawler that would have cited you because you confused it with one you meant to block. Understanding the landscape is how you avoid that.
What a user-agent is and why it matters
A user-agent is simply an identifying string that a crawler sends with every request it makes, announcing itself to your server. It is how your logs know that a particular request came from a particular bot, and it is how robots.txt rules target a specific crawler. Each AI company names its crawlers, and those names are the handles you use to allow or disallow them. Without knowing a crawler's user-agent, you cannot write a rule that applies to it specifically; with it, you can control it precisely.
This is the foundation of deliberate crawler control. Because every legitimate crawler identifies itself this way, you can see exactly which bots are reaching your site by looking at the user-agents in your logs, and you can write targeted instructions for each one. The user-agent is the common thread running through both halves of the task, diagnosis, seeing who visits, and control, telling them what they may read. Everything else builds on knowing these names and using them.
The crucial distinction: training versus retrieval
Here is the distinction that matters most, and that trips up the most people. AI companies typically run at least two kinds of crawler, and they do different jobs. One kind gathers content broadly to train or improve the AI models, an ongoing, large-scale data-collection effort. Another kind fetches your specific page in real time, because a user asked a question your page can answer, and the assistant wants to read your current content to respond, often citing you in the process.
These two activities are genuinely different, and you may well want to treat them differently. Blocking the training crawler is a choice some make to keep their content out of model training, and it is a defensible decision with its own trade-offs. But blocking the retrieval crawler is a very different matter: that is the bot that fetches your page to answer a live question and cite you, so blocking it directly costs you the citations you are presumably trying to earn. Confusing the two, and blocking retrieval when you only meant to block training, is a costly and common error.
How robots.txt controls them
robots.txt is a plain text file at the root of your site that gives crawlers instructions about what they may and may not access. It works by naming a user-agent and then specifying what that user-agent is allowed or disallowed to read. You can write rules that apply to all crawlers, or rules that target a specific one by its user-agent name, which is exactly what you need to treat different AI crawlers differently. This is where your decisions about each crawler get expressed.
The structure is straightforward: you identify the crawler you want to address by its user-agent, and you state what it may or may not crawl. Because you can target crawlers individually, you can, for example, allow one company's retrieval crawler full access while disallowing its training crawler, or permit some crawlers and restrict others entirely. The file is read by each crawler when it visits, and a cooperative crawler follows the instructions that apply to it. The precision comes from targeting the right user-agent with the right rule.
The honor-system caveat
There is an important limitation to understand: robots.txt is an honor-system standard, not a technical enforcement mechanism. It does not physically block anything. It is a set of instructions that well-behaved crawlers choose to read and respect, and the major, legitimate AI companies' crawlers generally do respect it. But a crawler that ignores robots.txt faces no technical barrier from the file itself, because the file only works through voluntary compliance.
This matters for setting expectations correctly. For controlling the legitimate AI crawlers you most care about, robots.txt is the right and sufficient tool, because those crawlers respect it. But if your goal is to hard-block a crawler regardless of whether it cooperates, robots.txt alone will not do it; that requires enforcement at the server or security layer. Knowing this distinction keeps you from relying on robots.txt for a job it cannot do, while still using it for the job it does well: instructing the cooperative crawlers that make up the ones whose behavior you are actually trying to shape.
Finding out which crawlers visit you
Before you write rules, it helps to know which crawlers are actually reaching your site, and your server logs tell you. Every request records the user-agent that made it, so by looking at your logs you can see which AI crawlers are visiting, how often, and what responses they are getting. This turns crawler control from guesswork into something grounded in reality: you are writing rules for the crawlers that genuinely visit you, and you can verify that your rules are having the intended effect.
This log-based view is also how you catch problems. If a crawler you want is not appearing in your logs at all, something upstream may be blocking it before it reaches your content. If a crawler is appearing but getting errors, you have a different problem to solve. The logs are the ground truth of who is visiting and how they fare, and consulting them makes every decision about robots.txt and crawler access an informed one rather than a shot in the dark.
Deciding your policy deliberately
With the landscape understood, the real task is deciding your policy on purpose rather than by accident. For most businesses that want to be cited by AI, the sensible default is to allow the retrieval crawlers, the ones that fetch your page to answer live questions and cite you, because blocking those directly undermines the goal. The decision about training crawlers is more genuinely a matter of preference and values, and reasonable organizations land in different places depending on how they feel about their content being used in model training.
The key word is deliberately. The worst outcomes come from crawler policies that happen by accident, a blanket block that catches retrieval crawlers along with training ones, a rule copied from somewhere without understanding what it does, or a default configuration no one examined. Making a conscious choice, allow these, restrict those, for reasons you understand, is how you ensure your robots.txt serves your actual goals rather than quietly working against them. The tool is precise; the responsibility is to use it with intent.
Common mistakes to avoid
A few errors show up repeatedly. The most damaging is the overly broad block: a rule intended to keep content out of training that, because it is not targeted to the right user-agent, ends up blocking retrieval crawlers too, silently costing citations. Another is the copied configuration: pasting someone else's robots.txt rules without understanding which crawlers they allow or block, and inheriting decisions that do not match your goals. Both come from acting without understanding what each rule actually does.
A subtler mistake is neglecting to revisit the file as the landscape changes. New crawlers appear, companies change their crawler names or add new ones, and a robots.txt written once and forgotten can become out of date, either failing to address a new crawler or restricting one under an old assumption. Treating the file as a living policy that you review periodically, rather than a one-time setup, keeps it aligned with both the current crawler landscape and your current intentions. Awareness and periodic review prevent most of these problems.
Why some sites block everything, and why that is usually wrong for visibility
Some sites take the blunt approach of disallowing all crawlers or all AI crawlers, reasoning that it is the safe, simple default. For a site whose goal is privacy or exclusivity, that can be a coherent choice. But for a business that actually wants to appear in AI answers, a blanket block is self-defeating, because it shuts out the retrieval crawlers that would otherwise fetch the page and cite it. The simplicity is appealing, but it trades away exactly the visibility the business is trying to build.
The better posture for a visibility-seeking site is openness by default to the crawlers that cite, with restrictions applied only where there is a specific reason. Rather than starting from a total block and reluctantly opening holes, start from allowing the retrieval crawlers you want and add restrictions deliberately, for instance on training crawlers if that matches your values. This keeps the door open to citation while still giving you control over the uses you genuinely want to limit, which is almost always the configuration that serves a business wanting to be found.
Keep the file simple and readable
robots.txt rewards simplicity. A file that is short, clearly organized, and easy to read is easier to get right and far easier to maintain than a sprawling one full of rules no one remembers the reason for. Complexity is where mistakes hide: an overly intricate file can contain a rule that contradicts another, or a stray directive that blocks something you meant to allow, and the more tangled it is, the harder those errors are to spot. Clarity in the file is clarity in your policy.
Aim to write rules you could explain to someone else in a sentence each, grouped so that it is obvious which crawler each rule addresses. When you revisit the file later, that readability is what lets you confirm quickly that it still does what you intend. A robots.txt that you can read and understand at a glance is one you can keep aligned with your goals; one that has grown into an inscrutable thicket is one that will eventually work against you without your noticing. Favor the simple version.
What robots.txt cannot do for you
It is worth being clear about the boundaries of this tool, so you do not expect it to solve problems it cannot. robots.txt controls whether cooperative crawlers are permitted to read your content; it does nothing about whether they can technically read it once permitted. If your content is invisible because it is built in the browser, or your server is blocking crawlers at the security layer, or your pages are too slow to crawl reliably, robots.txt does not address any of that. Permission and capability are separate things.
So robots.txt is one piece of a larger picture of crawler access, the piece that governs permission among cooperative crawlers. It sits alongside the rendering, security, and performance considerations that determine whether a permitted crawler actually succeeds in reading you. A complete approach to AI crawler access uses robots.txt to express your permission policy and handles the technical barriers separately. Expecting robots.txt alone to ensure you are read would be a mistake; it ensures you have given permission, which is necessary but certainly not sufficient.
Putting it together
The practical workflow is a short sequence. First, look at your server logs to see which AI crawlers are actually visiting and how they fare. Second, understand the distinction between training and retrieval crawlers for the companies you care about, so you know which user-agents do which job. Third, decide your policy deliberately, typically allowing retrieval crawlers and making a conscious choice about training ones. Fourth, express that policy in robots.txt using the correct user-agent names. Fifth, revisit it periodically as the landscape shifts.
Done this way, crawler control becomes a deliberate, informed part of your AI visibility rather than an accidental configuration working for or against you by chance. You know who is visiting, you understand what each crawler does, and you have made conscious decisions expressed precisely. That combination, grounded in your logs and guided by the training-versus-retrieval distinction, is what turns robots.txt from a mysterious file into a tool you use with intent to permit exactly the reading you want.
The takeaway
AI crawlers identify themselves with user-agents, and robots.txt is where you tell cooperative crawlers what they may read. The single most important thing to understand is that AI companies run different crawlers for different purposes, training versus real-time retrieval, and these deserve different decisions, because blocking the retrieval crawler that fetches your page to cite it is a direct, often accidental, cost to your visibility. Know which user-agent does which job before you ever write a single rule.
Use your server logs to see who actually visits, decide your policy deliberately rather than by inherited default, and express it with the correct user-agent names, remembering that robots.txt works by cooperation and needs server-level enforcement for anything you must hard-block. Handle the technical barriers, rendering, security, speed, separately, since robots.txt governs permission, not capability. Treat it as a living policy you review as crawlers change, and it becomes a precise instrument for permitting exactly the AI reading you want.
Frequently asked questions
What is a crawler user-agent?
A user-agent is a short identifying string a crawler sends with every request, announcing who it is. Each AI company's crawlers have their own user-agent names, and your robots.txt file and server logs use these names to identify and control them. Knowing the user-agent of a crawler lets you write rules that apply specifically to it, allowing or disallowing it independently of other crawlers. It is the handle by which you refer to a particular bot.
Do AI companies run more than one crawler?
Yes, commonly. A single AI company often operates several crawlers for different purposes, for example one that gathers data to train models and a separate one that fetches a page in real time to answer a user's live question. These different crawlers have different user-agents, and critically, you may want to treat them differently, since blocking a training crawler is a very different decision from blocking the one that fetches your page to cite it in an answer.
What is the difference between a training crawler and a retrieval crawler?
A training crawler gathers content to be used in training or improving AI models, a broad, ongoing data-collection activity. A retrieval crawler fetches your specific page in real time because a user asked something your page can answer, and it is the fetch that often leads directly to a citation. The distinction matters because blocking training while allowing retrieval is a common, deliberate choice, and confusing the two can cause you to block the crawler that would have cited you.
Does robots.txt actually stop crawlers?
robots.txt is an honor-system standard: well-behaved, legitimate crawlers read it and respect its instructions, but it is not a technical enforcement mechanism, so it does not physically block anything. The major AI companies' crawlers generally respect robots.txt. For crawlers you want to hard-block regardless of cooperation, you need enforcement at the server or security layer; robots.txt is for instructing cooperative crawlers, which includes the ones you most care about.