- Build a fixed panel of 20 to 40 buyer-style prompts, run each on every engine at least three times, and log mention rates instead of yes-or-no results.
- Treat Search Console as a trend tool, not a scoreboard: AI Overview activity is folded into its totals, and ChatGPT, Gemini and Perplexity never appear in it.
- Don't act on a swing until it clears the noise: with 120 answers per period, a 10-point change in mention rate is within the margin of error.
Most teams check their AI visibility the same way. They open ChatGPT, type the company name, read the answer, and feel reassured or quietly sick depending on what came back. That is one sample from a system that gives different answers to the same question on different days. AI visibility tracking replaces that single sample with a measurement you can repeat, compare over time and act on.
The counter-intuitive part is that your existing data isn’t useless. It just answers a different question. Search Console tells you how Google Search performed. It tells you almost nothing about whether ChatGPT recommended you to a buyer on Tuesday. You need a second instrument, and you have to build it on purpose.
Why AI visibility tracking is harder than rank tracking
Rank tracking works because a results page has fixed slots. You search “best project management software,” see yourself in position four, and check tomorrow to find roughly the same place. An answer engine has no slots. It writes a paragraph, and that paragraph can change between two runs of the same prompt for four reasons.
- Sampling. Language models generate text by sampling from probability distributions. The same prompt can produce different wording and a different list of brands with nothing else changed.
- Retrieval. Many answers are grounded in live search, so the pages a model reads depend on what its retrieval layer returned that time. Perplexity searches on nearly every question by design. ChatGPT searches when it decides a question needs current information. Google’s AI features draw largely on Google’s own index. Each engine works from a different pool, so being cited by one tells you little about the others.
- Context. Location, language, account history, saved memory and custom instructions all shape the output. The same prompt typed in Lisbon and in Chicago can return different lists.
- Versions. Engines change models and retrieval behavior without notice. A tracking series can break because the product changed, not because your visibility did.
The consequence is that “do we show up?” stops being a yes-or-no question. The honest version is a rate: across these prompts, on this engine, in this period, how often does the brand appear, and how often does a source link point to us? Everything else here is machinery for producing that rate reliably. For the inputs that earn citations, read how to get cited by AI engines.
What Search Console can and can’t tell you
Search Console is the instrument you already own, so start there. Just know where its reach ends.
What it shows you
Google has said that clicks and impressions from AI Overviews and AI Mode count toward the standard Performance report, rolled into the Web search totals. There is no separate filter to isolate them. Google changes this reporting often, so verify the current documentation before you rely on it.
The result is that you can see the effect without seeing the cause. Three patterns are worth checking every week:
- Impressions rising while clicks stay flat or fall, on question-style queries. Queries like “how do I” or “what is” are where AI answers appear most. This pattern fits an answer sitting above your result and resolving the question before anyone clicks. Treat it as a hypothesis to test against your prompt panel, not a conclusion.
- Growth in branded queries. When an answer recommends a brand, some buyers search that name next. A sustained rise in impressions for your brand term, with no matching campaign, is a useful lagging signal. It is correlation, not attribution, but few Search Console metrics move with AI mentions at all.
- New long-tail question queries reaching your pages. Conversational prompts are longer and rarer than keywords. When they start landing on a page, the engine is matching your content to a more specific question than before.
For the Google side of this, the traffic patterns from AI Overviews are the companion read.
What it hides
Search Console is blind to most of the picture:
- Nothing from ChatGPT, Gemini, Perplexity or Claude. The report measures Google Search. Answers from every other engine are invisible to it.
- Mentions without clicks. It records what happens on Google results pages. A brand named in a chat answer, with no click, leaves no trace.
- Privacy filtering. Google withholds some queries for privacy, and those tend to be the low-volume, conversational prompts most likely to represent new AI-driven demand.
- Position is not a citation metric. Average position doesn’t cleanly describe a citation inside an AI-generated block, so don’t use it as a stand-in for AI visibility.
Search Console’s job here is narrow. It tells you whether the trend in Google matches the trend in your panel, and it flags which queries deserve a prompt. It is not your scoreboard.
Building a prompt panel that means something
Prompt-based monitoring is the core of the method. You pick a fixed set of questions, run them on the engines your buyers use, and record what comes back.
Organize the panel into five buckets, because buyers ask different kinds of questions at different stages:
- Category discovery. “Best [category] for [constraint],” such as “best CRM for a five-person agency that doesn’t want per-seat pricing.”
- Problem-first. “How do I [job to be done],” where your product is one possible answer among several rather than the obvious one.
- Comparison. “[You] vs [competitor]” and “alternatives to [competitor].” These are often the highest-intent prompts on the list.
- Evaluation. “Is [you] worth it for [use case]” and “what are the downsides of [you].” Here the question is whether the answer is accurate, not only whether you’re named.
- Constraint-heavy buying prompts. Budget, region, team size, compliance requirement. Real buyers add constraints, and generic prompts tend to reward the largest brands.
Source the prompts from what buyers actually say: sales call transcripts, support tickets, onboarding questions, forum threads in your category, and the Search Console queries you’d expect to see typed into a chat window. Rewrite each one the way a person types into a chat box, constraints included, since a three-word keyword is a poor stand-in for a real prompt.
Start with 20 to 40 prompts. Under 20 and the rates jump around too much to read. Over 40 and you will stop running the panel by hand. Freeze the list for at least a quarter, because a series only means something if its questions stay the same. Keep new candidates in a separate list and merge them at the quarterly review.
Run it the same way every time
Consistency in the run protocol matters more than the number of prompts.
- Use a fresh session, or a clean account with memory and custom instructions turned off.
- Hold location and language constant. If you can’t, record the location used for each run.
- Run each prompt at least three times. Output varies, so a single run is an anecdote.
Log the fields that matter
For each answer, record the date, engine, prompt ID and run number. Then capture:
- Whether your brand is named, and its position if the answer is a list.
- Whether your domain is cited as a source, and which other domains are cited.
- Every competitor named.
- Accuracy: is what the answer says about you correct? Note wrong pricing, discontinued features and outdated claims.
Teams skip the accuracy field, and it is often the most valuable one. A mention with the wrong price or a feature you retired is a visibility problem with a cost attached. It usually points to an outdated page or a third-party listing the engine is reading, which tells you exactly what to correct.
If you script runs through an engine’s API, expect different results from the consumer app. Report API series separately and don’t splice them onto manual ones.
Share of voice across models
A mention rate tells you how often you appear. Share of voice tells you how much of the conversation you hold relative to the brands sharing the panel. Together they show whether you are gaining ground or losing it. This is what most teams mean by share of model.
Here is a worked example with illustrative numbers. Suppose you run 40 prompts three times each on one engine, giving you 120 answers. Across those answers, the tracked brands collect 150 mentions in total, counting each brand at most once per answer:
- Your brand: 36 mentions, a 30 percent mention rate, and a share of voice of 36 ÷ 150, or 24 percent.
- Competitor A: 48 mentions, a 40 percent mention rate, and a share of 32 percent.
- Competitor B: 36 mentions, a 30 percent mention rate, and a share of 24 percent.
- Competitor C: 30 mentions, a 25 percent mention rate, and a share of 20 percent.
Each metric answers a different question, and each has its own trap:
| Metric | Question it answers | How to calculate | Trap |
|---|---|---|---|
| Mention rate | How often does the engine name us? | Answers naming us ÷ total answers | Says nothing about whether the mention is accurate or favorable |
| Citation rate | How often does it link to our domain? | Answers citing our domain ÷ total answers | Engines often cite review sites and directories, so a mention can happen with no link to your pages |
| Share of voice | How much of the conversation do we hold? | Our mentions ÷ all tracked brand mentions | Moves when you add or remove a competitor, even if nothing about you changed |
| Accuracy rate | When we’re named, is it right? | Correct mentions ÷ all mentions | Needs a human checklist, because automated matching misses subtle factual errors |
Report each engine separately. A 40 percent mention rate on Perplexity and 10 percent on ChatGPT blend into a 25 percent figure that describes neither engine. A blended number is fine as a secondary line, but never as the headline.
The four metrics form a funnel, and each drop is a place where visibility leaks. Each leak points to a different fix:
Read the leaks like this:
- Named but not cited. The engine knows about you, often through third-party pages, but doesn’t retrieve your site for this question. Work on the page that answers the prompt directly, and on the outside sources that corroborate it. How to get cited by ChatGPT covers that half.
- Cited but inaccurate. The engine is reading your pages, or a page about you, and getting the facts wrong. Fix the pricing and product pages first, then the third-party listings it seems to rely on.
- Neither named nor cited, but ranking well in Google. The engine’s retrieval pool differs from Google’s. List the sources it does cite and compare them with your page. The gap is usually specific: a missing comparison, number or definition.
- Accurate and cited, but share falling. A competitor is gaining on this prompt set. Start with their pages and their recent third-party coverage.
Referral and crawler signals
Two server-side signals give you something closer to live evidence than Search Console does.
Referral traffic from AI apps
When a user clicks a source link in an AI answer, the visit carries a referrer. ChatGPT, Perplexity, Gemini, Copilot and Claude all show up as referral sources, though the exact strings change over time. Check your acquisition reports for what actually appears on your site instead of trusting a list.
Two things distort the numbers. Some clicks lose their referrer: links opened in a mobile app’s in-app browser, or from a browser with strict privacy settings, can arrive as direct traffic, which makes AI referrals look smaller than they are. And some apps append tracking parameters to outbound links. ChatGPT links have been observed carrying a utm_source=chatgpt.com parameter, which helps when it’s present, but it is not guaranteed on every click.
Put AI referrals in their own channel. In GA4, create a custom channel group under Admin with a rule that matches the session source against a pattern such as chatgpt\.com|perplexity\.ai|gemini\.google\.com|copilot\.microsoft\.com|claude\.ai. Check the source strings you actually see before you trust the pattern. The volume is usually small. Compare its conversion rate with organic search anyway, because these visitors often arrived with a recommendation already in mind.
Start the history now: trends need a past. Create the channel group before you need the data, so the line has months of history by the time someone asks about it.
Crawler logs
Server logs show which bots request your pages, and the major AI companies now split their crawlers by purpose. That split is the useful part:
- Training crawlers collect content for model training. OpenAI’s GPTBot is one example.
- Search crawlers build the index a product uses to retrieve sources. OpenAI’s OAI-SearchBot and PerplexityBot are examples.
- User-triggered fetchers request a page because a person asked the assistant to read it, or because the assistant needed it to answer a question. ChatGPT-User and Perplexity-User fall here.
The third category is the closest thing to a live signal you can get. A spike in user-triggered fetches on a page means someone’s question pulled that page in, recently. Pair the logs with your prompt panel. If a new comparison prompt starts pulling your comparison page, the fetch should appear in the logs in the same window.
Google works differently. Googlebot powers Search, including AI Overviews. Google-Extended is a robots.txt token that helps control training for Gemini, and it does not control inclusion in Search or AI Overviews.
User-agent strings are self-reported and can be spoofed, so verify the bots that matter against vendor-published IP ranges, or against reverse DNS for Google. Bot names also change, so check documentation before you write filters.
Reading noisy data without fooling yourself
This is where most AI visibility tracking goes wrong. Panels are small and answers vary, so the numbers move even when nothing real has changed.
Here’s the arithmetic. Say your mention rate is 30 percent across 120 answers in one period. The 95 percent margin of error on that rate is about 8 points (1.96 × √(0.30 × 0.70 ÷ 120) ≈ 0.08). Now compare two periods of 120 answers each. The margin on the difference widens to about 12 points. A move from 30 percent to 40 percent looks like progress, and it sits inside the noise. You can’t claim a win, and a drop back to 32 percent doesn’t prove a loss either.
One screenshot isn't data: a single chat answer naming your brand shows that it can happen, not that it happens. The number you need is the rate across the panel, with the sample size printed next to it.
The practical rules that follow:
- Print the sample size next to every rate. “30 percent (n=120)” is a claim you can check. “30 percent” is not.
- Don’t slice a 120-answer series by engine, prompt bucket and week, then celebrate the biggest slice. Each slice has fewer answers and wider error bars than the whole.
- Change the panel only at quarterly reviews, so movement comes from your work and not from rewording prompts.
- Judge direction across several periods. A consistent drift over three or four months means something. One good week doesn’t.
A lightweight weekly tracking routine
You don’t need a platform to start, and you don’t need to track everything. The routine below takes about two hours a week while you run the panel by hand, and it gets shorter once the runs are scripted.
- Monday, 20 minutes: Search Console. Pull branded impressions, the top 20 question-style queries, and pages where impressions rose while click-through fell.
- Monday, 60 to 90 minutes: prompt panel. Run the frozen panel under the protocol above, log every answer, and send accuracy problems to whoever owns the page.
- Tuesday, 10 minutes: referrals. Check the AI referral channel in GA4. Note sessions, conversions and landing pages.
- Friday, 15 minutes: the readout. Calculate mention rate, citation rate and share of voice for each engine, with sample sizes. Compare with the previous period and say plainly whether the change clears the noise.
- Monthly, 30 minutes: crawler logs. Group user-triggered fetches by URL and compare them with your citation log. Note any URL that appears in both.
- Quarterly: review. Replace up to a fifth of the panel with prompts from recent sales conversations, keeping a core set fixed. If you change more, start a new baseline and label it as one.
The honest summary
Measuring AI visibility is harder than measuring rankings, and the difficulty is mostly statistical. Answers vary, engines pull from different sources, and no platform shares data across them. The useful response is a fixed prompt panel, run consistently, read as rates with sample sizes, and cross-checked against the traffic and crawler signals you can observe.
The discipline is the same one behind every page worth ranking. Each prompt needs a reason to be in the panel. Each claim about your brand needs to be checkable. And each change needs a measurement that could have shown it failed. That loop is what AutoSEO Stack is built around: research what buyers actually ask, publish pages with a real reason to exist, and hold every one to the same quality bar so the results can be measured rather than guessed at. To see that loop running on your own site, start with the AI visibility section of AutoSEO Stack.