- Score each keyword by expected business value per click: estimate the share of clicks an AI answer leaves you, multiply by your position's share, then by conversion rate.
- Cluster keywords by the page type the top ten results already show, not by shared words, and assign each cluster exactly one target page.
- Pull your first candidate list from Search Console impressions, support tickets, sales calls and site search before opening a keyword tool.
Most keyword research still starts the same way: export a list, sort by search volume, and build pages for the biggest numbers. That method was always a shaky proxy, because a search is not a visit. It has gotten worse. When Google shows an AI Overview, or a buyer gets the answer from a chat assistant, the question can be fully answered before anyone clicks. Your tool still reports 2,000 searches a month. Your share of those searches may be close to zero. So keyword research in the AI era has a different job. The job is no longer finding the biggest number. It is finding demand that still turns into visits, and then into revenue.
Why search volume stopped being a proxy for clicks
The old arithmetic was simple enough to do on a napkin: visits equal search volume, times the click-through rate for your position, times whatever share of those clicks you win. Each factor was roughly stable. Position one got a predictable slice, and the rest was noise you could plan around.
The first factor is now being quietly split. Google’s AI Overviews, AI Mode and chat assistants answer definitional and how-to questions on the results page or inside the chat itself, so the reader often has no reason to visit anyone. Studies from 2024, including one by SparkToro and Datos, found that more than half of US Google searches ended without a click to an external website. That figure mixes every query type and is not a forecast for your niche. It does point in one direction.
The mechanism matters more than the statistic. A keyword tool counts searches. It has no field for “the searcher didn’t need to click.” Take the query “what is a churn rate.” The answer fits in two sentences, so the searcher gets it and leaves. The tool still shows the volume, and the volume is real. It just no longer tells you how many people will reach your site.
That gap splits one old question into two. How many people search this? And how many of them still visit a website when they do? Most keyword research workflows only ask the first.
Volume is an upper bound, not a forecast. Keyword tool figures are rounded, modeled from panels or ad data, and often lag by months. Treat anything under a few hundred searches as a rough hint, and never compare two tools' numbers as if they measured the same thing.
Sort demand by how much answers absorb it
Exposure is not uniform across informational queries, and treating them as one bucket is the first mistake. The useful question is what a complete answer requires. If it fits in a few sentences, the answer layer will take it. If it needs a tool, a template, a calculator, a live price or a judgment call, the reader still has a reason to click through.
Here is a working map. Use it as a hypothesis to check against the live results, not as a rule.
| Query type | AI answer exposure | Clicks likely left | Default move |
|---|---|---|---|
| Definition (“what is a churn rate”) | High | Few | Answer it briefly inside a larger page. Don’t build a page around it. |
| How-to with fixed steps | High | Some | Build only if you add a tool, template or worked example |
| Comparison (“X vs Y”) | Medium | Moderate | Strong candidate, with a clear verdict and your own data |
| Template or calculator | Lower | Most | Ship the template itself, since it is utility, not text |
| Specific problem (“IBAN rejected by bank”) | Lower | Most | Strong when it sits inside your product’s domain |
| Buying intent (“invoicing software for freelancers”) | Lower | Most | Priority. Sits closest to revenue |
The pattern is that the closer a query gets to a finished artifact or a decision, the more clicks survive. The closer it gets to a plain definition, the more the answer layer keeps. Some how-to queries still pull heavy clicks when the answer box is thin, and some comparison queries get summarized in full. The table tells you where to look first. It does not replace looking.
Cluster keywords by the intent the results reward
Keyword tools group terms by string similarity, and string similarity is a poor guide to intent. “Payroll” and “payroll software” share a word, but the first usually wants an explainer and the second wants a shortlist of vendors. Put them on one page and you will satisfy neither reader. If you’re unsure how the four main intent types differ, search intent: the four types and how to match them covers the classification in detail.
Google ranks pages to match the intent it infers for a query, and the results page is the most visible evidence of that inference. So cluster by the SERP, not by the words. The method is simple, if slightly tedious:
- Search each candidate in a clean session, or pull the top ten results from your rank tracker. Record the domain and the page type of each one: product page, guide, forum thread, video or tool.
- Compare the lists. If most of the top ten share a page type and several domains repeat across queries, those queries share an intent and belong on one page. I use that as a rough heuristic, not a published threshold.
- Where the page types split, say half forum threads and half product pages, the intent is mixed. Build for the type the top results reward most, then decide whether the minority intent deserves a page of its own.
- Name each cluster after the job it does for the reader, such as “compare payroll vendors for a 20-person company,” not after its head term. A clumsy name usually exposes a clumsy cluster.
Once the clusters are set, the next question is whether your site covers them in enough depth. That is the subject of how to build topical authority that ranks, and it is worth reading before you commit to a long list of pages.
Mine the questions buyers already ask
Keyword tools are good at expanding a term you already have. They are weak at showing the questions people ask in their own words, with the stakes attached. The best candidates usually come from sources that already contain that language.
- Search Console query exports. Filter for queries that get impressions but few clicks. Impressions show demand that already sees your site, and a low click-through rate shows where an answer or a competitor is taking the click. Also filter for question words to find phrasings you have not targeted yet.
- Support tickets and chat logs. Customers describe their situation in full, including details no search would contain. That phrasing sits closer to a real query than any suggestion list does.
- Sales call notes. Objections are keywords with a buying motive attached. “Does it sync with our accounting package” is a page waiting to be written, and it can be a high-value one.
- Site search logs. Every query someone types into your own search box is a visitor who did not find what they wanted. These are rare, and they are valuable.
- Forum and community threads. Reddit, niche forums and Q&A sites reveal wording, frustration and edge cases. Use them for phrasing and for understanding the problem, not as a volume source.
- Autocomplete and “People also ask” boxes. Good for spotting variants and related questions, and weak for judging value. They show what people type, not what they would pay to solve.
Take a hypothetical invoicing tool. A support ticket says a client’s bank bounced an invoice because the IBAN was written with spaces. A keyword tool shows almost nothing for that phrase. The ticket still gives you three things: the exact wording, the situation, and the stakes, since a bounced payment costs the customer money. A page that walks through the specific fix, and what to do when the bank rejects the payment anyway, is worth more to the business than a generic invoicing guide chasing a bigger term.
Don’t turn every question into a page, though. A question earns a page when it has a distinct job and when people ask it in several different ways across sources. One ticket is an anecdote. Five tickets, a Search Console cluster and a forum thread describing the same bounce is a pattern.
Score every keyword by value per click
Once candidates are clustered, you need a ranking that doesn’t start and end with volume. This is the model I use. It is crude, and it is good enough to change decisions.
Expected visits per month = search volume × share of clicks still available × your share of those clicks
Expected outcomes per month = visits × conversion rate
Two inputs are assumptions you should replace with your own data: how many clicks survive the answer layer, and what converts. The third input, your share of surviving clicks, depends mostly on position. A top-three result can win a meaningful slice. A result on page two wins close to nothing.
Here are two keywords for a subscription billing tool, using illustrative numbers rather than benchmarks:
- “churn rate formula”, 8,000 searches a month. An AI answer takes most of the clicks, so assume 10% survive. You win 30% of those from a strong position: 8,000 × 0.10 × 0.30 = 240 visits. The page converts 1% of visitors to a trial, which is about 2.4 trials a month.
- “invoice reminder template”, 210 searches a month. A reader wants to copy a template, and an answer does not replace that, so assume 70% of clicks survive. You win the same 30%: 210 × 0.70 × 0.30 = 44 visits. The page converts 8% of visitors, because the next step is sending reminders with the product. That is about 3.5 trials a month.
The second keyword has 1/38th of the volume and produces roughly 45% more trials. A volume-sorted list would have sent you the other way.
Get your own numbers first. In Search Console, compare the click-through rate you get on queries you already rank for with the typical rate for that position, then check the live results for the queries where the gap is widest. Don't assume the standard Performance report separates out AI answer clicks, so confirm what it shows before you build a plan on it.
A keyword research process for the AI era
Here is the sequence I’d follow when starting from scratch. Most of the time goes into steps four and five. Pulling volume takes minutes. Reading a results page properly takes longer, and that reading is where the decisions get made.
- Start from the pages that make money. List the jobs your product does for a buyer and the pages that convert those buyers today. A seed term drifts toward whatever is biggest. Starting from revenue keeps value in front of you from the first row.
- Gather candidates from questions before tools. Pull from Search Console, support tickets, sales notes and site search first. Keyword tools come second, to expand what you found rather than to decide what matters.
- Expand with a tool, and cap the volume. Use a keyword tool for related terms, rough volume and difficulty. Treat the volume as a ceiling, and keep it out of the decision itself.
- Cluster by SERP. Run the overlap check described above. Each cluster becomes one target page, and that page’s type matches what the top results already are.
- Score each cluster. Estimate the surviving click share from the live results and your own click-through data, then apply the value formula. Place the cluster on the quadrant.
- Write down the page’s job and its measure. For each cluster, name the format (guide, tool, comparison or template), the one query you will track as its primary measure, and the date you will review it.
Failure modes that waste a quarter
Most bad keyword plans fail in predictable ways. These are the ones I see most often.
- Chasing answer-heavy definitions. The traffic line rises for a few weeks and then flattens, because the reader got the answer without you. The page ends up thin, and it has no tool, template or verdict to hold a visitor once they arrive.
- Splitting one intent across many pages. “Keyword research tool,” “free keyword research tool” and “keyword research tool for beginners” usually share a SERP. Three pages then compete with each other, and none of them collects the full set of signals. Merge them into one page with sections that answer each variant.
- Mistaking a tool score for a fact. Difficulty scores are estimates built on link and ranking data, and for low-data terms they are often a guess. Use them to order your work, not to make yes-or-no calls.
- Treating the list as finished. SERPs change. A query that was a clean click opportunity last year may now be answered on the results page. Re-check your top clusters every quarter.
- Dismissing answer-heavy queries entirely. Some of them still matter for brand, because the answer may name your company as a source. Whether that produces measurable business value depends on your category, and the evidence is still thin. Don’t build a strategy on it, but don’t write off a well-sourced explainer as worthless either.
The honest summary
Keyword research still works. What changed is the unit of value. A keyword is no longer worth the searches it has. It is worth the visits it can still earn and the revenue those visits carry. That shifts effort toward clusters that match buying intent, toward the questions your customers already ask, and toward pages that do something a chat answer cannot, such as calculate, compare with a verdict or hand over a template.
It also means the finished list is shorter than it used to be. That is the right outcome. A long list of keywords that all return the same answer box is a plan for disappointment.
AutoSEO Stack is built around this discipline. It researches what your buyers actually search, decides which queries are worth a page, and writes each article in a human voice against a per-page quality bar. Start with the Google research section to see how the research side works, then check the features list for what gets published on your own site.