How to track AI search visibility without expensive tools

"Are we showing up in ChatGPT?" is now a standard question in marketing meetings, and the honest answer usually starts with "it depends when you ask". AI assistants give different answers to the same question from one minute to the next, so to track AI search visibility, a screenshot of one answer proves very little, and a single "AI ranking" proves even less.

This guide sets out a method that copes with that randomness without a paid tool: real customer questions, run several times each, scored as a mention rate with a margin of error, and checked against the free data Google, Microsoft and Google Analytics now provide. The sampling maths is worked through with numbers, so you can decide how much testing is enough.

Who this is for: small-business owners and in-house marketers who have been asked to report on AI visibility and want numbers they can defend. What you will be able to do:

• Build a prompt set that reflects how customers actually ask.
• Work out how many runs you need, and which changes are real.
• Combine Search Console, Bing Webmaster Tools and GA4 into one monthly view.
• Judge whether a paid tracker is worth it.

If you are still working on getting cited at all, start with our guide to getting your business cited in AI search; this page expands its measurement section.

What AI search visibility is and what you can measure

AI search visibility is how often AI assistants mention, recommend or link to your business when people ask questions you want to be part of. You can measure it at three levels: whether the answer names you, whether it cites your site as a source, and whether anyone clicks through.

The surfaces that matter for most businesses are Google's AI Overviews and AI Mode, ChatGPT search, Microsoft Copilot, Perplexity and Gemini. Each builds answers from its own retrieval system and model, so each needs its own numbers; one blended "AI" figure hides where you are winning or losing.

Mentions, citations and clicks are different things

• A mention is your brand named in the answer text, with or without a link. Only your own prompt testing sees it.
• A citation is a link to your site among the answer's sources. Search Console and Bing count some of these.
• A click is a visit from the answer, recorded by analytics if the referrer survives.

A brand can be recommended with no link, and a page can be cited for a fact while a competitor gets the recommendation. Report all three, and never treat one as a stand-in for another.

Why one check tells you almost nothing

One check tells you almost nothing because AI assistants generate each answer afresh: ask the same question again and you get a different set of brands, in a different order, with different sources. The only stable thing to measure is how often you appear across many runs.

The clearest evidence comes from a 2026 study by SparkToro and Gumshoe, in which 600 volunteers ran 12 prompts through ChatGPT, Claude and Google's AI Overviews and AI Mode a combined 2,961 times. The authors found less than a 1-in-100 chance that ChatGPT or Google's AI would give the same list of brands in any two responses, and the same list in the same order turned up roughly once in 1,000 runs. SparkToro discloses that Gumshoe, which ran the analysis, works in AI visibility tracking.

Other studies point the same way:

• Cited pages churn. When SE Ranking ran 10,000 keywords through AI Mode three times on the same day in 2025, the average exact-URL overlap across the three answers was 9.2%. Domain overlap was higher, at 14.7%, so AI Mode often returned to the same sites but picked different pages.
• Local answers churn too. BrightLocal's September 2026 study of more than 200,000 local searches found only 20–33% average overlap between result sets on repeat searches: 23% for ChatGPT, 21% for AI Mode and 33% for AI Overviews. A business that appeared once showed up again in about half of later runs.
• Apparent differences are often noise. A 2026 paper by Ronald Sielinski sampled Perplexity, OpenAI's search and Gemini repeatedly over nine days and found that many apparent differences between domains fell within measurement noise once confidence intervals were drawn.

A quick calculation: how often single checks disagree

Suppose the assistant names you in 40% of answers to a prompt. Two single checks a month apart will then disagree 48% of the time (2 × 0.4 × 0.6) even when nothing has changed. Run the prompt three times and you will see zero mentions about one time in five (0.6³ = 0.216), enough to convince a manager you have "disappeared from ChatGPT".

What this means for "AI rank tracking"

Position inside an AI answer is the least stable number of all: "third in ChatGPT" from one run is one draw from a lottery. SparkToro's authors concluded that a visibility percentage across many runs is the useful measure, and the rest of this guide takes that approach.

Metrics that survive the noise: mention rate, citation rate and share of voice

Three metrics hold up under randomness: mention rate, citation rate and share of voice. Each is a proportion counted across many runs, so each can be given a margin of error and compared fairly over time.

1. Mention rate is the share of runs whose answer names your brand: runs that mention you divided by total runs. Many trackers call this "visibility".
2. Citation rate is the share of runs that link to your domain in the sources. It tells you whether your own pages are doing the work, or whether third-party pages are carrying your name.
3. Share of voice is your mentions as a share of all mentions of the brands you track, across the same runs. If 100 runs contain 310 brand mentions and 42 of them are yours, your share of voice is 13.5%.

Calculate each metric per engine and per prompt group, because a blended figure can rise while your most valuable engine falls.

Secondary measures worth logging

• Accuracy: whether the answer states your services, location and prices correctly.
• Quality: recommended first, listed neutrally, or mentioned with a caveat.
• Sources cited when competitors are named: your outreach and listing targets.

Our free AI Citation Quality Scorer rates how favourable and specific a mention is when you paste in the answer, which keeps the quality column consistent when several people log runs. If answers keep getting your facts wrong, run your site through the AI Answer Consistency Checker: conflicting phone numbers, addresses or prices across your own pages are a common cause.

Leave average position out of headline reports. It only exists in runs where you appear, so it rests on the smallest sample of all.

Build a prompt set from real customer questions

Build your prompt set from the questions customers really ask, in their words, grouped by intent. Between 20 and 50 prompts is enough for most small businesses. Do not track one or two "ideal" prompts, because real people phrase the same need in very different ways.

In the SparkToro and Gumshoe study, the average semantic similarity of prompts volunteers wrote for the same need was 0.081, which the authors likened to the gap between Kung Pao chicken and peanut butter. Yet the assistants still returned a fairly consistent set of brands, which is why a broad prompt set measured as a rate beats a few perfect prompts.

Where to find the questions

1. Sales and support: enquiry forms, emails, call notes and chat logs. Copy the customer's wording, not your summary of it.
2. Search data: question-style queries in Search Console and the "People also ask" boxes for your topics. Our keyword research guide shows how to collect them.
3. Bing grounding queries: the phrases Bing's AI Performance report lists for your cited pages.
4. Reviews and forums, where customers describe the problem before they know your category's name.

Mix the intents

Split prompts into groups: category questions ("best X in Y"), problem questions ("how do I fix Z"), comparisons ("X vs Y") and brand questions ("is [brand] good for…"). Weight the set towards category and comparison prompts, because that is where recommendations happen. The free AI Prompt Intent Classifier sorts up to 50 questions by intent, from information to ready to buy, a quick way to check your set is not all informational.

Write prompts the way a person would, context included: "I run a small bakery and need accounting software that handles GST, what should I use?" Keep prompts that name a place in their own group. For how the assistant picks names, see how ChatGPT recommends businesses.

Then freeze the set. Editing prompts breaks month-on-month comparisons, so add new questions as a separate group and keep the core set unchanged for at least a quarter.

How many runs you need and how to sample fairly

Roughly 100 runs per engine give a mention rate within about ±10 percentage points, and 10 runs give about ±30. Spread those runs across many prompts rather than repeating a few, and keep the test conditions identical every time.

How we worked this out: we treated each run as a yes-or-no trial and used the binomial margin of error at 95% confidence, 1.96 × √(p(1 − p) ÷ n), the Wilson interval for likely ranges at small samples, and a design-effect adjustment for repeated runs of one prompt. You can reproduce every number with the spreadsheet formulas further down.

Margin of error by number of runs

Precision improves only with the square root of the sample, so halving the margin takes four times the runs. SparkToro's advice to ask "at least 60–100X" and average the results lands at ±10 to ±13 points in the worst case: enough to tell 20% from 50%, not 30% from 35%. A ±5-point margin needs about 385 runs (1.96² × 0.25 ÷ 0.05²).

More prompts beat more repeats

The table assumes independent runs, and runs of the same prompt are not: a prompt that names you once is more likely to name you again. Statisticians allow for this with a design effect of 1 + (m − 1) × ρ, where m is runs per prompt and ρ is how alike those runs are, from 0 (independent) to 1 (identical). Total runs divided by the design effect gives the effective sample.

We estimated ρ from BrightLocal's published figures. If a business appears in a share p of runs and, having appeared once, reappears in a share r of later runs, then ρ ≈ (r − p) ÷ (1 − p). With AI mention rates of 32–38% and reappearance rates of 50–58%, ρ comes out between about 0.2 and 0.4. Treat that as a rough, back-of-the-envelope figure for local prompts; the table uses 0.3.

The same effort buys a much better topic-level estimate when spread across prompts. Keep at least two runs per prompt so you can spot unstable ones, and run your three to five most valuable prompts 20 times or more if you need a reading for each. Rule of thumb: with three to five runs per prompt, halve your run count before calculating a margin of error.

A monthly sampling plan you can run by hand

Example (hypothetical): 25 prompts × 4 runs in each of ChatGPT and AI Mode is 200 runs a month, one run of every prompt each week. At about two minutes to run and log each (time your first batch), that is under seven hours a month. Each engine gets an effective sample of about 53 and a margin of roughly ±12 points: enough to catch a swing of around 20 points.

Sample fairly: location, personalisation and timing

What an assistant shows depends on where you are and what it knows about you. Control both, or your trend will measure your own browsing instead of your visibility.

• Location. OpenAI says ChatGPT search infers an approximate area from your IP address to make local results more relevant, may rewrite "restaurants near me" into a search naming a city, and that a VPN can change the location it infers. BrightLocal re-ran each prompt from several points on a map and found that the businesses named changed a lot from one point to the next. Run local prompts from the same place every time, or name the area in the prompt.
• Memory and history. OpenAI notes that when memory is on, saved location details can influence search results. Google's help page says Personal Intelligence in AI Mode draws on your previous searches and activity saved in your Search history when history and personalised recommendations are enabled. Test from a clean account, or with memory and personalisation switched off (checked October 2026).
• Timing and interface. Answers drift over days as well as between runs, so spread runs across the month and date each one. Use the same product and mode every time, and note announced model changes.

Local businesses face the widest swings, because AI answers name only a few businesses per question. Our guide to local SEO for AI search covers what shapes those answers.

Free first-party data from Search Console, Bing and GA4

Three free reports now measure parts of AI visibility directly: Search Console's generative AI report for Google, Bing Webmaster Tools' AI Performance report for Copilot, and GA4's AI Assistant channel for visits from chat assistants. None of them can see a mention without a link, which is why prompt runs remain necessary.

Search Console: impressions in AI Overviews and AI Mode

Google's generative AI performance report shows how often URLs from your site appeared in AI Overviews and AI Mode, by page, country, device and date. Google announced it on 3 June 2026, and a note on the post says it reached all websites worldwide on 31 August 2026. Because it counts URLs, unlinked brand mentions never appear. Our guide to the Search Console generative AI report explains how to read it page by page.

Bing Webmaster Tools: citations and grounding queries

Bing's AI Performance report shows how often your pages are cited in AI answers across Microsoft Copilot and Bing, with the grounding queries the AI used to retrieve them. Bing says the data is a sample and that a citation does not represent clicks or traffic. Grounding queries are generalised phrases, not the exact words people typed, but they are the closest thing to a prompt log any platform offers.

In June 2026 Bing added a Citation Share preview: your site's percentage of all citations shown for a grounding query. Bing describes it as observational rather than a ranking, and it does not reveal competitor domains. On 13 October 2026 Bing added a Topics Portfolio beta that groups your grounding queries into themes, estimates each topic's total citations across all sites and lists opportunity queries: queries in those topics where other sites are cited and yours is not. The tab only appears when your site has enough data (checked October 2026).

GA4: visits from AI assistants

On 13 May 2026 Google Analytics added an AI Assistant channel to its default channel group, naming ChatGPT, Gemini and Claude as examples. Google's channel definitions set the medium to "ai-assistant" when the referrer matches its list of AI assistants, and keep AI Overviews and AI Mode inside Organic Search. So GA4 cannot separate clicks from Google's own AI answers, and visits that arrive without a referrer are not classed as AI at all.

OpenAI's publisher FAQ says ChatGPT adds utm_source=chatgpt.com to referral links, which makes ChatGPT visits easy to find in landing-page reports. Pair those pages with conversions to see which AI-visible pages produce enquiries. What none of these reports shows is who was recommended instead of you, or what the answer said: that still needs your own prompt runs.

Run the tracking: a spreadsheet method step by step

A shared spreadsheet with one row per run is all you need: log each answer, score it the same way every time, and let formulas calculate the rates and their likely ranges.

The template: one row per run

The monthly cycle

1. Prepare a clean setup per engine: a test account or signed-out browser, memory and personalisation off, location fixed.
2. Run every prompt once a week in each engine, in a new chat, pasting the prompt exactly as written.
3. Log the row straight away and save the answer text or a dated screenshot.
4. At month end, calculate each engine's mention rate, citation rate and share of voice with a pivot table or AVERAGEIFS.
5. Add the 95% range, then copy the month's summary to a separate tab so history never changes.
6. Note what changed that month: new pages, reviews, press coverage, Google updates.

Spreadsheet formulas

With mentions in B2 and runs in C2, the mention rate is =B2/C2. The Wilson 95% range, which behaves well with small samples, is:

• Lower bound: =(B2/C2+1.96^2/(2*C2)-1.96*SQRT(B2/C2*(1-B2/C2)/C2+1.96^2/(4*C2^2)))/(1+1.96^2/C2)
• Upper bound: the same formula with the minus sign before 1.96*SQRT changed to a plus.

To allow for repeated runs of the same prompt, put the effective sample (runs divided by the design effect, or simply half your runs at three to five runs per prompt) in C2, and the matching number of mentions (rate × effective sample) in B2.

When an engine keeps naming a competitor instead of you, take one of the competitor pages it cites and your closest equivalent page to the free AI Citation Gap Checker. It compares the two factor by factor, so you know what to fix first.

Reading results: what counts as a real change

A change is real when it is bigger than the noise and it lasts. As a working rule, act when a two-proportion test on the effective sample gives a z-score of 2 or more, and the following month does not reverse it. Anything smaller is something to watch, not a win or a loss to report.

The test in one formula

For two months with rates p1 and p2 and effective samples n1 and n2, work out the pooled rate p̄ = (p1 × n1 + p2 × n2) ÷ (n1 + n2), then z = (p2 − p1) ÷ √(p̄ × (1 − p̄) × (1 ÷ n1 + 1 ÷ n2)). A z of 2 or more, either way, means the change is unlikely to be chance. Ranges that do not overlap at all also signal a real change, but overlap alone does not prove noise.

Worked example

Example (hypothetical): a bathroom renovation firm in Pune tracks 25 prompts × 4 runs in ChatGPT, which is 100 runs a month and an effective sample of 50.

October's eight-point rise looked like progress but sat inside the noise. November's 20-point rise passes the test, and December holding steady (z = 0.2 against November) confirms it. Without the design-effect adjustment, November would have looked far more certain (z = 3.0) than it was.

Cross-check every apparent change. If the ChatGPT mention rate jumps but Bing citations, Search Console AI impressions and GA4 AI Assistant sessions are flat, check your own test setup before you celebrate.

Decision table: what to do with a result

For a real loss, rule out access problems first: our guide to controlling AI crawlers shows how to confirm the search crawlers behind ChatGPT, Copilot and Google can still reach you. If the drop matches a Google update, use the core update recovery guide before changing pages for AI.

Should you pay for an AI visibility tracker?

Pay for a tracker when the sample you need is bigger than you can run by hand, or when you need many locations or languages; otherwise a spreadsheet and the free reports cover most small businesses. Before buying, ask how many times each prompt is run and what margin of error the tool reports.

Trackers usually charge by prompt or by answer, so compare the cost of each 100 answers with the hours the spreadsheet takes you.

Questions to ask any vendor

1. How many runs per prompt, per engine, per period? Peec's help centre, for example, says tracked prompts run about once every 24 hours (checked October 2026). That is about 30 answers per prompt per engine a month: useful as a monthly rate, misleading as a daily score.
2. What margin of error do you report? Gumshoe publishes one: it says its default setup of about 800 observations per run estimates overall visibility within ±5 points at 95% confidence, and recommends multiple runs before acting on topic-level differences. (The simple formula gives about ±3.5 points for 800 independent runs, so that figure is the cautious one.) A vendor that cannot answer is selling single runs.
3. How do you control location, language and personalisation? Ask for the settings behind each answer.
4. Which interface do you query? A consumer app, a signed-out session and a developer API can answer differently.
5. Can I export raw answers and cited URLs, and bring my own prompts?
6. Where do any "prompt volume" estimates come from, and how are model changes flagged?

Turn visibility into pipeline estimates

Turn visibility into money by working forwards from what you can measure: AI-referred sessions, their enquiry rate and your average customer value. Use mention rate as the leading indicator that explains why those sessions move, not as a revenue number on its own.

A simple model

Monthly AI pipeline = AI-referred sessions × enquiry rate × close rate × average customer value.

Example (hypothetical): a physiotherapy clinic in Bengaluru gets 120 sessions a month from GA4's AI Assistant channel, with a 5% enquiry rate. Half of its enquiries become patients worth ₹12,000 each on average, so 120 × 0.05 × 0.5 × ₹12,000 = ₹36,000 a month. If its ChatGPT mention rate rose from 22% to 42% and AI sessions rose in proportion, which is an assumption to test against your own GA4 trend, the same maths gives about 229 sessions and ₹68,700 a month.

The free AI Visibility to Pipeline Estimator runs this calculation from your own numbers and shows visits, leads and revenue side by side.

Count the influence you cannot click

Many AI answers end without a click: someone sees you recommended and searches your name later. Search Console's branded queries filter, introduced in November 2025 and open to all eligible sites since March 2026, splits clicks and impressions into branded and non-branded queries. If branded demand and mention rate rise together for several months, that supports the case; it is not proof, because press coverage, ads and seasons move branded search too.

What to put in the monthly report

• Mention rate, citation rate and share of voice per engine, each with its range.
• Search Console AI impressions, Bing citations, and GA4 AI Assistant sessions and conversions.
• Changes that passed the test, next to what you did that month.
• Wrong facts fixed, and the third-party pages cited for competitors.

For the rest of the report, see what to look for in a monthly SEO report. Measured this way, AI visibility becomes a number you can defend: a rate with a range, tracked under fixed conditions and tied to enquiries wherever the data allows.

If you want help setting up tracking and acting on what it shows, that is part of our AI search optimisation service. When you are ready, get in touch to talk it through.