Back to the blog

Measuring AI Visibility on a Growing Prompt Set: Fixed Panels, Pooled Windows and Error Bands

Jeremy Ashburn
By , founder of PushLeads
Two colleagues work at laptops in a modern office while wall monitors show abstract charts, suggesting data measurement and analysis.

When an AI visibility score is computed as a rate over “whatever prompts are currently tracked,” adding prompts changes the score even if nothing about the brand changed. We hit this on November 22, 2026: auto-expansion added 39 new buyer prompts, the buyer-intent prompt-engine pairs in our PushLeads panel went from 99 to 203, and the headline score fell from 30.7 to 23.0 overnight. On the prompts tracked since October 26, the same brand rose every week, from 27.2 to 36.8. This note documents how we rebuilt the score as a fixed-panel, 28-day pooled rate with a 95% error band, how new prompts are tracked without touching the headline, two data-quality faults the rebuild exposed, and what the major AI visibility tools and classic rank-tracking indices do about the same problem.

Key moments (19)
0:00 AI Visibility Scores: What Really Moves the Number
1:03 Who this is for
1:25 One Weekend, Two Opposite Stories
1:58 What Actually Happened That Weekend
2:26 What did the score look like on fixed questions only?
3:02 AI Answers Are Rewritten Every Time
3:47 The Batting Average Problem in AI Tracking
4:22 How Other Tools Handle This Problem
4:49 What exactly goes into an AI visibility score?
5:44 Three Rules for a Trustworthy Score
6:39 What the Method Switch Showed Across Clients
7:17 Two Bugs We Found — and Fixed
7:59 What are Google AI Overviews actually doing right now?
8:45 Fixed Score vs. Shifting Score: Side by Side
9:24 What Rand Fishkin’s Study Confirmed
9:56 Questions to Ask Any AI Visibility Report
10:38 Your Weekly AI Visibility Routine
11:17 Quick Recap
11:34 See Exactly Where Your Business Stands
Full transcript

0:00 AI Visibility Scores: What Really Moves the Number
Our own AI visibility score dropped seven-point-seven points in a single weekend — and the business did nothing wrong. The question list changed overnight. Your AI visibility score dropped. Before you change a single thing about your marketing, check one thing first: did your tracker add new questions? In this video we show you exactly what happened to our own score on a single weekend in November — a seven-point drop that had nothing to do with how well the business was performing. We will walk through why AI scores are harder to measure than Google rankings, the two bugs we found and fixed in our own system, and what Google AI Overviews are doing right now with YouTube links. By the end you will know what a trustworthy AI visibility report looks like — and what questions to ask if yours isn’t one. In the next few minutes, why scores drop without warning, the two bugs we found and fixed, and what a trustworthy AI visibility report looks like.

1:25 One Weekend, Two Opposite Stories
On Saturday, November twenty-first, twenty twenty-six, the PushLeads AI visibility score read thirty-point-seven. Sunday morning it read twenty-three-point-zero. A drop of seven-point-seven points in under twenty-four hours. If you saw that in a client report, you would call it a terrible week. It wasn’t. The questions changed, not the business. The number moved because the question list changed overnight — and that is a problem every AI visibility tracker has to solve.

1:58 What Actually Happened That Weekend
Here is what actually happened. Over that weekend the system added thirty-nine new buyer questions — the kind a restoration company owner types when shopping for marketing help. Those questions were asked for the very first time. Claude linked to PushLeads on about twelve percent of those new answers. Perplexity linked on about thirty-eight percent. Those are low first-week numbers, and when averaged into the headline score they dragged the whole number down. Not because visibility fell. Because the question set got harder.

2:26 What did the score look like on fixed questions only?
What did the score look like on fixed questions only? Now look at the same business using only questions tracked since October twenty-sixth. Week ending October twenty-sixth: twenty-seven-point-two. November second: twenty-nine-point-six. November ninth: thirty-two-point-nine. November sixteenth: thirty-five-point-seven. November twenty-second: thirty-six-point-eight — range plus or minus seven. The score rose every single week. Same business. Same weekend. Two completely opposite stories. The only difference is which questions got counted.

3:02 AI Answers Are Rewritten Every Time
To understand why this matters, you have to understand what makes AI visibility so much harder to measure than a Google ranking. A Google ranking is a position — third or ninth. An AI answer is a paragraph rewritten fresh every time someone asks. A study published in January twenty twenty-six had six hundred volunteers run twelve recommendation questions nearly three thousand times across ChatGPT, Claude, and Google’s AI features. Result: less than a one-in-one-hundred chance ChatGPT or Google’s AI returned the same brand list twice. One answer tells you almost nothing. What matters is how often you show up across many questions and many weeks — and that rate only means something if the questions stay the same.

3:47 The Batting Average Problem in AI Tracking
Think of a batting average. If the coach adds five games against the best pitcher in the league mid-season, the average drops. The hitter did not get worse. The schedule got harder. AI visibility works exactly the same way. New questions start low by definition. When you mix a low first-week score into a mature average, the average falls even if everything else is going up. That same Sunday, the score on our fixed question set read thirty-six-point-eight. The old mixed score read twenty-three.

4:22 How Other Tools Handle This Problem
This is not a problem unique to PushLeads. Peec AI’s research team tells users to freeze the prompt set. Profound’s help center says keep prompts consistent for the reporting window if you want fair comparisons. Classic rank trackers learned this years ago — visibility scores only mean something on a stable keyword list. PushLeads now follows the same rule: two-hundred-twenty-five fixed questions, reviewed once a quarter.

4:49 What exactly goes into an AI visibility score?
What exactly goes into an AI visibility score? Here is what goes into our AI visibility score. First: was the business named in the answer? Second: did the answer link to the website or one of its videos? We count both, on every engine, every week. Third: the question set. The headline uses a fixed set, and new questions get their own line. Fourth: the time window. We pool four weeks of answers and show a range. Fifth: the engines. Google AI Overviews, Google AI Mode, ChatGPT, Perplexity and Claude behave very differently, so each one is reported on its own. Sixth: what happens when Google shows no AI Overview at all. That should not count as a miss, and in our score it does not. Get a Free AI Visibility Check for your business — the link is in the description.

5:44 Three Rules for a Trustworthy Score
As of November twenty-second, twenty twenty-six, every PushLeads client dashboard runs on three rules. Rule one: the headline score only counts questions tracked since October twenty-sixth — two-hundred-twenty-five questions. The set changes only at a scheduled quarterly review, marked on the chart. Rule two: the score pools four weeks of answers and shows a range — for example, thirty-seven plus or minus seven. The range is the honest part. If your score moves three points and the range is seven, nothing has really changed. Rule three: new questions get their own line in the report. Ours reads nineteen on one-hundred-thirty-seven new questions. After four weeks of history they can join the main set at the next quarterly review.

6:39 What the Method Switch Showed Across Clients
When we switched every client account at once, here is what we found. Thirty-six of the fifty-three businesses with enough history moved less than five points — the method change barely touched them. Six went up by five or more points. Eleven went down by five or more. For those accounts, the old mixed number had been sitting higher than the fixed-question score. Five newer accounts are showing a provisional score until they build four weeks of history. A lower number after a method change is not bad news. It is a more honest starting line, and every week after it you are comparing like with like.

7:17 Two Bugs We Found — and Fixed
Rebuilding the score meant reading raw data line by line, and we found two bugs. The first: the service we use to read Google results returned server errors on a slice of AI Overview checks, so some questions were simply missing. The fix was to re-ask any question that errored. On the first run, the system recovered one-hundred-forty-two of the one-hundred-forty-four missing checks. The second bug: one engine’s name-check could not see the business name for some accounts, so it never counted a mention there. Fixed — and we corrected historical answers where the right result was certain. We are telling you about both because a report you cannot question is not worth much.

7:59 What are Google AI Overviews actually doing right now?
What are Google AI Overviews actually doing right now? Now let’s talk about what Google AI Overviews are actually doing right now. For the questions we track for PushLeads, Google showed an AI Overview ninety-seven to ninety-nine percent of the time every single week. They are not going away. What changed is what they link to. In late October, about forty percent of those answers linked to one of our YouTube videos. By mid-November that dropped to about twenty-eight percent. The average answer listed fewer YouTube links — about two per answer at the peak, down to about one-point-four in the latest week. Links to the website stayed in the same range the whole time. If you have been making videos, keep going. Watch the number, and don’t read one month as a trend.

8:45 Fixed Score vs. Shifting Score: Side by Side
Here is a direct comparison to evaluate any AI visibility report. A score worth acting on is built on a fixed question set, shows a range instead of a single number with false precision, tracks new questions separately, reports each AI engine on its own line, excludes no-overview weeks from the miss count, and discloses every method change with before-and-after numbers. A score that is measuring the question list instead of your business does the opposite on most of those. If you go through that checklist and most answers are no, the score may be moving because the tool changed — not because your visibility did.

9:24 What Rand Fishkin’s Study Confirmed
The research backs this up. In January twenty twenty-six, Rand Fishkin of SparkToro and Patrick O’Donnell of Gumshoe-dot-ai ran six hundred volunteers through twelve recommendation questions nearly three thousand times. Finding: less than a one-in-one-hundred chance ChatGPT or Google’s AI returned the same brand list twice. That is not a measurement problem you fix by checking your score more often. It is a reason to track rates across many questions and many weeks — on a stable prompt set so the rate actually means something.

9:56 Questions to Ask Any AI Visibility Report
Three questions to ask about any AI visibility report right now. First: is the headline built on a fixed question set, or does it shift every time questions are added? Second: does it show a range, or one number with false precision? A score of thirty-seven plus-or-minus seven is honest. A score of thirty-seven with no range implies certainty that AI tools simply cannot deliver. Third: are engines reported separately? Google and ChatGPT behave very differently. A single pooled number hides where you are winning and where you are weak. If the answers to those three questions are no, the report is measuring the tool — not your business.

10:38 Your Weekly AI Visibility Routine
A simple weekly routine that takes the guesswork out of this. First, check the fixed-question score — ignore the headline if new questions were recently added. Second, compare four-week trends, not single weeks. Single-week moves under about five points are mostly noise. Third, watch the video-link percentage if your visibility leans on YouTube — that twenty-eight percent share is the number to track. Fourth, review new questions on their own line to find weak areas without letting them rewrite history. Fifth, look at engine-level breakdowns monthly. Sixth, flag any method change immediately and ask for the before and after shown side by side.

11:17 Quick Recap
Quick recap. First, freeze your question set — mixing in new questions drags the score down even when visibility is rising. Second, demand a range, not a single number. Third, check each AI engine on its own line so you can see where you are winning and where you are weak.

11:34 See Exactly Where Your Business Stands
A good AI visibility report shows you the same questions over time, a range instead of false precision, and every AI engine on its own line. That is what we track for every client. See exactly how your business shows up in Google and AI search — find the link in the description, or call eight two eight, three four eight, seven six eight six.

Headline result

PushLeads, like-for-like on a fixed panel (225 prompts across Google AI Overviews, Google AI Mode, ChatGPT and Perplexity, plus a fixed AI Overview keyword sweep), 28-day pooled score with 95% band:

As of Fixed-panel score Band (95%) Old mixed-set score (as shown that day)
2026-10-26 27.2 ±5.2 26.9
2026-11-02 29.6 ±5.2 25.0
2026-11-09 32.9 ±5.9 30.0
2026-11-16 35.7 ±6.9 30.4
2026-11-22 36.8 ±7.1 23.0

The fixed-panel series rises in every interval. The mixed series zig-zags and ends with a 7.7-point “drop” that is entirely composition. The new prompts, scored separately, sit at 18.8 across 137 prompts.

Across the 53 client accounts with enough history to compute both, switching the headline from the mixed set to the fixed panel moved 36 accounts by less than 5 points, raised 6 by 5 or more and lowered 11 by 5 or more (median change 0). Five accounts onboarded after the panel was frozen show a provisional score until they have their own four weeks.

The problem: a ratio over a moving denominator

The previous score took the latest answer per engine and prompt from the last 30 days and computed cited and named rates over that set. Any prompt added to a client’s set entered the denominator on its first answer. Prompt-set expansion is deliberately biased toward gaps, so new prompts start low, and the blended rate falls. The November 22 example:

  • 39 new “vertical problem” buyer prompts went live on Claude and Perplexity (42 measured that week versus 3 the week before).
  • First-week results on that bucket: Claude cited the brand on 11.9% of answers, Perplexity on 38.1%.
  • Buyer-intent prompt-engine pairs in the 30-day window: 99 on November 21, 203 on November 22.

This is the same mix effect that same-store sales and price indices exist to remove. It is also why raw counts across weeks should never be compared when the basket changes.

How AI visibility tools and rank-tracking indices handle it

No vendor we reviewed ships an automatic chain-linked index for prompt-set changes. They keep the denominator stable by policy and segment everything else.

Tool or index How prompt or keyword changes are handled
Peec AI Research team advises freezing the prompt set; archive keeps history, delete erases it; tags separate old and new prompts (Peec).
Profound Totals change when prompts are added, removed or reworded; keep the set consistent for the reporting window; disable rather than delete (Profound help).
Otterly.AI Removing a prompt recalculates the brand report without it; no backfill for new prompts.
Semrush Position Tracking New keywords have no history; notes are added to the trend graph when keywords are added or deleted (Semrush KB).
Sistrix Visibility Index Fixed representative keyword set; a fixed proportion is exchanged regularly (Sistrix).
AccuRanker Per-keyword option to ignore a keyword in Share of Voice (AccuRanker help).

The statistical fix comes from official price statistics: introduce new items at a review point and link the old and new series through an overlap period (UNECE guidance on linking index series).

How much noise is in one AI answer

Single answers are close to useless as measurements. In a January 2026 study by Rand Fishkin (SparkToro) and Patrick O’Donnell (Gumshoe.ai), 600 volunteers ran 12 recommendation prompts 2,961 times across ChatGPT, Claude and Google’s AI features; there was less than a 1 in 100 chance that ChatGPT or Google’s AI returned the same brand list twice (summary). A University of St. Gallen preprint recommends at least 7 runs per prompt per day for brand-level monitoring and 2 to 4 week rolling windows (arXiv:2604.07585). Peec’s analysis of roughly 120 million chats argues for more prompts over more runs, because prompt-to-prompt variance dominates (Peec).

Our collection runs about one answer per prompt per engine per week, so the only defensible unit is a pooled multi-week rate with an explicit interval. At roughly 100 buyer pairs and a 30% rate, a single engine-week has a 95% half-width near 9 points; pooled across engines and weeks it narrows, but week-over-week moves under about 5 points for one brand should be read as noise.

Implementation

Panel definition

The baseline set (frozen October 26, 2026) holds every prompt measured since October 13 plus each client’s AI Overview keyword sweep. The headline panel, core_v1_2026_11_22, is the subset of that baseline still being measured after October 25: 3,001 prompt-engine pairs and 6,584 AI Overview sweep keywords across the client base. Membership is fixed, and history is recomputed backwards on the same membership.

The intersection matters. The raw baseline included prompts asked once in October and later rotated out of their engine’s schedule. Left in, they aged out of the rolling window and shrank the panel’s own denominator (PushLeads buyer pairs fell from 97 to 60 between the November 9 and November 16 windows), which is a composition change inside the “fixed” set.

Window and deduplication

All answers in the 28 days ending on the as-of date, one answer per client, engine, prompt and day (latest wins). Every weekly answer counts, so the rate is a pooled four-week rate rather than a single latest snapshot.

Score

Weights are unchanged from the previous score, so levels stay interpretable:

score = 100 * [ 0.75 * (0.45*buyer_cited + 0.30*buyer_named
                       + 0.15*head_cited + 0.10*all_cited)
              + 0.25 * aio_sweep_cited ]
(client with no AI Overview sweep: the bracketed block alone, whose weights sum to 1.0)

Error band

band = 1.96 * 100 * sqrt( sum_i  w_i^2 * p_i * (1 - p_i) / n_i )

Here w_i are the effective component weights, p_i the pooled rates, and n_i the number of distinct prompt-engine pairs (or distinct keywords for the sweep), not the number of answers. Repeated weekly answers to the same prompt are correlated, so counting answers would overstate precision. Cited and named rates are treated as independent, which slightly understates the band. For PushLeads on November 22 the result is 36.8 ± 7.1.

Expansion score

Prompts outside the panel get the same formula, without the sweep component, and are reported next to the headline as an expansion score with a prompt count. They never enter the headline.

Google AI Overview “no overview” state

An AI Overview check where Google showed no overview is excluded from the denominators rather than scored as a miss, so the AI Overview component measures presence when an overview appears, not Google’s trigger rate. For the PushLeads panel the trigger rate ran 96.8% to 99.4% per week, so the effect here is small; it matters more in niches where overviews trigger less often.

Storage and serving

A SQL function computes the panel and expansion scores for any as-of date. A daily job writes them to the history table (backfilled for October 26 to November 22, 1,607 client-day rows), and the dashboard, trend chart and client reports read the panel score, with the mixed score kept as a separate column for audit.

Two data-quality faults the rebuild exposed

Unretried SERP errors on AI Overview checks

The SERP provider returned “Internal SE Server Error” task statuses on a slice of live AI Overview requests in the November 22 run. Failed tasks were correctly not written as “no overview,” but the four fleet batches had no retry pass, so 144 prompts were simply missing that week. A retry driver now re-asks only prompts with no row since the batch started, for up to two passes; the first run recovered 142 of 144 for $1.15.

Empty alias lists silently zeroed a “named” signal

The Perplexity probe matched brand aliases against Perplexity Search API results, and 21 of 58 client prompt sets had no alias list, so “named” could never be true for those clients. For PushLeads it had been stuck at 0 since October 18 while “cited” ran around 70%. The probe now falls back to the client name. Past rows were corrected only where the outcome is deterministic: when the client name is a substring of the client’s domain, a cited result implies the name appears in the result URL, so named equals cited (109 rows across two clients). Other clients’ past Perplexity “named” values were not changed.

Side finding: YouTube citations in AI Overviews

Across all PushLeads AI Overview prompts, weekly, counting only answers where an overview appeared:

Week of Brand cited (any) Own site cited Own YouTube video cited YouTube links per answer
2026-10-19 43.9% 18.7% 34.4% 1.55
2026-10-26 42.8% 9.6% 39.6% 1.95
2026-11-02 37.4% 13.7% 29.4% 1.29
2026-11-09 28.5% 9.7% 25.3% 1.01
2026-11-16 31.6% 15.6% 27.8% 1.41

The decline in AI Overview citations for this brand is carried by video: own-video citation fell from a 39.6% peak to the high 20s while own-site citation stayed in a 10% to 19% band. That lines up with fewer YouTube links per answer overall, and with the drop in citations per answer we reported in 12 weeks of AI Overview data. It is not a trigger-rate artifact, since overviews appeared on 97% or more of these prompts.

Re-baselining plan

  • Quarterly review dates; the next is January 4, 2027.
  • A candidate prompt graduates only with at least 4 consecutive weeks of valid data on each tracked engine and at least 90% valid answers.
  • Graduation is capped at roughly 20% to 25% of the panel per review, so each step is modest.
  • At each review, the old and new panels are linked through their overlap period (ratio link, or an additive link for brands scoring under about 5 points, where ratios explode), and the break is annotated on the chart.
  • Claude was added after the October 26 freeze, so it is outside the current panel; it now has four weeks of history and is eligible at the first review.

Limitations

  • One answer per prompt per engine per week is below the run counts recommended in recent research; the pooled window and band compensate but do not replace repeated sampling.
  • The band ignores correlation between cited and named and between engines, so it is approximate.
  • Panel prompts were chosen by people and expansion jobs, not sampled from real query logs, so levels describe the panel, not the whole market.
  • The AI Overview sweep reads Google through an API provider, not a logged-in browser, and cannot see personal interface experiments.
  • Scores from different tools use different prompts, engines and weights and are not comparable with each other.

A plain-English version of this change for business owners is on PushLeads: why your AI visibility score jumps around, and how to read it.