Run-to-Run AI Visibility Variance: 1,371 Repeated Answers

A generated answer is an observation, not a rank. We tested that statement against Shareof.ai's production monitoring data: 307 prompt-model series with at least two completed runs, covering 1,371 repeated observations across ChatGPT, Claude, Gemini, Perplexity, Google AI, and Copilot.

Every repeated series produced more than one exact response hash. More important, the visible brand outcome sometimes flipped between present and absent even though the tracked prompt and model label stayed the same.

Across the repeated series, exact answer text changed in 100% of cases. Brand-presence flips ranged from 6.3% of ChatGPT series to 23.3% of Copilot series.

The cohort

The observation window ran from August 1 through August 23, 2026. The same project, prompt, and model formed a series. Only series with at least two completed observations were included.

“Answer hash changed” means at least two normalized response texts were not byte-identical. It does not mean the semantic conclusion reversed. “Brand mention flipped” is stricter: at least one run contained the tracked brand and at least one did not.

Finding one: verbatim stability is effectively zero

All 307 series generated more than one answer hash. That is expected in a generative system, but it has a practical consequence: screenshot-to-screenshot copy comparison is a poor monitoring primitive.

Some changes are harmless—sentence order, wording, or a different supporting example. Others change the shortlist, cited sources, or whether the tracked brand appears at all. A useful system therefore needs normalized entities and outcome fields in addition to archived response text.

Preserve the response for auditability. Do not use exact text equality as the outcome.

Finding two: mention volatility differs by surface

ChatGPT produced the lowest brand-presence flip rate in this cohort at 6.3%. Perplexity and Claude were in the middle at 16.7% and 14.3%. Gemini, Google AI, and Copilot ranged from 19.4% to 23.3%.

These are cohort rates, not permanent model rankings. The models covered different numbers of prompts, projects, and dates, and their retrieval behavior can change. Still, the spread is large enough to invalidate a universal “run it twice” rule.

For a prompt with a 20% series-level flip pattern, two observations can easily tell opposite stories. The correct sample size depends on the decision you are making and the volatility you observe.

A better reporting unit

For each prompt-model pair, report a distribution:

• mention rate across completed runs;
• median and range of recommendation position when present;
• source-domain frequency;
• sentiment distribution;
• number of completed and failed observations;
• first and last observation dates.

Then roll those distributions into prompt groups and business-weighted totals. This is the distinction behind share of model versus share of retrieval: the final mention is an outcome with sampling noise, while source presence can help diagnose why it moves.

How many runs are enough?

There is no magic number. For directional monitoring, five to eight repeated observations per prompt-model cell is a defensible starting range because it exposes obvious volatility without exploding cost. This cohort averaged 4.37 to 7.40 runs by model.

Use wider intervals when the observed mention rate sits near 50%, when a query carries high commercial value, or when the model is changing quickly. Use fewer runs only when the result is strongly one-sided and the consequence of error is low.

Do not hide the denominator. “Visibility 67%” means something different at two of three runs than at sixty-seven of one hundred.

Separate three kinds of change

A monitoring system should distinguish:

1. Generation change: different words, same entities and sources.
2. Evidence change: different visible sources, similar answer.
3. Outcome change: the brand enters or leaves the answer, position changes, or sentiment reverses.

The first is common and often unimportant. The second is an early signal. The third is the business outcome.

This hierarchy also prevents false alarms. An alert should not fire because a paragraph was rephrased. It should fire because a high-value prompt crossed an outcome threshold supported by enough completed observations.

Why model documentation does not promise determinism

Provider APIs expose tools and structured evidence, but they do not promise a stable shortlist for repeated open-web questions. OpenAI's web-search tool can retrieve and cite URLs; Google's grounded responses include search queries and grounding chunks; Perplexity returns current search results. Each run can encounter a different retrieval state before generation begins.

The web itself changes, and search-enabled systems can rewrite queries. Variance is therefore not merely “temperature.” It is a property of the whole answer pipeline.

Limitations

This operational cohort is uneven across models and projects. It is not a controlled laboratory experiment: observation dates, prompt mixes, retrieval surfaces, and model versions differ. Exact hashes overstate meaningful change because small wording differences count. Brand detection can miss ambiguous aliases or record false positives, though the stricter flip metric reduces that risk.

The study supports a measurement conclusion, not a league table: repeated sampling is necessary, and volatility should be reported per model and prompt family.

Reproducibility and data

The aggregate series counts and flip rates are published in the study JSON. No customer identifiers, full prompts, or response text are included.

Primary sources and evidence

• OpenAI API quickstart: web search tool↗
• Google: Grounding with Google Search↗
• Perplexity: structured search results↗
• Anthropic: Citations↗

Continue through the system

• A/B Testing Content for AI Citation
• Your AI “Rank” Is a Vanity Metric
• Citation-Domain Persistence After One Week

Run a measured baseline with the free AI visibility scan.