Measure Your Own Share of Retrieval With the Model APIs
Most people test whether AI cites their site by typing their brand into ChatGPT once. Here is how to measure it properly, with code you can run today.
Ask an engineer whether an AI assistant cites their website and you will usually get an anecdote. They opened ChatGPT, typed the company name, saw a link, felt good. Or they did not see a link and felt bad. Either way the sample size was one, the prompt was a vanity phrase no real buyer would type, and the result would have been different on the next run because these systems are stochastic. A single check tells you almost nothing about whether the machine that answers your customers' questions has ever heard of you.
The good news is that the real measurement is not hard to build. Some engines expose the exact set of sources they pulled before writing an answer, which is the layer that actually matters, and you can probe that layer over a realistic set of buyer questions with a short Python program. What follows is the do-it-yourself version of what a tool like shareof.ai runs continuously. The point is to separate two things most people collapse into one: how often you get retrieved, and how often you get named. They are different numbers with different fixes.
Retrieval is upstream of the citation you can see
A retrieval-augmented answer is built in stages. The model rewrites your question into several sub-queries, a step called fan-out. Each sub-query hits a search backend and comes back with a candidate set of documents. ChatGPT uses provider-managed web retrieval, Perplexity operates its own crawler and a provider-managed retrieval stack, and some systems query a vector store instead. A cross-encoder reranker then trims the pooled candidates down to a few passages, those passages go into the context window, and only then does the model write prose and attach citations.
The citation you see in the interface is the last step of that chain. It is the passages that survived every filter and then earned a mention. Your content can enter the candidate set for a sub-query and still get cut by the reranker, or survive the reranker and never get named in the text. So the honest question is not simply "did I get cited." It splits in two. Did my domain appear in the retrieved sources at all, the thing shareof.ai calls Share of Retrieval, and given that it appeared, did the answer actually name me. Measuring only the second number hides the reason you are losing.
Build a buyer prompt set, not a keyword list
The unit of measurement here is a prompt, meaning a full situational question of the kind a customer brings to an assistant. You want somewhere between 30 and 100 of them. Fewer than 30 and variance swamps your signal. More than 100 and the API bill climbs faster than the insight. Thirty to sixty is the sweet spot for a first pass.
Do not write these prompts by staring at a keyword tool. Pull them from where real problems get described in real words: discovery-call transcripts, support tickets, community threads, and your own team's testing. For each buying situation, write out its predictable shapes. A category shape ("best X for a small team"), a comparison shape ("X versus Y for our use case"), a switching shape ("moving off Z, what should we look at"), and a problem shape ("we keep hitting this, what fixes it"). Store them as plain data with a tag so you can group results later.
# prompts.py
# A buyer prompt set. Aim for 30-100 entries.
# The "intent" tag lets you roll results up by cluster later.
PROMPTS = [
{"id": "cat_01", "intent": "category",
"text": "What are the best AI visibility tracking tools for a B2B SaaS team?"},
{"id": "cmp_01", "intent": "comparison",
"text": "How does share-of-model tracking compare to traditional rank tracking?"},
{"id": "swi_01", "intent": "switching",
"text": "We use a keyword rank tracker today. What should we add to see AI answers?"},
{"id": "prb_01", "intent": "problem",
"text": "Our brand does not show up in ChatGPT answers. How do we find out why?"},
# ... continue to 30-100 entries across your real intents
]Keep the identifiers stable across runs. When you compare this month to last month, you want to line up the same prompt against itself, not guess which row moved.
Call the engines that expose their sources
Not every engine hands you its retrieved set, so be honest about what each surface gives you. The Perplexity API is the friendliest starting point because its chat completions responses return a list of the sources behind the answer, which is as close to a raw candidate view as a public API offers. ChatGPT Search exposes sources in its interface and, depending on your access, through its browsing-enabled responses. Gemini surfaces grounding metadata when grounding is switched on. Claude exposes cited URLs when its web search tool runs. Treat the retrieved-source list as the prize and the answer text as a second signal.
The wrapper below leaves the actual HTTP call to you, because keys and endpoints are yours to supply and should never sit in a shared script. Read the key from the environment, not from a literal in the file.
# engines.py
import os
# Load credentials from the environment. Never hardcode a key in source.
# e.g. export PERPLEXITY_API_KEY=... export OPENAI_API_KEY=...
PERPLEXITY_KEY = os.environ.get("PERPLEXITY_API_KEY")
def call_engine(engine, prompt_text):
"""Return (answer_text, source_urls) for one prompt on one engine.
You supply the real request. Each branch should:
1. POST the prompt to that engine's endpoint using its key.
2. Pull the answer string out of the response.
3. Pull the list of retrieved/cited source URLs out of the response.
Return an empty list of sources if the engine ran without retrieval.
"""
if engine == "perplexity":
# --- your Perplexity API call goes here ---
# POST https://api.perplexity.ai/chat/completions with PERPLEXITY_KEY.
# The response carries the answer plus a list of source URLs
# (the field that holds citations in the payload you receive).
answer_text = "" # extract the assistant message content
source_urls = [] # extract the list of citation URLs
return answer_text, source_urls
if engine == "chatgpt_search":
# --- your OpenAI browsing-enabled call goes here ---
# Use a browsing/web-search enabled response so sources come back,
# then read the URLs out of the tool or annotation output.
return "", []
# Add "gemini" (grounding metadata) and "claude" (web search citations)
# the same way, each returning its own answer plus source list.
raise ValueError("unknown engine: " + engine)Where a surface only shows sources in its interface and gives you nothing through an API, you have two options. Log those runs by hand into the same data shape, or drop that engine from the automated pass and note the gap. A partial measurement you trust beats a full one you fabricated.
Normalize domains before you count anything
Raw source URLs are messy. The same site shows up as https://www.example.com/guide?utm_source=x, http://example.com/guide/, and example.com. If you count strings, you will undercount yourself and everyone else. Reduce every URL to a registrable domain first, then compare domains, not URLs.
# normalize.py
from urllib.parse import urlparse
def to_domain(url):
"""Reduce a URL to a bare registrable domain for comparison."""
if not url:
return ""
netloc = urlparse(url).netloc.lower()
if not netloc: # handles bare "example.com/x" input
netloc = url.lower().split("/")[0]
if netloc.startswith("www."):
netloc = netloc[4:]
return netloc.split(":")[0] # strip any port
def mentions_domain(answer_text, domain):
"""True if the answer names your domain or brand token in prose."""
stem = domain.split(".")[0]
hay = (answer_text or "").lower()
return domain in hay or stem in hayThe mentions_domain helper is deliberately blunt, and you should tighten it for your own name. A brand token like "notion" will collide with the ordinary word, so match on something specific to you. This is the difference between retrieval and naming made concrete: to_domain measures whether you were in the sources, mentions_domain measures whether the prose said your name.
Sample each prompt N times to see the variance
Run every prompt once and you are back to the anecdote, just automated. These systems reroll fan-out and reranking on each call, so the same prompt can retrieve you on one run and miss you on the next. Sample each prompt several times per engine, five is a reasonable floor, and treat the appearance rate across those samples as your measurement rather than any single answer.
# run.py
import time
from prompts import PROMPTS
from engines import call_engine
from normalize import to_domain, mentions_domain
ENGINES = ["perplexity", "chatgpt_search"] # add others as you wire them up
SAMPLES = 5 # repeat each prompt this many times
MY_DOMAIN = "example.com" # your registrable domain
def run_all():
rows = []
for engine in ENGINES:
for prompt in PROMPTS:
for n in range(SAMPLES):
answer, urls = call_engine(engine, prompt["text"])
domains = {to_domain(u) for u in urls}
rows.append({
"engine": engine,
"prompt_id": prompt["id"],
"intent": prompt["intent"],
"sample": n,
"retrieved_me": MY_DOMAIN in domains,
"named_me": mentions_domain(answer, MY_DOMAIN),
"source_domains": sorted(domains),
})
time.sleep(1) # be polite to rate limits
return rowsEvery row now records two booleans that matter: whether your domain sat in the retrieved sources, and whether the answer named you. Keep the full list of source domains too, because your competitors' appearance rates are the benchmark that tells you whether a low number is a you-problem or a whole-category cold spot.
Compute Share of Retrieval, share of voice, and the ratio between them
The rollup is where the two numbers finally separate. Share of Retrieval is the fraction of all runs in which your domain appeared in the retrieved sources. Share of voice, in this narrow sense, is the fraction of runs in which the answer named you. Divide the second by the first and you get citation efficiency: given that you were retrieved, how often did that turn into a mention.
# score.py
from collections import defaultdict
def summarize(rows):
by_engine = defaultdict(lambda: {"runs": 0, "retrieved": 0, "named": 0})
for r in rows:
s = by_engine[r["engine"]]
s["runs"] += 1
s["retrieved"] += 1 if r["retrieved_me"] else 0
s["named"] += 1 if r["named_me"] else 0
for engine, s in by_engine.items():
runs = s["runs"] or 1
share_of_retrieval = s["retrieved"] / runs
share_of_voice = s["named"] / runs
# citation efficiency: naming given retrieval
efficiency = (s["named"] / s["retrieved"]) if s["retrieved"] else 0.0
print(engine)
print(" Share of Retrieval: %.0f%%" % (100 * share_of_retrieval))
print(" Share of voice: %.0f%%" % (100 * share_of_voice))
print(" Citation efficiency: %.0f%%" % (100 * efficiency))You can run the same rollup grouped by the intent tag instead of by engine, and that view is usually more actionable. It shows you which buying situations retrieve you and which never do, one cluster at a time.
Reading the three numbers
The whole exercise pays off in the shape the numbers make together, so learn to read the patterns.
High retrieval with high efficiency is the healthy state. Your content is in the candidate sets and the model is comfortable naming you from it. Nothing urgent to fix, so hold the position and watch the trend.
High retrieval with low efficiency is the most useful diagnosis the method produces. The engine is finding your pages, then the reranker or the model is choosing to cite something else from the same pool. That is a content-quality and framing problem sitting on pages that already get retrieved. The fixes are the ones the Princeton GEO study measured: add citations, add statistics, add direct quotation, and tighten the passage so a self-contained chunk answers the sub-query cleanly. You do not have a discovery problem here, you have a persuasion problem on discovered pages.
Low retrieval is the harder and more common failure, and it means the candidate set almost never contains you at all. Naming is moot when you were never in the pool. This is a Retrieval Gap: the distance between the pages that could answer these prompts and the smaller set the system actually pulls. Because most brand mentions in AI answers trace back to third-party pages rather than a company's own domain, the usual cause is that the third-party surfaces answering these questions do not mention you, and the usual fix lives out on those pages rather than in your title tags. Look at the source_domains you logged. The domains that keep appearing when you do not are your map of where the answer is actually being sourced.
One more read is worth the trouble. Compare Share of Retrieval across engines for the same prompts. A cluster that retrieves you well on Perplexity and never on ChatGPT usually points at a crawl or index difference rather than a content one, since the two systems draw from different backends. Check whether the relevant crawler can reach the pages in question before you rewrite a single word.
A note on measurement
Everything above is reproducible by hand, and running it once is genuinely clarifying. The maintenance is the real cost. Engines change their retrieval behavior without notice, prompt sets go stale as your category moves, and a five-sample pass across sixty prompts and four engines is over a thousand API calls you have to babysit and re-run on a schedule to see a trend rather than a snapshot. That upkeep is the part shareof.ai automates, running this same retrieval-versus-naming measurement continuously across ChatGPT, Claude, Gemini, and Perplexity so the numbers stay current. The method stands on its own regardless. Wire up call_engine for one engine, point it at thirty real buyer questions, sample each five times, and you will know more about whether AI cites your website than any amount of typing your own name into a chat box will ever tell you.
Primary sources and evidence
- GEO: Generative Engine Optimization (Aggarwal et al.) — the original paper and benchmark.
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — the foundational RAG paper.
- OpenAI web search tool — source annotations and grounded responses.
- Gemini grounding with Google Search — search grounding and citation metadata.
- Perplexity Search API — retrieving ranked web results programmatically.
Continue through the system
- Build a Brand-Mention Monitor Across Four LLMs in an Afternoon
- A/B Testing Content for AI Citation: A Methodology That Survives the Noise
- Reading Your AI Crawler Logs: GPTBot, ClaudeBot, PerplexityBot and Friends
Run the practical layer with the free AI visibility scan, explore the AI search library, or see the AI visibility benchmarks.