Technical playbooksPAPER 20

Reading Your AI Crawler Logs: GPTBot, ClaudeBot, PerplexityBot and Friends

Your access log already knows which AI engines can see you. Here is how to read it, and why the live-fetch agents matter more than the training crawlers.

Chris PelkinsResearch & editorial, Shareof.ai

Published July 1, 2026Updated August 26, 2026 · 12 min read

Every request an AI system makes to your server leaves a line in your access log, and most teams have never looked at those lines. The data is sitting there right now, timestamped and complete, in /var/log/nginx/access.log or wherever your reverse proxy writes. It records which bot arrived, which URL it asked for, when, and what status code it got back. That is a running account of which parts of your site the AI engines can actually reach, and it costs nothing to read. The reason to read it is that crawling is upstream of everything else. A page an engine has never fetched cannot enter a candidate set, cannot be reranked, cannot be cited. The log is the earliest place you can watch that gate open or stay shut.

Before you can measure anything you need to know which visitors are the ones you care about, and the AI bots do not hide. They announce themselves in the User-Agent string on every request. The problem is that there are more of them than most people expect, they belong to different companies, and two agents from the same company can mean completely different things about your visibility.

The user-agents, and the split that matters

Start with the real strings. OpenAI operates three: GPTBot, which crawls pages to build training data; OAI-SearchBot, which builds the index behind ChatGPT search; and ChatGPT-User, which fetches a page live when a user's question sends the assistant to the open web. Anthropic runs ClaudeBot, its general crawler, and Claude-User, which retrieves a page in real time to answer someone in a Claude session. Perplexity has PerplexityBot for indexing and Perplexity-User for live fetches tied to a specific question. Then there are the standalone crawlers: Google-Extended, which is a permission token in your robots.txt rather than a distinct fetcher, governing whether Google may use your content for Gemini and Vertex training; CCBot, the Common Crawl agent whose open dataset feeds the training pipelines of dozens of models; and Bytespider, ByteDance's aggressive crawler.

The distinction to burn into your parsing is between crawlers and fetchers. GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, CCBot, and Google-Extended are batch operations. They visit on their own schedule to build or refresh a corpus, and a hit from one tells you your page is eligible to be known, someday, in some future answer. The -User agents work differently. When you see ChatGPT-User, Claude-User, or Perplexity-User in your log, a human asked a question a few seconds earlier, the model decided your page was worth reading to answer it, and it went and got the page live. That request is a retrieval happening in real time rather than corpus maintenance, with a person waiting on the other end.

That second category is the closest thing you have to a direct visibility signal from inside the black box. A spike in Perplexity-User hits on your pricing page means Perplexity is pulling that page into live answers for real questions right now. If you have ever wanted proof that your content is being retrieved rather than merely indexed, the live-fetch agents in your log are it, and almost nobody is watching them.

Parsing the log by hand

You do not need a pipeline to start. A combined-format nginx or Apache log puts the user-agent in the last quoted field, so a single pass with grep gives you the raw counts. This pulls every AI agent hit and tallies them:

grep -aE 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|PerplexityBot|Perplexity-User|CCBot|Bytespider|Google-Extended' /var/log/nginx/access.log \
  | grep -oE 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|PerplexityBot|Perplexity-User|CCBot|Bytespider|Google-Extended' \
  | sort | uniq -c | sort -rn

The output is a leaderboard of who crawled you and how hard. To see which URLs an agent is actually reading, narrow to one bot and pull the request path, which sits in the seventh whitespace field of the combined format:

grep -a 'Claude-User' /var/log/nginx/access.log \
  | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

Run that for a -User agent and you are looking at your most-retrieved pages, ranked. Run it for ClaudeBot and you are looking at crawl coverage, which pages the indexer bothers to keep fresh. The gap between those two lists is worth staring at. Pages that get crawled but never live-fetched are indexed and ignored. Pages that get live-fetched often are carrying your visibility.

For status codes, add the ninth field. A crawler hammering a section that returns 404 or 301 is wasting its budget on you, and every one of those is a page an engine tried to read and could not use:

grep -a 'GPTBot' /var/log/nginx/access.log \
  | awk '{print $9}' | sort | uniq -c | sort -rn

A parser that separates training from live retrieval

Grep gets you started, but the moment you want the training-versus-live split over time you want a small script. This one reads a combined-format log, classifies every AI hit into a crawler bucket or a fetcher bucket, and reports the split per agent along with the top URLs each fetcher pulled:

import re
import sys
from collections import Counter, defaultdict

CRAWLERS = {"GPTBot", "OAI-SearchBot", "ClaudeBot", "PerplexityBot",
            "CCBot", "Bytespider", "Google-Extended"}
FETCHERS = {"ChatGPT-User", "Claude-User", "Perplexity-User"}
AGENTS = CRAWLERS | FETCHERS

# combined format: host - - [time] "METHOD /path HTTP/1.1" status size "ref" "ua"
LINE = re.compile(r'(?P<host>\S+) \S+ \S+ \[[^\]]+\] '
                  r'"\S+ (?P<path>\S+) [^"]*" (?P<status>\d{3}) \S+ '
                  r'"[^"]*" "(?P<ua>[^"]*)"')

hits = Counter()
paths = defaultdict(Counter)
hosts = defaultdict(set)

for line in open(sys.argv[1], errors="ignore"):
    m = LINE.match(line)
    if not m:
        continue
    ua = m.group("ua")
    agent = next((a for a in AGENTS if a in ua), None)
    if not agent:
        continue
    hits[agent] += 1
    paths[agent][m.group("path")] += 1
    hosts[agent].add(m.group("host"))

crawl_total = sum(hits[a] for a in CRAWLERS)
live_total = sum(hits[a] for a in FETCHERS)
print(f"training/index hits: {crawl_total}   live-fetch hits: {live_total}")

for agent in sorted(hits, key=hits.get, reverse=True):
    kind = "LIVE" if agent in FETCHERS else "crawl"
    print(f"\n{agent} [{kind}] {hits[agent]} hits from {len(hosts[agent])} IPs")
    for path, n in paths[agent].most_common(5):
        print(f"    {n:5d}  {path}")

Point it at a log and the first line alone reframes how you think about the traffic. A high crawl total with a near-zero live total means the engines know your pages exist but rarely reach for them when answering. A healthy live total, and a rising one, means your content is winning retrievals. The per-agent breakdown then tells you which engine and which pages, and the IP count per agent is the hook for the verification step you will need later.

What to actually measure

Raw hit counts are the start, not the destination. Four derived numbers turn the log into a visibility instrument.

Crawl frequency is how often each crawler returns to a given page or section. Compute it as hits per URL per week and watch the trend. When an engine increases its crawl rate on a section after you publish there, it is signalling that the section is worth re-indexing. A crawl rate that decays toward zero on pages that used to get attention is an early warning that you are drifting out of the corpus.

Coverage is the fraction of your important URLs that a given agent has fetched at all in a window. Take your list of pages that should be answering questions, intersect it with the paths each bot requested, and you have a coverage percentage per engine. Low coverage is a plumbing problem before it is a content problem: something in your robots rules, your sitemap, or your internal linking is keeping the crawler away from pages you want it to have.

The training-versus-live split is the ratio the parser prints. Track it per section over time. A section that is all crawl and no live-fetch is indexed but not retrieved, and that is precisely the shape of the Retrieval Gap: the pages could answer queries, the engine has them, and it still pulls something else. A section where live-fetches climb is a section moving up the funnel.

Live-fetch destinations are the single richest signal in the file. Log every -User hit with its path and its timestamp, and you are building a real-time map of which of your pages the assistants reach for, on which engine, and how that shifts week to week. Cross-reference a live-fetch spike with a content change you made and you can see retrieval respond to your edits, which is feedback no keyword tool will ever give you.

Where the log fits in the funnel

The AI Visibility Funnel runs Retrievability, Candidacy, Citation, Attribution. Your access log is the instrument for the first stage, and the first stage is the one everyone skips because it is invisible in every dashboard that watches final answers. A brand can obsess over whether ChatGPT names it while having no idea whether ChatGPT can even fetch its pages. Crawl data closes that blind spot. Coverage and crawl frequency measure Retrievability directly. The live-fetch agents give you a read on the boundary between Retrievability and Candidacy, because a live fetch means the engine judged your page relevant enough to a specific question to spend the round trip.

This is why crawl logs work as a leading indicator for Share of Retrieval, the fraction of candidate sets your content appears in. You cannot see the candidate sets from outside, but you can see the crawling that populates them and the live fetches that sample from them. When crawl coverage rises on a section and live-fetches follow a few weeks later, you are watching Retrievability convert into Candidacy in your own log. When you publish onto a third-party surface and start seeing ClaudeBot pick it up, you are watching your Citation Surface expand in real time, page by page, as the crawlers find the new material.

robots.txt, Google-Extended, and who you let in

Your log tells you who came. Your robots.txt decides who is allowed to, and the two files should be read together. Each of these agents obeys robots.txt by product policy, so a Disallow keyed to GPTBot keeps OpenAI's training crawler out while leaving OAI-SearchBot and ChatGPT-User free to index and fetch. That distinction is a real lever. You can decline to feed the training corpus while staying fully retrievable in search and live answers, or the reverse, by naming the specific agents rather than blanket-blocking.

Google-Extended is the odd one because you will never see it hitting your server. It works as a token you place in robots.txt that tells Google whether it may use content it already crawls with Googlebot for Gemini and Vertex model training. Blocking it changes nothing in your access log and everything in what Google is permitted to do with pages it fetches for ordinary search. Read your robots.txt against your log and check for the accidental own-goal: a stale Disallow that is quietly keeping OAI-SearchBot or PerplexityBot off the pages you most want retrieved. Teams inherit these rules from a template and never revisit them, and a single line can wall you out of an entire engine.

The pitfall: the user-agent string is a claim, not proof

Everything above trusts the User-Agent header, and that header is trivially forgeable. Anyone can send a request that says GPTBot in the user-agent, and scrapers routinely impersonate the well-behaved bots to slip past rate limits or to make their traffic look legitimate. If you act on raw user-agent counts alone you will overstate your AI visibility, sometimes badly, because a chunk of what claims to be an AI crawler is something else wearing the name.

Verification closes the hole, and the providers give you the means. OpenAI, Anthropic, Perplexity, and Google publish the IP ranges their agents fetch from, several of them as machine-readable JSON you can pull and refresh. The strong check is a reverse DNS lookup on the requesting IP followed by a forward lookup back to the same address, the same method Google has long recommended for verifying Googlebot. A real GPTBot request reverse-resolves into OpenAI's published domain and forward-resolves back to the original IP. A spoofed one does not. Verify against the published ranges before you trust a number, tag each hit as verified or unverified in your parser, and report on the verified traffic only. The unverified stream is worth keeping in view too, because a rise in fake ClaudeBot traffic is its own signal, just not one about your visibility.

Start with one command

The whole practice begins with a single grep against a log you already have. Run the leaderboard, find your live-fetch agents, and look at which pages they pull. Most teams discover two things in the first ten minutes: some section they care about is getting no AI crawl traffic at all, and some page they never think about is quietly being live-fetched by an assistant answering real questions. Automating the reverse-DNS verification and the coverage math, then rolling it up across every engine, is what a tool like shareof.ai does, and it connects the crawl signal to the retrieval and citation data downstream so the whole funnel reads as one picture. The log parsing stands on its own, though. You can build every number in this article with the tools already on your server, and the first time you watch a live-fetch spike track an edit you made, the crawl log stops being an afterthought and becomes the first place you look.

A note on measurement

Two cautions before you lean on these numbers. Crawl activity is necessary for visibility but not sufficient: a page can be fetched constantly and still never make it into an answer, so treat the log as a leading indicator to be confirmed against actual retrieval and citation data, never as the finish line. And the agent landscape moves. New user-agents appear, companies split one crawler into two, and published IP ranges change, so keep your agent list and your verification data current or your parser will quietly miss traffic. Read against those limits, your access log is the cheapest and earliest visibility signal you own, and it is already being written.

Primary sources and evidence

OpenAI web crawlers — GPTBot, OAI-SearchBot, and ChatGPT-User controls.
Perplexity crawler documentation — PerplexityBot and Perplexity-User guidance.
Google common crawlers — Googlebot and Google-Extended distinctions.

Continue through the system

Run the practical layer with the free AI visibility scan, explore the AI search library, or see the AI visibility benchmarks.