Why an AI Visibility Platform Must Log Whether Search Actually Ran

Your writer says the new comparison page got picked up by ChatGPT. Someone has a screenshot. By the end of the week, that screenshot could sit in the monthly report as proof the page worked.

If you lead content or SEO at a B2B software company, and your CMO now reads an AI visibility number every month, read this before that happens. That fresh citation most plausibly means a live search ran, found the page and quoted it. The model learning your page is the far weaker explanation. An AI visibility platform that can't tell those two events apart will credit the wrong work and blame the wrong pages.

The fix starts with one field. Log whether search ran on every answer you capture. Keep it beside two others, whether you were mentioned and whether your page was cited. Every honest claim you make to leadership rests on keeping those apart.

Why ‘our new page taught ChatGPT’ is usually wrong

So how does a page published last week turn up in a ChatGPT answer this week? Most plausibly through live retrieval. The assistant ran a search, read your page and used it. Model training runs on the provider's schedule, which you can't see, so a week-old page has no visible path into what the model already knows.

Two routes lead into an answer, and they behave differently. Retrieval happens at answer time. The assistant searches, pulls pages and often cites them. Learned association is older. The model carries a link between your brand and a category from data it was trained on, so it can name you without looking anything up.

Uncited mentions trip teams up most. An assistant names your brand with no link, and someone credits yesterday's refresh. That mention can't tell you which document caused it, if any did. Old brand familiarity could explain it. So could a search that drew on pages you've never seen. The screenshot can't separate them.

Blend the two routes and your reporting breaks in both directions. The team credits wins to work that didn't cause them, then repeats that work. Losses arrive with no explanation. If you've trialled a tool whose share-of-voice chart rose and fell without any link to what your team published, a measure that blends the two routes is the first thing to suspect, before you blame the content or the writers.

You feel the biggest cost in the publishing calendar. Once “new page, then ChatGPT picked it up” becomes the story, the obvious next move is more new pages. That's a plan built on a signal nobody checked.

What OpenAI's two crawlers reveal about attribution

OpenAI documents separate crawlers for separate jobs. OAI-SearchBot finds pages so ChatGPT can surface them in search answers. GPTBot crawls content that may be used to train OpenAI's foundation models. You can allow one in robots.txt and block the other [1]. OpenAI has already written your attribution model down for you.

OpenAI's crawler documentation also describes ChatGPT-User, an agent for certain visits a user triggers during a conversation. It says a site that opts out of OAI-SearchBot won't appear in ChatGPT search answers, apart from navigational links, and that one crawl may serve both uses when you allow both bots. That last detail matters. A single log line can't tell you which purpose a visit fed.

Take GPTBot first. A GPTBot entry in your logs shows that a request reached your server. It doesn't show that the page went into any model, and it certainly doesn't show that the model now knows your brand. OpenAI describes the content as something that may be used for training. Report it as a possible training crawl and stop there.

An OAI-SearchBot visit tells you more, and still less than people claim. It shows your page is reachable by the search pathway and that the crawler saw a version of it. That still doesn't show that any specific buyer's answer used the page. Your logs hold the crawler's request. They don't hold the prompt, the answer or the citation, so “the bot read our page, so it powered that answer” skips the one step nobody observed.

The controls also settle a strategic question. Blocking training and allowing search are independent choices. A B2B team can stay eligible for ChatGPT search while declining to feed training. Appearing in results still isn't guaranteed [1]. OpenAI keeps the two decisions apart, so your reporting should too. Crawler logs belong on the input side, where they tell you whether you're reachable and whether the refreshed version got seen. What the answers said belongs in a separate record.

Five AI surfaces, five different retrieval conditions

ChatGPT, Perplexity, Google AI Overviews, Gemini and Claude each reach the web by a different documented route. Some depend on ordinary search indexing, some search only when a setting is on, and Claude can decide per request whether to search at all. A single “AI channel” number hides which route produced a result.

Perplexity documents PerplexityBot as the crawler that surfaces and links sites in its search results, and Perplexity-User as the agent that can visit a page when a user asks. It doesn't describe either as a crawler for training foundation models [2]. A Perplexity citation is a search result, so read it as evidence about reachability and relevance for that question.

Google AI Overviews sit closer to the search you already report on. A supporting page has to be indexed and eligible to show a snippet in Google Search, and Google says there are no extra technical requirements for appearing [3]. If a page drops out of AI Overviews, start with the checks your SEO specialist already runs for organic visibility.

Gemini and Claude need more care, because search there is conditional. Grounding with Google Search connects Gemini to real-time search content and returns source citations when it's enabled [4]. Claude's web search tool supplies current results and citations, and the model decides whether to search based on the request [5]. A switched-on setting doesn't prove a search ran on a given response.

So apply one observable rule on every surface, ChatGPT included. Count an answer as searched only when the response shows it, through a visible search step or linked sources returned with the answer. Where the interface shows neither, log the search state as “not shown” rather than “no”. Keep those answers in their own group, and leave them out of any claim about this week's page.

Here's what that means in practice. The same buyer question can produce a citation on Perplexity, an uncited mention in Claude and nothing in an AI Overview, and each result can be a correct reading of a different system. Average them into one percentage and you throw that information away.

Three outcomes every AI visibility platform should record

For every captured answer, record whether a search ran, whether your brand was mentioned and whether your page was cited, as three separate fields. Each one answers a different question. Fold them into one score and you lose the information you need to explain any change, up or down.

Together, the fields give you a diagnostic split. Call it retrieval problem or memory problem. Here's how the combinations read:

The split also handles the message you'll get when an assistant repeats an old price or a retired feature. Start with search. If search ran and cited an outdated page, yours or someone else's, you have a source problem, and fixing that source is the lever. If the answer shows no sign of search, it most likely came from what the model already holds. Updating your page helps the answers where search runs, and it won't quickly correct the ones where it doesn't.

Now look at the blended visibility score. It mixes mention rate, share of voice, position and sentiment into one index. When it falls, the cause could be fewer searches, a competitor gaining citations or a shift in tone, and the index can't say which. That's why the question “why did it drop?” so often goes unanswered in the meeting.

The “cited but not recommended” result shows the distortion most clearly. Your comparison page appears as a source, and the answer recommends a competitor. A presence count scores that as a win. It may be the reverse, with your page supplying the facts behind someone else's recommendation. Log recommendation framing as its own field next to the citation.

Prompt wording can fake presence too. If a prompt names your brand, a mention proves very little. Keep brand-led prompts out of your visibility panel and use them only to check what assistants say about you.

Why one answer is a bad baseline

AI answers shift between runs, days and weeks, and the sources behind them shift faster than organic results. One screenshot records one event. It can't support a trend, a win or a loss, so your baseline has to be a sampled rate with its uncertainty shown.

A peer-reviewed study in Findings of ACL this year reran a query set two months apart. Only 18% of the pages behind AI Overviews appeared both times, against 45% for Google's organic results [6]. Put simply, the organic overlap was two and a half times larger. The same study found that the answers' broader coverage stayed relatively stable while the sources turned over. An answer can say much the same thing while citing different pages, which is why you track your URL separately from your brand.

Repeating the same question tells a similar story. A preprint published last month asked 50 questions 15 times each across six engines, for 4,500 responses. For each of the five engines answering without web retrieval, 86% to 92% of question cells were still adding an organisation they hadn't named before on the fifteenth run. The one engine that searched the web, Perplexity's Sonar, came closer to exhausting its brand list, yet 64% of its cells still named a new organisation at run 15 and 44% were still adding new cited domains [7].

Read the limit carefully. Only one engine searched, so the engine and the retrieval setting changed together, and the author calls for replication. The study can't tell you that retrieval causes stability. It does tell you that a first answer badly undersamples what an assistant might say, and that a narrower brand list can still sit on top of shifting sources.

So build a fixed panel of high-intent buyer prompts. Write each in an identical form and a naturally worded form, and keep your brand name out. Run them in fresh sessions, on each platform in the search mode your buyers would meet, across successive weeks, and record the date, the platform, the search state and every cited URL against each run so the rate you report has a visible basis. No study sets a universal number of repetitions. Choose a count your team can sustain and hold it fixed.

Your SEO specialist can run a lighter version by hand, logging prompts, search state, mentions and cited URLs from the consumer interfaces in a spreadsheet. A spreadsheet won't catch daily movement, and it won't scale to hundreds of prompts. What it does give you is answer-level evidence, which server logs never will.

Does the distinction matter if the brand appears?

It does. The buyer only sees the answer, but you have to decide what your team does next month. Without the split, you can't tell which work to repeat, which to stop and which result was luck. Budget follows your explanation, so the explanation has to hold up.

The objection is fair as far as it goes. A buyer reading a shortlist doesn't care how the assistant found you. Your next quarter depends on it, though. The retrieval pathway is the one your content work can reach in weeks, and that's where the tested levers sit.

The strongest is substantive content change. In a controlled benchmark presented at KDD two years ago, researchers rewrote source pages to add credible citations, quotations and statistics, and measured visibility in generated answers rose by up to 40% [8]. That figure tracks how much of an answer drew on the source, weighted by position. The benchmark didn't test recommendations or pipeline. Keyword stuffing scored roughly 10% worse than leaving the page alone in the researchers' Perplexity test.

Refreshes earn their place when facts change. Update a page when pricing, features or the set of credible alternatives moves, so a live search finds the current answer. A calendar reminder is a weak reason to touch a page, and a new date with no new information gives retrieval nothing new to find.

Distribution is the third lever with a comparison behind it. In March, Stacker and Scrunch reported on 87 stories from 30 clients and found that 97% of distributed stories earned at least one AI citation, against 82% of the owned versions [9]. That's a 15-point gap in how often a story got cited at all. Stacker ran the analysis on its own distributed stories and describes the data as observational, so treat it as a hypothesis to test on your own coverage.

Notice what none of these levers needs. None of these studies tested publishing frequency, and none supports a quota. The GEO gain came from changing what a page said, and the Stacker gap came from where a story appeared. When someone uses a visibility bump to argue for more output, ask which pathway the bump went through, and whether search even ran.

Choose an AI visibility platform on methodology

Judge a platform by how it collects, labels and qualifies its evidence, then look at features. Ask how it captures responses, whether it logs search mode on every answer, how it joins crawler evidence to answer evidence and how it reports uncertainty. Those answers decide whether its charts can carry an attribution claim.

Collection method comes first. Your buyer uses the consumer interface. A platform that queries a developer API and labels the result “ChatGPT” may be reporting a different model, different settings and different search behaviour from what your buyer sees. Ask which method it uses, engine by engine, and ask to see raw responses alongside the extraction rule that decided a brand was “mentioned”.

Search mode has to be a field on every record, beside mention, citation and framing. If a vendor can't tell you whether a given answer ran a search, every chart it produces blends the two pathways. Treat that as a disqualifying gap for attribution, even if the tool stays useful for broad monitoring. Rank the rest of the feature list the same way. Citation detail down to the exact URL, raw-response exports and crawler diagnostics you can set beside the answers come first, because they let you check the vendor's work. Engine coverage, sentiment and competitor tracking help once that record exists. Attribution claims come last and need the most proof.

Then check how input and output fit together. Crawler logs tell you about access, and captured answers tell you what assistants said. A sound report shows both side by side. If a vendor says it can see which page fed a particular buyer's answer, ask where that data comes from, because neither your server logs nor the documented crawler controls contain buyer prompts, and a confident join with no data source behind it is a guess.

Finally, run your own test. Compare refreshed or added pages against comparable pages you left alone, on the same prompt panel and over the same weeks. Tie the results to AI-referred visits, demo requests and qualified opportunities. An answer can put you into consideration, and the buyer still has reviews, a trial and their own buying group to get through before anyone signs.

That gives you a monthly line you can defend. “When ChatGPT searched our buyer questions, our pages were cited in this share of runs, and here's how refreshed pages compared with untouched ones. We can't show which page fed a particular buyer's answer.” Whatever AI visibility platform you choose, that's the line to hold, and a stated limit earns more trust than a confident chart. If you'd like to see how Eclipse keeps each article tied to the buyer question it answers, the proof behind its claims and a human approval before it goes live, ask us for a walkthrough.

Our Opinion

The search-ran field matters most because it decides next month's calendar. Once a report reaches the CMO, it stops being a record and becomes a plan. So we'd set the attribution rule before a page goes live. Name the buyer question the page exists to answer, the claims and sources it carries, and the result that would count as retrieval working. Then an uncited mention with no sign of search has nowhere to hide. It goes on a separate brand-presence line, and it never justifies another page.

Too many AI visibility conversations still treat the problem as one of output. The studies worth reading point elsewhere. The KDD gain came from changing what pages said, and the Stacker gap, which is observational, tracked where stories appeared. Neither tested how often anyone published. Yet blended scores keep getting used to defend quotas. Our position is blunt. If a platform can't show whether search ran on each answer, its number doesn't belong in the leadership report as an attribution claim. A smaller figure you can explain beats a bigger one you can't.

About the Author

Mark Ridgeon is the founder of Eclipse, the B2B content operating system for companies. He writes about evidence-led go-to-market content, client-scoped workflows and keeping every claim tied to proof from research through publish, a standard that matters most when AI visibility numbers reach leadership. Outside Eclipse, he also takes on fractional COO work through ridgeon.me.

References

1. Overview of OpenAI Crawlers
2. Perplexity Crawlers
3. AI Features and Your Website | Google Search Central | Documentation | Google for Developers
4. Grounding with Google Search | Gemini API | Google AI for Developers
5. Web search tool
6. Characterizing Web Search in the Age of Generative AI
7. Sampling Completeness in Generative Search: Brand and Cited-Domain Accumulation under Repeated Queries
8. GEO: Generative Engine Optimization
9. New Stacker Research: Earned Media Distribution Triples AI