WE PUBLISH WHAT WE LEARN

Why an AI Visibility Platform Must Log Whether Search Actually Ran

Your writer says the new comparison page got picked up by ChatGPT. Someone has a screenshot. By the end of the week, that screenshot could sit in the monthly report as proof the page worked.

If you lead content or SEO at a B2B software company, and your CMO now reads an AI visibility number every month, read this before that happens. That fresh citation most plausibly means a live search ran, found the page and quoted it. The model learning your page is the far weaker explanation. An AI visibility platform that can't tell those two events apart will credit the wrong work and blame the wrong pages.

The fix starts with one field. Log whether search ran on every answer you capture. Keep it beside two others, whether you were mentioned and whether your page was cited. Every honest claim you make to leadership rests on keeping those apart.

Why ‘our new page taught ChatGPT’ is usually wrong

So how…

Mark Ridgeon15 MIN READ

FROM THE DESK

Every article starts with evidence, keeps its sources attached and publishes only after human approval.

LATEST ARTICLES

7 ARTICLES IN THE ARCHIVE

Illustration of a glass jar with a black lid, filled with overlapping black and orange pills on a plain white background.

Why Your AI Visibility Platform Needs Repeated Prompt Tests

A single AI answer that names your brand is the weakest number you can bring to a budget review. It looks like proof. Teams screenshot it, report it as a content win, and move on before anyone checks whether it comes back the next day.

This article is for B2B content and SEO leads who defend a budget line by line. You're choosing between publishing more, refreshing what exists and earning coverage on other sites. If you pay for an AI visibility platform, or you're about to, one feature decides whether its numbers survive a CFO's questions. It's how the platform samples. ChatGPT, Perplexity, Gemini and Google's AI Overviews give different answers to the same question from one run to the next.

So here's the argument. Treat AI visibility as a rate you estimate from repeated tests of fixed buyer prompts, across surfaces and across weeks. Decide nothing from a snapshot. A tool that reports single runs is handing you anecdotes with decimal points.

Why is one favourable AI answer not a result?

One favourable answer is a single draw from a system built to vary. It proves your brand can appear. That's all. It can't tell you how often you appear, on which surface, or whether your content caused it. A content decision needs a rate built from many runs, and one citation is a sample of one.

Think about what happened inside that one answer. The model may or may not have searched the web. If it did, it pulled a set of candidate pages, picked a few and wrote around them. Ask the same question an hour later and each of those steps can land differently. You saw one outcome out of many it could have produced.

The trap gets worse when the win follows a change. You refresh a page on Monday and spot a citation on Thursday, and the story writes itself. Prompts you never touched may have moved that same week, up or down, for reasons unrelated to your edit. Without them in view, you're giving the refresh credit for the weather.

Now picture the conversation with finance. “ChatGPT mentioned us” is a story. “We were mentioned in this share of runs, across these prompts, on these surfaces, over these four weeks, and here's how sure we are” is a number someone can put in a cell and defend when the CFO pushes back. Only the second version tells you whether to fund the next round of refreshes.

Your measurement tool has to make that sample visible, across prompts, surfaces and dates. If it can't show you the denominator, its readout is a snapshot dressed as a trend.

Is answer churn a flaw or how these systems work?

Churn is how these systems work. A peer-reviewed study found the sources behind Google's AI Overviews turn over far faster than its organic results. In a separate repeated-query study, models were still naming brands they hadn't named before after fifteen runs. So stop waiting for answers to settle. Build a measurement method that expects variation and samples enough to see through it.

A peer-reviewed study in this year's Findings of ACL reran the same queries two months apart. Only 18% of the pages behind Google's AI Overviews appeared both times, against 45% of Google's organic results [1]. Fewer than one in five AI Overview sources survived the gap. Nearly half the blue links did.

The same work holds a more awkward result for anyone reporting yes-or-no outcomes. Repeated runs reversed the answer's decision in 9% to 28% of cases across the systems and time gaps tested, even with randomness set to zero where the systems allowed it [1]. That range counts flipped decisions, which is a separate measure from changed citations. It still tells you the conclusion of an answer can move between identical runs.

The second study asked 50 questions 15 times each across six engines. For the five engines answering without web retrieval, 86% to 92% of question and engine pairs were still producing an organisation never seen before on the fifteenth run [2]. Fifteen tries weren't enough to map the full set of brands those models might recommend.

Language models produce each ans…

Mark Ridgeon16 MIN READ
A cartoon-style illustration of a hand holding a pen, writing on paper, next to an open folder with lined pages and red circl

Reddit Marketing Tool Buying Guide for Agencies: The Six-Link Chain and the Client Isolation Test

The client email arrives just before lunch. It links to a Reddit thread where a buyer asked for recommendations in your client's category. It asks three short things. Who saw this, who replied, and what happens next.

That email is usually when an agency starts shopping for a Reddit marketing tool. The ranked lists it finds next assume one founder with one brand. You run several clients who must never see each other's material, each with its own approver and voice. The answers to that email already exist, scattered: an alert in a shared inbox, a draft in a doc, and either a post from the client's account or a thread left alone. The record sits across four systems and two people's memories.

If your next client call is within the hour, go straight to the sixty-minute test and the scorecard in the isolation section. The rest of this article explains why each check is there, in the order a thread travels. This is the multi-client version of the problem, built around six links, one last click and a client isolation test, rather than another guide to proving visibility to a single brand.

What is your client asking for?

Your client is asking for a record with six links. Someone spotted the thread, judged whether it deserved a reply, drafted in the right voice, got approval, posted from the right account and reported honestly afterwards. Judge any tool by which of those six links it holds, and for how many clients at once.

Read that chain again and the “best tools” framing falls apart. A keyword alert holds the first link and nothing else. A reply assistant holds drafting and perhaps posting. A listening suite holds discovery and some reporting. Ranking these products against each other is like ranking a stapler against a printer. Both touch paper. They do different work.

GreenRoom, a PR and digital marketing agency, shows how narrow the blocked job can be. Before it adopted Brand24, the team ran 34 Google Alerts and checked notifications by hand, and the listening tool then surfaced client conversations on Reddit and forums [1]. The tool fixed discovery. The case study says nothing about who approved each response or what went live.

That silence is where agencies get hurt, because your client's three questions sit in exactly the links the case never mentions. A tool can fix discovery and still leave you unable to say who approved what.

Start with the free native baseline

Start with what Reddit gives away. Reddit Pro lets a business account track keywords, set up a profile and measure performance, all for free [2]. A paid product earns its place only by naming a job those free tools cannot do, then doing it in a live demo on your data.

Give one client a week on it before any vendor call, and keep a short log. Each day, a strategist searches that client's subreddits by hand and marks every thread they would have replied to or escalated. That hand-marked list is a comparison set, not complete discovery coverage: it records only the threads one person found in the time they had. At the end of the week, compare that list with what Reddit Pro's keyword tracking surfaced. Note how each thread reached the strategist who owns that client, and which report fields the native numbers filled.

Set three pass marks before the week starts, and set the discovery number per client rather than copying mine. Four in five of the hand-marked threads is an illustrative acceptance criterion, fine for a client who can answer a thread a day later and far too loose for one in a regulated category who wants every complaint seen. Count a thread as missed only if the strategist would have acted on it. If the week yields fewer than ten actionable threads, run a second week before scoring, since three found out of four proves little. Reporting passes if three fields filled without a spreadsheet: threads found, links to them, and native performance for anything posted. Routing passes if every actionable thread reached that client's named strategist within one working day, with…

Mark Ridgeon25 MIN READ
An image of 1950's office variables

More AI Tools Are Making Agencies Less Scalable

Declared interest before you start: I build Eclipse, a content operating system for agencies, so weigh the argument below accordingly. Eclipse also ships a named AI Visibility component alongside its research canvas, editor and publishing pieces, which puts me inside the category this article is about. I come back to what that does and does not change near the end.

A client emails on a Tuesday. A competitor keeps turning up in ChatGPT answers about their category, and they do not. You already have the data. It sits in the research doc, the approval thread, the CMS, a chat session somebody closed without saving, and an AI visibility platform one of your strategists bought at the end of last year. Finding the answer takes forty minutes. The answer is one sentence long.

Assume the shape I know best for the rest of this: nine people, eleven B2B accounts, three regular freelancers. Every example below holds to that.

You reply, and the account lead asks where the number came from. Nobody can say. Your agency does not lack signal. It lacks a place where signal becomes a decision, and where the decision survives the person who made it.

Why another AI visibility platform does not add capacity

An AI visibility platform tracks where a brand gets mentioned or cited inside AI-generated answers, across engines like ChatGPT, Perplexity, Gemini, Claude and Google AI Overviews, against named competitors, over time. Underneath that headline the category is fairly uniform. You get prompt-level reporting, so you can see which buyer question produced which answer and which URLs got cited in it. You get trend lines on mentions, citations and rank, so this month reads against last. Most also score sentiment, compare you with named competitors on a defined prompt set, and audit whether your client's pages are readable by the crawlers feeding those answers. Where the tools actually differ is narrower, and worth asking about directly: how often each engine is sampled, whether you can see the cited source URLs behind a score or only the score itself, and whether anything exports into the place where the next article gets planned. The question to hold against all of it, and I come back to it at the end, is which decision the tool improves, who makes that decision, and where the resulting choice gets recorded. Useful. Buying one also adds a surface you have to govern, with its own copy of the client context and an export where you wanted a decision.

What you need is one persistent record per client, holding strategy, prompt set, competitors, evidence, decisions and approval history. That record is the reusable asset. Everything else in this article is about protecting it.

Think about what happens when a new tool lands. Someone configures it for all eleven accounts. Someone re-enters brand names, competitors, target questions. Someone builds a reporting view. Then a different person, three months later, has to work out which of the four places holds the current version of the truth.

Paid media at least counts the setup cost out loud. Fluency's 2026 benchmark study of more than 170 advertising organisations puts routine AdOps execution somewhere between roughly 40 and 46 hours per strategist per month, depending on which tasks get counted: PPC Land, reporting the same study, gives 39.75 hours for campaign optimisations and budget pacing, while MarTech Cube reports around 46 hours once account launches are included [1]. Take the top of that range and check the arithmetic yourself. Forty-six hours a month is about 550 hours a year, which against a 2,080-hour working year is roughly a quarter of it, or about three months per strategist. That is paid media operations, and it proves nothing about how long your editors spend arguing with a draft. The same disease shows up in a different department: configuration and re-entry eat the capacity you thought you were buying.

AI-assisted production makes this worse in one specific way. It raises the volume of material needing…

Mark Ridgeon28 MIN READ

7 ARTICLES ON THIS PAGE