AI Prompt Visibility Tracking: Strategies From The Community

this is something i’ve spent a lot of time developing, so happy to share:

tl;dr: if you just ask an AI to generate prompts for a given business, you’re gonna have a bad time. as in all things, you’ll get better results when you give the AI better inputs.

for a given client/business, i want to generate a list of queries that (1) reflect probable/realistic query patterns and (2) represent comprehensive coverage of all the prompt topics for which the business would absolutely want to be mentioned in the AI response.

to that end, we start by giving the AI the business’s website and target product (because businesses can have >1 product) with instructions to identify a list of “inputs” for the business’s product, including pain points solved, target customers, differentiated features, etc.

once we have those inputs, THEN we feed them to the AI to ask it to generate natural-sounding queries directly based on those inputs.

e.g.,
• input (target persona): grant manager who wants to save time with automation
• query: For a grant manager who wants to save time on busywork through automations, what’s the best grant management software?
we also create queries from different angles, e.g., problem-aware and solution-aware.

it’s a process grounded in seo/marketing fundamentals, where we think from the customer perspective about their challenges, goals, JTBDs, etc.

we’ve gotten pretty good results with this process; we even have an API (with a UI layer) that other startups in the ai visibility-tracking space use to generate (~50) solution-aware prompts for each of their b2b software customers.

you can play with it here: https://queries.contentsage.ai/dashboard

(you have to sign up and add a payment method, but each request is only $0.20)

Thanks Mike, I was hoping you’d weigh in here!

hope it was helpful!

Ask real clients. I have one client that invited 10 of them to the office and discuss their usage of LLMs

Thanks for posting @user just playing around with it now

Core prompts are built from real customer calls that reference they used an LLM in the process. Customer’s titles are grouped generalized to map to multiple personas per product. AI is used to summarize the call into a prompt format.

Job title, problem they have, request they did for an llm to help. This aligns to the loop marketing framework we’ve been pushing.

Note that the methodology has to be simple enough and memorable enough if you are communicating upwards in an enterprise.

I have a full separate thread on repurposing the concept of an index and how business should align prompts with goals and outcomes.

IMO companies need to start with their own brand -> topic before they can work on topic-> brand.

@user hitting the Save for Later on here. Thanks for this!

Love that @user mentioned "multiple" personas per product and per query.

Thank you @user and @user.

I remember reading the index-aligned thread; thanks for resurfacing!

Personally, I’d love to be in an easier industry where there are call recordings to tap into. That said, we do have frequent UXR sessions, so I think one way to start trying to dig in is through those.

@user I imagine you have some great thought on this topic also, tactically or not.

@user Well I am in alignment with Mark W on the subject of tracking visibility through llms being more squeeze than the juice for attribution.

With that said prediction of what prompts or conversational queries is definitely possible with cognitive and behavioral mapping. Persona stacking is a great way to at least get closer to the goal post.

Where I am going to steer away from the group is to say that in my opinion what you really want isn't the data from customers who have already purchased a product or service but the ones who have not. I call it strategic repulsion. To speak to the 1% within your copy or landing pages or any content really and repel the 99%. Yes, call data, customer support tickets, GSC data, big query all are great to establish a foundational psychological persona for who your customer is ...but what you really want is to attract which customers that haven't bought before that are within that target audience.

Here is where I start. Using also asked premium or free you can get an idea of the emotional and motivational patterns behind single topics to get the why before the buy to align content to meet a target audience where they are cognitively in their search journey. It's beyond what the best X for Y it's more why and is before who.

So for visibility within LLMs or attribution I stick with what's most accurate and that is simply asking or adding "how did you hear about us" in a contact form with choices. All I care about is the percentage of visibility from different properties.

Prompts are so personal and tracking attribution is a 80/20. Your chasing a running and sporadic moving target.

Now Victor has a point, everything above is shop talk but for C suite their known reality might be different and they may want attribution data and to that I say...no two people prompt the same way persona stacking proves more ROI than chasing who and what and how in LLMs got you here.

To make this simple enough or better yet to bring clarity for C suite buy in I have created the Quest method which will be published on Also Asked at some point.

Q – Quantum intent states of questions: Looking beyond surface-level keywords to capture the multidimensional ‘quantum intent states’ of search journeys.


U – User intent mapping: Deep analysis of the human context behind each question.


E – Emotional signal detection: Identifying and addressing the emotional and motivational drivers that influence decisions.


S – Search behavior profiling: Structuring insights into actionable behavioral pathways and user journeys.


T – Transformation prioritization: Identifying which behavioral patterns and emotional drivers to address first in your content strategy to create the strongest connection with your audience's transformation journey.

I knew you’d bring some gold; thank you @user!

@user you make it easy for C-suite by calling it quantum intent states?

@user yes with explaining that someone who searches can exist in two mental states at once...they actually love it for those that are science geeks otherwise I go with dynamic intent states. What you see in Quest is part of my use case and guide for the community in Also Asked.

Tap into the GSC Bulk Export and extracting all queries using various filters; so these prompts come from long tail queries on conversion generated pages or related and relevant because they are classified under a Search Intent model custom for the business that clusters queries from paid and organic....you can then layer any other external inputs to generate synthetic but there is daily impression demand that can be calculated from the deterministic GSC data that is highly underutilized imo

@user but how reliable is that GSC data? I've have seen a big chunk of that longtail query set being pure bot generate impressions. Or do you filter only the ones that generate clicks?

the bot data is a derivative of what users used on that platform, useful to query back into the LLM

But how do you separate queries done via a rank / LLM visibility tracker and an actual RAG triggered search?

Many of them show sudden spikes. Example: no way that a user put in this prompt over 2000x in one day. So when I apply logic to filter that kind of patterns out of it. Almost nothing is left.

And even if volatility for day to day comparison is lower, its still non valid data. Was this you? Haha

????

you can clean the data a few ways...in this case within a page worfklow can be nice to see what is in each respective index through the query/prompt used to access live index, these all intuitively make sense and come from GSC with no search volume

doing more executive level reporting would require more cleaning of the raw queries e.g. grammer check so its a natural phrase

Love the questions btw Jan! Also loving this thread cc @user to feature.

1. Visibility trackers don't click.
2. Visibility trackers (some) have set conventions/formulas and those are programmatic.
3. They queries are long, and fortunately, run like clockwork.
It's whack-a-mole but doable to filter out from people vs bots

Will we get perfect data? Absolutely not :joy:

Yeah, the problem is that marketeers are not clients. So saying "all intuitively make sense" doesn't do the job for me. Didn't finish university but I still rely on data, preferably :slightlysmilingface:

We need real trustworthy data and I haven't been able to get that from currently available data sources, so back to the initial question: ask real people haha.

I knew this would be a can of worms topic, and it was intentionally placed! Agree @user, really enjoying this thread.

Well, with the full export GSC is graciously giving us, we still have undefined queries. It's always been working with incomplete data.

Here's how I think about it:
1. Real people are doing this thing, not just marketers
2. How do we approximate a benchmark that is simple and understandable
3. We do the activities those people will feel an improvement in the experience
4. The benchmark should improve from those activities done
I'm suspicious of any volume metric right now because of that prior thread about panel data.

I think the gross can of worms here is your prompt tracking efforts will "pee in the pool" of your GSC data. Unless they are using Grounding with Google Search, scrapers are likely involved to get you your AI visibility data. It will drive up impressions and lower your CTR's.

@user my point is real people are asking alot in Google already, why would you make up the data? in this snapshot some of these look to be decent prompts others noise.... if there is a natural language question complete with a ? at the end and about the core topic space, wouldn't that be valid?

Another thought to consider: query fan outs are usually way shorter if you check ChatGPT for example. Are LLMs actually putting in those full prompts into search engines?

This will get into the https://theseocommunity.com/resources/best-of

I ran some testing based on Mike King's fan out tool and saw alot of overlap when you break apart the synthetic queries and lookup against the gsc bulk export https://x.com/jamesfgibbons/status/1989721370519810239?s=20

I think now we have to get into the web search tool (2-5) and QFO (deep thinking/research) which are a bit different - but we start getting different from the initial topic of prompt tracking... unless we're doing a sub-section on QFO tracking as well on prompts that trigger it.

This is also where it gets annoying because chat for many foundational models build in a web search tool, whereas defining "AI" inside Google Search is really messy (AI mode, AI Overviews, Using the Gemini app outright but searching)

Personally I think all of these topics are on the table here as they ladder back up to: what are we tracking for prompts?

The intent behind the question includes the thought of action - how then do we action on that tracking so we can improve the content we are creating to: better serve users and drive incremental visibility/performance/etc

> real people are asking alot in Google already, why would you make up the data?
100%! existing sources of organic search-pattern data (google, reddit, etc) are great for validating the nature of questions people are likely to be asking in LLMS, but i think it’s inherently risky to assume quantifiable search demand patterns on google will correlate in any meaningful way to what’s happening on LLMs. i.e., does “search volume” translate from one to the other. (they might correlate! but we just don’t have the data to suggest to what, if any, degree. meanwhile we’ve got a bunch of ai visibility tools out here acting like keyword search volume directly translates to ai search patterns.)

(substantiating the demand/validity of synthetically-generated prompts is something else i think a lot about and don’t have a reasonable solution for yet :upsidedownface:)

"substantiating the demand/validity of synthetically-generated prompts is something else"

I think this goes back to vector embeddings and probabilities. Web search tool vs not.

A synthetic prompt is useful if it approximates real user prompts/behaviors in aggregate on the same intent to embedding and is a good predictor of either brand mention or citation/referral.

If the prompt does not trigger web search, you learn a little more about training data bias

If the prompt does trigger a web search, there's a lot of talk about comparing the results on Google, Bing, or Exa.ai (which is likely scraping Google/Bing)

(Which ironically, includes queries polluting GSC data)

> A synthetic prompt is useful if it approximates real user prompts/behaviors in aggregate on the same intent to embedding and is a good predictor of either brand mention or citation/referral.
completely agree. the challenge is that clients/users don’t feel fully confident unless they have some sort of indication that the queries are grounded in real-world behavior. (which is why i hear about people asking for “search volume” of llm queries a lot.)

Yeah the "clustering" and "similarity" of prompts becomes issues as well because visibility scores being touted are often not weighted. Your visibility is therefore biased by the types of self-serving prompts you put in.

For those still following, your prompts can be converted to a vector embedding.

Like wondering if you should cluster keywords together, you could do the same with prompts. There's the dumb way (are the words similar in the prompt) - and then there's the harder ways (are the RAG queries the same/similar/overlapping) - and I'm sure there's others I'm forgetting off the top of my head.

we’ve found that a first clustering pass with embeddings followed by a refinement pass with AI (to break large clusters into smaller, more-focused ones) leads to best results from a human-evaluator perspective (i.e., clients find that contents of the resulting clusters make sense)

Suppose we have the best prompts and can track them…now what?

we move to tahiti and live on the beach ????

Are many people seeing trackable prompts as a truly effective proxy for / representative of a broader prompt set, and any correlation between visibility for these and performance which is beyond macro growth? Even with intelligent ways to cluster, I'm not sure if this won't just become a fool's errand quickly especially in verticals where the nature of the PMF lends itself to a more nuanced set of prompts, and aggregation is not particularly effective given the high variability of key information in prompts. I suspect this is particularly the case on the B2C side where users can be very diverse and can be looking for highly specific and personalised prompts that, even if can be clustered into some proxy prompt, the results for this won't necessarily correlate much to the specific responses individuals get for their more nuanced and specific prompts that change the underlying RAG queries enough to generate meaningfully different results

@user good points but how will you know when you have the best prompts? i view it as a continuum where you need all the data together clickstream referrals, GSC, log files... when visibility is improving are we seeing movement in the other deterministic data sets around more referral traffic and more crawl etc? unlike legacy keyword tracking its not always set it and forget it

@user my two cents is: if you follow the methodology i described above (building prompts based on the pain points, features, target audiences, etc. that are specific to your business) and create topics (i.e., groups of semantically similar prompts) around those prompts, then optimizing for those topics will dramatically increase your chances of appearing in a response to ANY prompt that’s very relevant to your business.

of course, easier said than done. when we generate a comprehensive list of problem-aware, solution-aware, and competitor-aware (i.e., “alternatives”) prompts and cluster them into topics for our b2b clients, there invariably are >100 topics. so at that point it becomes a matter of prioritization:
• for which of those topics is our brand visibility non-existent or weak? (i.e., biggest opportunities)
• which topics include responses with web searches / URL citations we can try to influence with outreach?
◦ of the topics with URL citations, which are the ones that aren’t dominated by competitor sites?
• focus on solution-aware and competitor-aware, first, bc people making those searches are closer to a buying decision

> optimizing for those topics will dramatically increase your chances of appearing in a response to ANY prompt that’s very relevant to your business
this part is unproven hypothesis at this point, but i have relatively high confidence just based on The Way Things Work:tm:

this is an example of a raw gsc query getting 83 impressions per month over a 90 day average in the US "i want to find a reliable and budget-friendly email service." this can be useful to track and present the ongoing trended GSC metrics + rank/llm tracking in a separate cut of data beyond qualitative synthetic prompts

if bots are the source of these, then it can be guidance to check the current index from any given llm