This is the first post in a new series pulling back the curtain on how NotionCue actually works. Most of what we publish on this blog is about AI search behavior generally. This series is specifically about our own product decisions, including the ones that failed before we found something that held up at scale.
The pitch for a citation tracker sounds deceptively simple: run a prompt, see if a brand gets mentioned, record it. The reality involves five completely different retrieval architectures, five different rate limit structures, five different response formats, and a genuinely hard problem sitting underneath all of it: how do you know if a citation today is meaningfully different from a citation last week, when the same prompt can legitimately produce a different answer purely from normal model variance?
Why Can't You Just Hit Each Engine's API on a Schedule?
The naive version of this product is a cron job: once a week, fire the same prompt at each engine's API, save the response, done. We built roughly that version first, and it broke in three distinct ways within the first month.
The first break was rate limiting. Each engine has its own throttling behavior, and none of them are designed around the use case of running the same handful of prompts across a large number of distinct brand accounts simultaneously. A naive scheduler firing every tracked prompt at the top of every hour looks, from the API provider's side, indistinguishable from abusive traffic. We had to build per-engine request pacing that spreads a customer's full prompt set across a rolling window rather than firing it all at once, with independent backoff logic for each engine since they each signal throttling differently — some return an explicit rate-limit error, some silently degrade response quality, and at least one occasionally returns a technically valid response that is clearly lower-effort than its normal output.
The second break was response format drift. An API response schema that worked cleanly in testing would occasionally change shape without a version bump, especially on the engines still iterating quickly on their search-grounding features. Our parsing layer now runs a schema validation step before extraction, and any response that doesn't match the expected shape gets flagged for manual review rather than silently producing a null or malformed citation record — a decision that costs us some short-term completeness but has saved us from shipping bad data to a customer's dashboard more than once.
The third break, and the hardest one conceptually, was distinguishing genuine week-over-week citation change from ordinary model variance. This is the problem the rest of this post is actually about.
How Do You Tell a Real Citation Change From Normal Model Noise?
Every engine we track has some baseline variance in how it answers the same question twice, for reasons covered elsewhere on this blog — approximate retrieval, live web content changing between runs, and in some cases genuine randomness in how a response is sampled. If we reported every single run's result as a fresh data point with no smoothing, a customer's dashboard would show citation rate bouncing around from week to week in a way that looks like noise even when nothing about their actual AEO performance has changed.
Our approach is to run each tracked prompt multiple times per tracking cycle rather than once, and report a citation rate across that cluster of runs rather than a single binary yes-or-no. A prompt that gets cited in four out of five runs this week and three out of five last week is a much more honest signal than either single run in isolation, and it's the difference between a dashboard a customer can actually trust and one that just adds anxiety without adding insight. This clustering approach is also why our reported citation data updates on a slight lag from when the underlying runs actually happen — we'd rather report a slightly delayed number we're confident in than a same-day number built on a single noisy sample.
Why Does Each Engine Need Its Own Parsing Logic Rather Than One Universal Parser?
The most tempting architectural shortcut, and the one we actively avoided after our first attempt at it, is building a single universal response parser that normalizes every engine's output into one common format immediately. It sounds efficient. It loses information that turns out to matter.
Perplexity's response structure includes explicit inline citation markers tied to specific claims, which lets us attribute a mention to a specific sentence rather than just knowing the brand appeared somewhere in the response. ChatGPT's search-grounded responses carry a different citation structure depending on whether the underlying answer drew from live retrieval or from the model's own training knowledge, and that distinction — covered in more depth elsewhere on this blog — is itself valuable data we would lose if we flattened everything into one generic schema too early. Google's AI Overview responses come through a different capture mechanism entirely, since there is no public API for AI Overview content the way there is for the conversational engines, which means that specific pipeline looks nothing like the other four under the hood even though the final dashboard presents it consistently.
We keep each engine's native response structure intact through the entire pipeline and only normalize at the very last step, right before it hits a customer's dashboard. This costs more engineering complexity than a universal parser would, but it means we can add new engine-specific features — like Perplexity's inline attribution — without redesigning the whole pipeline every time.
What Happens When an Engine Changes Its Behavior Overnight?
This has happened to us more than once, and it will keep happening, because every engine we track is under active, fast-moving development. The most disruptive version of this was when one major engine changed its default citation display behavior — the kind of change covered on this blog when it happened publicly — and our extraction logic, built around the previous behavior, started under-counting citations for a period of days before we caught it.
Our response since that incident has been building an internal drift-detection check: a small set of known-stable reference prompts, tracked continuously, where we have high confidence in what the expected citation pattern should look like. If those reference prompts suddenly show an unexpected pattern shift across the board, that is our signal to investigate an engine-side change before it silently corrupts customer-facing data. This is not a perfect system — it caught the incident above a day later than we would have liked — but it is a meaningfully faster detection loop than waiting for a customer to notice their dashboard looks wrong and reach out.
Why Do We Report a Delay Rather Than Real-Time Results?
A reasonable question customers occasionally ask is why NotionCue reports citation data on a scheduled weekly cadence rather than instantly whenever a prompt is checked. The honest answer is that instant, single-run results would be actively misleading given the variance problem described above. A dashboard that updates every time you refresh the page, showing a different citation status each time purely from normal model noise, trains a customer to distrust the tool — or worse, to make a real strategic decision based on a single noisy data point that happened to look favorable or unfavorable that particular moment.
The weekly cadence exists specifically to give the multi-run clustering approach enough data to produce a number worth trusting. We'd rather ship a slightly slower, more honest signal than a faster one that looks precise but isn't.
What's Next in This Series
Future posts in this series will cover the specific engineering decisions behind how we handle multilingual prompt tracking, why our schema audit tool checks the raw server response rather than the rendered DOM, and a full postmortem on a citation-tracking bug that taught us more about RAG variance than any of the academic papers we'd read up to that point.
If there's a specific part of how NotionCue works that you're curious about and we haven't covered, the feedback link in the product itself reaches the team directly — several posts in this series so far started as answers to a support conversation that turned out to be worth writing up properly.
If you're evaluating any AI citation tracking tool, not just ours, the single most important question to ask is how it handles the variance problem described in this post. A tool reporting single-run, same-day results without any averaging or clustering methodology is very likely giving you a noisier signal than it appears to be, regardless of how confident its dashboard looks.
Frequently Asked Questions About How NotionCue's Tracking Works
How many times does NotionCue actually run each tracked prompt?
We run a cluster of multiple attempts per prompt per tracking cycle rather than a single check, specifically to produce the averaged citation rate described in this post rather than a single noisy data point. The exact count varies slightly by engine based on each provider's rate limit constraints.
Does NotionCue use each engine's official public API, or does it simulate a browser?
This varies by engine and is driven by what each provider actually makes available. Where an official API with search-grounding exists, we use it. Where no public API exists for a specific surface, such as Google's AI Overview feature, we use a different, engine-specific capture approach designed around that surface's actual public availability.
Why does my citation rate sometimes change between two consecutive weekly reports even though I haven't changed anything on my site?
This is expected and is exactly the variance the multi-run clustering approach is built to smooth out, not eliminate entirely. Live retrieval engines pull from a web that is itself changing constantly, independent of anything you do, and a small week-to-week fluctuation with no clear underlying cause is normal. A sustained multi-week trend in one direction is the signal worth acting on; a single week's move usually is not.