How the AI Recommendation Index is measured.

Measured: how often a brand is recommended, mentioned without being recommended, and cited from its own website in answers to a fixed set of unbranded buyer questions. Results are reported for each engine and combined.

Not measured: personalized results in consumer apps, market share, product quality, causation, or future rankings. The index measures answers returned to API requests; it does not reproduce what one person sees in a chat window.

The unit of measurement is one eligible answer for one tracked brand. Mention, recommendation, first choice, citation and competitor displacement mean exactly what they mean in RecoProof's methodology, which defines every metric once for the index, the paid audit and the free check; the index does not invent new meanings. Every published number traces to retained runs, a parser version and a measurement date.

Version ari-v1.0 covers 6 B2B software categories, each tracking a frozen universe of up to 15 brands. A category must be a real shortlist purchase with a clear boundary and at least 8 established vendors. RecoProof does not index its own category, AI visibility tools.

Discovery pass. For a new category, the question set runs once on every engine and every recommended brand name is collected. Brands recommended in at least 3% of discovery answers, or on at least two engines, join the universe, up to 15 brands by discovery frequency.

Brands outside the universe. Every wave also counts recommended brands that are not tracked. One that passes 5% Shortlist Share in a wave joins the universe in the next version, and the change is logged.

Shared parent domains. Where one domain hosts several products, a citation counts as the brand's own only when the URL path starts with that product's prefix, such as a product section of a larger vendor's site.

Each category has 6 buyer intents. Every intent has 3 public paraphrases and one holdout paraphrase: 18 public questions and 6 holdout questions. Paraphrases are required because different wordings of the same intent often produce different shortlists; a ranking built on one wording would measure the wording.

Every question ends with the same sentence: “We are based in the United States.” No question names a brand or asks for alternatives to one, and none dictates a list length; only the shortlist intent asks for a shortlist. A second person reviews every question for neutrality before the version is frozen.

Holdout questions and their digest. The public questions are printed in full on every edition. Holdout questions are scored like the others, but their texts stay private until the version retires, at the earliest six months later. At freeze time we publish a SHA-256 digest of the holdout texts in every edition; when the texts are revealed, anyone can recompute the digest and confirm they were not changed. If a brand's Shortlist Share on public questions exceeds its holdout score by more than 20 points in either window, its leaderboard row, the leader summary and its evidence page show a warning that the result may reflect optimization for the published questions. A warning on the rolling three-wave window, where badges are decided, means no badge in that edition; an earlier badge is revoked only if manipulation is confirmed, and then with a dated correction.

Version ari-v1.0 measures 3 engines, each with its own web search, through the provider's direct API. Each engine is pinned to one model for the version; a retired model opens a new condition, run in parallel for one wave.

• No system prompt and no conversation history: every run is a fresh request.
• Temperature and search settings stay at the provider default, and the values are recorded. Reasoning settings also stay at the provider default.
• Each engine's output budget is in the table above. Where an engine reasons before it answers, the reasoning tokens count against the same budget, so it is a total budget rather than an answer length. An answer cut off at the budget is retried once at the larger retry budget; if it is cut off again, it is ineligible.
• Every question runs 2 times on every engine, on different days: the second repetition at least 20 hours after the first, and the whole wave within 72 hours. That is 144 runs per category per wave.
• Each run keeps the request, the raw response, the finish reason, the provider's citation metadata, the searches the engine made, the model version it returned, tokens, cost and latency.

Google AI Overviews and AI Mode have no official API and are not part of this version. Engine names describe the APIs queried; OpenAI, Google and Perplexity do not endorse or review the index.

An answer is eligible when it completed, is not empty, finished normally (finish status STOP), has parser confidence of at least 0.68, and offers options in the category. Ineligible runs are counted and reported, but they never count as “not recommended”.

Labels the parser cannot decide are marked REVIEW_REQUIRED and resolved by a person before publication. Any that remain unresolved are removed from that brand's denominator.

Minimum data. A brand needs at least 36 eligible answers on an engine for that engine's score, and at least 2 such engines for a combined score. Brands below that are listed as “insufficient data” and are not ranked.

A deterministic parser labels every answer: whether each brand is mentioned, recommended, the single explicit first choice, its position when the answer is an explicit ranking, and whether a cited source is the brand's own site. Every label carries the parser version, currently recommendation-gap-parser-v2.0-phase0b-frozen+ari-ambiguous-names-v2. The parser is frozen before a wave is collected.

Blind validation. Each wave, a person labels a random 10% of runs (at least 60, and at least 10 per category), stratified by category and engine, without seeing the parser's output. The wave passes when recommendation agreement is at least 95% with Cohen's kappa of at least 0.85, and first-choice agreement is at least 90%. If it fails, the parser is fixed under a new version, the wave is re-parsed (not re-collected) and validated again on a fresh sample.

For a brand on one engine, the denominator is that brand's eligible answers on that engine. Engines are scored separately. Mention Rate = Shortlist Share + Recommendation Gap: every mention is either a recommendation or a gap.

Each metric is defined once in RecoProof's methodology, which also gives its paid-audit and free-check variants. The index variant is below; every heading links to the shared definition.

Combined scores are the equal-weighted mean of the engine scores, never one pool of all answers. We do not know the engines' usage shares reliably, so equal weight is the most defensible choice. There is no composite “AI visibility score”: a number that blends recommendations, citations and sentiment with weights is only as meaningful as its weights.

A worked example with fictional counts:

The combined Shortlist Share is (62.5 + 76.6 + 44.4) ÷ 3 = 61.2, published as 61. Pooling all answers would give 61.4; the two methods drift further apart as engines' sample sizes differ.

Question bootstrap. Answers to the same question are not independent, so intervals resample questions, not answers: 2000 draws of the 24 questions with replacement, each drawn question bringing all of its engine and repetition answers. The 95% interval runs from the 2.5th to the 97.5th percentile. When every observation is 0 or 1 the bootstrap has no spread, and a Wilson interval with n equal to the number of questions is used instead.

The random seed is the SHA-256 digest of the version, edition, category and window, and it is published with every edition, so anyone with the data can reproduce the intervals. Points are rounded to whole numbers, lower bounds rounded down and upper bounds rounded up, so rounding never narrows an interval.

Ranks. Brands are ranked by Shortlist Share inside every bootstrap draw. A brand's rank range is the 2.5th to 97.5th percentile of its ranks, shown as “#1” or “#1–2”. A brand first in at least 97.5% of draws is the sole leader: its whole 95% rank range is #1, and it is shown as #1. Otherwise every brand whose rank range includes #1 is shown as tied for first. Overlapping Shortlist Share ranges alone do not decide a tie: two brands can have overlapping ranges while one of them still ranks first in nearly every draw. Tables list brands by the point estimate, ties by name.

Windows. Each edition reports the single wave and, once three waves of the same version exist, a rolling three-wave window that pools them with the question still the resampling unit. The rolling window is the default view and the basis for badges; pooling reduces month-to-month noise but not the uncertainty from using a fixed question set, so its intervals narrow only a little. At a Shortlist Share near 50%, a single wave's combined interval is roughly ±15 points.

Month-over-month change. The change in Shortlist Share is estimated with a paired bootstrap over the same question draws in both months. An arrow appears only when its 95% interval excludes zero; otherwise the table says there is no clear change. Change is measured against the category's most recent earlier published edition; when a month was not published after a failed gate, the table names the month compared with instead of saying “last month”. Change is shown only within one version.

Category context. Each edition also reports the average number of brands recommended per answer, the share of recommendation slots taken by the three most-recommended brands, and the share that went to brands outside the tracked universe.

Badges use the rolling three-wave window and the lower end of the 95% interval, not the point estimate, so one lucky month cannot earn one. Badges cannot be bought, and being a RecoProof customer never earns one.

Every tier also requires: a single-wave Shortlist Share point estimate at or above the tier's threshold in each of the last two waves; Shortlist Stability of at least 70%; and no public-versus-holdout warning. The highest qualifying tier wins.

• Edition badge. Each badge image belongs to one edition, prints its month, and never changes.
• Current badge. A second image is updated on every publication to the brand's most recent qualifying, unrevoked edition and always prints that edition's month, so an old result never looks new. A brand that never qualified has no current badge.
• Evidence page. Every badge's embed code links to the evidence page of the edition it shows, which shows the tier, Shortlist Share with its interval and n, the engines, the window, the parser version, and whether the badge is current, lapsed or revoked. That status is recomputed on every publication.
• Usage. A badge may not be altered and must show its category and month. Where a category name is too long for the badge's single line, the badge prints a shorter name: “Help desk software” for Help desk and customer support software. Only the Category Leader tier may say “#1”. It may not be presented as a claim that a product is better, and it may not use OpenAI, Google or Perplexity logos or imply their endorsement. Linking to the evidence page is not required, and nothing is offered in return for displaying a badge.
• Revocation. A badge is revoked after a data correction that removes the qualifying result, confirmed manipulation, or an altered or misdated badge. The revocation is published with its date and reason in the edition's corrections and on the evidence page, and the current badge then falls back to the brand's newest unrevoked edition, or is removed when there is none.

Gates are evaluated per category. A category that fails a gate is not published that month, while the other categories are. Every edition lists its gate results.

Measurement has not started yet, so no edition is scheduled. The first edition's collection and publication dates will be announced here before collection begins. Every edition is collected from the first Monday of its month, within 72 hours, and published on the Tuesday eight days after that Monday at 14:00 UTC. The index home lists the next twelve dates.

Every edition keeps its own permanent address, and the category page always shows the latest one. A number that has been cited never changes without a dated correction.

Corrections. Errors are corrected within 5 business days and logged on the edition with the date. There are no silent edits.

Data challenges. A brand that believes an answer was labelled wrongly can challenge it with the run id from the open data. Challenges are answered within 10 business days. An accepted challenge fixes the parser, and the fix is applied to every brand; no single brand's score is ever changed by hand.

Every edition's labels and scores are published under CC BY 4.0 at https://recoproof.com/data/ai-index/{edition}/, the canonical copy.

Not published: raw answer text, full source URLs (only domains in this version), and holdout question texts until the version retires. Data files and embeddable charts are licensed under CC BY 4.0; badge images are not, because they carry the usage rules above.

Suggested attribution: Source: RecoProof AI Recommendation Index, {Category}, {Month YYYY}. https://recoproof.com/ai-recommendation-index/{category}/{edition} (CC BY 4.0). Every chart and edition page has copy-ready embed code and citations in text, APA and BibTeX.

Every published page and file carries its version. Month-over-month comparisons are made only within one major version.

1. APIs, not consumer apps. Personalization, account history and interface differences are not reflected.
2. A fixed question set. 24 questions cannot represent every buyer question in a category, which is why intervals are wide.
3. Partial location. Location is sent only where the API supports it; elsewhere only the country sentence in the question carries it.
4. A recommendation is not success. Being recommended does not show that a product is better or that it wins sales.
5. No causation. The index records whether a brand was recommended, not why.
6. Engines change. Providers can change models and search without notice. The returned model version is recorded on every run, but invisible changes cannot be detected.