AI claims / Consumer evidence
Missing Outcomes and AI Accuracy Denominators
Keep unresolved, expired and selectively retained predictions visible beside any quoted accuracy or positive fraction.
Define the eligible prediction population
A reported percentage needs a stable population. Identify whether the claim covers all model outputs, emitted alerts, actionable instructions, closed hypothetical decisions or actual filled account trades. Those sets need not be identical. Preserve IDs, versions, timestamps and the chosen endpoint convention. The scoring reference defines metrics on supplied observations; it cannot establish that a commercial dataset includes every observation its advertisement claims to cover.
A missing outcome is not automatically positive, negative or harmless. Retain why it is missing and whether the observation was eligible under the frozen method. An open horizon, absent data, expired instruction and unfilled order are different states. Do not delete them because they complicate a clean figure. A complete publication history can still coexist with incomplete account execution evidence; report both populations rather than silently substituting one for the other.
Closed-only and all-issued questions differ
A closed-only fraction can describe its declared subset honestly. It becomes misleading when the heading implies that every issued output resolved under the same convention. Counts for unresolved observations help a reader understand coverage, but they do not justify guessing their future outcomes. A model-quality review should preserve the original cutoff and avoid choosing a later endpoint merely because the unresolved cases eventually look favourable.
Do not multiply update messages into extra wins
A partial target, an amendment or a repeated delivery may belong to one original idea. Keep its event history rather than expanding the number of independent predictions after a successful move. If the evaluated unit is an account quantity, weight actual executed components under the declared cash method. An abstract direction label cannot supply those quantities. No count reconciliation alone establishes net profit, a complete risk path or causal model evaluation.
A bounded result for a buyer
Report eligible count, observed outcome count, unresolved states, metric convention and evidence level together. A useful conclusion can reproduce the stated closed-only arithmetic while explaining why a larger advertised claim remains unverified. The source may later supply missing records; retain that as a dated update rather than replacing the original sample. The purpose is an inspectable population, not an invented rating that fills every gap with a preferred provider's marketing.
Completed fictional evidence case
An invented set contains ten issued predictions: four documented positives, two documented negatives and four unresolved. The observed closed-only positive fraction is 4/6, about 66.7%. Four unresolved predictions are 40% of the issued population. Counting those unresolved as wins would invent outcomes; deleting them from a claim about all ten would hide coverage. The completed review retains both counts and marks any all-issued success fraction unestablished. No account return or live provider record is represented, and the small invented sample supports no forecast of future reliability.
| Evidence object | Recorded value | Claim boundary |
|---|---|---|
| Issued population | 10 | Original IDs retained |
| Resolved population | 6 | 4 positive / 2 negative |
| Unresolved population | 4 | No guessed outcomes |
| Closed-only fraction | 4/6 | Not a universal hit rate |

Direct sources and reviewed scope
Public documents, not private vendor experiments. The worked cases are invented; source dates do not establish a model-validation engagement.