AI claims / Consumer evidence
Overlapping AI Forecasts and the Sample Unit
Distinguish many messages from many independent observations when forecasts share a horizon, instrument or underlying event.
Start with the unit that generated each row
Several forecasts can share a market event or overlapping outcome window. Their message count is not proof of independent statistical evidence. Preserve instrument, decision ID, horizon, target endpoint and any parent event before calculating a summary. The chronological split reference documents ordered samples and a gap parameter, but a sample-count gap does not automatically resolve every overlapping financial label. The actual target construction must decide which boundaries are relevant.
A buyer should also distinguish repeated delivery from distinct predictions. Five channels reposting one output create five observations of distribution, not five independently trained models. A provider may legitimately produce frequent updates, but the claim needs to state whether each update replaces or supplements the earlier forecast. Neither a high message count nor independent-looking repetition establishes sample independence or model quality.
Preserve shared information and endpoints
Record the time interval during which each outcome is measured. If several rows use the same later closing value, keep that common endpoint visible. An analyst can compare the rows under an explicit convention without treating them as unrelated trials. Uncertainty estimates and validation design require a suitable method; this chapter does not invent an effective sample size or a significance level from the number of overlaps.
A split must follow the target, not only its row date
A training row can have a later outcome endpoint than its displayed date. If that endpoint overlaps the evaluation interval, a calendar boundary alone may not establish separation. Retain any purge or gap rule as part of the experiment identity. The site's calendar checker tests one simple declared horizon and is labelled untested. It does not inspect multiple label windows, establish statistical independence or certify that a vendor implemented an adequate split.
What a comparison should retain
Publish row count, original-idea count, update handling, shared endpoint information and unresolved sample definitions. A stronger claim needs the complete original ledger, not a selected chart. Keep a historical research-fill convention separate from actual subscriber execution even when both use the same idea IDs. The review can explain an overlap without declaring every frequent-prediction model invalid; suitability of the evaluation method remains a specific question to resolve with evidence.
Completed fictional evidence case
A fictional system emits five updates during one ten-minute event, all evaluated against the same later endpoint. The record contains five outputs but one shared event under the stated construction. The reviewer cannot declare five independent strategy-validation samples. It preserves each version and asks how the method handles their relationship. Two delivery retries would enlarge delivery count, not original prediction count. No exact independence adjustment is asserted. The example identifies the unit problem without producing a model return, a confidence interval or a fraud allegation.
| Evidence object | Recorded value | Claim boundary |
|---|---|---|
| Outputs | 5 | Frequent updates |
| Shared endpoint | 1 declared event | Dependence question |
| Retries | 2 extra deliveries | Not new predictions |

Direct sources and reviewed scope
Public documents, not private vendor experiments. The worked cases are invented; source dates do not establish a model-validation engagement.