AI evidence benchmarks
Architecture, causality, experiment identity and execution are separate checks.
Identify the decision process
Determine what is trained and what is ordinary routing or fixed-rule logic. Product documentation can establish a capability without validating the selected strategy.
Causal and untouched evidence
Inspect feature availability, target horizon, split rules and the history of results seen during development. An outcome-guided change affects the status of the test period.
Bound each proof
A confirmed receipt concerns covered bytes. A backtest concerns its simulation convention. Account claims require real fills and costs. No blanket pass/fail matrix for whole product categories is published.
Sources
- QuantConnect optimization parameters
- ALFRED real-time and vintage data
- OpenTimestamps protocol documentation
Research guides
AI data leakage and holdout audit
Audit feature availability, outcome overlap and test exposure with a frozen experiment record and a worked calendar-boundary example.
Backtest, forward and live evidence
Reconcile simulated decisions, prospective instructions and account fills without promoting one evidence layer into another.
Algorithmic versus AI signal claims
Map fixed rules, trained predictions, AI explanations and order routing to the specific consumer evidence each needs.
Research guides
AI data leakage and holdout audit
Audit feature availability, outcome overlap and test exposure with a frozen experiment record and a worked calendar-boundary example.
Backtest, forward and live evidence
Reconcile simulated decisions, prospective instructions and account fills without promoting one evidence layer into another.
Algorithmic versus AI signal claims
Map fixed rules, trained predictions, AI explanations and order routing to the specific consumer evidence each needs.
Benchmarks by evidence object
Compare model definition, output meaning, population completeness, original timing and actual account reconciliation as separate lines. A documentation packet can be adequate for a product-function description while inadequate for a forecast-quality claim. Do not combine different evidence layers into a universal synthetic provider percentage.
A fictional comparison has one preserved model specification and one missing evaluation ledger. The first does not authenticate the second, but it remains useful. The new chapters deepen calibration, selection history, unavailable outcomes and runtime identity. The openly synthetic metric graphic is labelled educational; its values never become a score assigned to the vendors listed here.
Focused AI-claim investigations
- AI Probability Calibration: Frequencies and Small Samples
- AI Ranking Quality Is Not Probability Reliability
- AI Trading Accuracy and the Class-Balance Baseline
- AI Decision Thresholds and Untouched Evaluation
- Many AI Model Trials and the Selected Winner
- Walk-Forward AI Models: Identity and Information Cutoffs
- Missing Outcomes and AI Accuracy Denominators
- Overlapping AI Forecasts and the Sample Unit
- AI Features: Vintage, Publication and Receipt Clocks
- Training and Serving: The Transformation Mismatch
- AI Feature Importance Is Not a Causal Explanation
- AI Market Summaries: Citations, Versions and Timing
- QuantConnect, FreqAI and TrendSpider: AI Research Tasks
- AI Model Artefacts, Environments and Reproduction
- AI Drift Observations and Performance Claims
- Free AI Trading Tools, Data Rights and Compute Costs