Backtest, forward and live evidence
Reconcile simulated decisions, prospective instructions and account fills without promoting one evidence layer into another.
Identify the claim before collecting files
Write the advertised claim in a single sentence with its period, account or model identity and result convention. Historical model performance, prospective signal publication and subscriber profit require different records. If a seller says live, ask whether that means a running software process, an observation stream or actual funded execution. Do not choose the most favourable interpretation on behalf of the seller. Retain the original wording beside the review question.
Start separate folders or ledger fields for each evidence layer. A backtest report belongs with its dataset and execution convention. A prospective record belongs with its publication history. An account statement belongs with deposits, withdrawals, orders and fills. Related records can be reconciled through stable identifiers without being merged into one ambiguous percentage. This separation makes a missing fill or amendment visible. It also prevents a genuine historical simulation from being dismissed simply because it is not something it never claimed to be.
Reproduce the simulation convention, not the sales chart
Ask which instrument histories, universe membership, timezone and corporate-action conventions were used. Retain the model version, entry objective, chosen exit and treatment of absent observations. Costs need units and timing, not just a word such as realistic. A favourable research fill under a declared observation window remains a hypothetical reference; it cannot be described as the price every subscriber obtained. Preserve the frozen convention when comparing releases.
An invented example contains twenty closed hypothetical decisions, fourteen labelled positive and six negative, plus five still open. The closed-only positive fraction is fourteen divided by twenty, or seventy percent. Twenty-five total decisions answer a different coverage question. The example supplies no actual trading result and says nothing about net return, position weights or the final outcomes of the open items. Keep the unresolved rows instead of forcing them into wins or dropping them from the instruction count. A chart alone cannot reveal these denominators.
Count prospective instructions before counting profits
A forward observation should retain decisions when they appear, including those later cancelled, expired, rejected or left unresolved. Record a model deployment change at the time it happens rather than assigning an earlier prediction to a better later artefact. Distinguish source publication from subscriber receipt: a timestamp attached to one does not prove the other. Corrections should link to originals rather than overwrite them invisibly.
Suppose a fictional stream contains twelve messages: four commentaries, six actionable originals and two amendments to those originals. There are six original instructions, not twelve independent trades. If three of those six reached an account and three did not, execution coverage in this toy set is one half. That is not a profitability estimate. The amendment relationship and each unexecuted reason matter. A lost permission, stale instruction or missing acknowledgement belongs to an operating record, even if no profit can yet be calculated for it.
Reconcile execution states without assuming a fill
For every actionable instruction, map the requested order to acknowledgements, partial fills, cancellations and remaining quantity. A timeout is an unknown state until reconciled, not proof that an order never existed. An accepted order is not necessarily an executed order. Keep event ordering and venue identifiers where available, with private account information redacted in a public review. Do not place real orders to demonstrate this guide.
A hypothetical request for one unit can fill 0.4 units and cancel the remaining 0.6. The account still owns 0.4 units; cancellation is not an exit. A later confirmed sale of 0.25 leaves 0.15. These arithmetic states illustrate why a source signal and an account trade can diverge. Fees, financing, currency conversion and mark-to-market conventions must then be applied to the actual quantities and clocks. No generic platform description proves that every provider or subscriber records those states correctly.
Keep cash flow and model identity in the result
An account value can rise after a deposit even when the trading result is flat or negative. Retain externally contributed and withdrawn cash with its timing. Name the return convention used and avoid presenting a raw balance ratio as a cash-flow-adjusted result. An account-level measure and an unweighted sum of model percentages are not interchangeable. If the aggregation method is unavailable, report the published number as an attributed claim rather than independently reproduced performance.
Similarly, a family label can cover several model versions. Preserve the sequence of versions, retirement dates and evaluation rules. A later corrected dataset should not silently alter the old release receipt. If a source changes its record, keep both the old assessment and the dated correction. Do not infer a complete population from one selected successful account. The purpose of reconciliation is to expose what the records support and where they stop, not to manufacture a universally comparable league table.
Write an evidence-layer decision for the reader
End the assessment with the reviewed period, supplied artefacts, missing states and the strongest supported statement. Documentation-only research remains documentation-only even if the interface looks convincing. A preserved prospective stream can support publication chronology without proving funded execution. A reconciled account can establish a specific account experience without forecasting the next subscriber outcome. Say which boundary applies instead of using an unqualified verified label.
Use the checkpoint table to prioritise the next request. A consumer may need an actual cost ledger more urgently than another screenshot of gross profit. Someone comparing research platforms may instead need stable experiment identities and export formats. Preserve private information locally and share only redacted references. These pages have not run a paid model, subscribed a funded account or verified a live return. Their value is a reproducible way to distinguish evidence objects and prevent one valid object from being used to imply several unsupported conclusions.
Review checkpoints
| Object | Preserve | Do not infer |
|---|---|---|
| Simulation | Data, fill window, costs and model identity | A subscriber received that price |
| Prospective record | Original, amended and unresolved decisions | Every message became an order |
| Account outcome | Orders, fills, cash flows and equity basis | A screenshot is a complete ledger |
Download blank research record (CSV)
Headers only. No trades, account outcomes or operating receipts are supplied.
Sources
- scikit-learn: data leakage and preprocessing
- scikit-learn: probability calibration
- CFTC: AI trading-bot consumer advisory
- ALFRED: vintage-data help
Research guides
AI data leakage and holdout audit
Audit feature availability, outcome overlap and test exposure with a frozen experiment record and a worked calendar-boundary example.
Backtest, forward and live evidence
Reconcile simulated decisions, prospective instructions and account fills without promoting one evidence layer into another.
Algorithmic versus AI signal claims
Map fixed rules, trained predictions, AI explanations and order routing to the specific consumer evidence each needs.
The prospective stream retains unresolved states
A fictional stream has ten original decisions and four unresolved account endpoints. Its hypothetical model outcomes and six documented execution cases require separate populations. Adding the four chart-based results to the account ledger would invent transactions. The complete review can retain the model convention and explain the missing account evidence at the same time.
The original long-form population, timing, cash and quantity examples remain. The new missing-outcomes chapter deepens denominators, while walk-forward identity prevents the newest artefact from silently replacing original output ownership. Research LIVE, deployed code and actually certified subscriber execution are distinct states; this publication does not infer one from another.