AI data leakage and holdout audit
Audit feature availability, outcome overlap and test exposure with a frozen experiment record and a worked calendar-boundary example.
Begin with one decision and its information set
Choose an identifiable instruction rather than a polished aggregate chart. Record the decision clock, model artefact, feature specification and data identity used at that moment. Ask the seller which component actually produces the instruction: a fixed rule, trained predictor, explanatory language model or routing service. Unknown architecture should stay unknown, even when an application supports machine learning. A product capability and a particular deployed model are different objects.
Build a small availability ledger around that decision. For each input, retain what happened, when it was published, when the system received it and which revision was available. These clocks can disagree without anybody falsifying a record. A monthly observation label is not a release timestamp. A later corrected price file cannot silently become the historical file. This first audit step defines the question; it does not establish model accuracy, profitability or independent verification.
Inspect transformations before inspecting scores
A train/test separation can fail before the estimator is fitted. Ask how missing values, scaling, feature selection and category mappings were learned. The relevant distinction is between learning a transformation from development observations and applying that already learned transformation to evaluation observations. A retained pipeline specification helps identify the distinction, but its existence alone does not prove that a vendor followed it. Request run artefacts showing the actual data membership.
Consider an invented review where a seller selected indicators using a full history and only then divided the rows into training and evaluation periods. The chronological labels on the resulting chart do not repair the earlier selection exposure. Preserve that experiment as development evidence and state the exact flaw. Do not claim that every later model is therefore fraudulent or useless. A replacement experiment requires its own data boundary, identity and untouched evaluation plan, not a retrospective change of headings.
Work through an overlapping-label boundary
Here is a fictional calendar check, not a tested strategy. A last training row dated 31 December 2025 has a target horizon of two calendar days. Its declared target reaches 2 January 2026. If evaluation starts on 2 January, the target and evaluation touch under the worksheet convention requiring the label end to be strictly earlier than the test start. Moving that start to 3 January clears only this simple comparison. It says nothing about exchange sessions or other overlapping rows.
Write the actual label construction down before choosing a purge or gap. A target can involve a variable exit, a maximum observation window or another instrument clock; a fixed number of calendar days may describe none of them. The date checker deliberately exports an untested plan. It does not inspect samples, prove feature causality or certify adequate separation. Keep timezone, label endpoint and boundary equality visible so a reviewer can reproduce precisely the limited arithmetic being offered.
Preserve what each experiment exposed
Create an append-only experiment index containing the candidate identity, data periods viewed, questions asked and decisions influenced. A test has become development evidence when its outcomes guide another threshold, feature or model choice. A different filename or retraining date does not make that same period unseen again. Record unsuccessful experiments as well as the candidate later shown to consumers. Otherwise a final performance figure conceals the search process that produced it.
Freeze the entry objective and candidate identity before applying the predeclared exit evaluation. Exit-adjusted profit, holding drift and realised win rate can inform a later validation or rejection decision; they should not quietly redefine which entry candidates won the original search. Preserve the authorised hypothetical research-fill convention separately from broker execution. Changing the fill window after seeing outcomes creates a different experiment. A hash can identify retained bytes, but cannot prove that no unlogged experiment happened or that the declared access policy was obeyed.
Turn a missing artefact into a specific question
Use four distinct statuses: requested, supplied, inspected and unresolved. A link to an API manual is supplied documentation, not an inspected model run. A diagram describing training is not the training artefact. If the model weights are commercially sensitive, ask for a narrower audit object such as dated version identifiers, a feature-timing specification or a redacted experiment ledger. Explain what each object would answer without claiming that one object completes the entire investigation.
Preserve the seller response with its date and exact product scope. An honest inability to share a file is still missing evidence for the claim requiring that file. Avoid converting silence into an invented score or assuming that a founder credential supplies the missing experiment history. ALFRED can help investigate vintage differences, but it does not provide the intraday receipt log of an arbitrary system. Consumer review language should describe that limitation plainly enough that a reader can ask the next useful question.
Finish with a bounded audit conclusion
A useful conclusion names the strongest layer actually reviewed. For example: documentation describes a training boundary, a fictional date exercise reproduces the stated comparison, and real feature receipts remain unavailable. That is more actionable than a universal clean-data badge. Separate software correctness, historical simulation, prospective observation and executed account evidence in the final record. Passing a worksheet test demonstrates the worksheet behaviour, not an independent strategy-validation sample.
Before paying, compare the unresolved items with the capability you need. A consumer seeking a reproducible experiment may reasonably require different evidence from somebody purchasing market explanations only. Do not turn the checklist into an allocation recommendation. Save the blank CSV with private identifiers removed before sharing it, and date later corrections rather than rewriting the original assessment. This publication supplies a research framework and linked documentation; it has not inspected a private model dataset, established a new holdout or verified the results described by the seller in an advertisement.
Review checkpoints
| Object | Preserve | Do not infer |
|---|---|---|
| Feature row | Event, release, receipt and revision clocks | A row date proves availability |
| Training target | Last label observation and declared split | Earlier row means separated target |
| Test exposure | Experiment log and authorised readers | Renaming restores unseen evidence |
Download blank research record (CSV)
Headers only. No trades, account outcomes or operating receipts are supplied.
Sources
- scikit-learn: data leakage and preprocessing
- scikit-learn: probability calibration
- CFTC: AI trading-bot consumer advisory
- ALFRED: vintage-data help
Research guides
AI data leakage and holdout audit
Audit feature availability, outcome overlap and test exposure with a frozen experiment record and a worked calendar-boundary example.
Backtest, forward and live evidence
Reconcile simulated decisions, prospective instructions and account fills without promoting one evidence layer into another.
Algorithmic versus AI signal claims
Map fixed rules, trained predictions, AI explanations and order routing to the specific consumer evidence each needs.
A transformation can leak before the model fits
An invented experiment selects useful features from the entire history, then labels earlier rows training and later rows evaluation. The later row labels do not undo the earlier selection exposure. The reviewer retains that experiment as development evidence and requests a separately defined evaluation rather than simply renaming the final chart.
The original six-section protocol, chronological example and downloadable record remain. The new feature-vintage and walk-forward chapters deepen input availability and artefact transitions, while the threshold chapter isolates another source of test exposure. No website date comparison establishes that the vendor's complete pipeline avoided all those issues.