Skip to the evidence
AI Trading SignalsDefinition / Population / Proof

AI claims / Consumer evidence

Training and Serving: The Transformation Mismatch

Check whether a live prediction used the same learned preprocessing and feature contract as the claimed evaluated model.

A model artefact is not the whole input path

A trained estimator depends on its input transformations and feature definitions. Retain scaling, missing-value treatment, feature order, encodings and fitted transform identity alongside model weights. scikit-learn's pitfalls reference explains inconsistent preprocessing, while its persistence reference discusses retained models and compatibility. Those documents do not certify a particular vendor's runtime. A weights file alone cannot establish that the advertised evaluation and the emitted prediction used the same input space.

The live path may obtain a different source version, omit a feature or change units. Preserve the actual input contract and effective date. A score change can reflect a changed transform rather than a change in the underlying phenomenon. Conversely, a matching output label is not proof of matching internals. A consumer needs version bindings and retained examples before claiming equivalence or identifying a cause.

Unknown is a useful mismatch state

If the preprocessing artefact was not saved, record its identity as unavailable. Do not reconstruct a favourable scaler from the evaluation sample and attribute it to the original run. Data substitutions should be dated corrections with their own experiment status. A consistent pipeline can reduce a class of mistakes, but its presence in source code does not prove the deployed path used it. Research state, merge state and deployed output ownership remain separate.

Inspect units before interpreting a score

A feature in currency units can differ from a percentage or a normalised value despite having the same column name. Retain units and transformation order in the manifest. The API's accepted shape is not proof of semantic compatibility. Missing-value or fallback rules also need identities: replacing an absent observation with a later value can introduce information that was unavailable at the decision. Leave the resulting causal claim unverified until records resolve it.

End with the verified object

State whether documentation, a frozen artefact, an input example or an actual runtime binding was inspected. A corrected website test can establish its own software behaviour without validating the vendor's ML system. Do not load arbitrary untrusted model files merely to make a review look hands-on. Proprietary weights and deployment secrets can remain private; the claim still needs appropriate independent evidence rather than an invented universal AI quality score.

Completed fictional evidence case

A fictional evaluation uses feature x expressed in units and a stored transformation that divides x by ten. A later serving example supplies x already divided by ten but applies the transform again. The resulting input is one-tenth of the evaluated input under that constructed scenario. The reviewer documents both contracts and does not infer any real vendor error or resulting trading loss. If the actual serving transform were unavailable, the cause would remain unknown. A correct model filename would not resolve that missing input identity.

Evidence objectRecorded valueClaim boundary
Evaluation inputx in original unitsStored transform applies
Serving inputAlready transformedDifferent contract
Double applicationConstructed mismatchNot a measured vendor fault
An original claim, its available references and the bounded review
Illustrative record flow; no vendor experiment or customer account is pictured.

Direct sources and reviewed scope

Public documents, not private vendor experiments. The worked cases are invented; source dates do not establish a model-validation engagement.

Related evidence questions