Skip to the evidence
AI Trading SignalsDefinition / Population / Proof

AI claims / Consumer evidence

AI Decision Thresholds and Untouched Evaluation

Separate probability output from the threshold that turns it into a decision, and preserve the history of threshold selection.

The output and the action are different components

A model's numerical output can be converted into a decision by a selected threshold. That threshold has an identity and development history of its own. A probability model can remain unchanged while its emitted instructions change after a threshold revision. Preserve both versions and their effective times. The scikit-learn threshold reference describes this distinction; it does not establish that a vendor's chosen threshold was independent of the period advertised as an evaluation.

For an actionable trade, additional entry validity, quantity, exit and cost conventions are still needed. A classifier's default decision threshold is not a statement of financial suitability. Ask which objective selected the threshold and which data influenced that choice. If realised exit economics were used to redefine entry selection, retain that exposure as a different experiment rather than describing the raw entry ranking as unchanged.

Do not tune on the advertised final test

If a reviewer observes several thresholds on the same period and chooses the best one, that period helped select the decision rule. Its original untouched status is no longer available for that choice. Preserve all tested candidates and unsuccessful settings. A later file name or a newly fitted version cannot restore the missing independence. A replacement evaluation requires a separately declared boundary and method, not a change of captions under the same chart.

Costs do not come from a probability label

False alerts, missed events and actual implementation can have different consequences. Retain the outcome and any monetary convention separately. A target-touch classification does not prove that an account filled at the target. A transformed probability, confidence letter or weighted score cannot supply those prices and quantities. A threshold study may be informative as development evidence even when no comparable financial sample exists; state that scope rather than forcing a return percentage into the review.

What to request before a stronger verdict

Ask for model identifier, score definition, threshold version, development samples, selection objective, access history and final evaluation population. Keep the historical hypothetical fill convention frozen when analysing a research record. A correct website worksheet or sample metric calculation is not an independent strategy-validation observation. Product capability remains distinct from the chosen strategy's evidence, and unknown threshold history should stay an unresolved consumer question.

Completed fictional evidence case

A fictional score series is fixed, but a researcher tries thresholds 0.4, 0.5 and 0.6 after inspecting one evaluation period. Choosing 0.6 because it looks best makes that period development evidence for the selected threshold. The reviewer retains all three candidates and cannot certify the chosen decision rule as untouched on the same period. No actual trading return is claimed, and the example does not prescribe which threshold to use. A later independent record might evaluate the frozen choice; that record would have its own dates, original observations and unresolved implementation fields.

Evidence objectRecorded valueClaim boundary
Model outputFixed fictional seriesNot the whole decision
Threshold candidates0.4 / 0.5 / 0.6Viewed on one period
Selected threshold0.6That period influenced selection
An original claim, its available references and the bounded review
Illustrative record flow; no vendor experiment or customer account is pictured.

Direct sources and reviewed scope

Public documents, not private vendor experiments. The worked cases are invented; source dates do not establish a model-validation engagement.

Related evidence questions