Skip to the evidence
AI Trading SignalsDefinition / Population / Proof

AI claims / Consumer evidence

AI Probability Calibration: Frequencies and Small Samples

Inspect a probability claim against its defined outcome and evaluation population, using an openly synthetic eight-row illustration.

What does the reported probability describe?

A numerical output needs an event definition, horizon and observation unit before it can be assessed as a probability. A seller's score may describe a rank, direction, price interval or another quantity. Ask which, and retain the model version with its output. The existing score-versus-probability chapter introduces that distinction; this chapter follows a specific probability claim into the outcome record rather than converting a confidence label into a customer success chance.

scikit-learn's calibration reference discusses how predicted probabilities can be compared with observed frequencies. It supplies a statistical vocabulary, not evidence that a commercial model followed the process. A plotted point should state its population and outcome endpoint. A rounded percentage without counts, eligible observations and missing-state handling is not enough to reproduce the comparison.

Keep the development and evaluation roles visible

If calibration choices are learned from a sample, that sample has a development role. Evaluating the resulting choices requires its own declared method and separation. Do not tune bins or transformations after seeing a favourable plot and then call the same plot untouched evidence. Preserve the fitted model, calibration component, sample membership and dates. A library function correctly calculating a toy metric demonstrates only that metric computation, not a private vendor's causal training history.

Why tiny bins do not establish future reliability

Two observations in a bin can change its frequency sharply when just one outcome changes. Dependent decisions, overlapping horizons and selective missing outcomes add other limitations. Report the counts instead of smoothing away the uncertainty. A point near the diagonal can still come from very little evidence, and a point away from it may be noisy. Neither alone proves a future chance, an investable edge or misconduct. The appropriate conclusion names the sampled object and what remains unverified.

Read the synthetic worksheet with its exact boundary

The downloadable illustration contains eight deliberately chosen binary outcomes, not market data. scikit-learn computes its Brier score and ranking metric under the recorded library version. Its four bins each have two rows. The graphic and table describe those rows only, with no confidence guarantee or provider comparison. Keep the raw CSV, definitions and software version together so that another reader can reproduce the educational arithmetic without mistaking it for a strategy-validation sample.

Completed fictional evidence case

Synthetic scores are 0.2, 0.2, 0.4, 0.4, 0.6, 0.6, 0.8 and 0.8; outcomes are 0, 1, 0, 0, 1, 0, 1 and 1. Bin frequencies are 0.5, 0, 0.5 and 1 with two observations each. Brier score is 0.20 and the ROC ranking area is 0.75 for these invented rows. A constant 0.5 predictor has Brier 0.25 on the same rows. This comparison does not certify calibration, estimate a future win rate or measure returns. A different sample could give a different result, and no actual provider is assigned either score.

Evidence objectRecorded valueClaim boundary
Claimed probabilityDefined event and horizonNot supplied by a label
Each toy bin2 invented rowsVery small population
Toy Brier score0.20Not a trading result
Toy ROC area0.75Ranking, not probability reliability

Reproducible synthetic rows, not a model test

Four two-row synthetic bins have observed fractions 0.5, 0, 0.5 and 1
Eight deliberately invented binary rows. No market data, provider trial or future probability is measured.
Synthetic rowOriginal scoreTransformed scoreInvented outcome
toy-010.20.350
toy-020.20.351
toy-030.40.450
toy-040.40.450
toy-050.60.551
toy-060.60.550
toy-070.80.651
toy-080.80.651

scikit-learn 1.5.2 computes original Brier 0.20, transformed Brier 0.2125 and ROC area 0.75 for both sets. Raw JSON keeps floating-point output and software versions. These metrics do not certify calibration or an investable edge.

Direct sources and reviewed scope

Public documents, not private vendor experiments. The worked cases are invented; source dates do not establish a model-validation engagement.

Related evidence questions