AI claims / Consumer evidence
AI Trading Accuracy and the Class-Balance Baseline
Compare a claimed hit rate with its outcome definition and class population before treating a large percentage as model skill.
The class distribution changes the interpretation
An accuracy figure needs the number of observations and the distribution of the stated classes. If one label dominates, a rule that always chooses it can have a large correct fraction without distinguishing cases usefully. Identify whether positive means a price rise, a target event, a net account gain or something else. Those definitions can produce different populations. The scikit-learn scoring reference provides metric distinctions, but this review requests the actual vendor definitions and retained counts.
The baseline must answer the same question on the same eligible population. Do not choose a weaker comparator after inspecting the seller's results. A balanced educational example is not evidence of the class balance in live trading. Keep missing outcomes, cancellations and unresolved observations visible; a closed-only subset cannot silently replace all delivered predictions when a headline claims universal accuracy.
Inspect mistakes as well as successes
Request counts for each predicted and observed class, with a declared threshold where relevant. A confusion matrix organises those counts; it does not prove that inputs were causal or the test untouched. Precision and recall also answer different questions, so record their denominators rather than swapping names. For a consumer choosing a service, false alerts and missed events may have different practical consequences, but their financial costs are not supplied by the matrix alone.
Hold the target and sample definition fixed
A model evaluated on one horizon should not inherit an accuracy badge for another. If the provider changes the exit or loss definition after outcomes are seen, that is a different evaluation object. Preserve entry identity and its raw causal objective separately from later exit-adjusted economics. A classification check can validate an output definition without ranking entries by realised trade profit. This publication changes no model policy and trains no instrument strategy.
What a responsible buyer summary includes
Report the class definition, eligible population, baseline rule, confusion counts, threshold and evidence type. State unresolved fields plainly. A high accuracy number can be an attributed advertisement while its record remains unavailable; repeating it through several sites does not verify it. The consumer decision may be to request the missing matrix or avoid making a performance comparison, not to invent a poor score for one product and a winning score for a preferred alternative.
Completed fictional evidence case
A fictional sample has ninety class-0 outcomes and ten class-1 outcomes. Always predicting class 0 gives 90% accuracy but identifies none of the ten class-1 cases. Its class-1 recall is zero under that definition. An advertised 90% correct figure on this population therefore needs comparison with that baseline and its error distribution. The example is not a trading ledger and does not assign a live frequency to either class. No reward/loss magnitude, quantity or costs are present, so 90% accuracy cannot be converted into net profit or a subscriber recommendation.
| Evidence object | Recorded value | Claim boundary |
|---|---|---|
| Population | 90 zeros / 10 ones | Invented class balance |
| Constant-zero rule | 90% correct | No class-1 identification |
| Class-1 recall | 0% | Different denominator |
| Profitability | Unknown | No monetary evidence |

Direct sources and reviewed scope
Public documents, not private vendor experiments. The worked cases are invented; source dates do not establish a model-validation engagement.