When classification errors are asymmetric, accuracy is the wrong key performance indicator (KPI). A model can post a 93 percent accuracy score and still miss every defect it was built to find. The reported metric should be chosen from the error-cost matrix, the table that prices each kind of mistake, and the decision threshold should be treated as a business decision rather than a library default. A pre-registered evaluation on a public semiconductor dataset grounds that argument in numbers: across 30 evaluation folds, the accuracy-ranked-best random forest caught zero defects, while a cost-aligned threshold turned the same model into the cheapest of four. This post walks through the evaluation, the economics, and the review rule.
What Does the Exhibit Show?
The evaluation compares four classical scikit-learn classifiers on the public UCI SECOM semiconductor dataset, 1,567 units with a 6.6 percent fail rate, across 30 stratified folds from a repeated cross-validation design (scikit-learn documentation). The models are k-nearest neighbors (k-NN), a decision tree, a random forest, and logistic regression, all at library defaults (Pedregosa et al., 2011). The pre-registered primary endpoint is a sign test on per-fold fail-class recall differences between the two accuracy leaders; the sealed record is archived as Clouatre, 2026.
The headline is an inversion. The random forest posts the highest mean accuracy, 0.9336, while recording 0.0000 fail-class recall because its maximum predicted fail probability was 0.45, never crossing the default 0.5 threshold. The other three models trail it on accuracy yet lead it on recall; the accuracy gap between the top two was 0.0038. On the metric most dashboards reward they were indistinguishable; on the one that matters in a fab, they could not differ more.
The Sign Test Makes the Inversion Exact
In the pre-registered sign test comparing the top two models by mean accuracy, second-ranked k-NN (0.9298 accuracy, 0.0388 recall) achieved higher fail-class recall than the random forest in all 10 non-tied folds out of 30 (two-sided exact binomial p = 0.001953). The other 20 folds tie at zero recall for both models. Figure 1 plots the per-fold fail-class recall counts for both accuracy leaders: the random forest contributes all 30 of its folds to the 0.00 bucket, while k-NN breaks zero in 10.
Accuracy is not a consolation prize here either. On per-fold accuracy the random forest was better in 14 of 17 nonzero folds, a two-sided exact binomial p of about 0.013. The model is not weak; it answers the wrong KPI. The baseline makes the point bluntly: a do-nothing classifier that predicts pass for every unit scores exactly 0.9336 accuracy on this 6.6 percent fail split, matching the random forest’s winning score, while logistic regression (0.8917) and the decision tree (0.8726) fall below it. The accuracy leaderboard here is ranked by proximity to doing nothing.
Why Is Accuracy the Wrong KPI When Errors Are Asymmetric?
Accuracy weights every mistake equally. Operations do not. Which metric deserves the dashboard depends on what each error costs, not on habit (Google Developers, “Classification: Accuracy, recall, precision, and related metrics”). In defect inspection, a false positive sends one unit to a cheap manual check; a false negative lets a bad part escape into an assembly. In fraud screening, the same asymmetry holds.
Pricing the Confusion Matrix Cells
Price the cells explicitly: expected cost equals the false-negative (FN) cost times false negatives plus the false-positive (FP) cost times false positives.
14:1 ratio. The expected-cost line prices each error cell, and the comparison at the bottom is the whole argument: the same model’s cost falls from 4368 to 2991 once the threshold matches the matrix.from sklearn.metrics import confusion_matrix
# Illustrative ratio, not measured fab costs; one cost unit per
# inspection against 14 per escaped defect.
FN_COST = 14
FP_COST = 1
def expected_cost(y_true, y_pred) -> int:
tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()
return FN_COST * fn + FP_COST * fp
half = (p >= 0.5).astype(int) # default threshold
elkan = (p >= 1 / 15).astype(int) # Elkan threshold
print(expected_cost(y_pooled, half)) # 4368
print(expected_cost(y_pooled, elkan)) # 2991cost_matrix.pyThese snippets illustrate the arithmetic only; they are not a reproduction of the evaluation pipeline. The illustrative ratio prices a false-alarm inspection at one cost unit against 14 units per escaped defect, the ratio of the $100 inspection to the $1,400 escape. Costs in this post are expressed in those units. Under that ratio the ranking flips at the default threshold. The accuracy-best random forest is cost-worst among the four models at an expected cost of 4368 units, because 312 escaped defects dominate the arithmetic (Table 1).
How Does the Decision Threshold Change the Ranking?
The decision threshold converts a graded probability into a yes-or-no verdict, and the scikit-learn default is 0.5. Nothing in the cost matrix says 0.5 is right. Elkan’s cost-sensitive learning result gives the aligned threshold directly: p* equals the false-positive cost divided by the sum of the false-positive and false-negative costs (Elkan, 2001). At 14:1, the aligned threshold is 0.0667, or 1/15.
14:1 it lands on 1/15, and the thresholded predictions follow in the next two lines.import numpy as np
# Illustrative 14:1 ratio, not measured fab costs.
FP_COST, FN_COST = 1, 14
# Elkan (2001): aligned threshold, 1/15 at a 14:1 ratio.
p_star = FP_COST / (FP_COST + FN_COST) # 0.0667
proba = forest.predict_proba(X_pooled)[:, 1]
preds = (proba >= p_star).astype(int)
# Default escapes 312 defects; aligned escapes 83.
print(p_star, preds.sum())threshold.pyThe pooled cost table at the 14:1 ratio makes the reversal concrete; pooling weights each unit exactly three times, once per repeat, so the numbers cover 4,701 unit evaluations and are comparable across models.
| Model | Mean accuracy | Mean fail-class recall | Cost at 0.5 | Cost at p* = 1/15 |
|---|---|---|---|---|
| random forest | 0.9336 | 0.0000 | 4368 | 2991 |
| k-NN | 0.9298 | 0.0388 | 4230 | 3556 |
| logistic regression | 0.8917 | 0.2182 | 3681 | 3380 |
| decision tree | 0.8726 | 0.1603 | 4005 | 4005 |
Both Thresholds Side by Side
Under a 14:1 false-negative to false-positive cost ratio pooled across all 30 folds, the random forest ranked last among the four models at the 0.5 threshold (cost 4368 units) but ranked first when the threshold was lowered to p* = 1/15 (cost 2991 units). The 229 averted escapes at 14 units of cost each outweigh the 1,829 extra inspections at 1 each. The lowered threshold flags 2,058 of the 4,701 pooled unit evaluations for manual inspection, 229 true defects plus 1,829 false alarms, about 44 percent of production flow. Those flags catch 229 of the 312 pooled defects, 73 percent, at the inspection volume where random screening would catch 44 percent. Whether inspection capacity absorbs that volume is an operations question, not a modeling one. The k-NN model falls from 4230 to 3556, logistic regression from 3681 to 3380. The decision tree stays at 4005 at both thresholds, and Figure 2 explains why.
Why Threshold Tuning Helps Only Some Models
Threshold tuning only pays on models that emit graded probabilities. The unpruned decision tree outputs pure 0 and 1 leaf values, so it is threshold-invariant at 4005 in both columns. The k-NN vote fractions move in discrete steps, so its threshold response is coarse. The random forest and logistic regression capture the full gain.
Sensitivity to the Cost Ratio
The evaluation also ran a sensitivity check at a 10:1 ratio, with the threshold still held at p* = 1/15. At 10:1 the Elkan-aligned threshold would be 1/11; holding the threshold at 1/15 keeps the predictions fixed and isolates the effect of the ratio itself. At that ratio logistic regression takes the lead at 2620 with the random forest at 2659, an effective tie that swaps the 14:1 top-two order. The ordering at the top is ratio-sensitive within a narrow band, which is itself a reason to measure the cost ratio rather than assume it.
| Model | Cost at 0.5, 14:1 | Cost at p* = 1/15, 14:1 | Cost at p* = 1/15, 10:1 |
|---|---|---|---|
| logistic regression | 3681 | 3380 | 2620 |
| random forest | 4368 | 2991 | 2659 |
| k-NN | 4230 | 3556 | 2808 |
| decision tree | 4005 | 4005 | 2957 |
One objection deserves an answer: if random forests miss everything, is the lesson to avoid random forests? No. Scikit-learn defaults on a 93 percent majority class push nearly any model toward predicting the majority. The failure is the combination of an accuracy metric and an untuned default threshold, not one model family; any reasonably calibrated classifier wins once the threshold matches the matrix.
Who Should Own the Decision Threshold?
The threshold encodes a cost ratio, and cost ratios are business facts. Engineering proposes and business owns. Engineering calibrates the probabilities and computes the cost curve at each candidate threshold; the business owner of escaped-defect and false-alarm costs sets the ratio, and the threshold follows by Elkan’s formula. When finance cannot price an escape exactly, the workflow inverts: engineering plots the inspection-volume curve across thresholds, and once operations sets an inspection cap, Elkan’s formula back-solves the implied cost ratio the organization is accepting. That framing rhymes with the routing argument in When Should an AI Agent Enumerate, and When Delegate?, where the decision about who or what renders the verdict is separated from the quality of the candidate answers. Here the graded probability is the confidence signal, and the threshold is the delegation decision about when that signal becomes action.
If nobody owns the ratio, the library default owns it silently: a threshold inherited from a code example is a business decision made by nobody.
What Caveats Bound the Claim?
Several limits deserve equal billing with the headline. The sign test p-value is nominal: the 30 folds come from repeated cross-validation and are not independent, and with only 10 nonzero folds and 20 ties the test has limited sensitivity.
The result is a single dataset with no generality claim beyond UCI SECOM. All models ran at scikit-learn defaults within one seed family; k-NN is seed-free by construction, so a single pinned seed is not claimed for all four models.
Elkan’s p* is exact only for calibrated probabilities, and random forest probabilities are known to squash toward the majority class on imbalanced data. The record shows that compression directly: a median predicted fail probability of 0.06 and a maximum of 0.45. A calibrated model on a rare-class dataset can legitimately top out below 0.5, so the numbers alone do not prove miscalibration; what they do establish is that 1/15 is the operating point on these models’ actual probability scales, not a conversion from calibrated ones. A deployment should calibrate first, for example with cross-fitted Platt (sigmoid) scaling on held-out folds, then recompute p* or read its threshold directly from its own cost curve. Sigmoid is the safer default here: scikit-learn advises against isotonic regression below roughly 1,000 calibration samples because it overfits, and this dataset carries only 104 fail labels.
The cost ratios are illustrative, not measured, and any deployment should substitute its own. Receiver operating characteristic and precision-recall areas are not part of the pre-registered record, so no threshold-free ranking claims are made; the probability spread appears descriptively in Figure 2 only.
What Should a Model Review Ask?
An auditable rule: any classifier KPI ships with its error-cost matrix. A model review should ask five questions.
- What are the four confusion-matrix cells for this model in this deployment, and which error is expensive?
- What cost ratio prices the two error types, who measured it, and who owns it?
- What threshold converts probabilities to verdicts, was it chosen from the cost curve or inherited from a default, can downstream operations absorb the added inspections, and what do those inspections buy?
- Does the model emit graded probabilities, or is it threshold-invariant so that tuning cannot help?
- What happens to the ranking under a different plausible cost ratio, and does the headline survive the sensitivity check?
A model that answers all five is measurable; a model that reports accuracy alone is answering a question nobody in the business asked.
References
- Clouatre, H., “Classifier Cost Exhibit: accuracy vs fail-class recall inversion under asymmetric error costs (UCI SECOM)” (2026) — https://doi.org/10.5281/zenodo.24259668
- Elkan, C., “The Foundations of Cost-Sensitive Learning” (2001) — https://cseweb.ucsd.edu/~elkan/rescale.pdf
- Google Developers, “Classification: Accuracy, recall, precision, and related metrics” (2026) — https://developers.google.com/machine-learning/crash-course/classification/accuracy-precision-recall
- Pedregosa, F., et al., “Scikit-learn: Machine Learning in Python” (2011) — https://jmlr.csail.mit.edu/papers/v12/pedregosa11a.html
- scikit-learn developers, “sklearn.model_selection.RepeatedStratifiedKFold” (2026) — https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.RepeatedStratifiedKFold.html
- UCI Machine Learning Repository, “SECOM” — https://archive.ics.uci.edu/dataset/179/secom