AI Visual Inspection: Balance Missed Defects and False Rejects

Define which class triggers a reject

Here positive means predicted defective. A true positive correctly flags a defective part, a false negative misses one, a false positive flags a good part and a true negative passes a good part. State that convention before reading a confusion matrix.

Scikit-learn separates model scoring from the decision threshold and warns against tuning that threshold using the training data. The library supplies analysis methods, not a manufacturing release rule. Define what the plant does with each outcome and how reference labels are established.

Compare two thresholds on the same fictional parts

The illustrative held-out cohort contains 10000 parts:100 truly defective and 9900 good according to a stated reference. At the lower threshold, 95 defects are flagged, 5 missed and 198 good parts falsely flagged. At the higher threshold, 80 defects are flagged, 20 missed and 49 good parts falsely flagged.

Both matrices use the same underlying parts and labels. The higher threshold reduces flags, including both false and true defect flags. It is not evidence of a better model; only the decision rule changed.

Calculate the metrics and the assumed consequence

Recall is 95% at the lower threshold and 80% at the higher. Precision is 32.42% and 62.02% respectively. Accuracy rises from 97.97% to 99.31% at the higher threshold, even though missed defects increase from 5 to 20.

For a teaching cost comparison, assume EUR 100 per missed defect and EUR 2 per false reject, with no other costs. The lower threshold totals 5×100+198×2 = EUR 896. The higher totals 20×100+49×2 = EUR 2098. These are invented decision inputs; safety, customer and regulatory requirements cannot be priced away with this example.

Check what the inspection team can actually review

The lower threshold flags 293 parts; the higher flags 129. If flags enter manual review, capacity and turnaround matter. A growing review queue can delay good production, while a rushed review can miss the same defects again.

Clarify whether a flag means review, rework, scrap or automatic release blocking. Do not call all false positives scrap cost if the good parts are reviewed and returned to production. Use attribute agreement analysis to investigate human reference and review consistency.

1. Define outcomes and consequences

   State positive class, reference method, defect severity and response for each outcome.

2. Tune on appropriate validation data

   Use a declared objective and keep training and final test records separate.

3. Evaluate the final rule

   Check held-out per-part errors, uncertainty, subgroups and review workload.

4. Validate operation and monitor

   Confirm the decision under representative line conditions and retain approved fallback behavior.

Protect the evaluation from easy shortcuts

Keep repeated images of the same part and related lots from leaking across training and evaluation when that would not represent deployment. Include relevant lighting, products and defect severities. Manufacturing AI data leakage explains the split boundary.

The 1% defect prevalence here is assumed. Precision and workload can change when prevalence changes, even if some class-conditional rates stay similar. Report uncertainty and enough defect examples for the actual decision; a large count of good images cannot supply missing rare-defect evidence.

Keep the threshold version with the operating decision

Record model, preprocessing, camera setup, threshold, reference method and effective date. Monitor actual outcomes when labels arrive, including those from parts that initially passed. Reviewing only flagged parts can hide false negatives.

The download includes both fictional confusion matrices and a blank decision record. Review the missed defects and the manual-review workload together before proposing a threshold change. Product release and inspection qualification remain with the responsible site team.