Attribute agreement analysis checks whether inspectors classify the same items consistently, agree with one another and match an established reference. Those are different questions. Inspectors can agree and still be wrong. Use blinded repeat assessments of representative items, including borderline conditions, with a defensible reference method. This small fictional example calculates simple agreement percentages and shows why a clean average cannot qualify an inspection system or replace a suitable study design.
Separate repeatability, between-inspector agreement and reference accuracy.
Trace a disagreement to an identified item and trial.
Avoid using a numerical Gage R&R for categorical decisions.
Match the study to the kind of decision
An attribute decision assigns a category such as pass/fail or a severity grade. Minitab distinguishes within-appraiser agreement, between-appraiser agreement and agreement with a known standard. A continuous Gage R&R studies numerical measurements; it answers a different question.
Define the defect and inspection conditions. For a visual surface decision, retain reference images or artifacts, lighting, viewing distance and permitted aids. A confident reference label without an approved basis simply moves the disagreement into the reference column.
Inspect the same items twice without revealing the first result
The illustrative record has 10 items, two inspectors A and B and two rounds, giving 40 ratings. Items 1 to 5 have reference Pass and 6 to 10 reference Fail. The sample is deliberately small and balanced for arithmetic, not a recommended qualification sample.
A matches the reference except on item 10 in round 2. B calls item 5 Fail in both rounds and item 6 Pass only in round 1. All other ratings match the reference. Randomize presentation, hide prior ratings and make sure each round is a fresh assessment rather than a memory test.
| Items | Reference | A round 1 | A round 2 | B round 1 | B round 2 |
|---|---|---|---|---|---|
| 1-4 | Pass | Pass | Pass | Pass | Pass |
| 5 | Pass | Pass | Pass | Fail | Fail |
| 6 | Fail | Fail | Fail | Pass | Fail |
| 7-9 | Fail | Fail | Fail | Fail | Fail |
| 10 | Fail | Fail | Pass | Fail | Fail |
Calculate the three comparisons separately
A repeats 9 of 10 decisions, so within-inspector agreement is 90%. B also repeats 9 of 10. Between inspectors, 8 of 10 ratings match in each round, giving 16 of 20 paired agreements or 80%. Do not divide that paired count by all 40 ratings.
Against the reference, A has 19 correct ratings out of 20, or 95%. B has 17 of 20, or 85%. The overall 36 of 40 correct ratings give 90%, but that average hides B’s repeated false rejection of item 5 and A’s missed defect on item 10.
Reference accuracy: A 19/20 =95%; B 17/20 =85%. All 40 ratings include two rounds on the same 10 items, not 40 independent parts.
Look at error direction and borderline conditions
A false acceptance passes a reference-fail item; a false rejection fails a reference-pass item. Their consequences can differ sharply. Keep the direction and item evidence rather than reporting only total agreement.
Chance-corrected statistics such as kappa may add context, but category prevalence, study design and uncertainty affect interpretation. This article does not assign a universal acceptable percentage or compute a qualification from 10 items. Use a study appropriate to the decision and consequences.
- Define categories and reference
Agree operational criteria and how the reference decisions were established.
- Select representative items
Include the relevant variation and difficult conditions without presenting a contrived easy set.
- Run blinded repeats
Randomize items and preserve each appraiser, round and decision.
- Investigate disagreements
Review item evidence, method, aids and training before repeating a changed study.
Improve the method before blaming the inspector
A vague defect boundary, unstable lighting or inaccessible surface can produce disagreements that another training slide will not solve. Examine the work method and reference together. Retain old results when the method changes so the next study has a clear version boundary.
Use Gage R&R when the underlying question concerns numerical measurements. Use acceptance sampling for the separate lot decision. A sampling plan cannot repair an inspector’s inability to identify the defect.
Keep inspection evidence usable for downstream decisions
A pass/fail field can feed yield reports or an AI training set. If the definition changes between shifts, the resulting percentage or label set may describe a changing measurement rule rather than a changing process.
The download preserves all 40 fictional ratings and a blank collection record. If these inspection results will train a model, resolve the reference disagreements first. See predictive quality for the next set of checks.
A related example from Abolfazl
Inspect a repeated-measurement example before blaming the machining process.
In English, on founder Abolfazl’s personal blog. This is related reading, not an independent technical source.
Practical checklist
- Define each category operationally.
- Establish a defensible reference.
- Include representative borderline items.
- Keep rounds blind and presentation randomized.
- Preserve each individual rating.
- Use the correct denominator for each comparison.
- Separate false acceptance from false rejection.
- Review uncertainty and method changes before qualification.
FAQ
FAQ
Sources and further reading
Authoritative references used to research and verify this guide.