Reading focus
Syllabus: What Is a Mental Disorder?
Compare observed diagnostic agreement with agreement expected from how frequently the labels are used.
Why you’re reading it. Learn why agreement needs a defined comparison and why it cannot substitute for validity.
A question to consider. Two clinicians almost always choose the most common diagnosis. Why could their high percentage agreement be less informative than it first appears?
Why this reading
A short paper that changed the field's standard of evidence. Earlier studies of diagnostic agreement reported percentages that could not be compared across studies or corrected for chance. Spitzer and Fleiss re-express six of them as kappa, a chance-corrected agreement statistic, and conclude that reliability was satisfactory for only a few broad categories and poor for most. By making the reliability problem quantifiable, the paper motivated the explicit, criterion-based diagnoses of DSM-III, which Spitzer went on to lead.
Before you read
Understand what chance agreement is and why raw percentage agreement overstates reliability when some diagnoses are common. Kappa is the proportion of agreement beyond what chance would produce; values near zero mean agreement is no better than chance.
Keep in mind that the studies being reanalysed used the diagnostic systems of the 1950s and 1960s, with brief definitions and no explicit criteria.
Read for
- Why the authors regard kappa as the right statistic, and what it corrects for.
- Which categories reached acceptable agreement and which did not, and what the pattern suggests about where the problem lay.
- The argument's structure: from a measurement point (agreement must be chance-corrected) to a substantive conclusion (diagnosis was unreliable) to a proposal (explicit criteria).
- What the paper does not claim. Reliability is treated as necessary for validity, not as evidence of it.
Consider afterwards
- If two clinicians agree perfectly on a diagnosis, what have you learned about whether the diagnosis is valid?
- The paper's remedy was explicit criteria. What might be lost when diagnosis becomes a matter of counting criteria?
Editorial context
This reanalysis makes agreement an explicit object of evaluation. Chance correction does not make kappa independent of diagnostic prevalence, samples, raters, or interview design. The DSM-5 field trials provide a later comparison using separate assessments, but their values cannot simply be ranked against older studies without examining those design differences. Reliability is necessary to many uses of diagnosis; it is not sufficient evidence of diagnostic validity.