Reading focus
Syllabus: What Is a Mental Disorder?
Use the earlier reliability distinction to interpret separate assessments. Follow the uncertainty around estimates as well as differences between diagnoses.
Why you’re reading it. Examine how the assessment procedure affects what a reliability result means.
A question to consider. Would two clinicians rating the same recorded interview answer the same reliability question as two clinicians interviewing a person on different days? Explain what changes between the designs.
Why this reading
The DSM-5 field trials assess test-retest agreement using separate clinical interviews. Results vary across diagnoses: some showed good or very good agreement under the authors' criteria, while several common diagnoses had lower agreement. The paper lets the reader examine both the measurement of reliability and the proposed standards for judging it.
Before you read
Refresh kappa, a chance-corrected agreement statistic. Separate interviews can differ because of raters, elicited information, changes over time, and criteria. Joint ratings of one interview address a different reliability question. Examine the authors' interpretive bands and the uncertainty around each estimate.
Read for
- The design choices that make this study closer to routine practice than earlier reliability studies, and how they affect the numbers.
- The standards the authors propose for what counts as acceptable reliability, and the reasoning behind them.
- Which diagnoses did well and which did poorly. Is there a pattern in what distinguishes them?
- How the authors interpret low agreement for common diagnoses: as a problem with the criteria, with the raters, with the constructs, or with the design.
Consider afterwards
- If a common diagnosis has questionable test-retest reliability in expert settings, what does that imply for research that uses the diagnosis as an outcome or a grouping variable?
- Compare the standards proposed here with the implicit standards in Spitzer and Fleiss. Has the bar moved? Should it have?
Editorial context
The findings do not justify saying that explicit criteria solved diagnostic reliability. Nor do they establish that every diagnosis performs poorly. Interpret each estimate in relation to the sample and assessment design, and distinguish disagreement from a demonstration that a diagnostic construct is invalid. Comparisons with Spitzer and Fleiss (1974) are useful when they illuminate changed methods and standards rather than imply a controlled historical comparison.