Reading focus
Syllabus: How Do We Know What a Psychological Test Measures?
Examine the evidence needed to use task performance as an individual-differences measure, rather than assuming a repeatable condition effect is enough.
Why you’re reading it. Extend measurement questions beyond questionnaires and interpret convergence between tasks and self-reports carefully.
A question to consider. A persistence task shows an average effect of distraction but unstable participant rankings. How should that affect interpretation of its weak correlation with the questionnaire?
Why this reading
A task may repeatedly produce an average experimental contrast while poorly distinguishing individuals. These test–retest studies make the difference consequential for attempts to correlate cognitive-task scores with other attributes.
Before you read
Within-person contrasts compare conditions for the same person. Between-person variation concerns differences among people. Test–retest reliability asks how consistently a measure distinguishes people across occasions. Difference scores can have properties unlike either component score.
Read for
- Separate mean experimental effects from the stability of participants’ scores.
- Track the roles of between-person variation and measurement error.
- Check task versions, samples, intervals, and uncertainty before generalizing a reliability estimate.
Consider afterwards
- Everyone slows down under distraction, but their rankings change between sessions. Which claim does this support, and which claim needs further evidence?
Editorial context
The reported tasks and student samples do not establish that all cognitive measures are unsuitable for individual-differences work. Avoid equating within-person condition effects with between-group comparisons: Karvelis and Diaconescu (2025) clarify how measurement error can also attenuate standardized between-group differences. Reliability must be evaluated for the intended inference.