Reading focus
Syllabus: How Do We Describe and Measure Individual Differences?
Follow the workflow from justifying a new task to reliability, factor structure, item selection, and a second validation sample, then the argument for measuring each ability with several tasks.
Why you’re reading it. Understand why measuring differences between people demands more than a task that produces a robust average effect.
A question to consider. The team plans to use each learner’s slowdown under distraction—the difference between two conditions—as a measure of self-control. What reliability evidence should it seek before correlating that score with conscientiousness?
Why this reading
Experimental psychologists increasingly use their tasks to study differences between people, but tasks designed to produce dependable average effects often distinguish individuals poorly. This tutorial sets out how to build and evaluate performance measures of individual differences, from reliability and norms to factor structure, item selection, and independent validation.
Before you read
Reliability is the consistency with which a measure distinguishes people, across occasions (test–retest reliability) or items (internal consistency, estimated by coefficients such as alpha and omega). Norms describe score distributions in defined groups. In item response theory, each item’s difficulty and discrimination describe how the probability of a correct answer changes with ability. A latent variable is an unobserved construct estimated from several measures; confirmatory factor analysis and structural equation modelling test models of such variables. The R code is included for readers who want to reproduce the analyses; following the reasoning does not require running it.
Read for
- Follow the argument that correlational research depends on reliable scores, and why a difference between two highly correlated conditions often makes an unreliable score.
- Trace the development workflow—justifying a new task, securing a large sample, checking reliability and factor structure, selecting items, and validating in a second sample—and the reasons given for each step.
- Examine Kane’s levels of validity argument and the case for measuring each construct with several tasks, including the worked examples in which latent correlations exceed correlations between single tests.
Consider afterwards
- A task produces a large, replicable average effect, yet individual scores correlate only .3 across two sessions. What can the task still show, and what should it not be used for?
- Why might two tasks that correlate weakly with each other still measure the same underlying ability, and what evidence would distinguish that possibility from their measuring different things?
Editorial context
This is a practical tutorial by an experimental psychologist for researchers without psychometric training. Its numerical guidance—at least 200 participants, reliability above .7, retaining items by item–rest correlations of roughly .2 to .4, and a tentative minimum validity correlation of .2—offers rules of thumb rather than universal thresholds; suitable values depend on how scores will be used and in which population. Under classical assumptions, the ceiling on an observed correlation depends on the reliabilities of both measures, so the tutorial’s shorthand that a variable cannot correlate with another more than with itself is not a precise rule. As the author notes, latent-variable estimates built from unreliable tasks involve considerable extrapolation, and a model revised in response to data needs fresh validation.