Reading focus
Syllabus: How Do We Describe and Measure Individual Differences?
Follow how four item types were evaluated for difficulty, structure, and reliability, then compared with self-reported admissions-test scores and a brief commercial test.
Why you’re reading it. Compare the construction of a performance test with the questionnaire in the previous unit, including the tools—item response theory and a general-factor model—used to evaluate it.
A question to consider. The team wants a reasoning test that learners can take online, untimed, at home. Which parts of the ICAR evidence support that plan, and what would the team still need to establish for its own learners?
Why this reading
Most cognitive ability tests are proprietary, costly, and administered individually. This paper reports the development of a free, public-domain alternative designed for online research, showing how a performance test’s items, structure, reliability, and validity are evaluated.
Before you read
The first four item types were letter and number series, matrix reasoning, verbal reasoning, and three-dimensional rotation. Each online participant answered a random subset of items, a form of matrix sampling, so analyses combine responses across people. Item response theory estimates item difficulty and discrimination; omega hierarchical estimates how much of a test’s variance reflects a general factor. Correlations corrected for unreliability or for range restriction—reduced variation in a selected sample—estimate what a relationship would be under specified assumptions.
Read for
- Examine the case for a public-domain test and the safeguards proposed when items cannot be kept secret.
- Follow how the four item types were evaluated for difficulty, factor structure, general-factor saturation, and reliability across the range of ability.
- Compare the two validity studies—self-reported admissions-test scores in a large online sample and a brief commercial test in a small university sample—and notice how corrections change the correlations.
Consider afterwards
- Most online participants took part in exchange for feedback about their personalities. How might this recruitment shape the sample, and which conclusions does it affect?
- The correlation with the commercial test was about .8 after corrections. What do those corrections assume, and why would a reader also want the uncorrected values?
Editorial context
The evidence comes largely from a self-selected online sample recruited through a personality-feedback website, with untimed and unproctored administration. The validity analyses relied on self-reported admissions-test scores and on 137 students at one selective university. Several reported correlations are corrected for unreliability, range restriction, or selective reporting, and they depend on the assumptions of those corrections. The authors call for more difficult items, clearer structure for the matrix items, and tests of whether items function the same way across groups. They describe the resource as under continuing development, so evidence for a particular item set and population should be checked before use.