Reading focus
Syllabus: What Can Individual Differences Predict?
Follow the training, leaderboard, and holdout separation and compare each outcome with the corrected benchmark.
Why you’re reading it. Examine forecasting directly and identify the limits of transporting a result from one dataset and task to another.
A question to consider. How could the programme compare a complex model with a simple baseline on a later cohort, and what would it learn from large errors even if the complex model performed best?
Why this reading
Many research teams attempt the same prediction tasks and face the same withheld outcomes. This collaboration makes predictive performance observable, exposing the gap between rich background information and accurate forecasts of particular lives.
Before you read
The challenge divides families into training, leaderboard, and holdout sets. Teams use childhood information and available training-family outcomes to predict six adolescent or family outcomes for other families. A simple benchmark provides a comparison. The 2021 correction updates benchmark code and supplementary materials.
Read for
- Identify which information is available during model development and which outcomes remain withheld until evaluation.
- Compare models with the benchmark for each outcome rather than treating all outcomes as equally predictable.
- Distinguish limitations of these data and tasks from a universal limit on predicting human behavior.
Consider afterwards
- A complex model outperforms a simple benchmark slightly but remains inaccurate for many families. What has it demonstrated, and what would you need to know before using its predictions to allocate support?
Editorial context
The predictors include broad family and developmental information, not only personality and ability measures. The correction addresses a mismatch between the described and implemented benchmark for binary outcomes, a supplementary data-file issue, and software versions; the authors state that the conclusions remain unchanged. Performance here cannot establish how well different measures, outcomes, populations, or interventions would fare.