Reading focus
Syllabus: What Would Justify a Causal Claim?
Track both stages of allocation and compare what the different sets of adjustment variables accomplish.
Why you’re reading it. Evaluate assignment and adjustment in an actual methodological experiment.
A question to consider. If the same training comparison had substantial dropout and no pretest of relevant skills, which part of its causal argument would become weaker?
Why this reading
Shadish and colleagues compare assigned and chosen training within one study. Holding many features constant helps isolate how selection and adjustment affect the estimates, while exposing how much depends on measuring the right preexisting differences.
Before you read
A covariate is a measured characteristic used in an analysis. A propensity score describes the probability of receiving a condition given measured characteristics. Balance on those characteristics does not establish balance on unmeasured causes. A within-study comparison places competing design approaches in a common setting.
Read for
- Separate allocation to the assignment-method arms from allocation or self-selection into mathematics and vocabulary training.
- Compare adjustment using carefully chosen pretests with adjustment using a limited set of convenient demographic variables.
- Examine what the setting, brief interventions, outcome measures and near-complete follow-up contribute to this comparison.
Consider afterwards
- If an adjustment method matches the randomized estimate in this study, what would you need to know before expecting it to work for a long-term community intervention?
Editorial context
The findings are conditional on this design, sample and covariates; they do not establish equivalence between observational studies and experiments in general. The study used month of birth to allocate participants to assignment-method arms, a practical procedure worth examining rather than accepting solely from the title. Rich pretests and limited attrition distinguish this setting from many field studies. Statistical balance cannot verify the absence of unmeasured confounding.