Reading focus
Syllabus: How Do We Combine Evidence Across Studies?
Connect each simulated research environment to the performance of the competing adjustments.
Why you’re reading it. Evaluate sensitivity to assumptions instead of treating a corrected estimate as the recovered truth.
A question to consider. A review’s conclusion survives three adjustment methods but changes under a fourth. What would you examine before calling it robust?
Why this reading
A correction method can introduce error as well as reduce it. Carter and colleagues simulate research literatures under known conditions, making it possible to examine when different methods recover an effect and when their conclusions become misleading.
Before you read
Publication selection changes which results enter a literature; flexible analysis can change results before selection. A simulation specifies a data-generating process, so its true effect is known. Bias, false-positive rates, interval coverage and failure to return an estimate are different aspects of performance.
Read for
- Connect the simulated effect sizes, heterogeneity, study counts and selection rules to the conclusions about each method.
- Distinguish performance conditional on producing an estimate from how often a method fails to do so.
- Examine the proposed sensitivity analysis and the assumptions used in the ego-depletion illustrations.
Consider afterwards
- Two correction methods disagree. How would you assess their assumptions and performance without choosing the result you prefer or taking a majority vote?
Editorial context
The findings concern the methods, implementations and simulated conditions examined. The authors explicitly acknowledge limits on transfer to other processes and the possibility of choosing favorable assumptions. The ego-depletion examples illustrate that problem; this article alone does not settle every claim about self-control or rank all later methods.