Reading focus
Syllabus: What Can Statistical Evidence Tell Us?
Translate a comparison of labels into the effect contrast that needs estimating.
Why you’re reading it. Assess claims that an intervention works differently across groups or settings.
A question to consider. One school reports a significant benefit and another does not. What estimate and uncertainty would address whether their benefits differ?
Why this reading
Gelman and Stern expose a common mistake in comparisons: treating different significance labels as evidence that two effects differ. Numerical and published examples show how to ask the comparison directly.
Before you read
An estimate and its standard error describe different things: magnitude and sampling uncertainty. Comparing two estimates requires uncertainty for their difference, including their dependence when they come from the same data.
Read for
- Follow how sample size changes significance while leaving an estimated effect similar.
- Compare the threshold-based graph with the display of estimates and uncertainty.
Consider afterwards
- One intervention has p = .03 and another has p = .20. What information would you need to judge whether their effects differ?
Editorial context
The title does not mean that differently labeled effects can never differ. It means the labels alone do not answer that question. Nor does a nonsignificant contrast establish equivalence. The biological examples illustrate statistical interpretation; the paper does not independently validate their proposed mechanisms or settle the underlying substantive literatures.