Reading focus
Syllabus: How to Read Psychological Science
Connect the demonstrations, simulations, and proposed disclosure requirements. Track what changes across analyses and why it changes the inference. You can understand the argument without reproducing every calculation.
Why you’re reading it. Understand how selecting among analyses can change the meaning of a nominal significance threshold.
A question to consider. Choose a measurement decision from Flake and Fried's questions. How could making that decision after inspecting the results create additional analytic flexibility?
Why this reading
Selective decisions can make a nominal significance threshold misleading. Simmons, Nelson and Simonsohn combine demonstrations and simulations to show how choices in data collection, analysis and reporting can produce an apparently persuasive result under a false claim.
Before you read
A p-value describes how unusual a test statistic at least as extreme as the observed one would be under a specified null model and its assumptions. It is not the probability that the claim is false. A procedure’s false-positive rate concerns repeated use when the null is true. Selective choices can make that rate exceed its nominal level.
Researcher degrees of freedom are choices about collecting, analyzing and reporting data. Their consequences depend on how they are made and disclosed; flexibility does not itself establish misconduct.
Read for
- Trace the particular choices combined in the simulations and the conditions under which error rates increase.
- Identify which decisions make the song demonstration look persuasive when omitted from its report.
- Connect each proposed disclosure requirement to a problem it could reveal, and ask what it would leave unresolved.
Consider afterwards
- For a paper you know, identify one analysis decision whose timing and rationale would affect your interpretation.
- What could a precise preregistration clarify, and what weaknesses in measurement or design would it leave intact?
Editorial context
The high false-positive rate in the simulations depends on the particular combination of flexible choices being investigated; it is not an estimate of the percentage of all psychology papers that are false. The paper's contribution is to show why nominal significance levels cannot be interpreted independently of data collection, analysis, and reporting decisions. Distinguish that mechanism from what a replication project can tell us about a selected body of published results.