Reading focus
Syllabus: How Can Psychological Science Become More Reliable?
Connect the demonstrations and simulations to the disclosures proposed by the authors.
Why you’re reading it. Identify a specific way in which selection changes the meaning of reported evidence.
A question to consider. If a researcher tries several outcome measures after seeing the results, what would you need to know to interpret the selected analysis?
Why this reading
Selective decisions can make a nominal significance threshold misleading. Simmons, Nelson and Simonsohn combine demonstrations and simulations to show how choices in data collection, analysis and reporting can produce an apparently persuasive result under a false claim.
Before you read
A p-value describes how unusual a test statistic at least as extreme as the observed one would be under a specified null model and its assumptions. It is not the probability that the claim is false. A procedure’s false-positive rate concerns repeated use when the null is true. Selective choices can make that rate exceed its nominal level.
Researcher degrees of freedom are choices about collecting, analyzing and reporting data. Their consequences depend on how they are made and disclosed; flexibility does not itself establish misconduct.
Read for
- Trace the particular choices combined in the simulations and the conditions under which error rates increase.
- Identify which decisions make the song demonstration look persuasive when omitted from its report.
- Connect each proposed disclosure requirement to a problem it could reveal, and ask what it would leave unresolved.
Consider afterwards
- For a paper you know, identify one analysis decision whose timing and rationale would affect your interpretation.
- What could a precise preregistration clarify, and what weaknesses in measurement or design would it leave intact?
Editorial context
The high false-positive rate in the simulations depends on the particular combination of flexible choices being investigated; it is not an estimate of the percentage of all psychology papers that are false. The paper's contribution is to show why nominal significance levels cannot be interpreted independently of data collection, analysis, and reporting decisions. Distinguish that mechanism from what a replication project can tell us about a selected body of published results.