How Can Psychological Science Become More Reliable?
Making claims easier to evaluate, reproduce and correct
- Level
- Intermediate
- Reading load
- 10 core readings in 6 units
- Prerequisites
- Familiarity with study design, statistical uncertainty and the distinction between association and causation. The introductory reading and statistical-evidence paths provide useful preparation. No programming experience is assumed. The simulation papers require attention to rules and assumptions rather than reproducing the software.
- Last reviewed
- September 2026
Introduction
Science grows more trustworthy when its claims can be examined, challenged, and corrected. Replication is one part of that work. Measurement choices, analytical decisions, the responsible sharing of data, and institutional rewards also shape what enters the record. This path asks how proposed reforms address those particular problems, and what evidence would show that the reforms themselves help.
This path begins with selective decisions, then asks what replication can establish. It pairs an argument for prospective planning with an evaluation of Registered Reports, a format in which the study question and methods are reviewed before results are known. It then examines variation among analysts and connects usable data with ethical obligations. It ends with a model of scientific incentives and an independent reconstruction of the model's code that exposed a coding error. The same scrutiny applies to proposals for reform as to the research they seek to improve.
By the end, you should be able to connect a proposed safeguard to a specific problem, identify what it leaves unresolved and propose evidence for evaluating its benefits and burdens. The objective is a more informative and correctable record, with room for exploration, uncertainty and disagreement.
Unit 1 of 6
How can research decisions distort the record?
Begin with a mechanism rather than a verdict on a field. A simulation can show what particular choices permit without telling us how often those choices occur. Keep error, selective reporting and evidence of intent distinct.
Reading
1 of 10False-Positive Psychology
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359–1366. https://doi.org/10.1177/0956797611417632
Methodological contribution · Historical turning point · Accessible · Full work: 8 pages
Connect the demonstrations and simulations to the disclosures proposed by the authors.
Why you’re reading it. Identify a specific way in which selection changes the meaning of reported evidence.
A question to consider. If a researcher tries several outcome measures after seeing the results, what would you need to know to interpret the selected analysis?
Unit 2 of 6
What would a replication tell us?
Nosek and Errington offer a definition centered on claims and possible outcomes. The Open Science Collaboration supplies a large empirical investigation to examine through that lens. Ask both what each replication tests and what the collection can represent.
Reading
2 of 10What Is Replication?
Nosek, B. A., & Errington, T. M. (2020). What is replication? PLOS Biology, 18(3), Article e3000691. https://doi.org/10.1371/journal.pbio.3000691
Foundational argument · Methodological contribution · Accessible · Full work: 8 pages
Keep the claim being tested visible when considering changes in procedures or context.
Why you’re reading it. Distinguish informative confrontation with evidence from literal repetition.
A question to consider. A study supports a broad theory, but a contrary outcome is dismissed as a task failure. What would make the test genuinely diagnostic?
Reading
3 of 10Estimating the Reproducibility of Psychological Science
Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716. https://doi.org/10.1126/science.aac4716
Replication · Historical turning point · Moderate · Full work: 8 pages
Compare the project’s selection decisions and outcome criteria with the preceding conceptual argument.
Why you’re reading it. Interpret a coordinated replication project without treating its headline as a field-wide reliability score.
A question to consider. Which conclusion would require several replications of one effect rather than single replications of many effects?
Unit 3 of 6
What changes when plans and publication decisions come first?
Recording a plan and securing prospective review address related but different decisions. Compare the rationale for each with evidence about published outcomes, keeping the quality of the underlying question and design in view.
Reading
4 of 10The Preregistration Revolution
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences, 115(11), 2600–2606. https://doi.org/10.1073/pnas.1708274114
Methodological contribution · Foundational argument · Moderate · Full work: 7 pages
Distinguish commitments about predictions, analyses and publication from one another.
Why you’re reading it. Understand what a prospective record makes assessable while preserving transparent exploration.
A question to consider. If an exploratory result becomes a new prediction, what would a subsequent study need to make that progression clear?
Reading
5 of 10An Excess of Positive Results
Scheel, A. M., Schijen, M. R. M. J., & Lakens, D. (2021). An excess of positive results: Comparing the standard psychology literature with Registered Reports. Advances in Methods and Practices in Psychological Science, 4(2), Article 25152459211007467. https://doi.org/10.1177/25152459211007467
Methodological contribution · Moderate · Full work: 12 pages
Trace the selection and coding decisions before interpreting the comparison between publication formats.
Why you’re reading it. Evaluate a reform with the same causal caution applied to other observational comparisons.
A question to consider. What alternative explanation for the difference in positive results would you investigate before crediting the format alone?
Unit 4 of 6
How much does a conclusion depend on its analysis?
A prespecified analysis still represents a choice among possible approaches. The many-analyst study opens those choices to inspection. Judge whether analyses answer comparable questions before deciding what agreement or disagreement means.
Reading
6 of 10Many Analysts, One Data Set
Silberzahn, R., Uhlmann, E. L., Martin, D. P., Anselmi, P., Aust, F., Awtrey, E., Bahník, Š., Bai, F., Bannard, C., Bonnier, E., Carlsson, R., Cheung, F., Christensen, G., Clay, R., Craig, M. A., Dalla Rosa, A., Dam, L., Evans, M. H., Flores Cervantes, I., … Nosek, B. A. (2018). Many analysts, one data set: Making transparent how variations in analytic choices affect results. Advances in Methods and Practices in Psychological Science, 1(3), 337–356. https://doi.org/10.1177/2515245917747646
Methodological contribution · Moderate · Full work: 20 pages
Compare the analytic questions, models and uncertainty behind the teams’ conclusions.
Why you’re reading it. Extend scrutiny beyond whether a single analysis was planned in advance.
A question to consider. If most teams obtain a significant association, what must still be examined before interpreting it as a causal effect?
Unit 5 of 6
What makes evidence inspectable—and sharing responsible?
Hardwicke and colleagues test whether shared data can actually support reconstruction of results. Meyer asks how sharing relates to consent and risk. Together they invite a plan that makes scrutiny possible while respecting legitimate access restrictions.
Reading
7 of 10From Available Data to Reproducible Analyses
Hardwicke, T. E., Mathur, M. B., MacDonald, K., Nilsonne, G., Banks, G. C., Kidwell, M. C., Hofelich Mohr, A., Clayton, E., Yoon, E. J., Henry Tessler, M., Lenne, R. L., Altman, S., Long, B., & Frank, M. C. (2018). Data availability, reusability, and analytic reproducibility: Evaluating the impact of a mandatory open data policy at the journal Cognition. Royal Society Open Science, 5(8), Article 180448. https://doi.org/10.1098/rsos.180448
Methodological contribution · Moderate · Full work: 18 pages
Separate a working data link, usable documentation and recovered numerical results.
Why you’re reading it. Translate transparency into concrete conditions for inspection.
A question to consider. What would you add to a study archive if another researcher could open its files but could not reconstruct the reported exclusions?
Reading
8 of 10Practical Tips for Ethical Data Sharing
Meyer, M. N. (2018). Practical tips for ethical data sharing. Advances in Methods and Practices in Psychological Science, 1(1), 131–144. https://doi.org/10.1177/2515245917747656
Methodological contribution · Foundational argument · Moderate · Full work: 14 pages
Examine the ethical premises of sharing recommendations, including decisions about previously collected material.
Why you’re reading it. Make participant obligations part of research quality rather than an afterthought.
A question to consider. For sensitive interviews, what combination of documentation, permission and access conditions could support justified scrutiny?
Unit 6 of 6
How can institutions support correction?
A model proposes one way institutional rewards could shape scientific practices. Its reimplementation then subjects that proposal to scrutiny. Separate three questions: whether the code follows the model, whether the model supports an argument and whether the argument describes real institutions.
Reading
9 of 10The Natural Selection of Bad Science
Smaldino, P. E., & McElreath, R. (2016). The natural selection of bad science. Royal Society Open Science, 3(9), Article 160384. https://doi.org/10.1098/rsos.160384
Foundational argument · Methodological contribution · Challenging · Full work: 17 pages
Trace the model’s selection mechanism and assess its scope alongside the linked correction.
Why you’re reading it. Consider why individual intentions alone may not explain patterns in a scientific record.
A question to consider. Which institutional practice would you change to test the proposed mechanism, and what outcome would count against it?
Reading
10 of 10Replicating a Model of Scientific Incentives
Kohrt, F., Smaldino, P. E., McElreath, R., & Schönbrodt, F. (2023). Replication of the natural selection of bad science. Royal Society Open Science, 10(2), Article 221306. https://doi.org/10.1098/rsos.221306
Replication · Critical reappraisal · Moderate · Full work: 5 pages
Compare the stated replication criterion with the changed dynamics and preserved conclusions.
Why you’re reading it. Use an actual correction to evaluate what scientific self-correction requires.
A question to consider. How should a public correction distinguish a changed result, a changed interpretation and a conclusion that remains supported?
Scope and omissions
This path concerns the production, evaluation and correction of a scientific record. It does not replace training in theory, measurement, causal inference or statistical modeling. The examples mainly concern quantitative psychology and simulation; qualitative traditions, community-led research and large clinical infrastructures have additional standards and constraints. The optional CARE paper adds collective governance to a discussion otherwise centered largely on individual consent.
The readings do not provide a timeless percentage of reliable psychology or a universal checklist for good science. The model papers demonstrate implications of assumptions, and observational evaluations of reforms cannot isolate every cause. Ethical and regulatory discussions belong to their historical and jurisdictional settings. This is not current legal advice, a misconduct-detection manual or comprehensive research-integrity training.
Synthesis
Reliability develops through the relation between claims, decisions open to inspection, and opportunities for correction. Prospective commitments, new evidence, transparent analyses, and institutional changes address different weaknesses. Theory, measurement, and judgment remain essential. Reform is therefore part of the inquiry it seeks to improve: its benefits and burdens need evaluation, and its corrections should make clear exactly what has changed.
Consider an invented consortium studying student belonging. Different teams use different measures, some analyses are exploratory, interview transcripts contain identifying experiences, and staff advancement depends heavily on publication counts. A replication estimates a smaller effect than the original, while a separate code check finds an exclusion error that changes one subgroup result.
Propose a research and publication plan for the consortium. Identify the claim each study can test, explain how prospective and exploratory work will be distinguished, specify what can be inspected and under which access conditions, and describe how the error and replication will be reported. Recommend one institutional change and a feasible way to evaluate both its benefits and burdens. End by identifying an uncertainty that the plan cannot resolve.
To examine what makes a claim explanatory in the first place, continue with How Do We Build and Test Psychological Theories?.
Further reading
Optional. Not part of the core sequence.
- Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1(1), Article 0021. Paper. Compare a broad program of methodological and institutional proposals with the specific problems examined in the core path.
- Chambers, C. D., & Tzavella, L. (2022). The past, present and future of Registered Reports. Nature Human Behaviour, 6(1), 29–42. Paper. Extend the discussion of prospective review, practical implementation and limitations; distinguish the proposed benefits from their evaluation.
- Carroll, S. R., Garba, I., Figueroa-Rodríguez, O. L., Holbrook, J., Lovett, R., Materechera, S., Parsons, M., Raseroka, K., Rodriguez-Lonebear, D., Rowe, R., Sara, R., Walker, J. D., Anderson, J., & Hudson, M. (2020). The CARE Principles for Indigenous Data Governance. Data Science Journal, 19, Article 43. Paper. Examine collective benefit, authority to control, responsibility and ethics as a distinct contribution to data governance.
- Ledgerwood, A. (2018). The preregistration revolution needs to distinguish between predictions and analyses. Proceedings of the National Academy of Sciences, 115(45), E10516–E10517. Paper. Sharpen the distinction between how a hypothesis is formed and how evidence is produced; the core guide also links the authors’ reply.
- Nosek, B. A., Spies, J. R., & Motyl, M. (2012). Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7(6), 615–631. Paper. Examine an earlier institutional reform argument and compare proposed incentives with the modeled mechanisms in the final unit.
- Stewart, A. J., & Plotkin, J. B. (2021). The natural selection of good science. Nature Human Behaviour, 5(11), 1510–1518. Paper. Explore an alternative model in which investment in theory changes the selection of research practices; compare its assumptions with the original incentives model.