How Do We Combine Evidence Across Studies?

From a review question to a warranted conclusion

Level: Intermediate
Reading load: 8 core readings in 5 units
Prerequisites: Comfort with study designs, effect sizes and uncertainty; the statistical-evidence path provides a useful foundation. No meta-analysis software experience is assumed. The modeling and simulation papers are demanding; their reasoning can be followed without reproducing every calculation.
Last reviewed: September 2026

Introduction

A literature is an assembly of inquiries, each conducted under its own conditions. Their findings may concern different people, use different measures, share observations, or reach publication selectively. Bringing them together requires a question clear enough to govern both inclusion and interpretation. Precision is valuable only when we understand what the more precise answer is an answer to.

This path starts with the design of a review, then examines statistical models and the problems that arise when findings share participants or other sources of information before applying those ideas to hiring-discrimination studies. It contrasts recovering unpublished trials with statistical attempts to adjust for missing evidence. Finally, qualitative synthesis makes a different contribution: developing concepts and interpretations across accounts.

By the end, you should be able to trace a review’s conclusion back to its selection and analytic decisions, distinguish variation from bias, and explain what additional evidence could change its interpretation. The longer reviews and simulation paper reward several sittings. The examples are material for methodological appraisal, not current treatment recommendations.

Unit 1 of 5

What evidence could answer the review question?

A review begins by defining its claim and the evidence that could bear on it. Follow how eligibility, retrieval and appraisal shape the eventual synthesis; systematic procedures make choices inspectable without removing the need for judgment.

Reading
1 of 8
How to Do a Systematic Review
Siddaway, A. P., Wood, A. M., & Hedges, L. V. (2019). How to do a systematic review: A best practice guide for conducting and reporting narrative reviews, meta-analyses, and meta-syntheses. Annual Review of Psychology, 70(1), 747–770. https://doi.org/10.1146/annurev-psych-010418-102803
Methodological contribution · Major synthesis · Moderate · Full work: 24 pages
Trace the connections between the review question, eligible studies, searching and the form of synthesis.
Why you’re reading it. Treat a review as a designed investigation with a defensible scope.
A question to consider. For a review of whether group study improves learning, what differences in participants, comparison activities and outcomes would you specify before searching?

Unit 2 of 5

What are we combining, and what does the model assume?

First distinguish a common effect from an average of varying effects. Then examine why several estimates can contain overlapping information. Both the target and the dependence structure determine what a combined result can mean.

Reading
2 of 8
What Does a Meta-Analytic Average Estimate?
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2010). A basic introduction to fixed-effect and random-effects models for meta-analysis. Research Synthesis Methods, 1(2), 97–111. https://doi.org/10.1002/jrsm.12
Methodological contribution · Moderate · Full work: 15 pages
Compare the model assumptions with the changing weights and intervals in the worked examples.
Why you’re reading it. Understand the quantity a pooled estimate represents before interpreting its precision.
A question to consider. If study-skills interventions differ in format and duration, what would an average effect describe, and what would it leave unresolved about a particular course?
Reading
3 of 8
When Effect Sizes Are Not Independent
Cheung, M. W.-L. (2019). A guide to conducting a meta-analysis with non-independent effect sizes. Neuropsychology Review, 29(4), 387–396. https://doi.org/10.1007/s11065-019-09415-6
Methodological contribution · Challenging · Full work: 10 pages
Identify shared participants, shared comparisons and nested samples in the examples.
Why you’re reading it. Recognize why counting estimates is not the same as counting independent evidence.
A question to consider. If five reports come from two cohorts and each gives several outcomes, what information would you collect to avoid treating every result as a new study?

Unit 3 of 5

What can a pattern across studies establish?

A substantive synthesis lets us apply the earlier distinctions. Hiring audits provide contrasts within studies, while trends across years depend on comparisons between studies conducted under changing conditions. Those are related but different inferential tasks.

Reading
4 of 8
Meta-analysis of Field Experiments Shows No Change in Racial Discrimination in Hiring Over Time
Quillian, L., Pager, D., Hexel, O., & Midtbøen, A. H. (2017). Meta-analysis of field experiments shows no change in racial discrimination in hiring over time. Proceedings of the National Academy of Sciences, 114(41), 10870–10875. https://doi.org/10.1073/pnas.1706255114
Major synthesis · Methodological contribution · Moderate · Full work: 6 pages
Follow the search, coding and callback measure before examining the time-trend analyses.
Why you’re reading it. Connect eligibility and comparability to a substantive claim from a literature.
A question to consider. How could changes in sampled jobs or audit methods affect a time trend even if each individual study provides a credible comparison?

Unit 4 of 5

How could missing evidence change the conclusion?

Driessen and colleagues trace trials from funding records to recover results outside the journal literature. Carter and colleagues investigate statistical adjustments under simulated selection. Compare what direct recovery can establish with what an adjustment must assume.

Reading
5 of 8
What Happens When Unpublished Trials Are Included?
Driessen, E., Hollon, S. D., Bockting, C. L. H., Cuijpers, P., & Turner, E. H. (2015). Does publication bias inflate the apparent efficacy of psychological treatment for major depressive disorder? A systematic review and meta-analysis of US National Institutes of Health-funded trials. PLOS ONE, 10(9), Article e0137864. https://doi.org/10.1371/journal.pone.0137864
Methodological contribution · Major synthesis · Moderate · Full work: 23 pages
Compare the published-only and combined evidence within the same treatment contrasts, keeping unavailable data and outcome reporting in view.
Why you’re reading it. Understand why searching a registry or funding record can reveal a different evidence base from searching papers alone.
A question to consider. If all recoverable unpublished trials were added, what remaining problem could still bias the review’s conclusion?
Reading
6 of 8
Can Statistical Methods Correct Publication Bias?
Carter, E. C., Schönbrodt, F. D., Gervais, W. M., & Hilgard, J. (2019). Correcting for bias in psychology: A comparison of meta-analytic methods. Advances in Methods and Practices in Psychological Science, 2(2), 115–144. https://doi.org/10.1177/2515245919847196
Methodological contribution · Critical reappraisal · Challenging · Full work: 30 pages
Connect each simulated research environment to the performance of the competing adjustments.
Why you’re reading it. Evaluate sensitivity to assumptions instead of treating a corrected estimate as the recovered truth.
A question to consider. A review’s conclusion survives three adjustment methods but changes under a fourth. What would you examine before calling it robust?

Unit 5 of 5

How can studies contribute to a shared interpretation?

Some questions ask how people understand a process rather than how large an effect is. Thomas and Harden explain one qualitative synthesis method. Leamy and colleagues offer a different narrative synthesis to examine in practice: compare how each constructs an interpretation without assuming identical procedures.

Reading
7 of 8
Synthesizing Meanings Across Qualitative Studies
Thomas, J., & Harden, A. (2008). Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Medical Research Methodology, 8, Article 45. https://doi.org/10.1186/1471-2288-8-45
Methodological contribution · Moderate · Full work: 10 pages
Follow the transition from reported findings to descriptive and analytical themes, including how context is preserved or lost.
Why you’re reading it. See what synthesis can add when an average effect is not the intended result.
A question to consider. If a review moves from accounts of belonging to recommendations for a support program, which step requires an additional inference?
Reading
8 of 8
A Conceptual Framework for Personal Recovery in Mental Health
Leamy, M., Bird, V., Le Boutillier, C., Williams, J., & Slade, M. (2011). Conceptual framework for personal recovery in mental health: Systematic review and narrative synthesis. The British Journal of Psychiatry, 199(6), 445–452. https://doi.org/10.1192/bjp.bp.110.083733
Major synthesis · Moderate · Full work: 8 pages
Trace the recovery framework through source selection, coding, cultural comparison and consultation.
Why you’re reading it. Apply the question of evidential warrant to a conceptual synthesis.
A question to consider. What would a frequently mentioned recovery theme establish, and what further evidence would be required to claim that promoting it causes recovery?

Scope and omissions

This is a path for evaluating and designing an argument from a literature, not comprehensive training in meta-analytic computation. It introduces common/random effects, dependence and sensitivity without covering every estimator, network meta-analysis, individual-participant-data synthesis or Bayesian model. The qualitative unit demonstrates two approaches rather than an exhaustive map of synthesis traditions.

The examples concern U.S. hiring audits, funded depression trials and accounts of personal recovery, with methodological illustrations from other domains. Their populations, periods and eligibility criteria limit their claims. A framework developed mainly from English-language material cannot stand for every cultural understanding. No single reporting checklist, bias adjustment or quality score makes a review conclusive.

Synthesis

Synthesis makes a connected argument from a dispersed literature. Its question must remain visible through the choice, appraisal, and interpretation of evidence. Combining estimates depends on assumptions about their meaning and relation; variation may teach us something, and missing studies may alter the account. A new interpretation likewise remains responsible to the material from which it was developed.

Consider an invented review of peer-support programs for university students. It finds small trials with several outcomes, multiple papers from the same cohorts, two completed but unpublished studies, and interviews in which students describe both belonging and exclusion. Some trials compare peer support with no program, while others compare it with an established support service.

Write a short review plan that separates the questions of effectiveness and lived experience. Explain which results you would combine, which comparisons you would keep distinct, how you would handle dependent estimates and missing evidence, and how the qualitative material could contribute. End with a provisional conclusion, the assumptions on which it depends, and one feasible source of evidence that could change it.

For a closer look at the question of treatment benefit and harm, continue with How Do We Know When Psychotherapy Helps—or Harms?.

For the practices that shape the research record being synthesized, continue with How Can Psychological Science Become More Reliable?.