What Can Statistical Evidence Tell Us?

Introduction

A school is considering a programme intended to improve students' performance. A study reports better scores among participants. Before deciding what to make of the result, we need to ask how large the difference was, how precisely it was estimated, and whether the study's design supports the comparison. A single number cannot answer all of these questions.

This syllabus begins with the quantity being estimated and the units used to express it. It then examines uncertainty and the problem of comparing findings: why might two studies appear to disagree, and what would show that their results really differ? The aim is to connect statistical summaries with the particular claim they can support.

The next readings ask what a study of a given design can reasonably detect or estimate, and how selective reporting can change the picture that reaches readers. The final unit considers two further questions: which specified model better predicted the observations, and whether the evidence can rule out effects large enough to matter. Failing to establish an effect is not automatically evidence that any effect must be negligible.

Return to the school example throughout. By the end, you should be able to explain the meaning and limits of a reported result, identify an assumption that matters to its interpretation, and say what additional information would change your judgment. The guides introduce the statistical terms where the questions require them; the goal is informed interpretation rather than proficiency with a software package.

Scope and omissions

The eight core articles span six units and 97 full-work pages. This intermediate path combines conceptual arguments, numerical demonstrations, an empirical survey and reanalyses. The guides explain what the calculations mean without assuming that every reader can implement each procedure. Access copies with cover sheets or manuscript pagination are identified separately.

The central skills are defining an effect, interpreting uncertainty, comparing estimates, evaluating design sensitivity and distinguishing support for a model from a judgment about meaningful magnitude. The neuroscience survey is a historical case with sampling limitations. The school example is a recurring reasoning exercise, not evidence that a particular intervention works.

This is not a catalogue of statistical tests or a comprehensive course in estimation, Bayesian computation, multiplicity, sequential designs, missing data or decision theory. Statistics cannot repair invalid measurement or identify a causal effect without a suitable design. Different groups may reasonably assign different costs to errors, burdens and benefits; a threshold does not settle those choices.

Synthesis

Statistical evidence becomes useful when it joins a clear question to an effect, an estimate, uncertainty, and stated assumptions. It must also distinguish a planned inquiry from choices made after the results were seen. More observations may improve precision; they cannot alone supply valid measurement, identify a cause, or decide whether an effect matters for the choice at hand.

Keep the questions separate. A p value is not a probability that a hypothesis is true. Different significance labels do not establish different effects. Power describes a design under assumed effects; it is not the observed result’s credibility score. A Bayes factor compares specified models, while equivalence testing evaluates justified bounds. Neither eliminates the need for substantive reasoning.

Prepare a short evidence statement for the school. Define the outcome, comparison and units. Explain the estimated benefit and its uncertainty, identify a plausible source of bias, and describe how the design was chosen. State what predictions would enter a Bayesian comparison and what bounds would define a meaningful gain. Explain what remains uncertain, whose costs and benefits matter, and which additional observation would most change the decision.

For the design assumptions behind a causal effect, return to What Would Justify a Causal Claim?. To examine predictions for consequential outcomes, explore What Can Individual Differences Predict?.

For the design and interpretation of repeated observations, continue with How Can We Study Change Within People?.