People often say “A small p-value means strong evidence.” That sounds intuitive, but it is incorrect and can easily lead to misunderstanding research results.

What a p-value actually means

A p-value answers a very specific question: If there were actually no real effect, how surprising would the data we observed be?
So a small p-value simply means: the results would be unusual if there were truly no effect. But it does not tell us that the effect is real or important.

Why a small p-value is not strong evidence

It does not tell whether the effect is important. A study can find a very tiny difference that doesn’t matter in practice, but still produce a very small p-value.
This happens especially in very large studies where even trivial differences can appear “statistically significant.”

Random chance can still produce small p-values

If researchers carries out several tests, some will appear significant purely by chance.

Example:

A scientist tests 20 different outcomes. Even if nothing works, about one result will look significant by chance. This is why statistical adjustments for multiple comparisons are needed.

Results with small p-values may not be reproducible

If the same study is repeated, the p-value can change a lot. A study with p = 0.04 today might produce p = 0.20 tomorrow simply because of random variation. This is one reason some published findings fail to replicate.

Instead of saying: Small p-value = strong evidence
It is preferable to say: A small p-value means the results would be surprising if there were truly no effect.
But deciding whether something is convincing also requires looking at:
– How large the clinical/biological/engineering/practical effect is and its uncertainty
– How well the study was conducted
– Whether other studies corroborate the results.

A small p-value is just one piece of information. By itself, it does not prove that a result is real, important, or reliable.

This is exactly why many statisticians emphasise estimation over dichotomous testing. Confidence intervals provide a range of values compatible with the data rather than a binary “significant / not significant” decision (Altman 2000; Greenland et al. 2016). The ASA has also recommended focusing on effect estimates and uncertainty rather than p-value thresholds (Wasserstein et al. 2019).