Context

The paper argues that many, perhaps most, published research findings are false positives rather than true discoveries.

The likelihood that a published claim is actually true depends not just on “statistical significance,” but on several factors including the prior probability that the hypothesis is true, the statistical power of the study, and biases (in design, analysis, reporting).

Key Factors Influencing Truth of Findings

The author outlines conditions that tend to make published findings less reliable:

  • Small sample size / low power: Studies with too few subjects have lower chance to detect real effects and higher chance to produce false positives.
  • Small effect sizes: When the true effect is modest, it’s harder to detect reliably. As effect-size decreases, the proportion of false claims rises.
  • Large number of hypotheses tested (exploratory, “fishing expeditions”): Fields where many relationships are tested — e.g. genetics, exploratory epidemiology — have low “pre‑study odds” of being true, so many “significant” findings are just chance.
  • Flexibility in methods, definitions, outcomes, analyses: This includes multiple ways to slice data, choose endpoints, run analyses etc. This flexibility makes it easier (intentionally or inadvertently) to find statistically “significant” results even if no real effect.
  • Bias — in design, analysis, publication, or reporting: Selective reporting, analysis manipulation, publication bias toward “positive” findings all reduce the chance that claimed results are true.
  • Competitive / “hot” research fields with many independent teams: When many researchers probe the same questions, at least some are likely to report significant — but possibly false — results.

Quantitative Argument (simplified model)

  • The paper uses a probabilistic model (2 × 2 table of “true relationships” vs “observed findings”) to show that the positive predictive value (PPV) — the probability that a “significant” finding is actually true — depends heavily on pre‑study odds, power and bias.
  • Under realistic conditions for many research fields (low pre‑study odds, small effects, moderate power, some bias), the model suggests PPV may be well below 50% — meaning more findings are false than true.

Implications & Recommendations

The paper draws several important consequences:

  • “Significant” p‑values (e.g. p < 0.05) should not be treated as strong evidence of truth by themselves.
  • Scientists and readers should be more skeptical — particularly about findings from small, flexible, exploratory, or highly competitive fields.
  • To improve reliability: more well-powered studies; pre‑study thinking about prior plausibility (pre‑study odds); less flexibility/bias in design and reporting; replication; meta-analysis and large confirmatory trials rather than overemphasis on single small studies.
  • Recognize that many “discoveries” may reflect bias + noise, not true effects — so replicate before acting on new findings.

Conclusion

This work was foundational for the field of metascience (science about science) — exposing how methodological flaws, publication practices, and incentives distort the reliability of published research.

It remains widely cited, used to explain issues such as replication crisis, irreproducible results, and inflated claims especially in biomedical, genetic, and social‑science research.

“Low-powered studies are not only less likely to detect true effects, but they also tend to exaggerate effect sizes and greatly reduce the likelihood that a statistically significant result reflects a true effect.” Button, K. S., et al. (2013)

References

https://doi.org/10.1371/journal.pmed.1004085

Button, K. S., et al. (2013). Power failure: why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14, 365–376.