Analysis
The reproducibility crisis: how many studies hold up?
Large replication projects have found that many published results do not repeat: 36% of 100 psychology replications were statistically significant against 97% of the originals, and cancer-biology replications showed effects a median 85% smaller.[1][2] A 2019 National Academies report says failed replications do not necessarily mean something is wrong, while pointing to the misuse of statistics and lagging training in computational practice.[3][4][5] Whether this amounts to a "crisis" is itself debated; the proposed fixes include sharing data and code, and peer review of study plans before data are collected.[6][7][8]
The question
If a study is repeated, should it give the same answer? In a 2019 report commissioned by the US National Science Foundation at Congress’s direction, the National Academies of Sciences, Engineering, and Medicine separated two ideas.[9] Reproducibility means getting consistent results from the same data, code and methods. Replicability means getting consistent results across studies that each collect their own data to answer the same question.[10] The report noted that when results cannot be confirmed, some see a problem and others see a natural part of science that can lead to discoveries.[6]
What replication projects found
Psychology. In a 2015 Science paper the Open Science Collaboration repeated 100 studies from three psychology journals, using high-powered designs and original materials where available. Ninety-seven percent of the originals had statistically significant results, against 36% of the replications, and replication effects were on average half as large.[1] Success was better predicted by the strength of the original evidence than by who ran the studies.[11]
Social science in top journals. A 2018 project replicated 21 social-science experiments published in Nature and Science between 2010 and 2015, with samples about five times larger. It found a significant effect in the same direction for 13 (62%), with effects averaging about half the original size.[12] Researchers’ beliefs about which studies would replicate were strongly related to the outcomes.[13]
Cancer biology. The Reproducibility Project: Cancer Biology set out to repeat 193 experiments from 53 high-impact papers but completed only 50 experiments from 23 papers.[14] The data needed to compute effect sizes were publicly available for just 4 of the 193 experiments, and none was described in enough detail to repeat without help from the original authors.[15] For positive effects, the median replication effect was 85% smaller than the original, and 46% of effects replicated on more criteria than they failed.[2]
Well before these projects, a 2005 PLOS Medicine essay by John Ioannidis argued from simulations that for most study designs and settings a research claim is more likely to be false than true.[16]
Why results fail to replicate
The National Academies report said non-replicability occurs for many reasons that do not necessarily mean something is wrong. Some cases help science by revealing previously unknown effects or sources of variability; others, from simple mistakes and methodological errors to bias and fraud, do not.[3] It said statistical inference has had an outsized role in replicability discussions because of the frequent misuse of statistics, such as the p-value and the threshold for statistical significance.[4] It also said the training of scientists in best computational practices has not kept pace with the growth of computation in science, which likely contributes to a surprisingly low rate of computational reproducibility.[5]
When papers are found to be seriously flawed or fraudulent, they can be retracted; see why scientific papers are retracted.[17][18]
How to judge a failed replication
The report argued that replication may not yield a pass or fail answer and is better expressed as the degree to which one result replicates another. It added that focusing on single replications is an inefficient way to establish reliable knowledge, and that reviews of cumulative evidence on a subject are often a more useful guide.[19]
The projects above measure different things: one tests whether a result reaches significance again, another compares effect sizes. A finding can “fail” by one test and partly hold by another. That is one reason headline replication rates vary so much between fields and projects. It also means a single failed replication says less about any one finding than a body of repeated studies does.
Proposed fixes
The National Academies recommended that NSF and other funders endorse or create repositories that preserve research code and data over the long term.[7] Registered Reports, a format in which peer review happens before data are collected, provisionally accept studies on the strength of their design.[8] The Center for Open Science, which promotes the format, says it removes questionable practices such as low statistical power, selective reporting and publication bias.[20]
How journal review works today, and how newer models differ, is covered in how peer review works.
Competing views
There is a serious problem
Replication projects in psychology, social science and cancer biology found that many published effects shrank or failed to reach significance again, and a 2005 essay argued from simulations that for most study designs a research claim is more likely to be false than true.[1][12][2][16]
Read next
Next in the crash course
Why scientific papers are retracted
What a retraction is, how it differs from a correction or expression of concern, how often misconduct is the cause, and how to check a paper.
Questions readers ask
What is the difference between reproducibility and replicability?
In the National Academies' 2019 definitions, reproducibility means getting consistent results from the same data and code, and replicability means getting consistent results in a new study that collects its own data to answer the same question.[10]
How many psychology studies replicate?
In the 2015 Open Science Collaboration project, 97% of 100 original studies had statistically significant results, against 36% of the replications, and the replication effects were on average half as large.[1]
Is this only a problem in psychology?
No. A 2018 replication of 21 social-science experiments in Nature and Science found significant effects for 13 (62%), and a cancer-biology project found replication effects a median 85% smaller than the originals.[12][2]
Does a failed replication mean the original was fraud?
No. The National Academies report says non-replicability occurs for many reasons that do not necessarily mean something is wrong; some cases reveal unknown effects, while others stem from mistakes, methodological errors, bias or fraud.[3]
Can scientists predict which studies will replicate?
To a degree. The 2018 social-science project found that researchers' beliefs about which studies would replicate were strongly related to the actual outcomes.[13]
Sources
· 20 claims from 9 publishers
Each numbered claim is a statement we checked against the sources listed with it. Tap a number in the text to see its sources in place.
- [1]
In a 2015 Science paper, the Open Science Collaboration repeated 100 psychology studies; 97% of the originals had statistically significant results but 36% of the replications did, and replication effects were on average half as large. confirmedas of 2026-11-08
- Estimating the reproducibility of psychological science (Open Science Collaboration, Science 2015): abstract · Science (abstract via NCBI PubMed) · Primary source · 2015-08-28 (retrieved 2026-11-08)
- Estimating the reproducibility of psychological science (Open Science Collaboration, Science 2015): abstract · Science (abstract via NCBI PubMed) · Primary source · 2015-08-28 (retrieved 2026-11-08)
- Estimating the reproducibility of psychological science (Open Science Collaboration, Science 2015): abstract · Science (abstract via NCBI PubMed) · Primary source · 2015-08-28 (retrieved 2026-11-08)
- [2]
For positive effects, the median replication effect size in the cancer-biology project was 85% smaller than in the originals, and 46% of effects replicated on more criteria than they failed. confirmedas of 2026-11-08
- Investigating the replicability of preclinical cancer biology (Errington et al., eLife 2021): abstract · eLife (abstract via NCBI PubMed) · Primary source · 2021-12-07 (retrieved 2026-11-08)
- Reproducibility Project: Cancer Biology · Center for Open Science · Secondary source (retrieved 2026-11-08)
- [3]
The report said non-replicability occurs for many reasons that do not necessarily mean something is wrong; some cases help science by revealing unknown effects or sources of variability, while others, from simple mistakes and methodological errors to bias and fraud, do not. confirmedas of 2026-11-08
- Reproducibility and Replicability in Science: Summary · National Academies of Sciences, Engineering, and Medicine · Primary source · 2019-05-07 (retrieved 2026-11-08)
- Reproducibility and Replicability in Science: Summary · National Academies of Sciences, Engineering, and Medicine · Primary source · 2019-05-07 (retrieved 2026-11-08)
- Reproducibility and Replicability in Science: Summary · National Academies of Sciences, Engineering, and Medicine · Primary source · 2019-05-07 (retrieved 2026-11-08)
- [4]
The report said statistical inference has had an outsized role in replicability discussions because of the frequent misuse of statistics such as the p-value and the threshold for statistical significance. confirmedas of 2026-11-08
- Reproducibility and Replicability in Science: Summary · National Academies of Sciences, Engineering, and Medicine · Primary source · 2019-05-07 (retrieved 2026-11-08)
- [5]
The report said the training of scientists in best computational research practices has not kept pace with the growth of computation in science, which likely contributes to a surprisingly low rate of computational reproducibility. confirmedas of 2026-11-08
- Reproducibility and Replicability in Science: Summary · National Academies of Sciences, Engineering, and Medicine · Primary source · 2019-05-07 (retrieved 2026-11-08)
- Reproducibility and Replicability in Science: Summary · National Academies of Sciences, Engineering, and Medicine · Primary source · 2019-05-07 (retrieved 2026-11-08)
- [6]
The National Academies report noted that when results cannot be confirmed, some see a problem and others see a natural part of science that can lead to discoveries. confirmedas of 2026-11-08
- Reproducibility and Replicability in Science: Summary · National Academies of Sciences, Engineering, and Medicine · Primary source · 2019-05-07 (retrieved 2026-11-08)
- [7]
The report recommended that NSF and other funders endorse or create code and data repositories for long-term preservation. confirmedas of 2026-11-08
- Reproducibility and Replicability in Science: Summary · National Academies of Sciences, Engineering, and Medicine · Primary source · 2019-05-07 (retrieved 2026-11-08)
- [8]
In the Registered Reports format, peer review happens before data are collected, and protocols that pass are provisionally accepted for publication if the authors follow the registered method. confirmedas of 2026-11-08
- Registered Reports · Center for Open Science · Primary source (retrieved 2026-11-08)
- Registered Reports · Center for Open Science · Primary source (retrieved 2026-11-08)
- Criteria for Publication · PLOS One · Primary source (retrieved 2026-11-08)
- [9]
The National Academies committee was set up by NSF at Congress's direction to define reproducing and replicating a study and to assess effects on public trust in science. confirmedas of 2026-11-08
- Reproducibility and Replicability in Science: Summary · National Academies of Sciences, Engineering, and Medicine · Primary source · 2019-05-07 (retrieved 2026-11-08)
- Reproducibility and Replicability in Science: Summary · National Academies of Sciences, Engineering, and Medicine · Primary source · 2019-05-07 (retrieved 2026-11-08)
- [10]
A 2019 National Academies report defined reproducibility as getting consistent results from the same data and code, and replicability as getting consistent results across studies that collect their own data to answer the same question. confirmedas of 2026-11-08
- Reproducibility and Replicability in Science: Summary · National Academies of Sciences, Engineering, and Medicine · Primary source · 2019-05-07 (retrieved 2026-11-08)
- Reproducibility and Replicability in Science: Summary · National Academies of Sciences, Engineering, and Medicine · Primary source · 2019-05-07 (retrieved 2026-11-08)
- [11]
The 2015 psychology project found replication success was better predicted by the strength of the original evidence than by characteristics of the research teams. confirmedas of 2026-11-08
- Estimating the reproducibility of psychological science (Open Science Collaboration, Science 2015): abstract · Science (abstract via NCBI PubMed) · Primary source · 2015-08-28 (retrieved 2026-11-08)
- [12]
A 2018 project replicated 21 social-science experiments published in Nature and Science between 2010 and 2015 with samples about five times larger and found a significant effect in the same direction for 13 (62%), with effects averaging about half the original size. confirmedas of 2026-11-08
- Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015 (Camerer et al., Nature Human Behaviour 2018): abstract · Nature Human Behaviour (abstract via NCBI PubMed) · Primary source · 2018-09-01 (retrieved 2026-11-08)
- Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015 (Camerer et al., Nature Human Behaviour 2018): abstract · Nature Human Behaviour (abstract via NCBI PubMed) · Primary source · 2018-09-01 (retrieved 2026-11-08)
- Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015 (Camerer et al., Nature Human Behaviour 2018): abstract · Nature Human Behaviour (abstract via NCBI PubMed) · Primary source · 2018-09-01 (retrieved 2026-11-08)
- [13]
The 2018 team found that researchers' beliefs about which studies would replicate were strongly related to the actual outcomes. confirmedas of 2026-11-08
- Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015 (Camerer et al., Nature Human Behaviour 2018): abstract · Nature Human Behaviour (abstract via NCBI PubMed) · Primary source · 2018-09-01 (retrieved 2026-11-08)
- [14]
The Reproducibility Project - Cancer Biology set out to repeat 193 experiments from 53 high-impact papers but completed only 50 experiments from 23 papers. confirmedas of 2026-11-08
- Challenges for assessing replicability in preclinical cancer biology (Errington et al., eLife 2021): abstract · eLife (abstract via NCBI PubMed) · Primary source · 2021-12-07 (retrieved 2026-11-08)
- Challenges for assessing replicability in preclinical cancer biology (Errington et al., eLife 2021): abstract · eLife (abstract via NCBI PubMed) · Primary source · 2021-12-07 (retrieved 2026-11-08)
- [15]
The data needed to compute effect sizes were publicly available for only 4 of the 193 cancer-biology experiments, and none was described in enough detail to repeat without asking the original authors. confirmedas of 2026-11-08
- Challenges for assessing replicability in preclinical cancer biology (Errington et al., eLife 2021): abstract · eLife (abstract via NCBI PubMed) · Primary source · 2021-12-07 (retrieved 2026-11-08)
- Challenges for assessing replicability in preclinical cancer biology (Errington et al., eLife 2021): abstract · eLife (abstract via NCBI PubMed) · Primary source · 2021-12-07 (retrieved 2026-11-08)
- [16]
In a 2005 PLOS Medicine essay, John Ioannidis argued from simulations that for most study designs and settings a research claim is more likely to be false than true. confirmedas of 2026-11-08
- Why Most Published Research Findings Are False · PLOS Medicine · Primary source · 2005-08-30 (retrieved 2026-11-08)
- [17]
Under ICMJE guidance, if an institutional investigation proves scientific misconduct the editor should publish a retraction. confirmedas of 2026-11-08
- Recommendations for the Conduct, Reporting, Editing, and Publication of Scholarly Work in Medical Journals · International Committee of Medical Journal Editors (ICMJE) · Primary source (retrieved 2026-11-08)
- [18]
A 2012 PNAS study of 2,047 retracted biomedical and life-science articles in PubMed found 21.3% were retracted for error and 67.4% for misconduct, including fraud or suspected fraud (43.4%), duplicate publication (14.2%) and plagiarism (9.8%). confirmedas of 2026-11-08
- Misconduct accounts for the majority of retracted scientific publications (Fang, Steen and Casadevall, PNAS 2012): abstract · PNAS (abstract via NCBI PubMed) · Primary source · 2012-10-16 (retrieved 2026-11-08)
- Misconduct accounts for the majority of retracted scientific publications (Fang, Steen and Casadevall, PNAS 2012): abstract · PNAS (abstract via NCBI PubMed) · Primary source · 2012-10-16 (retrieved 2026-11-08)
- [19]
The report said replication may best be judged as a matter of degree rather than pass or fail, and that reviews of cumulative evidence are often a better guide than single replications. confirmedas of 2026-11-08
- Reproducibility and Replicability in Science: Summary · National Academies of Sciences, Engineering, and Medicine · Primary source · 2019-05-07 (retrieved 2026-11-08)
- Reproducibility and Replicability in Science: Summary · National Academies of Sciences, Engineering, and Medicine · Primary source · 2019-05-07 (retrieved 2026-11-08)
- Reproducibility and Replicability in Science: Summary · National Academies of Sciences, Engineering, and Medicine · Primary source · 2019-05-07 (retrieved 2026-11-08)
- [20]
The Center for Open Science says Registered Reports remove questionable research practices such as low statistical power, selective reporting and publication bias. confirmedas of 2026-11-08· interpretation
- Registered Reports · Center for Open Science · Primary source (retrieved 2026-11-08)
Revision history (1)
- Page created.
Created Nov 8, 2026. Last reviewed by an editor on Nov 8, 2026. Next scheduled review: May 8, 2027.
Cite this page
"The reproducibility crisis: how many studies hold up?." ContentLora, updated Nov 8, 2026. https://contentlora.com/analysis/reproducibility-crisis-debate
Spotted an error? Suggest a correction or emailcorrections@contentlora.com.