EDITORIAL

Reproducibility

Science  17 Jan 2014:
Vol. 343, Issue 6168, pp. 229
DOI: 10.1126/science.1250475
CREDIT: STACEY PENTLAND PHOTOGRAPHY

Science advances on a foundation of trusted discoveries. Reproducing an experiment is one important approach that scientists use to gain confidence in their conclusions. Recently, the scientific community was shaken by reports that a troubling proportion of peer-reviewed preclinical studies are not reproducible. Because confidence in results is of paramount importance to the broad scientific community, we are announcing new initiatives to increase confidence in the studies published in Science. For preclinical studies (one of the targets of recent concern), we will be adopting recommendations of the U.S. National Institute of Neurological Disorders and Stroke (NINDS) for increasing transparency.* Authors will indicate whether there was a pre-experimental plan for data handling (such as how to deal with outliers), whether they conducted a sample size estimation to ensure a sufficient signal-to-noise ratio, whether samples were treated randomly, and whether the experimenter was blind to the conduct of the experiment. These criteria will be included in our author guidelines.

There are a number of reasons why peer-reviewed preclinical studies may not be reproducible. The system under investigation may be more complex than previously thought, so that the experimenter is not actually controlling all independent variables. Authors may not have divulged all of the details of a complicated experiment, making it irreproducible by another lab. It is also expected that through random chance, a certain number of studies will produce false positives. If researchers are not alert to this possibility and have not set appropriately stringent significance tests for their results, the outcome is a study with irreproducible results. Although there is always the possibility that an occasional study is fraudulent, the number of preclinical studies that cannot be reproduced is inconsistent with the idea that all irreproducibility results from misconduct in such research.

CREDIT: ONEO2/ISTOCKPHOTO.COM

It is unlikely that the issues with irreproducibility are confined to preclinical studies (social science has been equally noted, for example). Unfortunately, there are no equivalents to the NINDS recommendations for other disciplines that provide a basis for requiring transparency across all fields. For the next 6 months, we will be asking reviewers and editors to identify papers submitted to Science that demonstrate excellence in transparency and instill confidence in the results. This will inform the next steps in implementing reproducibility guidelines. Science Translational Medicine, a sister journal of Science, already enforces the NINDS guidelines for preclinical studies. Both journals also are open to improving on the NINDS recommendations for preclinical studies.

There is also a wide range of sophistication in the application of statistics displayed in research analysis, ranging from practically no statistics, to the routine use of generic software packages, to the application of advanced methods that extract subtle signals from noise. Because reviewers who are chosen for their expertise in the subject matter of a study may not be authorities in statistics as well, statistical errors in manuscripts may slip through undetected. For that reason, with the advice of the American Statistical Association and others, we are adding new members to our Board of Reviewing Editors from the statistics community to ensure that manuscripts receive appropriate scrutiny in their methods of data analysis.

Science's standards have always been high, and these measures add to steps we have already taken to increase transparency, such as requiring data accessibility. Nevertheless, journals can only do so much to assure readers of the validity of the studies they publish. The ultimate responsibility lies with authors to be completely open with their methods, all of their findings, and the possible pitfalls that could invalidate their conclusions.

  • * S. C. Landis et al., Nature 490, 187 (2012).

Subjects

Navigate This Article