EDITORIAL

Raising the bar

See allHide authors and affiliations

Science  04 Jul 2014:
Vol. 345, Issue 6192, pp. 9
DOI: 10.1126/science.1257891
IMAGE: STACEY PENTLAND PHOTOGRAPHY

Numbers. Lots and lots of numbers. It is hard to find a paper published in Science or any other journal that is not full of numbers. Interpretation of those numbers provides the basis for the conclusions, as well as an assessment of the confidence in those conclusions. But unfortunately, there have been far too many cases where the quantitative analysis of those numbers has been flawed, causing doubt about the authors' interpretation and uncertainty about the result. Furthermore, it is not realistic to expect that a technical reviewer, chosen for her or his expertise in the topical subject matter or experimental protocol, will also be an expert in data analysis. For that reason, with much help from the American Statistical Association, Science has established, effective 1 July 2014, a Statistical Board of Reviewing Editors (SBoRE), consisting of experts in various aspects of statistics and data analysis, to provide better oversight of the interpretation of observational data.

“Readers must have confidence in the conclusions published in our journal.”

IMAGE: SORBETTO/ISTOCKPHOTO.COM

For those familiar with the role of Science's Board of Reviewing Editors (BoRE), the function of the SBoRE will be slightly different. Members of the BoRE perform a rapid quality check of manuscripts and recommend which should receive indepth review by technical specialists. Members of the SBoRE will receive manuscripts that have been identified by editors, BoRE members, or possibly reviewers as needing additional scrutiny of the data analysis or statistical treatment. The SBoRE member assesses what the issue is that requires screening and suggests experts from the statistics community to provide it.

So why is Science taking this additional step? Readers must have confidence in the conclusions published in our journal. We want to continue to take reasonable measures to verify the accuracy of those results. We believe that establishing the SBoRE will help avoid honest mistakes and raise the standards for data analysis, particularly when sophisticated approaches are needed. But even when taking added precautions, no review system is infallible, and no combination of reviewers can be expected to uncover all of the ways in which the interpretation of results may have gone wrong. In particular, it is difficult for reviewers to detect whether authors have approached the study with a lack of bias in their data collection and presentation.

I recall a study that I conducted years ago involving a global analysis of some oceanographic features that I was modeling to understand the rheology of oceanic plates on million-year time scales. I had only a handful of data points—perhaps a dozen or so—and the fit to my model failed a significance test. Clearly, throwing out a few of the data points by declaring them “outliers” would have improved the fit dramatically, and in fact I even recall a reviewer of the paper commenting: “Can't you make the data fit the model better?”

Really?

The editor published the paper despite the lousy fit of the model to the data. It was not too long before it was realized that those “outliers” were the key to a more complete understanding of the long-term rheological behavior of the oceanic plates. Although the model in the earlier paper needed an overhaul, the fundamental observations, because they were presented without bias, inspired much further progress in the field.

In the years since, I have been amazed at how many scientists have never considered that their data might be presented with bias. There are fundamental truths that may be missed when bias is unintentionally overlooked, or worse yet, when data are “massaged.” Especially as we enter an era of “big data,” we should raise the bar ever higher in scrutinizing the analyses that take us from observations to understanding.

Navigate This Article