Measurement error and the replication crisis

See allHide authors and affiliations

Science  10 Feb 2017:
Vol. 355, Issue 6325, pp. 584-585
DOI: 10.1126/science.aal3618

eLetters is an online forum for ongoing peer review. Submission of eLetters are open to all. eLetters are not edited, proofread, or indexed.  Please read our Terms of Service before submitting your own eLetter.

Compose eLetter

Plain text

  • Plain text
    No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.
Author Information
First or given name, e.g. 'Peter'.
Your last, or family, name, e.g. 'MacMoody'.
Your email address, e.g.
Your role and/or occupation, e.g. 'Orthopedic Surgeon'.
Your organization or institution (if applicable), e.g. 'Royal Free Hospital'.
Statement of Competing Interests

This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.

Vertical Tabs

  • Correcting Measurement Error to Build Scientific Knowledge
    • Brenton M. Wiernik, Postdoctoral Fellow, Ghent University
    • Other Contributors:
      • Deniz S. Ones, Professor, University of Minnesota

    Loken and Gelman (1) describe problems of null-hypothesis significance testing, selective publishing, and imperfect measures distorting the scientific literature. They raise questions about the validity of widely-accepted, well-understood methods for statistically correcting measurement error (2) and the studies that have applied them. Relationships and effects under study are affected by “noise.” But fortunately, the “noise” can be separated into two types, and there are well-known solutions for each. The first type is systematic error in measurements, which predictably biases observed relations downward. The second type is random (sampling) error, which unpredictably obscures true relations, unsystematically making observed relations smaller or larger. On average, random error is zero, but it can be large in small samples (3). Both types of error obscure true relations and must be corrected to draw accurate conclusions. Systematic error can be addressed in single samples using well-known statistical corrections (2). Random error cannot be corrected in single samples because the direction and size of the error is unknown. However, random error asymptotes to zero as sample size increases. Its impact can be mitigated either by gathering much larger samples or, more practically, by combining results from smaller studies using meta-analysis (4). When studies are meta-analytically pooled, their random errors cancel out. Furthermore, their systematic errors can be statistically...

    Show More
    Competing Interests: None declared.
  • RE: Measurement error and the replication crisis

    As noted in a personal email from the lead author, the article could have made a clearer distinction between sampling error and random measurement error. We show that random measurement error always attenuates population effect sizes and statistical power, which reduces the chance of obtaining a significant result.

    If sampling error (due to luck or questionable research practices) inflates observed effect sizes enough to produce a significant result, the median amount of inflation is inversely related to power of a study. Thus, conditional on selection for significance, random measurement error leads to more inflation, but the estimated effect sizes are never larger than the effect sizes that would have been obtained with a more reliable measure. In sum, consistent with statistical theories, random measurement error always attenuates observed effect sizes, even when studies are selected for significance.

    For a detailed comment that could not be published in Science, you can read the commentary on my blog.

    Ulrich Schimmack & Rickard Carlsson

    Competing Interests: None declared.

Stay Connected to Science