Report

Evaluating replicability of laboratory experiments in economics

See allHide authors and affiliations

Science  25 Mar 2016:
Vol. 351, Issue 6280, pp. 1433-1436
DOI: 10.1126/science.aaf0918
  • Fig. 1 Replication results.

    (A) Plotted are 95% CIs of replication effect sizes (standardized to correlation coefficients). The standardized effect sizes are normalized so that 1 equals the original effect size (fig. S1 shows a nonnormalized version). Eleven replications have a significant effect in the same direction as in the original study [61.1%; 95% CI = (36.2%, 86.1%)]. The 95% CI of the replication effect size includes the original effect size for 12 replications [66.7%; 95% CI = (42.5%, 90.8%)]; if we also include the study in which the entire 95% CI exceeds the original effect size, this increases to 13 replications [72.2%; 95% CI = (49.3%, 95.1%)]. AER denotes the American Economic Review and QJE denotes the Quarterly Journal of Economics. (B) Meta-analytic estimates of effect sizes, combining the original and replication studies. Plotted are 95% CIs ofcombined effect sizes (standardized to correlation coefficients). The standardized effect sizes are normalized as in (A) (fig. S1 shows a nonnormalized version). Fourteen studies have a significant effect in the same direction as the original study in the meta-analysis [77.8%; 95% CI = (56.5%, 99.1%)].

  • Fig. 2 Prediction market and survey beliefs.

    A plot of prediction market beliefs and survey beliefs, in relation to whether the original result was replicated with P < 0.05 in the original direction. The mean prediction market belief in a successful replication is 75.2% [range, 59% to 94%; 95% CI = (69.7%, 80.6%)], and the mean survey belief is 71.1% [range, 54% to 86%; 95% CI = (66.4%, 75.8%)]. The prediction market beliefs and survey beliefs are highly correlated (Spearman correlation coefficient = 0.79, P < 0.001, n = 18). Both the prediction market beliefs (Spearman correlation coefficient = 0.30, P = 0.232, n = 18) and the survey beliefs (Spearman correlation coefficient 0.52, P = 0.028, n = 18) are positively correlated with the ranked degree of replication success.

  • Fig. 3 Correlations between P values and sample sizes in original studies and replicability indicators.

    (A) The original P value is negatively correlated with all six replicability indicators, and five of these correlations are significant. (B) The original sample size is positively correlated with all six replicability indicators, and five of these correlations are significant. Spearman correlation coefficients are shown on the vertical axes. *P < 0.05; **P < 0.01.

  • Fig. 4 A comparison of replicability indicators in experimental economics (this study) and psychological sciences (RPP).

    The graph shows means ± SE for replicability indicators. All six replicability indicators are higher for experimental economics; this difference is significant for three of the replicability indicators. The average difference in replicability across the six indicators is 19 percentage points. Details about the statistical tests are included in the supplementary materials. *P < 0.05; **P < 0.01.

Supplementary Materials

  • Evaluating replicability of laboratory experiments in economics

    Colin F. Camerer, Anna Dreber, Eskil Forsell, Teck-Hua Ho, Jürgen Huber, Magnus Johannesson, Michael Kirchler, Johan Almenberg, Adam Altmejd, Taizan Chan, Emma Heikensten, Felix Holzmeister, Taisuke Imai, Siri Isaksson, Gideon Nave, Thomas Pfeiffer, Michael Razen, Hang Wu

    Materials/Methods, Supplementary Text, Tables, Figures, and/or References

    Download Supplement
    • Materials and Methods
    • Figs. S1 to S6
    • Tables S1 to S5
    • Full Reference List

Navigate This Article