df/N refers to the information on which the test of the effect was based (for example, df of t test, denominator df of F test, sample size –3 of correlation, and sample size for z and χ2). Four original results had P values slightly higher than 0.05 but were considered positive results in the original article and are treated that way here. Exclusions (explanation provided in supplementary materials, A3) are “replications P < 0.05” (3 original nulls excluded; n = 97 studies); “mean original and replication effect sizes” (3 excluded; n = 97 studies); “meta-analytic mean estimates” (27 excluded; n = 73 studies); “percent meta-analytic (P < 0.05)” (25 excluded; n = 75 studies); and, “percent original effect size within replication 95% CI” (5 excluded, n = 95 studies).
Effect size comparison | Original and replication combined | ||||||||||
Replications P < 0.05 in original direction | Percent | Mean (SD) original effect size | Median original df/N | Mean (SD) replication effect size | Median replication df/N | Average replication power | Meta-analytic mean (SD) estimate | Percent meta-analytic (P < 0.05) | Percent original effect size within replication 95% CI | Percent subjective “yes” to “Did it replicate?” | |
Overall | 35/97 | 36 | 0.403 (0.188) | 54 | 0.197 (0.257) | 68 | 0.92 | 0.309 (0.223) | 68 | 47 | 39 |
JPSP, social | 7/31 | 23 | 0.29 (0.10) | 73 | 0.07 (0.11) | 120 | 0.91 | 0.138 (0.087) | 43 | 34 | 25 |
JEP:LMC, cognitive | 13/27 | 48 | 0.47 (0.18) | 36.5 | 0.27 (0.24) | 43 | 0.93 | 0.393 (0.209) | 86 | 62 | 54 |
PSCI, social | 7/24 | 29 | 0.39 (0.20) | 76 | 0.21 (0.30) | 122 | 0.92 | 0.286 (0.228) | 58 | 40 | 32 |
PSCI, cognitive | 8/15 | 53 | 0.53 (0.2) | 23 | 0.29 (0.35) | 21 | 0.94 | 0.464 (0.221) | 92 | 60 | 53 |