CaltechAUTHORS
  A Caltech Library Service

Predicting the replicability of social science lab experiments

Altmejd, Adam and Dreber, Anna and Forsell, Eskil and Huber, Juergen and Imai, Taisuke and Johannesson, Magnus and Kirchler, Michael and Nave, Gideon and Camerer, Colin (2019) Predicting the replicability of social science lab experiments. PLoS ONE, 14 (12). Art. No. e0225826. ISSN 1932-6203. PMCID PMC6894796. https://resolver.caltech.edu/CaltechAUTHORS:20191218-084443166

[img] PDF - Published Version
Creative Commons Attribution.

799Kb
[img] PDF - Submitted Version
Creative Commons Attribution.

493Kb
[img] PDF (S1 Table. Variable descriptions.) - Supplemental Material
Creative Commons Attribution.

64Kb
[img] PDF (S2 Table. Summary statistics.) - Supplemental Material
Creative Commons Attribution.

46Kb
[img] PDF (S3 Table. Prediction market accuracy.) - Supplemental Material
Creative Commons Attribution.

32Kb
[img] PDF (S4 Table. PCA—Explained variance.) - Supplemental Material
Creative Commons Attribution.

31Kb
[img] PDF (S1 Text. Data management and decisions.) - Supplemental Material
Creative Commons Attribution.

11Kb
[img] PDF (S2 Text. Predicting a noisy outcome.) - Supplemental Material
Creative Commons Attribution.

21Kb
[img] Postscript (S1 Fig. Full correlation plot.) - Supplemental Material
Creative Commons Attribution.

69Kb
[img] Postscript (S2 Fig. PCA plot.) - Supplemental Material
Creative Commons Attribution.

116Kb
[img] Postscript (S3 Fig. Performance comparison of algorithms.) - Supplemental Material
Creative Commons Attribution.

37Kb
[img] Postscript (S4 Fig. SSRP prediction evaluation—Stage 1.) - Supplemental Material
Creative Commons Attribution.

51Kb

Use this Persistent URL to link to this item: https://resolver.caltech.edu/CaltechAUTHORS:20191218-084443166

Abstract

We measure how accurately replication of experimental results can be predicted by black-box statistical models. With data from four large-scale replication projects in experimental psychology and economics, and techniques from machine learning, we train predictive models and study which variables drive predictable replication. The models predicts binary replication with a cross-validated accuracy rate of 70% (AUC of 0.77) and estimates of relative effect sizes with a Spearman ρ of 0.38. The accuracy level is similar to market-aggregated beliefs of peer scientists [1, 2]. The predictive power is validated in a pre-registered out of sample test of the outcome of [3], where 71% (AUC of 0.73) of replications are predicted correctly and effect size correlations amount to ρ = 0.25. Basic features such as the sample and effect sizes in original papers, and whether reported effects are single-variable main effects or two-variable interactions, are predictive of successful replication. The models presented in this paper are simple tools to produce cheap, prognostic replicability metrics. These models could be useful in institutionalizing the process of evaluation of new findings and guiding resources to those direct replications that are likely to be most informative.


Item Type:Article
Related URLs:
URLURL TypeDescription
https://doi.org/10.1371/journal.pone.0225826DOIArticle
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6894796PubMed CentralArticle
https://doi.org/10.17605/OSF.IO/4FN73Related ItemData
https://doi.org/10.1371/journal.pone.0225826.s001DOIS1 Table
https://doi.org/10.1371/journal.pone.0225826.s002DOIS2 Table
https://doi.org/10.1371/journal.pone.0225826.s003DOIS3 Table
https://doi.org/10.1371/journal.pone.0225826.s004DOIS4 Table
https://doi.org/10.1371/journal.pone.0225826.s005DOIS1 Text
https://doi.org/10.1371/journal.pone.0225826.s006DOIS2 Text
https://doi.org/10.1371/journal.pone.0225826.s007DOIS1 Fig.
https://doi.org/10.1371/journal.pone.0225826.s008DOIS2 Fig.
https://doi.org/10.1371/journal.pone.0225826.s009DOIS3 Fig.
https://doi.org/10.1371/journal.pone.0225826.s010DOIS4 Fig.
https://doi.org/10.31222/osf.io/zamryDOIDiscussion Paper
ORCID:
AuthorORCID
Altmejd, Adam0000-0002-4248-0677
Dreber, Anna0000-0003-3989-9941
Imai, Taisuke0000-0002-0610-8093
Johannesson, Magnus0000-0001-8759-6393
Nave, Gideon0000-0001-6251-5630
Camerer, Colin0000-0003-4049-1871
Additional Information:© 2019 Altmejd et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Received: June 20, 2019; Accepted: November 13, 2019; Published: December 5, 2019. Data Availability: All data and code files are available from the OSF project repository (https://osf.io/4fn73/, DOI: 10.17605/OSF.IO/4FN73). Generous support was provided by the Alfred P. Sloan Foundation (G-201513929, to CFC), the Chen Center for Social and Decision Neuroscience (to CFC), the Austrian Science Fund FWF (SFB F63 to MK and JH, P29362-G27 to JH), The Jan Wallander and Tom Hedelius Foundation (P2015-0001:1 to MJ, P14-0214 and P13-0156 to AD), The Knut and Alice Wallenberg Foundation (Wallenberg Academy Fellows grant to AD), The Swedish Foundation for Humanities and Social Sciences (NHS14--1719 to MJ and AD), and Forte (2016-07099 to AA). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. The authors have declared that no competing interests exist. We thank Isak Roth and Dylan Manfredi for outstanding research assistance. Author Contributions: Conceptualization: Adam Altmejd. Data curation: Adam Altmejd, Anna Dreber, Eskil Forsell, Juergen Huber, Taisuke Imai, Magnus Johannesson, Michael Kirchler, Gideon Nave. Formal analysis: Adam Altmejd, Eskil Forsell, Taisuke Imai, Gideon Nave. Funding acquisition: Anna Dreber, Magnus Johannesson, Colin Camerer. Investigation: Adam Altmejd. Methodology: Adam Altmejd, Taisuke Imai, Gideon Nave. Project administration: Adam Altmejd. Software: Adam Altmejd. Supervision: Anna Dreber, Magnus Johannesson, Colin Camerer. Validation: Adam Altmejd. Visualization: Adam Altmejd. Writing – original draft: Adam Altmejd. Writing – review & editing: Adam Altmejd, Taisuke Imai, Magnus Johannesson, Michael Kirchler, Gideon Nave, Colin Camerer.
Group:Tianqiao and Chrissy Chen Institute for Neuroscience
Funders:
Funding AgencyGrant Number
Alfred P. Sloan FoundationG-201513929
Tianqiao and Chrissy Chen Institute for NeuroscienceUNSPECIFIED
FWF Der WissenschaftsfondsSFB F63
FWF Der WissenschaftsfondsP29362-G27
Jan Wallander and Tom Hedelius FoundationP2015-0001:1
Jan Wallander and Tom Hedelius FoundationP14-0214
Jan Wallander and Tom Hedelius FoundationP13-0156
Knut and Alice Wallenberg FoundationUNSPECIFIED
Riksbankens JubileumsfondNHS14-1719
Forskningsrådet om Hälsa, Arbetsliv och Välfärd2016-07099
Issue or Number:12
PubMed Central ID:PMC6894796
Record Number:CaltechAUTHORS:20191218-084443166
Persistent URL:https://resolver.caltech.edu/CaltechAUTHORS:20191218-084443166
Official Citation:Altmejd A, Dreber A, Forsell E, Huber J, Imai T, Johannesson M, et al. (2019) Predicting the replicability of social science lab experiments. PLoS ONE 14(12): e0225826. https://doi.org/10.1371/journal.pone.0225826
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:100340
Collection:CaltechAUTHORS
Deposited By: Tony Diaz
Deposited On:18 Dec 2019 20:00
Last Modified:17 Jan 2020 21:07

Repository Staff Only: item control page