CaltechAUTHORS
  A Caltech Library Service

Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015

Camerer, Colin F. and Dreber, Anna and Holzmeister, Felix and Ho, Teck-Hua and Huber, Jürgen and Johannesson, Magnus and Kirchler, Michael and Nave, Gideon and Nosek, Brian A. and Pfeiffer, Thomas and Altmejd, Adam and Buttrick, Nick and Chan, Taizan and Chen, Yiling and Forsell, Eskil and Gampa, Anup and Heikensten, Emma and Hummer, Lily and Imai, Taisuke and Isaksson, Siri and Manfredi, Dylan and Rose, Julia and Wagenmakers, Eric-Jan and Wu, Hang (2018) Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nature Human Behaviour, 2 (9). pp. 637-644. ISSN 2397-3374. http://resolver.caltech.edu/CaltechAUTHORS:20181120-072245155

[img] PDF (Supplementary Methods, Supplementary References, Supplementary Tables 1–7 and Supplementary Figures 1–9) - Supplemental Material
See Usage Policy.

880Kb
[img] PDF (Reporting Summary) - Supplemental Material
See Usage Policy.

67Kb

Use this Persistent URL to link to this item: http://resolver.caltech.edu/CaltechAUTHORS:20181120-072245155

Abstract

Being able to replicate scientific findings is crucial for scientific progress. We replicate 21 systematically selected experimental studies in the social sciences published in Nature and Science between 2010 and 2015. The replications follow analysis plans reviewed by the original authors and pre-registered prior to the replications. The replications are high powered, with sample sizes on average about five times higher than in the original studies. We find a significant effect in the same direction as the original study for 13 (62%) studies, and the effect size of the replications is on average about 50% of the original effect size. Replicability varies between 12 (57%) and 14 (67%) studies for complementary replicability indicators. Consistent with these results, the estimated true-positive rate is 67% in a Bayesian analysis. The relative effect size of true positives is estimated to be 71%, suggesting that both false positives and inflated effect sizes of true positives contribute to imperfect reproducibility. Furthermore, we find that peer beliefs of replicability are strongly related to replicability, suggesting that the research community could predict which results would replicate and that failures to replicate were not the result of chance alone.


Item Type:Article
Related URLs:
URLURL TypeDescription
https://doi.org/10.1038/s41562-018-0399-zDOIArticle
https://rdcu.be/bbEDAPublisherFree ReadCube access
ORCID:
AuthorORCID
Camerer, Colin F.0000-0003-4049-1871
Holzmeister, Felix0000-0001-9606-0427
Ho, Teck-Hua0000-0001-5210-4977
Johannesson, Magnus0000-0001-8759-6393
Nave, Gideon0000-0001-6251-5630
Nosek, Brian A.0000-0001-6797-5476
Pfeiffer, Thomas0000-0002-0592-577X
Altmejd, Adam0000-0002-4248-0677
Imai, Taisuke0000-0002-0610-8093
Additional Information:© 2018 Springer Nature Limited. Received 06 March 2018; Accepted 06 July 2018; Published 27 August 2018. Neither Nature Human Behaviour nor the publisher had any involvement with the conduct of this study prior to its submission to the journal. For financial support we thank: the Austrian Science Fund FWF (SFB F63, START-grant Y617-G11), the Austrian National Bank (grant OeNB 14953), the Behavioral and Neuroeconomics Discovery Fund (C.F.C.), the Jan Wallander and Tom Hedelius Foundation (P2015-0001:1 and P2013-0156:1), the Knut and Alice Wallenberg Foundation (Wallenberg Academy Fellows grant to A.D.), the Swedish Foundation for Humanities and Social Sciences (NHS14-1719:1), the Netherlands Organisation for Scientific Research (Vici grant 016.Vici.170.083 to E.-J.W.), the Sloan Foundation (G-2015-13929) and the Singapore National Research Foundation’s Returning Singaporean Scientists Scheme (grant NRF-RSS2014-001 to T.-H.H.). The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript. We thank the following people for assistance with the experiments and analyses: D. van den Bergh, P.-C. Bindra, J. van Doorn, C. Huber, A. Ly, M. Marsman and J. Zambre. Code availability: The analysis codes for both the aggregate data and each individual replication are available at the project’s OSF repository (https://osf.io/pfdyw/). Data availability: The data reported in this paper and in the Supplementary Information are tabulated in Supplementary Tables 3–6. The replication reports (pre-data collection and final versions) and the data and analysis code for each individual replication are available in subprojects organized in the same repository (https://osf.io/pfdyw/). Author Contributions: C.F.C., A.D., F.H., J.H., T.-H.H., M.J., M.K., G.N., B.A.N. and T.P. designed the research. C.F.C., A.D., F.H., T.-H.H., J.H., M.J., M.K., D.M., G.N., B.A.N., T.P. and E.-J.W. wrote the paper. T.C., A.D., E.F., F.H., T.-H.H., M.J., T.P. and Y.C. helped to design the prediction market part. F.H. and E.-J.W. analysed the data. A.A., N.B., A.G., E.H., F.H., L.H., T.I., S.I., D.M., J.R. and H.W. carried out the replications (including re-estimating the original estimate with the replication data). All authors approved the final manuscript. The authors declare no competing interests.
Funders:
Funding AgencyGrant Number
FWF Der WissenschaftsfondsY617-G11
Austrian National BankOeNB 14953
Behavioral and Neuroeconomics Discovery FundUNSPECIFIED
Jan Wallander and Tom Hedelius FoundationP2015-0001:1
Jan Wallander and Tom Hedelius FoundationP2013-0156:1
Knut and Alice Wallenberg FoundationUNSPECIFIED
Swedish Foundation for Humanities and Social SciencesNHS14-1719:1
Nederlandse Organisatie voor Wetenschappelijk Onderzoek (NWO)016.Vici.170.083
Alfred P. Sloan FoundationG-2015-13929
National Research Foundation (Singapore)NRF-RSS2014-001
Record Number:CaltechAUTHORS:20181120-072245155
Persistent URL:http://resolver.caltech.edu/CaltechAUTHORS:20181120-072245155
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:91063
Collection:CaltechAUTHORS
Deposited By: Tony Diaz
Deposited On:20 Nov 2018 16:06
Last Modified:05 Apr 2019 21:17

Repository Staff Only: item control page