A Caltech Library Service

Evaluating replicability of laboratory experiments in economics

Camerer, Colin F. and Dreber, Anna and Forsell, Eskil and Ho, Teck-Hua and Huber, Jürgen and Johannesson, Magnus and Kirchler, Michael and Almenberg, Johan and Altmejd, Adam and Chan, Taizan and Heikensten, Emma and Holzmeister, Felix and Imai, Taisuke and Isaksson, Siri and Nave, Gideon and Pfeiffer, Thomas and Razen, Michael and Wu, Hang (2016) Evaluating replicability of laboratory experiments in economics. Science, 351 (6280). pp. 1433-1436. ISSN 0036-8075. doi:10.1126/science.aaf0918.

[img] PDF (Materials and Methods, Figs. S1 to S6, Tables S1 to S5, Full Reference List) - Supplemental Material
See Usage Policy.


Use this Persistent URL to link to this item:


The reproducibility of scientific findings has been called into question. To contribute data about reproducibility in economics, we replicate 18 studies published in the American Economic Review and the Quarterly Journal of Economics in 2011-2014. All replications follow predefined analysis plans publicly posted prior to the replications, and have a statistical power of at least 90% to detect the original effect size at the 5% significance level. We find a significant effect in the same direction as the original study for 11 replications (61%); on average the replicated effect size is 66% of the original. The reproducibility rate varies between 67% and 78% for four additional reproducibility indicators, including a prediction market measure of peer beliefs.

Item Type:Article
Related URLs:
URLURL TypeDescription DOIArticle material
Camerer, Colin F.0000-0003-4049-1871
Dreber, Anna0000-0003-3989-9941
Ho, Teck-Hua0000-0001-5210-4977
Johannesson, Magnus0000-0001-8759-6393
Altmejd, Adam0000-0002-4248-0677
Imai, Taisuke0000-0002-0610-8093
Nave, Gideon0000-0001-6251-5630
Additional Information:© 2016 American Association for the Advancement of Science. Received 16 December 2015; accepted 19 February 2016; Published online 3 March 2016. For financial support we thank: Austrian Science Fund FWF (START-grant Y617-G11), Austrian National Bank (grant OeNB 14953), Behavioral and Neuroeconomics Discovery Fund (CFC), Jan Wallander and Tom Hedelius Foundation (P2015-0001:1 and P2013-0156:1), Knut and Alice Wallenberg Foundation (Wallenberg Academy Fellows grant to A. Dreber), Swedish Foundation for Humanities and Social Sciences (NHS14-1719:1), and Sloan Foundation (G-2015-13929). We thank the following experimental labs for kindly allowing us to use them for replication experiments: Center for Behavioral Economics at National University of Singapore, Center for Neuroeconomics Studies at Claremont Graduate University, Frankfurt Laboratory for Experimental Economic Research, Harvard Decision Science Laboratory, Innsbruck ECONLAB, and Nuffield Centre for Experimental Social Sciences. We thank the following persons for assistance with the experiments: Jorge Barraza, Agneta Berge, Rahul Bhui, Andreas Born, Nina Cohodes, Ho Kinh Dat, Christoph Dohmen, Zayan Faiayd, Malte Heissel, Austin Henderson, Gabe Mansur, Jutta Preussler, Lukas Schultze, Garrett Thoelen, and Elizabeth Warner. The data reported in this paper are tabulated in tables S1, S3 and S4 and the Replication Reports, analyses code, and the data from the replications are available at and at OSF ( The authors report no potential conflicts of interest. No MTAs, patents or patent applications apply to methods or data in the paper. CC, AD, JH, TH, MJ, and MK designed research; CC, AD, EF, JH, TH, MJ, and MK wrote the paper; EF, JA, TC, TH, TP helped design the prediction market part; EF, FH, JH, MK, MR, TP, and HW analyzed data; AA, EH, FH, TI, SI, GN, MR, and HW carried out the replications (including re-estimating the original estimate with the replication data); all authors approved the final manuscript.
Funding AgencyGrant Number
FWF Der WissenschaftsfondsY617-G11
Austrian National BankOeNB 14953
Behavioral and Neuroeconomics Discovery Fund (CFC)UNSPECIFIED
Jan Wallander and Tom Hedelius FoundationP2015-0001:1
Jan Wallander and Tom Hedelius FoundationP2013-0156:1
Knut and Alice Wallenberg Foundation (Wallenberg Academy Fellows)UNSPECIFIED
Swedish Foundation for Humanities and Social SciencesNHS14-1719:1
Sloan FoundationG-2015-13929
Issue or Number:6280
Record Number:CaltechAUTHORS:20160303-090305606
Persistent URL:
Official Citation:Evaluating replicability of laboratory experiments in economics By Colin F. Camerer, Anna Dreber, Eskil Forsell, Teck-Hua Ho, Jürgen Huber, Magnus Johannesson, Michael Kirchler, Johan Almenberg, Adam Altmejd, Taizan Chan, Emma Heikensten, Felix Holzmeister, Taisuke Imai, Siri Isaksson, Gideon Nave, Thomas Pfeiffer, Michael Razen, Hang Wu Science 25 Mar 2016: Vol. 351, Issue 6280, pp. 1433-1436 DOI: 10.1126/science.aaf0918
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:64988
Deposited By: Katherine Johnson
Deposited On:03 Mar 2016 20:49
Last Modified:10 Nov 2021 23:38

Repository Staff Only: item control page