Examining the replicability of online experiments selected by a decision market

Creators: Holzmeister, Felix¹; Johannesson, Magnus²; Camerer, Colin F.³; Chen, Yiling⁴; Ho, Teck-Hua⁵; Hoogeveen, Suzanne⁶; Huber, Juergen¹; Imai, Noriko⁷; Imai, Taisuke⁷; Jin, Lawrence⁸; Kirchler, Michael¹; Ly, Alexander^{9, 10}; Mandl, Benjamin¹¹; Manfredi, Dylan¹²; Nave, Gideon¹²; Nosek, Brian A.^{13, 14}; Pfeiffer, Thomas¹⁵; Sarafoglou, Alexandra⁹; Schwaiger, Rene¹; Wagenmakers, Eric-Jan⁹; Waldén, Viking¹⁶; Dreber, Anna^{1, 2}

1. Universität Innsbruck
2. Stockholm School of Economics
3. California Institute of Technology
4. Harvard University
5. Nanyang Technological University
6. Utrecht University
7. Osaka University
8. National University of Singapore
9. University of Amsterdam
10. Centrum Wiskunde & Informatica
11. Independent Researcher, Vienna, Austria
12. University of Pennsylvania
13. University of Virginia
14. Center for Open Science
15. Massey University
16. Swedish National Bank

Abstract

Here we test the feasibility of using decision markets to select studies for replication and provide evidence about the replicability of online experiments. Social scientists (n = 162) traded on the outcome of close replications of 41 systematically selected MTurk social science experiments published in PNAS 2015–2018, knowing that the 12 studies with the lowest and the 12 with the highest final market prices would be selected for replication, along with 2 randomly selected studies. The replication rate, based on the statistical significance indicator, was 83% for the top-12 and 33% for the bottom-12 group. Overall, 54% of the studies were successfully replicated, with replication effect size estimates averaging 45% of the original effect size estimates. The replication rate varied between 54% and 62% for alternative replication indicators. The observed replicability of MTurk experiments is comparable to that of previous systematic replication projects involving laboratory experiments.

Copyright and License

© 2024, The Author(s). This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Acknowledgement

We thank A. Andevall for helping with the data collection and programming of experiments and R. Willer for helpful advice on defining the IP address check and exclusion criteria used to exclude individuals from participating to minimize low-quality participant data. For financial support, we thank the Austrian Science FWF (grant SFB F63 to J.H. and M.K.), Jan Wallander and Tom Hedelius Foundation (grants P21-0091 and P23-0098 to A.D.), Knut and Alice Wallenberg Foundation and Marianne and Marcus Wallenberg Foundation (Wallenberg Scholar grant to A.D.) and Riksbankens Jubileumsfond (grant P21-0168 to M.J.). The funders had no role in study design, data collection and analysis, decision to publish or preparation of the paper. One author (V.W.) is currently employed by Sveriges Riksbank but did this work before being employed by Sveriges Riksbank; the opinions expressed in this article are the sole responsibility of the authors and should not be interpreted as reflecting the views of Sveriges Riksbank.

Data Availability

The data reported in this paper are tabulated in Supplementary Tables 1–8. The replication reports (both the pre-replication and the post-replication versions), the pre-analysis plan, the data from the survey and the decision market, and the data for each of the 26 replications are available at the project’s OSF repository (https://osf.io/sk82q).

Code Availability

The analysis scripts, generating all results, figures and tables reported in the main text and the Supplementary Information, are available at the project’s OSF repository (https://osf.io/sk82q).

Supplemental Material

Supplementary Information:
Supplementary Notes, references, Fig. 1 and Tables 1–8.

Contributions

A.D., F.H., J.H., M.J., M.K., B.A.N. and T.P. designed the study. A.D., F.H. and M.J. managed the study. Y.C., A.D., F.H., M.J. and T.P. designed and implemented the decision market. A.D., F.H., M.J., B.M. and V.W. selected articles and critical findings for (potential) replication. A.D., C.F.C., F.H., T.-H.H., S.H., J.H., N.I., T.I., L.J., M.J., M.K., B.M., D.M., G.N., A.S., R.S., E.-J.W. and V.W. designed the replications and collected replication data. F.H., S.H., T.I., L.J., A.S., R.S. and V.W. conducted the preregistered statistical tests on the individual replications. A.L. computed the Bayes factors. F.H. conducted all analyses reported in the paper. A.D., F.H. and M.J. wrote the paper. All authors reviewed and approved the final paper.

Files

Name	Size	Download all
s41562-024-02062-9.pdf md5:dd456410ab5d3f3183bc97eb0a0549e1	10.5 MB	Preview Download
41562_2024_2062_MOESM1_ESM.pdf md5:f24173758c7a793442ce6876ac01e05a	856.5 kB	Preview Download

	All versions	This version
Views	0	0
Downloads	0	0
Data volume	0 Bytes	0 Bytes