CaltechAUTHORS
  A Caltech Library Service

Megastudies improve the impact of applied behavioural science

Milkman, Katherine L. and Gromet, Dena and Ho, Hung and Kay, Joseph S. and Lee, Timothy W. and Pandiloski, Pepi and Park, Yeji and Rai, Aneesh and Bazerman, Max and Beshears, John and Bonacorsi, Lauri and Camerer, Colin and Chang, Edward and Chapman, Gretchen and Cialdini, Robert and Dai, Hengchen and Eskreis-Winkler, Lauren and Fishbach, Ayelet and Gross, James J. and Horn, Samantha and Hubbard, Alexa and Jones, Steven J. and Karlan, Dean and Kautz, Tim and Kirgios, Erika and Klusowski, Joowon and Kristal, Ariella and Ladhania, Rahul and Loewenstein, George and Ludwig, Jens and Mellers, Barbara and Mullainathan, Sendhil and Saccardo, Silvia and Spiess, Jann and Suri, Gaurav and Talloen, Joachim H. and Taxer, Jamie and Trope, Yaacov and Ungar, Lyle and Volpp, Kevin G. and Whillans, Ashley and Zinman, Jonathan and Duckworth, Angela L. (2021) Megastudies improve the impact of applied behavioural science. Nature, 600 (7889). pp. 478-483. ISSN 0028-0836. doi:10.1038/s41586-021-04128-4. https://resolver.caltech.edu/CaltechAUTHORS:20211216-65471400

[img] PDF (Supplementary Figures, Tables and Notes) - Supplemental Material
See Usage Policy.

3MB
[img] PDF (Reporting Summary) - Supplemental Material
See Usage Policy.

84kB
[img] Image (JPEG) (Extended Data Fig. 1: Measured vs. predicted change in likelihood of gym visit in a given week) - Supplemental Material
See Usage Policy.

271kB
[img]
Preview
Image (JPEG) (Extended Data Table 1 Regression-estimated effects of each experimental condition on whether participants visited the gym in a given week during the four-week intervention period relative to the Planning, Reminders and Micro-Incentives to Exercise conditi) - Supplemental Material
See Usage Policy.

135kB
[img]
Preview
Image (JPEG) (Extended Data Table 2 Regression-estimated effects of each experimental condition on whether participants visited the gym in a given week during the four-week post-intervention period relative to the Placebo Control condition) - Supplemental Material
See Usage Policy.

136kB
[img]
Preview
Image (JPEG) (Extended Data Table 3 The percentage of other conditions that each experimental condition outperformed for our dependent variable measuring whether participants visited the gym in a given week at p < .05 during the four-week intervention period) - Supplemental Material
See Usage Policy.

708B
[img]
Preview
Image (JPEG) (Extended Data Table 4 Participants’ mean age (in years), gender, length of gym membership (in weeks), and mean weekly gym visits in the four-week pre-intervention period across the 54 study conditions) - Supplemental Material
See Usage Policy.

701B
[img]
Preview
Image (JPEG) (Extended Data Table 5 Percentage of significant p-values and absolute difference in coefficients from pairwise comparisons of the 54 study conditions in our megastudy on each variable listed (alpha = .05)) - Supplemental Material
See Usage Policy.

32kB
[img]
Preview
Image (JPEG) (Extended Data Table 6 Regression-estimated effects of each experimental condition on total weekly gym visits during the four-week intervention period relative to the Placebo Control condition) - Supplemental Material
See Usage Policy.

134kB
[img]
Preview
Image (JPEG) (Extended Data Table 7 Regression-estimated effects of each experimental condition on whether participants visited the gym in a given week during the four-week intervention period relative to the Placebo Control condition) - Supplemental Material
See Usage Policy.

136kB
[img]
Preview
Image (JPEG) (Extended Data Table 8 Regression-estimated effects of each experimental condition on total weekly gym visits during the four-week intervention period relative to the Planning, Reminders, and Micro-Incentives to Exercise condition) - Supplemental Material
See Usage Policy.

135kB
[img]
Preview
Image (JPEG) (Extended Data Table 9 Regression-estimated effects of each experimental condition on total weekly gym visits during the four-week post-intervention period relative to the Placebo Control condition) - Supplemental Material
See Usage Policy.

136kB

Use this Persistent URL to link to this item: https://resolver.caltech.edu/CaltechAUTHORS:20211216-65471400

Abstract

Policy-makers are increasingly turning to behavioural science for insights about how to improve citizens’ decisions and outcomes. Typically, different scientists test different intervention ideas in different samples using different outcomes over different time intervals. The lack of comparability of such individual investigations limits their potential to inform policy. Here, to address this limitation and accelerate the pace of discovery, we introduce the megastudy—a massive field experiment in which the effects of many different interventions are compared in the same population on the same objectively measured outcome for the same duration. In a megastudy targeting physical exercise among 61,293 members of an American fitness chain, 30 scientists from 15 different US universities worked in small independent teams to design a total of 54 different four-week digital programmes (or interventions) encouraging exercise. We show that 45% of these interventions significantly increased weekly gym visits by 9% to 27%; the top-performing intervention offered microrewards for returning to the gym after a missed workout. Only 8% of interventions induced behaviour change that was significant and measurable after the four-week intervention. Conditioning on the 45% of interventions that increased exercise during the intervention, we detected carry-over effects that were proportionally similar to those measured in previous research. Forecasts by impartial judges failed to predict which interventions would be most effective, underscoring the value of testing many ideas at once and, therefore, the potential for megastudies to improve the evidentiary value of behavioural science.


Item Type:Article
Related URLs:
URLURL TypeDescription
https://doi.org/10.1038/s41586-021-04128-4DOIArticle
https://rdcu.be/cDmGNPublisherFree ReadCube access
https://osf.io/9av87/?view_only=8bb9282111c24f81a19c2237e7d7eba3Related ItemData/Code
ORCID:
AuthorORCID
Milkman, Katherine L.0000-0002-9706-4830
Rai, Aneesh0000-0001-9633-2568
Camerer, Colin0000-0003-4049-1871
Chang, Edward0000-0003-0502-2957
Dai, Hengchen0000-0001-7640-6558
Gross, James J.0000-0003-3624-3090
Horn, Samantha0000-0003-2716-5860
Karlan, Dean0000-0003-0164-1398
Kirgios, Erika0000-0002-7891-0499
Ladhania, Rahul0000-0002-7902-7681
Loewenstein, George0000-0003-2790-0474
Duckworth, Angela L.0000-0002-2644-3729
Additional Information:© 2021 Nature Publishing Group. Received 22 October 2020; Accepted 13 October 2021; Published 08 December 2021. Support for this research was provided in part by the Robert Wood Johnson Foundation, the AKO Foundation, J. Alexander, M. J. Leder, W. G. Lichtenstein, the Pershing Square Fund for Research on the Foundations of Human Behavior from Harvard University and by Roybal Center grants (P30AG034546 and 5P30AG034532) from the National Institute on Aging. The views expressed here do not necessarily reflect the views of any of these individuals or entities. We thank 24 Hour Fitness for partnering with the Behavior Change for Good Initiative at the University of Pennsylvania to make this research possible. Data availability: The data analysed in this paper were provided by 24 Hour Fitness and we have their legal permission to share the deidentified data. We have therefore made deidentified data available at https://osf.io/9av87/?view_only=8bb9282111c24f81a19c2237e7d7eba3. Furthermore, tables of all of the preregistration links for each of the substudies with the interventions and the prediction studies are available in Supplementary Tables 2 and 30. Code availability: The code to replicate the analyses and figures in the paper and Supplementary Information is available online (https://osf.io/9av87/?view_only=8bb9282111c24f81a19c2237e7d7eba3). Author Contributions: K.L.M., D.G., A.R., M.B., J.B., L.B., E.C., G.C., R.C., H.D., L.E.-W., A.F., J.J.G., S.H., A.H., S.J.J., D.K., E.K., J.K., A.K., G.L., B.M., S.M., S.S., G.S., J.H.T., J.T., Y.T., L.U., K.G.V., A.W., J.Z. and A.L.D. designed the research. K.L.M., D.G., J.S.K., P.P., Y.P., A.L.D. and A.R. performed the research. H.H., T.W.L., P.P. and Y.P. analysed the data. K.L.M. and A.L.D wrote the paper. D.G., H.H., J.S.K., T.W.L., P.P., Y.P., A.R., M.B., J.B., C.C., G.C., H.D., A.F., J.J.G., D.K., T.K., E.K., J.K., R.L., J.L., B.M., S.M., S.S., J.S., A.W. and J.Z. provided feedback on the paper. K.L.M., D.G., J.S.K., T.K., R.L. and S.M. supervised data analysis. K.L.M., D.G., H.H., J.S.K. and T.W.L. prepared the Supplementary Information. The authors declare no competing interests. The authors did not receive commercial benefits from the fitness chain or speaking/consulting fees related to any of the interventions presented here. Peer review information: Nature thanks Charles Shearer and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.
Funders:
Funding AgencyGrant Number
Robert Wood Johnson FoundationUNSPECIFIED
AKO FoundationUNSPECIFIED
J. AlexanderUNSPECIFIED
M. J. LederUNSPECIFIED
W. G. LichtensteinUNSPECIFIED
Harvard UniversityUNSPECIFIED
Roybal CenterUNSPECIFIED
NIHP30AG034546
NIH5P30AG034532
Subject Keywords:Decision making; Human behaviour
Issue or Number:7889
DOI:10.1038/s41586-021-04128-4
Record Number:CaltechAUTHORS:20211216-65471400
Persistent URL:https://resolver.caltech.edu/CaltechAUTHORS:20211216-65471400
Official Citation:Milkman, K.L., Gromet, D., Ho, H. et al. Megastudies improve the impact of applied behavioural science. Nature 600, 478–483 (2021). https://doi.org/10.1038/s41586-021-04128-4
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:112488
Collection:CaltechAUTHORS
Deposited By: Tony Diaz
Deposited On:17 Dec 2021 17:11
Last Modified:25 Jul 2022 23:15

Repository Staff Only: item control page