CaltechAUTHORS
  A Caltech Library Service

Variability in the analysis of a single neuroimaging dataset by many teams

Botvinik-Nezer, Rotem and Camerer, Colin F. (2020) Variability in the analysis of a single neuroimaging dataset by many teams. Nature, 582 (7810). pp. 84-88. ISSN 0028-0836. https://resolver.caltech.edu/CaltechAUTHORS:20191118-092814513

[img] PDF - Submitted Version
Creative Commons Attribution Non-commercial No Derivatives.

1267Kb
[img] PDF - Supplemental Material
See Usage Policy.

667Kb
[img] PDF (Reporting Summary) - Supplemental Material
See Usage Policy.

119Kb
[img] MS Excel (Source Data Fig. 1) - Supplemental Material
See Usage Policy.

1336b
[img] MS Word (Source Data Fig. 2) - Supplemental Material
See Usage Policy.

77Kb
[img] Image (JPEG) (Extended Data Fig. 1: Voxels overlap) - Supplemental Material
See Usage Policy.

250Kb
[img] Image (JPEG) (Extended Data Fig. 2: Variability of whole-brain unthresholded maps for hypotheses 2 and 4–9) - Supplemental Material
See Usage Policy.

631Kb
[img] Image (JPEG) (Extended Data Fig. 3: Variability and consensus of unthresholded statistical maps) - Supplemental Material
See Usage Policy.

545Kb
[img] Image (JPEG) (Extended Data Fig. 4: Results of the consistent thresholding and ROI selection analysis) - Supplemental Material
See Usage Policy.

114Kb
[img] Image (JPEG) (Extended Data Fig. 5: Prediction markets over time) - Supplemental Material
See Usage Policy.

268Kb
[img] Image (JPEG) (Extended Data Table 1: Hypotheses and results) - Supplemental Material
See Usage Policy.

58Kb
[img] Image (JPEG) (Extended Data Table 2: Results submitted by analysis teams) - Supplemental Material
See Usage Policy.

214Kb
[img] Image (JPEG) (Extended Data Table 3: Data links and analysis-related tables) - Supplemental Material
See Usage Policy.

119Kb
[img] Image (JPEG) (Extended Data Table 4: Variability of statistical maps across teams) - Supplemental Material
See Usage Policy.

45Kb
[img] Image (JPEG) (Extended Data Table 5: Results of prediction markets and additional data) - Supplemental Material
See Usage Policy.

93Kb

Use this Persistent URL to link to this item: https://resolver.caltech.edu/CaltechAUTHORS:20191118-092814513

Abstract

Data analysis workflows in many scientific domains have become increasingly complex and flexible. Here we assess the effect of this flexibility on the results of functional magnetic resonance imaging by asking 70 independent teams to analyse the same dataset, testing the same 9 ex-ante hypotheses. The flexibility of analytical approaches is exemplified by the fact that no two teams chose identical workflows to analyse the data. This flexibility resulted in sizeable variation in the results of hypothesis tests, even for teams whose statistical maps were highly correlated at intermediate stages of the analysis pipeline. Variation in reported results was related to several aspects of analysis methodology. Notably, a meta-analytical approach that aggregated information across teams yielded a significant consensus in activated regions. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset. Our findings show that analytical flexibility can have substantial effects on scientific conclusions, and identify factors that may be related to variability in the analysis of functional magnetic resonance imaging. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for performing and reporting multiple analyses of the same data. Potential approaches that could be used to mitigate issues related to analytical variability are discussed.


Item Type:Article
Related URLs:
URLURL TypeDescription
https://doi.org/10.1038/s41586-020-2314-9DOIArticle
https://rdcu.be/b4CQwPublisherFree ReadCube access
https://doi.org/10.1101/843193DOIDiscussion Paper
ORCID:
AuthorORCID
Botvinik-Nezer, Rotem0000-0003-2669-1877
Camerer, Colin F.0000-0003-4049-1871
Additional Information:© 2020 The Author(s), under exclusive licence to Springer Nature Limited. Received 14 November 2019; Accepted 07 April 2020; Published 20 May 2020. Neuroimaging data collection, performed at Tel Aviv University, was supported by the Austrian Science Fund (P29362-G27), the Israel Science Foundation (ISF 2004/15 to T. Schonberg) and the Swedish Foundation for Humanities and Social Sciences (NHS14-1719:1). Hosting of the data on OpenNeuro was supported by a National Institutes of Health (NIH) grant (R24MH117179). We thank M. C. Frank, Y. Assaf and N. Daw for comments on an earlier draft; the Texas Advanced Computing Center for providing computing resources for preprocessing of the data; the Stanford Research Computing Facility for hosting the data; and D. Roll for assisting with data processing. T. Schonberg thanks The Alfredo Federico Strauss Center for Computational Neuroimaging at Tel Aviv University; A.D. thanks the Knut and Alice Wallenberg Foundation and the Marianne and Marcus Wallenberg Foundation (A.D. is a Wallenberg Scholar), the Austrian Science Fund (FWF, SFB F63) and the Jan Wallander and Tom Hedelius Foundation (Svenska Handelsbankens Forskningsstiftelser); F. Holzmeister, J. Huber and M. Kirchler thank the Austrian Science Fund (FWF, SFB F63); D.W. was supported by the Research Foundation Flanders (FWO) and the European Union’s Horizon 2020 research and innovation programme (https://ec.europa.eu/programmes/horizon2020/en) under the Marie Skłodowska-Curie grant agreement no. 665501; L. Tisdall was supported by the University of Basel Research Fund for Junior Researchers; C.B.C. was supported by grant 12O7719N from the Research Foundation Flanders; E.L. was supported by grant 12T2517N from the Research Foundation Flanders and Marie Skłodowska-Curie Actions under COFUND grant agreement 665501; A. Eed was supported by a predoctoral fellowship La Caixa-Severo Ochoa from Obra Social La Caixa and also acknowledges Comunidad de Cálculo Científico del CSIC for the high-performance computing (HPC) use; C.L. was supported by the Vienna Science and Technology Fund (WWTF VRG13-007) and Austrian Science Fund (FWF P 32686); A.B.L.V. was supported by the Vienna Science and Technology Fund (WWTF VRG13-007); L.Z. was supported by the Vienna Science and Technology Fund (WWTF VRG13-007), the National Natural Science Foundation of China (no. 71801110), MOE (Ministry of Education in China) Project of Humanities and Social Sciences (no. 18YJC630268) and China Postdoctoral Science Foundation (no. 2018M633270); D.P. is currently supported by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany’s Excellence Strategy ‘Science of Intelligence’ (EXC 2002/1, project number 390523135); P.H. was supported in part by funding provided by Brain Canada, in partnership with Health Canada, for the Canadian Open Neuroscience Platform initiative; J.-B.P. was partially funded by the NIH (NIH-NIBIB P41 EB019936 (ReproNim), NIH-NIMH R01 MH083320 (CANDIShare) and NIH RF1 MH120021 (NIDM)) and the National Institute Of Mental Health of the NIH under award number R01MH096906 (Neurosynth), as well as the Canada First Research Excellence Fund, awarded to McGill University for the Healthy Brains for Healthy Lives initiative and the Brain Canada Foundation with support from Health Canada; S.B.E. was supported by the European Union’s Horizon 2020 Research and Innovation Programme under grant agreement no. 785907 (HBP SGA2); G.M. was supported by the Max Planck Society; S. Heunis has received funding from the Dutch foundation LSH-TKI (grant LSHM16053-SGF); J.F.G.M. was supported by a Graduate Research Fellowship from the NSF and T32 Predoctoral Fellowship from the NIH; B.M. was supported by the Deutsche Forschungsgemeinschaft (grant CRC1193, subproject B01); A.R.L. was supported by NSF 1631325 and NIH R01 DA041353; M.E.H., T.J. and D.J.W. were supported by the Australian National Imaging Facility, a National Collaborative Research Infrastructure Strategy (NCRIS) capability; P.M.I. was supported by VIDI grant 452-17-013 from the Netherlands Organisation for Scientific Research; B.M.B. was supported by the Max Planck Society; J.P.H. was supported by a grant from the Swedish Research Council; R.W.C. and R.C.R. were supported by NIH IRP project number ZICMH002888; D.M.N., R.W.C., and R.C.R. used the computational resources of the National Institutes of Health High Performance Computing Biowulf cluster (http://hpc.nih.gov); D.M.N. was supported by NIH IRP project number ZICMH002960; C.F.C. was supported by the Tianqiao and Chrissy Center for Social and Decision Neuroscience Center Leadership Chair; R.G.B. was supported by the Max Planck Society; R.M.W.J.B. was supported by the Max Planck Society; M.B., O.C. and R.G. were supported by the Belgian Excellence of Science program (EOS project 30991544) from the FNRS-Belgium; O.C. is a research associate at the FRS-FNRS of Belgium; A.D.L. was supported by grant R4195 “Repimpact” of EraNET Neuron; Q.S. was funded by grant no. 71971199,71602175 and 71942004 from the National Natural Science Foundation of China and no. 16YJC630103 of the Ministry of Education of Humanities and Social Science; and T.E.N. was supported by the Wellcome Trust award 100309/Z/12/Z. Data availability: The full fMRI dataset is publicly available on OpenNeuro (https://doi.org/10.18112/openneuro.ds001734.v1.0.4) and is described in detail in a Data Descriptor1. The results reported by all teams are presented in Extended Data Table 2. A table describing the methods used by the analysis teams is available with the analysis code. NeuroVault collections containing the submitted statistical maps are available via the links provided in Extended Data Table 3a. Source data for Figs. 1, 2 are provided with the paper. Readers may obtain access to the data and run the full analysis stream on the team submissions by following the directions at https://github.com/poldrack/narps/tree/master/ImageAnalyses. Access to the raw data requires specifying a URL for the dataset, which is: https://zenodo.org/record/3528329/files/narps_origdata_1.0.tgz. Results (automatically generated figures, results and output logs) for image analyses are available for anonymous download at https://doi.org/10.5281/zenodo.3709275. Code availability: Code for all analyses of the reports and statistical maps submitted by the analysis teams is openly shared in GitHub (https://github.com/poldrack/narps). Image-analysis code was implemented within a Docker container, with software versions pinned for reproducible execution (https://hub.docker.com/r/poldrack/narps-analysis/tags). Python code was automatically tested for quality using the flake8 static analysis tool and the codacy.com code quality assessment tool, and the results of the image-analysis workflow were validated using simulated data. The image-analysis code was independently reviewed by an expert who was not involved in writing the original code. Prediction market analyses were performed using R v.3.6.1; packages were installed using the checkpoint package, which reproducibly installs all package versions as of a specified date (13 August 2019). Analyses reported in this manuscript were performed using code release v.2.0.3 (https://doi.org/10.5281/zenodo.3709273). Although not required to, several analysis teams publicly shared their analysis code. Extended Data Table 3d includes these teams along with the link to their code. Author Contributions: NARPS management team: R.B.-N., F. Holzmeister, C.F.C., A.D., J. Huber, M.J., M. Kirchler, R.A.P. and T. Schonberg. fMRI dataset (experiment design): R.I., J. Durnez, R.A.P. and T. Schonberg. fMRI dataset (data collection): R.I. and T. Schonberg. fMRI dataset (preprocessing, quality assurance and data sharing): R.B.-N., K.J.G., R.A.P. and T. Schonberg. Analysis teams (recruitment, point of contact and management): R.B.-N., R.A.P. and T. Schonberg. Analysis teams (analysis of the submitted results and statistical maps): R.A.P., T.E.N., J.A.M., J.-B.P., A.P., R.B.-N. and T. Schonberg. Code review: T.G. and K.D. Prediction markets (design and management): F. Holzmeister, C.F.C., A.D., J. Huber, M.J. and M. Kirchler. Prediction markets (analysis): F. Holzmeister, R.B.-N., C.F.C., A.D., J. Huber, M.J., M. Kirchler, S.K., R.A.P. and T. Schonberg. Writing the manuscript: R.B.-N., F. Holzmeister, A.D., J. Huber, M.J., M. Kirchler, T.E.N., R.A.P. and T. Schonberg. Participated as members of analysis teams and reviewed and edited the manuscript: R.A.A., P.A., B.M.B., A. Bajracharya, L.B., S. Ball, M.B., N.B., D.B., J.B., R.G.B., R.M.W.J.B., J.P.B., B.B.B., S.B.-S., T.B., K.L.B., A. Bowring, S. Braem, H.R.B., E.G.B., C.B.C., J.A.C., J.J.C., L.C., E.C.C., Z.J.C., O.C., R.W.C., W.A.C., S.C., K.D., C.P.D., A.D.L., M.R.D., L.D., J.B.D., X.D., E.W.D., E.D., C.L.D., J. Dukart, N.W.D., A. Eed, S.B.E., A. Erhart, L.F., G.M.F., S.F., A.G., R.G., S.G., E.G., J.J.G., S.A.E.G., C.G.-G., K.J.G., C.L.G., M.A.G., J.F.G.M., O.G., S. Hakimi, J.P.H., R.H., G. Handjaras, B.B.H., C.H., P.H., G. Herman, S. Heunis, F. Hoffstaedter, J. Hogeveen, S. Holmes, C.-P.H., S.A.H., M.E.H., V.I., A.D.I., P.M.I., A.I.I., A.J., M.R.J., T.J., M.J.E.J., A.C.J., J.W.K., M. Kassinopoulos, C.K., X.-Z.K., T.R.K., N.E.K., B.A.K., A.R.L., C.L., R.L., N.L., H.L., S.L., A. Leemans, A. Leo, E.L., F.L., M.Y.C.L., P.C.L., E.N.L., S.W.L., A.B.L.V., B.C.L., M.L.M., N.M., T.M., C.M., K.M., J.T.M., H.M., A.S.M.L., B.M., K.N.M., G.M., G.D.M., J.M., T.E.N., D.M.N., G.N., M.P.N., E.O., A.I.O., P.P., K.R.P., J.E.P., D.P., Y.P., S.R., P.A.R.-L., R.C.R., E.R., J.R.R., A.M.R.-T., A.R., T. Salo, G.R.S.-L., E.S.-M., M.L.S., D.H.S., Q.S., M.A.S., J.A.S., K.S., A.S., D.V.S., P.S.-H., S.R.S., S.M.T., B.T., J.N.T., G.T., L. Tisdall, S.H.T., C.T.-S., J.J.T.T., L. Tozzi, V.T., L. Turella, A.E.v.V., T.V., J.M.V., S.V., K.V., M.B.W., W.D.W., S.W., D.J.W., D.W., A.X.-P., E.A.Y., S.Y., R.Y., K.S.L.Y., L.Z., X.Z. and J.E.Z. The authors declare no competing interests.
Group:Tianqiao and Chrissy Chen Institute for Neuroscience
Funders:
Funding AgencyGrant Number
FWF Der WissenschaftsfondsP29362-G27
Israel Science Foundation2004/15
Swedish Foundation for Humanities and Social SciencesNHS14-1719:1
NIHR24MH117179
Knut and Alice Wallenberg FoundationUNSPECIFIED
Marianne and Marcus Wallenberg FoundationUNSPECIFIED
FWF Der WissenschaftsfondsSFB F63
Jan Wallander and Tom Hedelius FoundationUNSPECIFIED
Marie Curie Fellowship665501
University of BaselUNSPECIFIED
Fonds Wetenschappelijk Onderzoek (FWO)12O7719N
Fonds Wetenschappelijk Onderzoek (FWO)12T2517N
Obra Social La CaixaUNSPECIFIED
Viennese Science and Technology FundWWTF VRG13-007
FWF Der WissenschaftsfondsP 32686
National Natural Science Foundation of China71801110
Ministry of Education (Taipei)18YJC630268
China Postdoctoral Science Foundation2018M633270
Deutsche Forschungsgemeinschaft (DFG)EXC 2002/1
Deutsche Forschungsgemeinschaft (DFG)390523135
Health CanadaUNSPECIFIED
NIHP41 EB019936
NIHR01 MH083320
NIHRF1 MH120021
NIHR01MH096906
Canada First Research Excellence FundUNSPECIFIED
Brain Canada FoundationUNSPECIFIED
European Research Council (ERC)785907
Max Planck SocietyUNSPECIFIED
Dutch FoundationLSHM16053-SGF
NSF Graduate Research FellowshipUNSPECIFIED
NIH Predoctoral FellowshipUNSPECIFIED
Deutsche Forschungsgemeinschaft (DFG)CRC1193
NSFIIS-1631325
NIHR01 DA041353
Australian National Imaging FacilityUNSPECIFIED
Nederlandse Organisatie voor Wetenschappelijk Onderzoek (NWO)452-17-013
Swedish Research CouncilUNSPECIFIED
NIHZICMH002888
NIHZICMH002960
Tianqiao and Chrissy Chen Institute for NeuroscienceUNSPECIFIED
Belgian Excellence of Science30991544
Fonds de la Recherche Scientifique (FNRS)UNSPECIFIED
EraNET NeuronR4195
National Natural Science Foundation of China71971199
National Natural Science Foundation of China71602175
National Natural Science Foundation of China71942004
Ministry of Education (China)16YJC630103
Wellcome Trust100309/Z/12/Z
Subject Keywords:Decision; Decision making; Human behaviour; Scientific community
Issue or Number:7810
Record Number:CaltechAUTHORS:20191118-092814513
Persistent URL:https://resolver.caltech.edu/CaltechAUTHORS:20191118-092814513
Official Citation:Botvinik-Nezer, R., Holzmeister, F., Camerer, C.F. et al. Variability in the analysis of a single neuroimaging dataset by many teams. Nature 582, 84–88 (2020). https://doi.org/10.1038/s41586-020-2314-9
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:99892
Collection:CaltechAUTHORS
Deposited By: Tony Diaz
Deposited On:18 Nov 2019 17:54
Last Modified:04 Jun 2020 18:26

Repository Staff Only: item control page