Variability in the analysis of a single neuroimaging dataset by many teams

Rotem Botvinik-Nezer; Felix Holzmeister; Colin Camerer; Anna Dreber Almenberg; Jürgen Huber; Magnus Johannesson; Michael Kirchler; Roni Iwanir; Jeanette Mumford; R. Alison Adcock; Paolo Avesani; Blazej Baczkowski; Aahana Bajracharya; Leah Bakst; Sheryl Ball; Marco Barilari; Nadège Bault; Derek Beaton; Julia Beitner; Ruud Berkers; Roland Benoit; Jamil Bhanji; Bharat Biswal; Sebastian Bobadilla-Suarez; Tiago Bortolini; Katherine Bottenhorn; Alexander Bowring; Senne Braem; Hayley Brooks; Emily Brudner; Cristian Calderon; Julia Camilleri; Jaime Castrellon; Luca Cecchetti; Edna Cieslik; Zachary Cole; Olivier Collignon; Robert Cox; William Cunningham; Stefan Czoschke; Kamalaker Dadi; Charles Davis; Alberto De Luca; Mauricio Delgado; Lysia Demetriou; Jeffrey Dennison; Xin Di; Erin Dickie; Ekaterina Dobryakova; Claire Donnat; Juergen Dukart; Niall Duncan; Joke Durnez; Amr Eed; Simon Eickhoff; Andrew Erhart; Laura Fontanesi; G. Matthew Fricke; Shiguang Fu; Adriana Galván; Remi Gau; Sarah Genon; Tristan Glatard; Enrico Glerean; Jelle Goeman; Sergej Golowin; Carlos González-García; Krzysztof Gorgolewski; Cheryl Grady; Mikella Green; João Guassi Moreira; Olivia Guest; Shabnam Hakimi; J. Paul Hamilton; Roeland Hancock; Giacomo Handjaras; Bronson Harry; Colin Hawco; Peer Herholz; Gabrielle Herman; Stephan Heunis; Felix Hoffstaedter; Jeremy Hogeveen; Susan Holmes; Chuan-Peng Hu; Scott Huettel; Matthew Hughes; Vittorio Iacovella; Alexandru Iordan; Peder Isager; Ayse Isik; Andrew Jahn; Matthew Johnson; Tom Johnstone; Michael Joseph; Anthony Juliano; Joseph Kable; Michalis Kassinopoulos; Cemal Koba; Xiang-Zhen Kong

doi:10.1038/s41586-020-2314-9

Back

Variability in the analysis of a single neuroimaging dataset by many teams

Journal article

Open access

Peer reviewed

Variability in the analysis of a single neuroimaging dataset by many teams

Rotem Botvinik-Nezer, Felix Holzmeister, Colin Camerer, Anna Dreber Almenberg, Jürgen Huber, Magnus Johannesson, Michael Kirchler, Roni Iwanir, Jeanette Mumford, R. Alison Adcock, …

Nature, Vol.582(7810), pp.84-88

2020-06-04

DOI: https://doi.org/10.1038/s41586-020-2314-9

PMID: 32483374

Abstract

Cognitive Science

Neuroscience

Data analysis workflows in many scientific domains have become increasingly complex and flexible. Here we assess the effect of this flexibility on the results of functional magnetic resonance imaging by asking 70 independent teams to analyse the same dataset, testing the same 9 ex-ante hypotheses1. The flexibility of analytical approaches is exemplified by the fact that no two teams chose identical workflows to analyse the data. This flexibility resulted in sizeable variation in the results of hypothesis tests, even for teams whose statistical maps were highly correlated at intermediate stages of the analysis pipeline. Variation in reported results was related to several aspects of analysis methodology. Notably, a meta-analytical approach that aggregated information across teams yielded a significant consensus in activated regions. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset2-5. Our findings show that analytical flexibility can have substantial effects on scientific conclusions, and identify factors that may be related to variability in the analysis of functional magnetic resonance imaging. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for performing and reporting multiple analyses of the same data. Potential approaches that could be used to mitigate issues related to analytical variability are discussed.

Files and links (1)

url

View repository versionView

Open

Metrics

9 Record Views

542 Times Cited - Web of Science

Details

Title: Variability in the analysis of a single neuroimaging dataset by many teams
Creators: Rotem Botvinik-Nezer - Tel Aviv University
Felix Holzmeister - Universität Innsbruck
Colin Camerer - California Institute of Technology
Anna Dreber Almenberg - Stockholm School of Economics, Department of Economics
Jürgen Huber - University of Innsbruck
Magnus Johannesson - Stockholm School of Economics, Department of Economics
Michael Kirchler - Leopold Franzens Universität Innsbruck - University of Innsbruck
Roni Iwanir - Tel Aviv University
Jeanette Mumford - University of Wisconsin-Madison
R. Alison Adcock - Duke University [Durham]
Paolo Avesani - Fondazione Bruno Kessler [Trento, Italy]
Blazej Baczkowski - Max Planck Society
Aahana Bajracharya - Washington University in Saint Louis
Leah Bakst - Boston University [Boston]
Sheryl Ball - Virginia Tech [Blacksburg]
Marco Barilari - Université Catholique de Louvain = Catholic University of Louvain
Nadège Bault - Plymouth University
Derek Beaton - Rotman Research Institute at the Baycrest Centre
Julia Beitner - University of Amsterdam [Amsterdam]
Ruud Berkers - Max Planck Institute for Human Cognitive and Brain Sciences [Leipzig]
Roland Benoit - Max Planck Society
Jamil Bhanji - Rutgers University [Newark]
Bharat Biswal - New Jersey Institute of Technology [Newark]
Sebastian Bobadilla-Suarez - University College of London [London]
Tiago Bortolini - D'Or Institute for Research and Education [Rio de Janeiro]
Katherine Bottenhorn - Florida International University
Alexander Bowring - University of Oxford
Senne Braem - Vrije Universiteit Brussel
Hayley Brooks - University of Denver
Emily Brudner - Rutgers, The State University of New Jersey
Cristian Calderon - Ghent University
Julia Camilleri - Heinrich Heine University Düsseldorf
Jaime Castrellon - Duke University
Luca Cecchetti - School for Advanced Studies Lucca
Edna Cieslik - Heinrich Heine University Düsseldorf
Zachary Cole - University of Nebraska–Lincoln
Olivier Collignon - UCLouvain
Robert Cox - National Institutes of Health
William Cunningham - University of Toronto
Stefan Czoschke - Goethe-University Frankfurt am Main
Kamalaker Dadi - Modelling brain structure, function and variability based on high-field MRI data
Charles Davis - University of Connecticut
Alberto De Luca - University Medical Center [Utrecht]
Mauricio Delgado - Rutgers, The State University of New Jersey
Lysia Demetriou - University of Oxford
Jeffrey Dennison - Temple University
Xin Di - New Jersey Institute of Technology
Erin Dickie - University of Toronto
Ekaterina Dobryakova - Kessler Foundation [East Hanover]
Claire Donnat - Stanford University
Juergen Dukart - Heinrich Heine University Düsseldorf
Niall Duncan - Taipei Medical University
Joke Durnez - Stanford University
Amr Eed - Instituto de Neurociencias de Alicante
Simon Eickhoff - Heinrich Heine University Düsseldorf
Andrew Erhart - University of Denver
Laura Fontanesi - University of Basel
G. Matthew Fricke - The University of New Mexico [Albuquerque]
Shiguang Fu - Zhejiang University of Technology
Adriana Galván - University of California [Los Angeles]
Remi Gau - UCLouvain
Sarah Genon - Heinrich Heine University Düsseldorf
Tristan Glatard - Concordia University [Montreal]
Enrico Glerean - Aalto University
Jelle Goeman - Leiden University Medical Center (LUMC)
Sergej Golowin - Taipei Medical University
Carlos González-García - Ghent University
Krzysztof Gorgolewski - Stanford University
Cheryl Grady - University of Toronto
Mikella Green - Duke University
João Guassi Moreira - University of California [Los Angeles]
Olivia Guest - University College of London [London]
Shabnam Hakimi - Duke University
J. Paul Hamilton - Linköping University
Roeland Hancock - University of Connecticut
Giacomo Handjaras - School for Advanced Studies Lucca
Bronson Harry - University of Sydney
Colin Hawco - Centre for Addiction and Mental Health [Toronto]
Peer Herholz - McGill University = Université McGill [Montréal, Canada]
Gabrielle Herman - University of Toronto
Stephan Heunis - Eindhoven University of Technology [Eindhoven]
Felix Hoffstaedter - Jülich Research Centre
Jeremy Hogeveen - University of New Mexico
Susan Holmes - Stanford University
Chuan-Peng Hu - Leibniz-Institut für Resilienzforschung [Mainz]
Scott Huettel - Duke University
Matthew Hughes - Swinburne University of Technology (Hawthorn campus)
Vittorio Iacovella - University of Trento
Alexandru Iordan - University of Michigan [Ann Arbor]
Peder Isager - Eindhoven University of Technology
Ayse Isik - Max Planck Society
Andrew Jahn - University of Michigan–Ann Arbor
Matthew Johnson - University of Nebraska–Lincoln
Tom Johnstone - Swinburne University of Technology
Michael Joseph - University of Toronto
Anthony Juliano - Kessler Foundation
Joseph Kable - University of Pennsylvania [Philadelphia]
Michalis Kassinopoulos - McGill University
Cemal Koba - IMT School for Advanced Studies Lucca
Xiang-Zhen Kong - Max Planck Institute for Psycholinguistics
Publication Details: Nature, Vol.582(7810), pp.84-88
Publisher: Nature Publishing Group
Academic Unit: Department of Economics
Language: English
Resource Type: Journal article

Variability in the analysis of a single neuroimaging dataset by many teams

Abstract

Files and links (1)

Metrics

Details

Stockholm School of Economics Social media