Consensus-based guidance for conducting and reporting multi-analyst studies

Balázs Aczél; Barnabás Szászi; Gustav Nilsonne; Olmo van den Akker; Casper Albers; Marcel A.L.M. Van Assen; Jojanneke Bastiaansen; Daniel J. Benjamin; Udo Boehm; Rotem Botvinik-Nezer; Laura Bringmann; Niko A Busch; Emmanuel Caruyer; Andrea M. Cataldo; Nelson Cowan; Andrew Delios; Noah van Dongen; Chris Donkin; Johnny Van Doorn; Anna Dreber Almenberg; Gilles Dutilh; Gary F. Egan; Ann Gernsbacher Morton; Rink Hoekstra; Sabine Hoffmann; Felix Holzmeister; Jürgen Huber; Magnus Johannesson; Kai J. Jonas; Alexander Kindel; Michael Kirchler; Yoram K. Kunkels; D. Stephen Lindsay; Jean-Francois Mangin; Dora Matzke; Marcus R. Munafo; Ben R. Newell; Brian A. Nosek; Russell Alan Poldrack; Don van Ravenzwaaij; Jorg Rieskamp; Matthew J. Salganik; Alexandra Sarafoglou; Tom Schonberg; Martin Schweinsberg; David Shanks; Raphael Silberzahn; Daniel J. Simons; Barbara A Spellman; Samuel St-Jean; Jeffrey J. Starns; Eric Luis Uhlmann; Eric Jan M. Wagenmakers; Jelte Wicherts

doi:10.7554/eLife.72185

Back

Consensus-based guidance for conducting and reporting multi-analyst studies

Journal article

Open access

Peer reviewed

Consensus-based guidance for conducting and reporting multi-analyst studies

Balázs Aczél, Barnabás Szászi, Gustav Nilsonne, Olmo van den Akker, Casper Albers, Marcel A.L.M. Van Assen, Jojanneke Bastiaansen, Daniel J. Benjamin, Udo Boehm, Rotem Botvinik-Nezer, …

eLIFE, Vol.10, e72185

2021

DOI: https://doi.org/10.7554/eLife.72185

Abstract

multi-analyst

metascience

statistical practice

science forum

expert consensus

analytical variability

None

Any large dataset can be analyzed in a number of ways, and it is possible that the use of different analysis strategies will lead to different results and conclusions. One way to assess whether the results obtained depend on the analysis strategy chosen is to employ multiple analysts and leave each of them free to follow their own approach. Here, we present consensus-based guidance for conducting and reporting such multi-analyst studies, and we discuss how broader adoption of the multi-analyst approach has the potential to strengthen the robustness of results and conclusions obtained from analyses of datasets in basic and applied research.

Files and links (2)

pdf

2021_JournalArticle_DreberAnna_ConsensusBased467.78 kBDownload View

Published (Version of record)CC BY V4.0, Open Access

url

https://doi.org/10.7554/eLife.72185View

Published (Version of record)CC BY V4.0, Open

Metrics

8 Record Views

28 Times Cited - Web of Science

Details

Title: Consensus-based guidance for conducting and reporting multi-analyst studies
Creators: Balázs Aczél - Eötvös Loránd University
Barnabás Szászi - Eötvös Loránd University
Gustav Nilsonne - Karolinska Institutet (SE)
Olmo van den Akker - Tilburg University (NL)
Casper Albers - University of Groningen (NL)
Marcel A.L.M. Van Assen - Tilburg University (NL)
Jojanneke Bastiaansen - University of Groningen (NL)
Daniel J. Benjamin - National Bureau of Economic Research (US)
Udo Boehm - University of Amsterdam (NL)
Rotem Botvinik-Nezer - Dartmouth College
Laura Bringmann - University of Groningen (NL)
Niko A Busch - University of Münster
Emmanuel Caruyer - University of Rennes
Andrea M. Cataldo - Harvard Medical School
Nelson Cowan - University of Missouri System
Andrew Delios - National University of Singapore
Noah van Dongen - University of Amsterdam (NL)
Chris Donkin - University of New South Wales (AU)
Johnny Van Doorn - University of Amsterdam
Anna Dreber Almenberg - Stockholm School of Economics, Department of Economics
Gilles Dutilh - University Hospital of Basel
Gary F. Egan - Monash University
Ann Gernsbacher Morton - University of Wisconsin–Madison
Rink Hoekstra - University of Groningen (NL)
Sabine Hoffmann - Ludwig Maximilians University München (DE)
Felix Holzmeister - University of Innsbruck (AT)
Jürgen Huber - University of Innsbruck
Magnus Johannesson - Stockholm School of Economics, Department of Economics
Kai J. Jonas - Maastricht University
Alexander Kindel - Princeton University (US)
Michael Kirchler - University of Innsbruck (AT)
Yoram K. Kunkels - University of Groningen (NL)
D. Stephen Lindsay - University of Victoria
Jean-Francois Mangin - Paris-Saclay University
Dora Matzke - University of Amsterdam (NL)
Marcus R. Munafo - Bristol University (GB)
Ben R. Newell - University of New South Wales (AU)
Brian A. Nosek - University of Virginia (US)
Russell Alan Poldrack - Stanford University (US)
Don van Ravenzwaaij - University of Groningen (NL)
Jorg Rieskamp - University of Basel
Matthew J. Salganik - Princeton University (US)
Alexandra Sarafoglou - University of Amsterdam (NL)
Tom Schonberg - Tel Aviv University (IL)
Martin Schweinsberg - European School of Management and Technology
David Shanks - University College London (GB)
Raphael Silberzahn - University of Sussex (GB)
Daniel J. Simons - University of Illinois (US)
Barbara A Spellman - University of Virginia (US)
Samuel St-Jean - University of Alberta (CA)
Jeffrey J. Starns - University of Massachusetts
Eric Luis Uhlmann - INSEAD (FR)
Eric Jan M. Wagenmakers - University of Amsterdam (NL)
Jelte Wicherts - Tilburg University (NL)
Publication Details: eLIFE, Vol.10, e72185
Publisher: eLife Sciences Publications
Copyright: Copyright Aczel et al. This article is distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use and redistribution provided that the original author and source are credited.
Academic Unit: Department of Economics
Language: English
Resource Type: Journal article

Consensus-based guidance for conducting and reporting multi-analyst studies

Abstract

Files and links (2)

Metrics

Details

Stockholm School of Economics Social media