Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results

R. Silberzahn; E. L. Uhlmann; D. P. Martin; P. Anselmi; F. Aust; E. Awtrey; S. Bahnik; F. Bai; C. Bannard; E. Bonnier; R. Carlsson; F. Cheung; G. Christensen; R. Clay; M. A. Craig; A. Dalla Rosa; L. Dam; M. H. Evans; I. Flores Cervantes; N. Fong; M. Gamez-Djokic; A. Glenz; S. Gordon-McKeon; T. J. Heaton; K. Hederos; M. Heene; A. J. Hofelich Mohr; F. Hogden; K. Hui; M. Johannesson; J. Kalodimos; E. Kaszubowski; D. M. Kennedy; R. Lei; T. A. Lindsay; S. Liverani; C. R. Madan; D. Molden; E. Molleman; R. D. Morey; L. B. Mulder; B. R. Nijstad; N. G. Pope; B. Pope; J. M. Prenoveau; F. Rink; E. Robusto; H. Roderique; A. Sandberg; E. Schlueter; F. D. Schoenbrodt; M. F. Sherman; S. A. Sommer; K. Sotak; S. Spain; C. Spoerlein; T. Stafford; L. Stefanutti; S. Tauber; J. Ullrich; M. Vianello; E-J Wagenmakers; M. Witkowiak; S. Yoon; B. A. Nosek

doi:10.1177/2515245917747646

Back

Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results

Journal article

Open access

Peer reviewed

Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results

R. Silberzahn, E. L. Uhlmann, D. P. Martin, P. Anselmi, F. Aust, E. Awtrey, S. Bahnik, F. Bai, C. Bannard, E. Bonnier, …

Advances in Methods and Practices in Psychological Science, Vol.1(3), pp.337-356

2018-09

DOI: https://doi.org/10.1177/2515245917747646

Abstract

Crowdsourcing science

Open data

Open materials

Scientific transparency

Data Analysis

Twenty-nine teams involving 61 analysts used the same data set to address the same research question: whether soccer referees are more likely to give red cards to dark-skin-toned players than to light-skin-toned players. Analytic approaches varied widely across the teams, and the estimated effect sizes ranged from 0.89 to 2.93 (Mdn = 1.31) in odds-ratio units. Twenty teams (69%) found a statistically significant positive effect, and 9 teams (31%) did not observe a significant relationship. Overall, the 29 different analyses used 21 unique combinations of covariates. Neither analysts' prior beliefs about the effect of interest nor their level of expertise readily explained the variation in the outcomes of the analyses. Peer ratings of the quality of the analyses also did not account for the variability. These findings suggest that significant variation in the results of analyses of complex data may be difficult to avoid, even by experts with honest intentions. Crowdsourcing data analysis, a strategy in which numerous research teams are recruited to simultaneously investigate the same research question, makes transparent how defensible, yet subjective, analytic choices influence research results.

Files and links (2)

pdf

2018_JournalArticle_JohannessonMagnus_ManyAnalysts247.53 kBDownload View

Published (Version of record)CC BY-NC V4.0, Open Access

url

https://doi.org/10.1177/2515245917747646View

Published (Version of record)CC BY-NC V4.0, Open

Metrics

2 Record Views

320 Times Cited - Web of Science

Details

Title: Many Analysts, One Data Set
Creators - without role: R. Silberzahn - Univ Sussex Business Sch, Org Behav, Brighton, E Sussex, England
E. L. Uhlmann - INSEAD Asia Campus, Org Behav Area, Singapore, Singapore
D. P. Martin - Univ Virginia, Dept Psychol, 918 Emmet St N, Charlottesville, VA 22903 USA
P. Anselmi - University of Padua
F. Aust - University of Cologne
E. Awtrey - Univ Cincinnati, Dept Management, Cincinnati, OH 45221 USA
S. Bahnik - Univ Econ, Fac Business Adm, Dept Management, Prague, Czech Republic
F. Bai - Hong Kong Polytechnic University
C. Bannard - University of Liverpool
E. Bonnier - Stockholm Sch Econ, Dept Econ, Stockholm, Sweden
R. Carlsson - Linnaeus University
F. Cheung - Univ Hong Kong, Sch Publ Hlth, Hong Kong, Peoples R China
G. Christensen - University of California, Berkeley
R. Clay - CUNY Coll Staten Isl, Dept Psychol, Staten Isl, NY 10314 USA
M. A. Craig - New York University
A. Dalla Rosa - Univ Padua, Dept Philosophy Sociol Educ & Appl Psychol, Padua, Italy
L. Dam - University of Groningen
M. H. Evans - University of Manchester
I. Flores Cervantes - Westat
N. Fong - Temple University
M. Gamez-Djokic - Northwestern Univ, Kellogg Sch Management, Dept Management & Org, Evanston, IL 60208 USA
A. Glenz - University of Zurich
S. Gordon-McKeon
T. J. Heaton - University of Sheffield
K. Hederos - Stockholm University
M. Heene - Ludwig-Maximilians-Universität München
A. J. Hofelich Mohr - University of Minnesota
F. Hogden - University of Cologne
K. Hui - Xiamen Univ, Sch Management, Xiamen, Peoples R China
M. Johannesson - Stockholm School of Economics, Department of Economics
J. Kalodimos - Oregon State University
E. Kaszubowski - Univ Fed Santa Catarina, Dept Psychol, Florianopolis, SC, Brazil
D. M. Kennedy - Univ Washington Bothell, Sch Business, Bothell, WA USA
R. Lei - New York University
T. A. Lindsay - University of Minnesota
S. Liverani - Queen Mary Univ London, Sch Math Sci, London, England
C. R. Madan - Univ Nottingham, Sch Psychol, Nottingham, England
D. Molden - Northwestern Univ, Dept Psychol, Evanston, IL 60208 USA
E. Molleman - University of Groningen
R. D. Morey - Cardiff University
L. B. Mulder - University of Groningen
B. R. Nijstad - University of Groningen
N. G. Pope - University of Maryland, College Park
B. Pope - Brigham Young University
J. M. Prenoveau - Loyola Univ Maryland, Dept Psychol, Baltimore, MD USA
F. Rink - University of Groningen
E. Robusto - University of Padua
H. Roderique - University of Toronto
A. Sandberg - Stockholm Univ, Swedish Inst Social Res SOFI, Stockholm, Sweden
E. Schlueter - Justus Liebig Univ, Inst Sociol, Dept Social Sci & Cultural Studies, Giessen, Germany
F. D. Schoenbrodt - Ludwig Maximilian University of Munich
M. F. Sherman - Loyola University Maryland
S. A. Sommer - US Mil Acad West Point, West Point, NY USA
K. Sotak - State University of New York at Oswego
S. Spain - Concordia University
C. Spoerlein - University of Bamberg
T. Stafford - University of Sheffield
L. Stefanutti - University of Padua
S. Tauber - University of Groningen
J. Ullrich - University of Zurich
M. Vianello - Univ Padua, Dept Philosophy Sociol Educ & Appl Psychol, Padua, Italy
E-J Wagenmakers - University of Amsterdam
M. Witkowiak
S. Yoon - Temple Univ, Dept Mkt & Supply Chain Management, Philadelphia, PA 19122 USA
B. A. Nosek - Univ Virginia, Dept Psychol, 918 Emmet St N, Charlottesville, VA 22903 USA
Publication Details: Advances in Methods and Practices in Psychological Science, Vol.1(3), pp.337-356
Publisher: Sage
Number of pages: 20
Grant note: 283876 / European Research Council; European Research Council (ERC); European Commission P2015-0001:1 / Jan Wallander and Tom Hedelius Foundation R305B090002 / Institute of Education Sciences, U.S. Department of Education; US Department of Education ECF-2011-576; RPG2013-326 / Leverhulme Trust NHS14-1719:1 / Swedish Foundation for Humanities and Social Sciences CGSD2-426287-2012 / Natural Sciences and Engineering Research Council of Canada; Natural Sciences and Engineering Research Council of Canada (NSERC); CGIAR
Identifiers: 991001489599406056
Copyright: © The Author(s) 2018. This article is distributed under the terms of the Creative Commons Attribution-NonCommercial 4.0 License (http://www.creativecommons.org/licenses/by-nc/4.0/) which permits non-commercial use, reproduction and distribution of the work without further permission provided the original work is attributed as specified on the SAGE and Open Access pages (https://us.sagepub.com/en-us/nam/open-access-at-sage).
Academic Unit: Department of Economics
Language: English
Resource Type: Journal article

Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results

Abstract

Files and links (2)

Metrics

Details

Stockholm School of Economics Social media