Same data, different conclusions: Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis

Martin Schweinsberg; Michael Feldman; Nicola Staub; Olmo R. van den Akker; Robbie C.M. van Aert; Marcel A.L.M. van Assen; Yang Liu; Tim Althoff; Jeffrey Heer; Alex Kale; Zainab Mohamed; Hashem Amireh; Vaishali Venkatesh Prasad; Abraham Bernstein; Emily Robinson; Kaisa Snellman; S. Amy Sommer; Sarah M.G. Otner; David Robinson; Raphael Silberzahn; Nikhil Madan; Pavel Goldstein; Warren Tierney; Toshio Murase; Benjamin Mandl; Domenico Viganola; Carolin Strobl; Catherine B.C. Schaumans; Stijn Kelchtermans; Chan Naseeb; S. Mason Garrison; Tal Yarkoni; C.S. Richard Chan; Prestone Adie; Paulius Alaburda; Casper Albers; Sara Alspaugh; Jeff Alstott; Andrew A. Nelson; Eduardo Ariño de la Rubia; Adbi Arzi; Štěpán Bahník; Jason Baik; Laura Winther Balling; Sachin Banker; David AA Baranger; Dale J. Barr; Brenda Barros-Rivera; Matt Bauer; Enuh Blaise; Lisa Boelen; Katerina Bohle Carbonell; Robert A. Briers; Oliver Burkhard; Miguel-Angel Canela; Laura Castrillo; Timothy Catlett; Olivia Chen; Michael Clark; Brent Cohn; Alex Coppock; Natàlia Cugueró-Escofet; Paul G. Curran; Wilson Cyrus-Lai; David Dai; Giulio Valentino Dalla Riva; Henrik Danielsson; Rosaria de F.S.M. Russo; Niko de Silva; Curdin Derungs; Frank Dondelinger; Carolina Duarte de Souza; B. Tyson Dube; Marina Dubova; Ben Mark Dunn; Peter Adriaan Edelsbrunner; Sara Finley; Nick Fox; Timo Gnambs; Yuanyuan Gong; Erin Grand; Brandon Greenawalt; Dan Han; Paul H.P. Hanel; Antony B. Hong; David Hood; Justin Hsueh; Lilian Huang; Kent N. Hui; Keith A. Hultman; Azka Javaid; Lily Ji Jiang; Jonathan Jong; Jash Kamdar; David Kane; Gregor Kappler; Erikson Kaszubowski; Christopher M. Kavanagh; Madian Khabsa; Bennett Kleinberg; Jens Kouros; Heather Krause; Angelos-Miltiadis Krypotos; Dejan Lavbič; Rui Ling Lee; Timothy Leffel; Wei Yang Lim; Silvia Liverani; Bianca Loh; Dorte Lønsmann; Jia Wei Low; Alton Lu; Kyle MacDonald; Christopher R. Madan; Lasse Hjorth Madsen; Christina Maimone; Alexandra Mangold; Adrienne Marshall; Helena Ester Matskewich; Kimia Mavon; Katherine L. McLain; Amelia A. McNamara; Mhairi McNeill; Ulf Mertens; David Miller; Ben Moore; Andrew Moore; Eric Nantz; Ziauddin Nasrullah; Valentina Nejkovic; Colleen S Nell; Gustav Nilsonne; Rory Nolan; Christopher E. O'Brien; Patrick O'Neill; Kieran O'Shea; Toto Olita; Jahna Otterbacher; Diana Palsetia; Bianca Pereira; Ivan Pozdniakov; John Protzko; Jean-Nicolas Reyt; Travis Riddle; Amal (Akmal) Ridhwan Omar Ali; Ivan Ropovik; Joshua M. Rosenberg; Stephane Rothen; Michael Schulte-Mecklenbeck; Nirek Sharma; Gordon Shotwell; Martin Skarzynski; William Stedden; Victoria Stodden; Martin A. Stoffel; Scott Stoltzman; Subashini Subbaiah; Rachael Tatman; Paul H. Thibodeau; Sabina Tomkins; Ana Valdivia; Gerrieke B. Druijff-van de Woestijne; Laura Viana; Florence Villesèche; W. Duncan Wadsworth; Florian Wanders; Krista Watts; Jason D Wells; Christopher E. Whelpley; Andy Won; Lawrence Wu; Arthur Yip; Casey Youngflesh; Ju-Chi Yu; Arash Zandian; Leilei Zhang; Chava Zibman; Eric Luis Uhlmann

doi:10.1016/j.obhdp.2021.02.003

Back

Same data, different conclusions: Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis

Journal article

Open access

Peer reviewed

Same data, different conclusions: Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis

Martin Schweinsberg, Michael Feldman, Nicola Staub, Olmo R. van den Akker, Robbie C.M. van Aert, Marcel A.L.M. van Assen, Yang Liu, Tim Althoff, Jeffrey Heer, Alex Kale, …

Organizational Behavior and Human Decision Processes, Vol.165, pp.228-249

2021-07

DOI: https://doi.org/10.1016/j.obhdp.2021.02.003

Abstract

Analysis-contingent results

Crowdsourcing data analysis

Research reliability

Researcher degrees of freedom

Scientific robustness

Scientific transparency

•A new platform (DataExplained) helps analysts justify preferred and rejected analytical paths in real time.•Independent analysts used DataExplained to test two hypotheses on the same dataset in a crowdsourced initiative.•Analysts conducted radically different analyses and reported dispersed effect sizes, including significant effects in the opposite direction for the same hypothesis.•A BOBA multiverse analysis highlights the importance of variable operationalizations beyond statististical choices. In this crowdsourced initiative, independent analysts used the same dataset to test two hypotheses regarding the effects of scientists’ gender and professional status on verbosity during group meetings. Not only the analytic approach but also the operationalizations of key variables were left unconstrained and up to individual analysts. For instance, analysts could choose to operationalize status as job title, institutional ranking, citation counts, or some combination. To maximize transparency regarding the process by which analytic choices are made, the analysts used a platform we developed called DataExplained to justify both preferred and rejected analytic paths in real time. Analyses lacking sufficient detail, reproducible code, or with statistical errors were excluded, resulting in 29 analyses in the final sample. Researchers reported radically different analyses and dispersed empirical outcomes, in a number of cases obtaining significant effects in opposite directions for the same research question. A Boba multiverse analysis demonstrates that decisions about how to operationalize variables explain variability in outcomes above and beyond statistical choices (e.g., covariates). Subjective researcher decisions play a critical role in driving the reported empirical results, underscoring the need for open data, systematic robustness checks, and transparency regarding both analytic paths taken and not taken. Implications for organizations and leaders, whose decision making relies in part on scientific findings, consulting reports, and internal analyses by data scientists, are discussed.

Files and links (2)

pdf

2021_JournalArticle_MandlBenjamin_SameData3.32 MBDownload View

Published (Version of record)CC BY V4.0, Open Access

url

https://doi.org/10.1016/j.obhdp.2021.02.003View

Published (Version of record)CC BY V4.0, Open

Metrics

1 Record Views

52 Times Cited - Web of Science

Details

Title: Same data, different conclusions: Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis
Creators - without role: Martin Schweinsberg - ESMT Berlin, Germany
Michael Feldman - University of Zurich, Switzerland
Nicola Staub - University of Zurich, Switzerland
Olmo R. van den Akker - Tilburg University, Netherlands
Robbie C.M. van Aert - Tilburg University, Netherlands
Marcel A.L.M. van Assen - Tilburg University and Utrecht University, Netherlands
Yang Liu - University of Washington, United States
Tim Althoff - University of Washington, United States
Jeffrey Heer - University of Washington, United States
Alex Kale - University of Washington, United States
Zainab Mohamed - ESMT Berlin and Indiana University, Germany
Hashem Amireh - ESMT Berlin and Humboldt University Berlin, Germany
Vaishali Venkatesh Prasad - ESMT Berlin, Germany
Abraham Bernstein - University of Zurich, Switzerland
Emily Robinson
Kaisa Snellman - INSEAD
S. Amy Sommer - Marshall School of Business, University of Southern California, United States
Sarah M.G. Otner - Imperial College London
David Robinson
Raphael Silberzahn - University of Sussex
Nikhil Madan - Indian School of Business, India
Pavel Goldstein - School of Public Health, University of Haifa, Israel
Warren Tierney - INSEAD
Toshio Murase - Waseda University, Japan
Benjamin Mandl - Stockholm School of Economics, Department of Economics
Domenico Viganola - Stockholm School of Economics, Department of Economics
Carolin Strobl - University of Zurich, Switzerland
Catherine B.C. Schaumans - Independent researcher
Stijn Kelchtermans - KU Leuven, Belgium
Chan Naseeb - IBM
S. Mason Garrison - Wake Forest University
Tal Yarkoni - University of Texas at Austin, United States
C.S. Richard Chan - Stony Brook University
Prestone Adie - University of Nairobi, Kenya
Paulius Alaburda
Casper Albers - University of Groningen, Netherlands
Sara Alspaugh - University of California, Berkeley, United States
Jeff Alstott - Massachusetts Institute of Technology, United States
Andrew A. Nelson - University of Kentucky, United States
Eduardo Ariño de la Rubia - California State University-Dominguez Hills, United States
Adbi Arzi - INSEAD, France
Štěpán Bahník - Prague College of Psychosocial Studies
Jason Baik
Laura Winther Balling - Copenhagen Business School, Denmark
Sachin Banker - University of Utah, United States
David AA Baranger - University of Pittsburgh
Dale J. Barr - University of Glasgow, United Kingdom
Brenda Barros-Rivera - Texas A&M University, United States
Matt Bauer - Illinois Institute of Technology, United States
Enuh Blaise - Eskisehir Osmangazi University, Turkey
Lisa Boelen - Imperial College London, United Kingdom
Katerina Bohle Carbonell - Northwestern University, United States
Robert A. Briers - Edinburgh Napier University, United Kingdom
Oliver Burkhard
Miguel-Angel Canela - Universidad de Navarra
Laura Castrillo
Timothy Catlett
Olivia Chen
Michael Clark - University of Michigan, United States
Brent Cohn
Alex Coppock - Yale University
Natàlia Cugueró-Escofet - Universitat Oberta de Catalunya, Spain
Paul G. Curran - Michigan State University
Wilson Cyrus-Lai - INSEAD, France
David Dai - St. Michael's Hospital, University of Toronto, Canada
Giulio Valentino Dalla Riva - University of Canterbury
Henrik Danielsson - Linköping University, Sweden
Rosaria de F.S.M. Russo - Universidade Nove de Julho
Niko de Silva - ESMT Berlin, Germany
Curdin Derungs - Lucerne University of Applied Sciences and Arts, Switzerland
Frank Dondelinger - Lancaster University, United Kingdom
Carolina Duarte de Souza - Universidade Federal de Santa Catarina
B. Tyson Dube
Marina Dubova - Indiana University, United States
Ben Mark Dunn - University of Glasgow
Peter Adriaan Edelsbrunner - ETH Zurich, Switzerland
Sara Finley - Pacific Lutheran University, United States
Nick Fox - Rutgers University, United States
Timo Gnambs - Leibniz Institute for Educational Trajectories, Germany, & Johannes Kepler University Linz, Austria
Yuanyuan Gong - Okayama University, Japan
Erin Grand
Brandon Greenawalt - University of Notre Dame, United States
Dan Han
Paul H.P. Hanel - University of Bath
Antony B. Hong - INSEAD, France
David Hood
Justin Hsueh
Lilian Huang - University of Chicago, United States
Kent N. Hui - Xiamen University
Keith A. Hultman - Elmhurst College, United States
Azka Javaid - Columbia University Medical Center, United States
Lily Ji Jiang - Indiana University
Jonathan Jong - University of Oxford & Coventry University, United Kingdom
Jash Kamdar
David Kane - Harvard University, United States
Gregor Kappler - University of Vienna
Erikson Kaszubowski - Universidade Federal de Santa Catarina
Christopher M. Kavanagh
Madian Khabsa
Bennett Kleinberg - University College London, United Kingdom
Jens Kouros
Heather Krause - York University, United Kingdom
Angelos-Miltiadis Krypotos - Utrecht University
Dejan Lavbič - University of Ljubljana, Slovenia
Rui Ling Lee - Nanyang Technological University, Singapore
Timothy Leffel - University of Chicago, United States
Wei Yang Lim - University of Colorado, Colorado Springs, United States
Silvia Liverani - Queen Mary University of London
Bianca Loh - INSEAD, France
Dorte Lønsmann - University of Copenhagen
Jia Wei Low - Singapore Management University, Singapore
Alton Lu - University of Washington, United States
Kyle MacDonald - McD Tech Labs, United States
Christopher R. Madan - University of Nottingham
Lasse Hjorth Madsen - Novo Nordisk, Denmark
Christina Maimone - Northwestern University
Alexandra Mangold
Adrienne Marshall - University of Idaho
Helena Ester Matskewich - University of Washington
Kimia Mavon - Harvard University
Katherine L. McLain - European School of Management and Technology
Amelia A. McNamara - University of St. Thomas - Minnesota
Mhairi McNeill - Edinburgh Napier University
Ulf Mertens - Heidelberg University
David Miller - Northwestern University
Ben Moore - University of Edinburgh
Andrew Moore
Eric Nantz - Eli Lilly
Ziauddin Nasrullah - European School of Management and Technology
Valentina Nejkovic - University of Nis
Colleen S Nell - George Washington University
Gustav Nilsonne - Karolinska Institutet and Stockholm University, Sweden
Rory Nolan - University of Oxford
Christopher E. O'Brien
Patrick O'Neill - University of Maryland, Baltimore County
Kieran O'Shea - University of Glasgow
Toto Olita - Western University
Jahna Otterbacher - Open University of Cyprus
Diana Palsetia - Northwestern University
Bianca Pereira
Ivan Pozdniakov - National Research University Higher School of Economics
John Protzko - University of California, Santa Barbara
Jean-Nicolas Reyt - McGill University
Travis Riddle - National Institutes of Health
Amal (Akmal) Ridhwan Omar Ali - University of Sheffield
Ivan Ropovik - Charles University
Joshua M. Rosenberg - University of Tennessee at Knoxville
Stephane Rothen
Michael Schulte-Mecklenbeck - University of Bern
Nirek Sharma - Washington University in St. Louis
Gordon Shotwell - Dalhousie University
Martin Skarzynski - Booz Allen Hamilton
William Stedden
Victoria Stodden - University of Illinois Urbana-Champaign
Martin A. Stoffel - University of Edinburgh
Scott Stoltzman - Colorado State University
Subashini Subbaiah - Colorado State University
Rachael Tatman - Rasa Technologies, United States
Paul H. Thibodeau - Oberlin College
Sabina Tomkins - Stanford University
Ana Valdivia - Universidad de Granada
Gerrieke B. Druijff-van de Woestijne - Radboud University Nijmegen
Laura Viana - University of Hawaii System
Florence Villesèche - Copenhagen Business School
W. Duncan Wadsworth - Rice University
Florian Wanders - University of Amsterdam
Krista Watts - United States Military Academy
Jason D Wells - Dartmouth College
Christopher E. Whelpley - College of Charleston
Andy Won
Lawrence Wu - University of California, Berkeley
Arthur Yip - National Renewable Energy Laboratory
Casey Youngflesh - University of California, Los Angeles
Ju-Chi Yu - The University of Texas at Dallas
Arash Zandian - KTH Royal Institute of Technology
Leilei Zhang
Chava Zibman - United States Food and Drug Administration
Eric Luis Uhlmann - INSEAD, Singapore
Publication Details: Organizational Behavior and Human Decision Processes, Vol.165, pp.228-249
Publisher: Elsevier Inc
Identifiers: 991001489298606056
Academic Unit: Department of Economics
Language: English
Resource Type: Journal article

Same data, different conclusions: Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis

Abstract

Files and links (2)

Metrics

Details

Stockholm School of Economics Social media