The same dataset can be analysed in different justifiable ways to answer the same research question, potentially challenging the robustness of empirical science. In this crowd initiative, we investigated the degree to which research findings in the social and behavioural sciences are contingent on analysts’ choices. We examined a stratified random sample of 100 studies published between 2009 and 2018, in which, for one claim per study, at least five reanalysts independently reanalysed the original data. The statistical appropriateness of the reanalyses was assessed in peer evaluations, and the robustness indicators were inspected along a range of research characteristics and study designs. We found that 34% of the independent reanalyses yielded the same result (within a tolerance region of ±0.05 Cohen’s d) as the original report; with a four times broader tolerance region, this indicator increased to 57%. Of the reanalyses conducted, 74% reached the same conclusion as the original investigation, 24% yielded no effects or inconclusive results and 2% reported the opposite effect. This exploratory study indicates that the common single-path analyses in social and behavioural research should not be simply assumed to be robust to alternative analyses. Therefore, we recommend the development and use of practices to explore and communicate this neglected source of uncertainty.
- Investigating the analytical robustness of the social and behavioural sciences
- Balazs Aczel - Eötvös Loránd UniversityBarnabas Szaszi - Eötvös Loránd UniversityHarry T Clelland - Eötvös Loránd UniversityMarton Kovacs - Eötvös Loránd UniversityFelix Holzmeister - University of InnsbruckDon van Ravenzwaaij - University of GroningenHannah Schulz-Kümpel - Ludwig-Maximilians-Universität MünchenSabine Hoffmann - Ludwig-Maximilians-Universität MünchenGustav Nilsonne - Karolinska InstitutetLivia Kosa - Eötvös Loránd UniversityZoltan A Torma - Eötvös Loránd UniversityYousuf Abdelfatah - Princeton UniversityChristopher L Aberson - Pacific Oaks CollegeOguz A Acar - King's College LondonEnsar Acem - Kadir Has UniversityMatus Adamkovic - Slovak Academy of SciencesTimofey Adamovich - Psychological Institute of the Russian Academy of EducationKrisna Adiasto - Radboud University NijmegenLove Ahnström - Karolinska InstitutetAtakan M Akil - Eötvös Loránd UniversityAdil S Al-Busaidi - Sultan Qaboos University HospitalAli H Al-Hoorie - Royal Commission for Jubail and YanbuCasper J Albers - University of GroningenPeter J Allen - University of BristolTaym Alsalti - Leipzig University of Applied SciencesMicah Altman - Massachusetts Institute of TechnologyShilaan Alzahawi - Stanford UniversityEttore Ambrosini - University of PaduaSaule Anafinova - Budapest University of Technology and EconomicsRahul Anand - Aarhus UniversityMartin Angerer - University of LiechtensteinAriadna Angulo-Brunet - Universitat Oberta de CatalunyaAlberto Antonietti - Politecnico di MilanoJozsef Arato - University of ViennaAndreu Arenas - Universitat de BarcelonaMarco M Aviña - Harvard UniversityFlavio Azevedo - Utrecht UniversityMarko Bachl - Freie Universität BerlinBence Bago - Tilburg UniversityŠtěpán Bahník - Prague University of Economics and BusinessBradley J Baker - Temple UniversityElza Balayan - Universität InnsbruckCassandra L Baldwin - Georgia Southern UniversityBenjamin Banai - Banai Analitika, Osijek, CroatiaKasia Banas - University of EdinburghFrantišek Bartoš - University of AmsterdamErnest Baskin - Saint Joseph's UniversityJojanneke A Bastiaansen - Department of Psychiatry, University of Groningen, University Medical Center Groningen, Groningen, The NetherlandsNadège Bault - University of PlymouthChristopher W Bauman - University of California, IrvineQuintin H Beazer - Florida State UniversityMaciej Behnke - Adam Mickiewicz University in PoznańTheiss Bendixen - Aarhus UniversitySebastian Berger - Bern University of Applied SciencesAnna Bernard - Universidade Católica PortuguesaUrsa Bernardic - ETH ZurichPaul A Bloom - Columbia UniversityAnnika Boldt - University College LondonCiril Bosch-Rosa - Technische Universität BerlinRotem Botvinik-Nezer - Tel Aviv UniversityAdam Bouyamourn - Princeton UniversityOzge Bozkurt - Anadolu UniversityLaurel Brehm - University of California, Santa BarbaraJohannes Breuer - Center for Advanced Internet StudiesRyan Briggs - University of GuelphHilmar Brohmer - Department of Psychology, University of Graz, Graz, AustriaErin Buchanan - Harrisburg University of Science and TechnologyJohannes Buckenmaier - University of ZurichJeffrey Buckley - Technological University of the Shannon: Midlands MidwestJacek Buczny - Sopot University of Applied ScienceMatthias Burghart - Max Planck Institute for the Study of Crime, Security and LawBilal H Butt - Institute of Business Administration KarachiNick Byrd - Stevens Institute of TechnologyValentina Cafarelli - Independent Researcher, New York, NY, USAPatrick Callahan - Ludwig-Maximilians-Universität MünchenTabaré Capitán - Swedish University of Agricultural SciencesKevin Carriere - Stonehill CollegeAndrea M Cataldo - McLean HospitalGabriel Cepaluni - University of Notre DameEugene Chan - Toronto Metropolitan UniversityJesse J Chandler - Institute for Social Research, University of Michigan, Ann Arbor, MI, USAChia-Chen Chang - National University of SingaporeXi Chen - Shanghai Open UniversityShirley Shuo Chen - Wilfrid Laurier UniversityFadong Chen - Zhejiang LabHao Chen - Technische Universität DresdenValerii Chirkov - Humboldt-Universität zu BerlinDaniela Cialfi - Centro Ricerche Enrico FermiBeth Clarke - The University of MelbourneSophie G Coelho - York UniversityClara Cohen - University of GlasgowJason Collins - University of Technology SydneySusan W Cook - University of IowaGaia Corlazzoli - Université Libre de BruxellesJamie Cummins - University of BernChristian Czymara - Goethe University FrankfurtJonathan D'hondt - Vrije Universiteit AmsterdamAnna Dalla Rosa - University of PaduaAbi M B Davis - University of LincolnAnna Dreber Almenberg - Stockholm School of Economics, Department of EconomicsCharles P Davis - University of Connecticut
- Nature, Vol.652(8108), pp.135-142
- Springer Nature
- Department of Economics
- English
- Journal article