Many Labs 5: Testing Pre-Data-Collection Peer Review as an Intervention to Increase Replicability

Charles R. Ebersole; Maya B. Mathur; Erica Baranski; Diane-Jo Bart-Plange; Nicholas R. Buttrick; Christopher R. Chartier; Katherine S. Corker; Martin Corley; Joshua K. Hartshorne; Hans IJzerman; Ljiljana B. Lazarevic; Hugh Rabagliati; Ivan Ropovik; Balazs Aczel; Lena F. Aeschbach; Luca Andrighetto; Jack D. Arnal; Holly Arrow; Peter Babincak; Bence E. Bakos; Gabriel Banik; Ernest Baskin; Radomir Belopavlovic; Michael H. Bernstein; Michal Bialek; Nicholas G. Bloxsom; Bojana Bodroza; Diane B. Bonfiglio; Leanne Boucher; Florian Bruehlmann; Claudia C. Brumbaugh; Erica Casini; Yiling Chen; Carlo Chiorri; William J. Chopik; Oliver Christ; Antonia M. Ciunci; Heather M. Claypool; Sean Coary; Marija Colic; W. Matthew Collins; Paul G. Curran; Chris R. Day; Benjamin Dering; Anna Dreber; John E. Edlund; Filipe Falcao; Anna Fedor; Lily Feinberg; Ian R. Ferguson; Maire Ford; Michael C. Frank; Emily Fryberger; Alexander Garinther; Katarzyna Gawryluk; Kayla Ashbaugh; Mauro Giacomantonio; Steffen R. Giessner; Jon E. Grahe; Rosanna E. Guadagno; Ewa Halasa; Peter J. B. Hancock; Rias A. Hilliard; Joachim Hueffmeier; Sean Hughes; Katarzyna Idzikowska; Michael Inzlicht; Alan Jern; William Jimenez-Leal; Magnus Johannesson; Jennifer A. Joy-Gaba; Mathias Kauff; Danielle J. Kellier; Grecia Kessinger; Mallory C. Kidwell; Amanda M. Kimbrough; Josiah P. J. King; Vanessa S. Kolb; Sabina Kolodziej; Marton Kovacs; Karolina Krasuska; Sue Kraus; Lacy E. Krueger; Katarzyna Kuchno; Caio Ambrosio Lage; Eleanor Langford; Carmel A. Levitan; Tiago Jesse Souza de Lima; Hause Lin; Samuel Lins; Jia E. Loy; Dylan Manfredi; Lukasz Markiewicz; Madhavi Menon; Brett Mercier; Mitchell Metzger; Venus Meyet; Ailsa E. Millen; Jeremy K. Miller; Andres Montealegre; Don A. Moore; Rafal Muda; Gideon Nave; Austin Lee Nichols; Sarah A. Novak; Christian Nunnally; Ana Orlic; Anna Palinkas; Angelo Panno; Kimberly P. Parks; Ivana Pedovic; Emilian Pekala; Matthew R. Penner; Sebastiaan Pessers; Boban Petrovic; Thomas Pfeiffer; Damian Pienkosz; Emanuele Preti; Danka Puric; Tiago Ramos; Jonathan Ravid; Timothy S. Razza; Katrin Rentzsch; Juliette Richetin; Sean C. Rife; Anna Dalla Rosa; Kaylis Hase Rudy; Janos Salamon; Blair Saunders; Przemyslaw Sawicki; Kathleen Schmidt; Kurt Schuepfer; Thomas Schultze; Stefan Schulz-Hardt; Astrid Schuetz; Ani N. Shabazian; Rachel L. Shubella; Adam Siegel; Ruben Silva; Barbara Sioma; Lauren Skorb; Luana Elayne Cunha de Souza; Sara Steegen; L. A. R. Stein; R. Weylin Sternglanz; Darko Stojilovic; Daniel Storage; Gavin Brent Sullivan; Barnabas Szaszi; Peter Szecsi; Orsolya Szoke; Attila Szuts; Manuela Thomae; Natasha D. Tidwell; Carly Tocco; Ann-Kathrin Torka; Francis Tuerlinckx; Wolf Vanpaemel; Leigh Ann Vaughn; Michelangelo Vianello; Domenico Viganola; Maria Vlachou; Ryan J. Walker; Sophia C. Weissgerber; Aaron L. Wichman; Bradford J. Wiggins; Daniel Wolf; Michael J. Wood; David Zealley; Iris Zezelj; Mark Zrubka; Brian A. Nosek

doi:10.1177/2515245920958687

Back

Many Labs 5: Testing Pre-Data-Collection Peer Review as an Intervention to Increase Replicability

Journal article

Open access

Peer reviewed

Many Labs 5: Testing Pre-Data-Collection Peer Review as an Intervention to Increase Replicability

Charles R. Ebersole, Maya B. Mathur, Erica Baranski, Diane-Jo Bart-Plange, Nicholas R. Buttrick, Christopher R. Chartier, Katherine S. Corker, Martin Corley, Joshua K. Hartshorne, Hans IJzerman, …

Advances in Methods and Practices in Psychological Science, Vol.3(3), pp.309-331

2020-09

DOI: https://doi.org/10.1177/2515245920958687

Abstract

metascience

open data

peer review

preregistered

Registered Reports

replication

reproducibility

Replication studies in psychological science sometimes fail to reproduce prior findings. If these studies use methods that are unfaithful to the original study or ineffective in eliciting the phenomenon of interest, then a failure to replicate may be a failure of the protocol rather than a challenge to the original finding. Formal pre-data-collection peer review by experts may address shortcomings and increase replicability rates. We selected 10 replication studies from the Reproducibility Project: Psychology (RP:P; Open Science Collaboration, 2015) for which the original authors had expressed concerns about the replication designs before data collection; only one of these studies had yielded a statistically significant effect (p < .05). Commenters suggested that lack of adherence to expert review and low-powered tests were the reasons that most of these RP:P studies failed to replicate the original effects. We revised the replication protocols and received formal peer review prior to conducting new replication studies. We administered the RP:P and revised protocols in multiple laboratories (median number of laboratories per original study = 6.5, range = 3-9; median total sample = 1,279.5, range = 276-3,512) for high-powered tests of each original finding with both protocols. Overall, following the preregistered analysis plan, we found that the revised protocols produced effect sizes similar to those of the RP:P protocols (Delta r = .002 or .014, depending on analytic approach). The median effect size for the revised protocols (r = .05) was similar to that of the RP:P protocols (r = .04) and the original RP:P replications (r = .11), and smaller than that of the original studies (r = .37). Analysis of the cumulative evidence across the original studies and the corresponding three replication attempts provided very precise estimates of the 10 tested effects and indicated that their effect sizes (median r = .07, range = .00-.15) were 78% smaller, on average, than the original effect sizes (median r = .37, range = .19-.50).

Files and links (1)

url

https://hdl.handle.net/11567/1072738View

Open

Metrics

9 Record Views

42 Times Cited - Web of Science

Details

Title: Many Labs 5: Testing Pre-Data-Collection Peer Review as an Intervention to Increase Replicability
Creators: Charles R. Ebersole - Univ Virginia, Dept Psychol, 485 McCormick Rd, Charlottesville, VA 22904 USA
Maya B. Mathur - Stanford Univ, Quantitat Sci Unit, Stanford, CA 94305 USA
Erica Baranski - Univ Houston, Dept Psychol, Houston, TX 77004 USA
Diane-Jo Bart-Plange - Univ Virginia, Dept Psychol, 485 McCormick Rd, Charlottesville, VA 22904 USA
Nicholas R. Buttrick - University of Virginia
Christopher R. Chartier - Ashland Univ, Dept Psychol, Ashland, OH USA
Katherine S. Corker - Grand Valley State Univ, Dept Psychol, Allendale, MI 49401 USA
Martin Corley - Univ Edinburgh, Sch Philosophy Psychol & Language Sci, Psychol, Edinburgh, Midlothian, Scotland
Joshua K. Hartshorne - Boston Coll, Psychol & Neurosci Dept, Chestnut Hill, MA 02167 USA
Hans IJzerman - Univ Grenoble Alpes, LIP PC2S, Grenoble, France
Ljiljana B. Lazarevic - Univ Belgrade, Fac Philosophy, Inst Psychol, Belgrade, Serbia
Hugh Rabagliati - Univ Edinburgh, Sch Philosophy Psychol & Language Sci, Psychol, Edinburgh, Midlothian, Scotland
Ivan Ropovik - Charles Univ Prague, Fac Educ, Inst Res & Dev Educ, Prague, Czech Republic
Balazs Aczel - Eotvos Lorand Univ, Inst Psychol, Budapest, Hungary
Lena F. Aeschbach - Univ Basel, Dept Psychol, Basel, Switzerland
Luca Andrighetto - Univ Genoa, Dept Educ Sci, Genoa, Italy
Jack D. Arnal - McDaniel Coll, Psychol Dept, Westminster, MD USA
Holly Arrow - Univ Oregon, Dept Psychol, Eugene, OR 97403 USA
Peter Babincak - Univ Presov, Fac Arts, Inst Psychol, Presov, Slovakia
Bence E. Bakos - Eotvos Lorand Univ, Inst Psychol, Budapest, Hungary
Gabriel Banik - Univ Presov, Fac Arts, Inst Psychol, Presov, Slovakia
Ernest Baskin - St Josephs Univ, Haub Sch Business, Dept Food Mkt, Philadelphia, PA 19131 USA
Radomir Belopavlovic - University of Novi Sad
Michael H. Bernstein - Brown University
Michal Bialek - University of Wrocław
Nicholas G. Bloxsom - Ashland University
Bojana Bodroza - University of Novi Sad
Diane B. Bonfiglio - Ashland University
Leanne Boucher - Nova Southeastern University
Florian Bruehlmann - University of Basel
Claudia C. Brumbaugh - Queens College, CUNY
Erica Casini - University of Milano-Bicocca
Yiling Chen - Harvard Univ, John A Paulson Sch Engn & Appl Sci, Cambridge, MA 02138 USA
Carlo Chiorri - University of Genoa
William J. Chopik - Michigan State University
Oliver Christ - Hagen
Antonia M. Ciunci - University of Rhode Island
Heather M. Claypool - Miami University
Sean Coary - Loyola University Chicago
Marija Colic - University of Belgrade
W. Matthew Collins - Nova Southeastern University
Paul G. Curran - Grand Valley State Univ, Dept Psychol, Allendale, MI 49401 USA
Chris R. Day - Coventry University
Benjamin Dering - University of Stirling
Anna Dreber - Stockholm School of Economics, Department of Economics
John E. Edlund - Rochester Institute of Technology
Filipe Falcao - Universidade do Porto
Anna Fedor - Centre for Ecological Research
Lily Feinberg - Boston College
Ian R. Ferguson - Virginia Commonwealth University
Maire Ford - Loyola Marymount University
Michael C. Frank - Stanford Univ, Dept Psychol, Stanford, CA 94305 USA
Emily Fryberger - Pacific Lutheran University
Alexander Garinther - University of Oregon
Katarzyna Gawryluk - Kozminski University
Kayla Ashbaugh - Rose–Hulman Institute of Technology
Mauro Giacomantonio - Sapienza University of Rome
Steffen R. Giessner - Erasmus University Rotterdam
Jon E. Grahe - Pacific Lutheran University
Rosanna E. Guadagno - Stanford University
Ewa Halasa - Maria Curie-Skłodowska University
Peter J. B. Hancock - University of Stirling
Rias A. Hilliard - Rose–Hulman Institute of Technology
Joachim Hueffmeier - TU Dortmund University
Sean Hughes - Ghent University
Katarzyna Idzikowska - Kozminski University
Michael Inzlicht - Univ Toronto, Dept Psychol, Toronto, ON, Canada
Alan Jern - Rose–Hulman Institute of Technology
William Jimenez-Leal - Univ Los Andes, Dept Psychol, Bogota, Colombia
Magnus Johannesson - Stockholm School of Economics, Department of Economics
Jennifer A. Joy-Gaba - Virginia Commonwealth Univ, Dept Psychol, Richmond, VA 23284 USA
Mathias Kauff - Supreme Council Of Health
Danielle J. Kellier - University of Pennsylvania
Grecia Kessinger - Brigham Young University - Idaho
Mallory C. Kidwell - Univ Utah, Dept Psychol, Salt Lake City, UT 84112 USA
Amanda M. Kimbrough - The University of Texas at Dallas
Josiah P. J. King - University of Edinburgh
Vanessa S. Kolb - University of Rhode Island
Sabina Kolodziej - Kozminski University
Marton Kovacs - Eötvös Loránd University
Karolina Krasuska - Maria Curie-Skłodowska University
Sue Kraus - Fort Lewis College
Lacy E. Krueger - Texas A&M University – Commerce
Katarzyna Kuchno - Maria Curie-Skłodowska University
Caio Ambrosio Lage - Pontifical Catholic University of Rio de Janeiro
Eleanor Langford - University of Virginia
Carmel A. Levitan - Occidental Coll, Dept Cognit Sci, Los Angeles, CA 90041 USA
Tiago Jesse Souza de Lima - Universidade de Brasília
Hause Lin - University of Toronto
Samuel Lins - Universidade do Porto
Jia E. Loy - University of Edinburgh
Dylan Manfredi - Univ Penn, Wharton Sch, Mkt Dept, Philadelphia, PA 19104 USA
Lukasz Markiewicz - Kozminski University
Madhavi Menon - Nova Southeastern University
Brett Mercier - University of California, Irvine
Mitchell Metzger - Ashland University
Venus Meyet - Brigham Young University - Idaho
Ailsa E. Millen - University of Stirling
Jeremy K. Miller - Willamette Univ, Dept Psychol, Salem, OR USA
Andres Montealegre - Cornell Univ, Dept Psychol, Ithaca, NY 14853 USA
Don A. Moore - Univ Calif Berkeley, Haas Sch Business, Berkeley, CA 94720 USA
Rafal Muda - Maria Curie-Skłodowska University
Gideon Nave - Univ Penn, Wharton Sch, Mkt Dept, Philadelphia, PA 19104 USA
Austin Lee Nichols - Duy Tan University
Sarah A. Novak - Hofstra University
Christian Nunnally - Rose–Hulman Institute of Technology
Ana Orlic - University of Belgrade
Anna Palinkas - Eötvös Loránd University
Angelo Panno - Sapienza University of Rome
Kimberly P. Parks - University of Virginia
Ivana Pedovic - University of Nis
Emilian Pekala - Maria Curie-Skłodowska University
Matthew R. Penner - Western Kentucky University
Sebastiaan Pessers - KU Leuven
Boban Petrovic - University of Belgrade
Thomas Pfeiffer - Massey Univ, New Zealand Inst Adv Study, Palmerston North, New Zealand
Damian Pienkosz - Maria Curie-Skłodowska University
Emanuele Preti - University of Milano-Bicocca
Danka Puric - University of Belgrade
Tiago Ramos - Universidade do Porto
Jonathan Ravid - Boston College
Timothy S. Razza - Nova Southeastern University
Katrin Rentzsch - Psychologische Hochschule Berlin
Juliette Richetin - University of Milano-Bicocca
Sean C. Rife - Murray State University
Anna Dalla Rosa - University of Padua
Kaylis Hase Rudy - Brigham Young University - Idaho
Janos Salamon - Eötvös Loránd University
Blair Saunders - University of Dundee
Przemyslaw Sawicki - Kozminski University
Kathleen Schmidt - Southern Illinois Univ Carbondale, Sch Psychol & Behav Sci, Carbondale, IL USA
Kurt Schuepfer - Miami University
Thomas Schultze - University of Göttingen
Stefan Schulz-Hardt - University of Göttingen
Astrid Schuetz - University of Bamberg
Ani N. Shabazian - Loyola Marymount University
Rachel L. Shubella - Rose–Hulman Institute of Technology
Adam Siegel - Cultivate Labs, Chicago, IL USA
Ruben Silva - Universidade do Porto
Barbara Sioma - Maria Curie-Skłodowska University
Lauren Skorb - Boston College
Luana Elayne Cunha de Souza - Universidade de Fortaleza
Sara Steegen - Univ Leuven, Psychol & Educ Sci, Leuven, Belgium
L. A. R. Stein - University of Rhode Island
R. Weylin Sternglanz - Nova Southeastern University
Darko Stojilovic
Daniel Storage - Univ Denver, Dept Psychol, Denver, CO 80208 USA
Gavin Brent Sullivan - Coventry Univ, Ctr Trust Peace & Social Relat, Coventry, W Midlands, England
Barnabas Szaszi - Eotvos Lorand Univ, Inst Psychol, Budapest, Hungary
Peter Szecsi - Eötvös Loránd University
Orsolya Szoke - Eötvös Loránd University
Attila Szuts - Eötvös Loránd University
Manuela Thomae - MEU Multiversitat, Magdeburg, Germany
Natasha D. Tidwell - Fort Lewis College
Carly Tocco - Queens College, CUNY
Ann-Kathrin Torka - TU Dortmund University
Francis Tuerlinckx - Univ Leuven, Psychol & Educ Sci, Leuven, Belgium
Wolf Vanpaemel - Univ Leuven, Psychol & Educ Sci, Leuven, Belgium
Leigh Ann Vaughn - Ithaca Coll, Dept Psychol, Ithaca, NY 14850 USA
Michelangelo Vianello - University of Padua
Domenico Viganola - Stockholm School of Economics, Department of Economics
Maria Vlachou - KU Leuven
Ryan J. Walker - Miami University
Sophia C. Weissgerber - University of Kassel
Aaron L. Wichman - Western Kentucky University
Bradford J. Wiggins - Brigham Young University - Idaho
Daniel Wolf - University of Bamberg
Michael J. Wood - Univ Winchester, Dept Psychol, Winchester, Hants, England
David Zealley - Brigham Young University - Idaho
Iris Zezelj - University of Belgrade
Mark Zrubka - University of Amsterdam
Brian A. Nosek - University of Virginia
Publication Details: Advances in Methods and Practices in Psychological Science, Vol.3(3), pp.309-331
Publisher: Sage
Number of pages: 23
Academic Unit: Department of Economics
Language: English
Resource Type: Journal article

Many Labs 5: Testing Pre-Data-Collection Peer Review as an Intervention to Increase Replicability

Abstract

Files and links (1)

Metrics

Details

Stockholm School of Economics Social media