Peer Review: Practices and Controversies
Learn more about the peer review processes and ideas from the field about how to improve it. Find a variety of perspectives in this collection of open-access articles.
Clase, C. M., Dicks, E., Holden, R., Sood, M. M., Levin, A., Kalantar-Zadeh, K., ... & Hartwig, S. (2022). Can peer review be kinder? Supportive peer review: a re-commitment to kindness and a call to action. Canadian Journal of Kidney Health and Disease, 9, 20543581221080327.
Peer review aims to select articles for publication and to improve articles before publication. We believe that this process can be infused by kindness without losing rigor. In 2014, the founding editorial team of the Canadian Journal of Kidney Health and Disease (CJKHD) made an explicit commitment to treat authors as we would wish to be treated ourselves. This broader group of authors reaffirms this principle, for which we suggest the terminology “supportive review.”
Fitzpatrick, K. (2012). Revising peer review. Contexts, 11(4), 80-80.
Abstract. Kathleen Fitzpatrick argues that as online platforms for scholarly publishing foster increasingly fluid means of communication amongst researchers, the principles on which such publishing is founded—including, most crucially, peer review—must become more flexible.
Horbach, S.P.J.M.S., Halffman, W.W. The changing forms and expectations of peer review. Research Integrity and Peer Review 3, 8 (2018). https://doi.org/10.1186/s41073-018-0051-5
Abstract. The quality and integrity of the scientific literature have recently become the subject of heated debate. Due to an apparent increase in cases of scientific fraud and irreproducible research, some have claimed science to be in a state of crisis. A key concern in this debate has been the extent to which science is capable of self-regulation. Among various mechanisms, the peer review system in particular is considered an essential gatekeeper of both quality and sometimes even integrity in science.
Karhulahti, VM., Backe, HJ. Transparency of peer review: a semi-structured interview study with chief editors from social sciences and humanities. Research Integrity and Peer Review 6, 13 (2021). https://doi.org/10.1186/s41073-021-00116-4
Abstract. Open peer review practices are increasing in medicine and life sciences, but in social sciences and humanities (SSH) they are still rare. We aimed to map out how editors of respected SSH journals perceive open peer review, how they balance policy, ethics, and pragmatism in the review processes they oversee, and how they view their own power in the process.
We conducted 12 pre-registered semi-structured interviews with editors of respected SSH journals. Interviews consisted of 21 questions and lasted an average of 67 min. Interviews were transcribed, descriptively coded, and organized into code families.
Kaltenbrunner, W., Birch, K., & Amuchastegui, M. (2022). Editorial work and the peer review economy of STS journals. Science, Technology, & Human Values, 47(4), 670-697.
Abstract. In this paper, we analyze the role of science and technology studies (STS) journal editors in organizing and maintaining the peer review economy. We specifically conceptualize peer review as a gift economy running on perpetually renewed experiences of mutual indebtedness among members of an intellectual community. While the peer review system is conventionally presented as self-regulating, we draw attention to its vulnerabilities and to the essential curating function of editors. Aside from inherent complexities, there are various shifts in the broader political–economic and sociotechnical organization of scholarly publishing that have recently made it more difficult for editors to organize robust cycles of gift exchange. This includes the increasing importance of journal metrics and associated changes in authorship practices; the growth and differentiation of the STS journal landscape; and changes in publishing funding models and the structure of the publishing market through which interactions among authors, editors, and reviewers are reconfigured. To maintain a functioning peer review economy in the face of numerous pressures, editors must balance contradictory imperatives: the need to triage intellectual production and rely on established cycles of gift exchange for efficiency, and the need to expand cycles of gift exchange to ensure the sustainability and diversity of the peer review economy.
Powell, J. L. (2022). Peer review and the pillar of salt: a case study. Research Ethics, 0(0). https://doi.org/10.1177/17470161221131491
Abstract. Peer review has long been regarded as the gold standard of scientific publication, essential to the integrity of science itself. But, as any publishing scientist knows, peer review has its downside, including long delays and reviewer bias. Until the coming of the Internet, there appeared to be no alternative. Now, articles appear online as preprints almost immediately upon submission. But they lack peer review and thus their scientific standing can be questioned. Post-publication discussion platforms such as PubPeer have proven useful, but are no substitute for pre-publication peer review. Nevertheless, some may be tempted to believe that peer review can now be done without. This article challenges that view by analyzing a recent, non-peer-reviewed article in Skeptical Enquirer, a magazine published by the Committee for Skeptical Enquiry (CSI). The article, “Sodom Meteor Strike Claims Should Be Taken With a Pillar of Salt,” casts doubt on one of the most widely read scientific articles of the last decade and provides a stern warning of the cost of abandoning peer review.
Roumbanis, L. (2021). Disagreement and agonistic chance in peer review. Science, Technology, & Human Values, 01622439211026016.
Abstract. The purpose of grant peer review is to identify the most excellent and promising research projects. However, sociologists of science and STS scholars have shown that peer review tends to promote solid low-risk projects at the expense of more original and innovative projects that often come with higher risk. It has also been shown that the review process is affected by significant measures of chance. Against this background, the aim of this study is to theorize the notions of academic judgment and agonistic chance and to present and analyze situations in which expert reviewers are faced with the challenge of trying to decide which grant proposals to select when there is strong disagreement. The empirical analysis is based on ethnographic observations of ten panel groups at the Swedish Research Council in the areas of natural and engineering sciences. By focusing on disagreement, the study provides a more in-depth understanding of how agonistic chance creeps into the peer-review process and becomes part of the consensus that is created.
Tennant, J.P., Ross-Hellauer, T. The limitations to our understanding of peer review. Research Integrity and Peer Review 5, 6 (2020). https://doi.org/10.1186/s41073-020-00092-1
Abstract. Peer review is embedded in the core of our knowledge generation systems, perceived as a method for establishing quality or scholarly legitimacy for research, while also often distributing academic prestige and standing on individuals. Despite its critical importance, it curiously remains poorly understood in a number of dimensions. In order to address this, we have analysed peer review to assess where the major gaps in our theoretical and empirical understanding of it lie. We identify core themes including editorial responsibility, the subjectivity and bias of reviewers, the function and quality of peer review, and the social and epistemic implications of peer review. The high-priority gaps are focused around increased accountability and justification in decision-making processes for editors and developing a deeper, empirical understanding of the social impact of peer review. Addressing this at the bare minimum will require the design of a consensus for a minimal set of standards for what constitutes peer review, and the development of a shared data infrastructure to support this. Such a field requires sustained funding and commitment from publishers and research funders, who both have a commitment to uphold the integrity of the published scholarly record. We use this to present a guide for the future of peer review, and the development of a new research discipline based on the study of peer review.
Winsett, R. P. (2022). What Editors Want to See in a Good Review. Progress in Transplantation, 15269248221089816.
Abstract. Whether a novice or experienced investigator or clinician, reviewing for a peer review journal helps you to understand the transition of designing and implementing a research study to writing for consumption by your peers. Peer reviewing can give you insights on how to critically assess another scholar’s work while learning to improve your own writing skills. From time to time even experienced reviewers can use an overview of what editors want to see in a good review. Every editor wishes to have a solid review and may have specific requests from a reviewer, but there is a basic structure to critiquing and writing a review. This guideline is an overview of that basic structure.
More Methodspace posts about peer review and research ethics
Many Methodspace researchers conduct independent research, or are in situations where they do not have access to an Institutional Review Board or other ethics review options. Working with a private agency is an option, as described in this guest post.