The documented low levels of reliability of the peer review process present a serious challenge to editors who must often base their publication decisions on conflicting referee recommendations. The purpose of this article is to discuss this process and examine ways to produce a more reliable and useful peer review system.

Peer review is an important component of scholarly research. Long a black box whose practical mechanisms were unknown to researchers and readers, peer review is increasingly facing demands for accountability and improvement. Numerous studies address empirical aspects of the peer review process. Much less consideration is typically given to normative dimensions of peer review. This paper considers what authors, editors, reviewers, and readers ought to expect from the peer review process. Integrity in the review process is vital if various parties (...) are to have trust, or faith, in the credibility of peer review mechanisms. Trust in the quality of peer review can increase or diminish in response to numerous factors. Five core elements of peer review are identified. Constitutive elements of scholarly peer review include: fairness in critical analysis of manuscripts; the selection of appropriate reviewers with relevant expertise; identifiable, publicly accountable reviewers; timely reviews, and helpful critical commentary. The F.A.I.T.H. model provides a basis for linking conceptual analysis of the core norms of peer review with empirical research into the adequacy and effectiveness of various processes of peer review. The model is intended to describe core elements of high-quality peer review and suggest what factors can foster or hinder trust in the integrity of peer review. (shrink)

In some recent articles, Dr. Leigh Turner [Doffing the Mask: Why Manuscript Reviewers Ought to Be Identifiable,” Journal of Academic Ethics, 1 (2003), pp. 41–48; “Promoting F.A.I.T.H. in Peer Review: Five Core Attributes in Effective Peer Review,” Journal of Academic Ethics, 1 (2003), pp. 181–188.] makes some rather critical observations regarding the processes of peer-review in academic journals. I shall note them in turn, note wherein I concur and wherein I disagree, and discuss some of Turner's suggestions to resolve such (...) difficulties. It is hoped that my comments on Turner's much-appreciated points will engage readers of this august and well-edited journal to take more seriously Turner's arguments for the sake of the betterment of academic research. (shrink)

Background: Editorial peer review is universally used but little studied. We examined the relationship between external reviewers' recommendations and the editorial outcome of manuscripts undergoing external peer-review at the Journal of General Internal Medicine. Methodology/Principal Findings: We examined reviewer recommendations and editors' decisions at JGIM between 2004 and 2008. For manuscripts undergoing peer review, we calculated chance-corrected agreement among reviewers on recommendations to reject versus accept or revise. Using mixed effects logistic regression models, we estimated intra-class correlation coefficients at the (...) reviewer and manuscript level. Finally, we examined the probability of rejection in relation to reviewer agreement and disagreement. The 2264 manuscripts sent for external review during the study period received 5881 reviews provided by 2916 reviewers; 28% of reviews recommended rejection. Chance corrected agreement on rejection among reviewers was 0.11. In mixed effects models adjusting for study year and manuscript type, the reviewer-level ICC was 0.23 and the manuscript-level ICC was 0.17. The editors' overall rejection rate was 48%: 88% when all reviewers for a manuscript agreed on rejection and 20% when all reviewers agreed that the manuscript should not be rejected. Conclusions/Significance: Reviewers at JGIM agreed on recommendations to reject vs. accept/revise at levels barely beyond chance, yet editors placed considerable weight on reviewers' recommendations. Efforts are needed to improve the reliability of the peer-review process while helping editors understand the limitations of reviewers' recommendations. © 2010 Kravitz et al. (shrink)