The principles and practices of research peer review are described. While the principles are fundamentally generic and apply to peer review across the full spectrum of performing institutions as well as to manuscript/proposal/program peer review, the focus of this paper is peer review of proposed and ongoing programs in federal agencies. The paper describes desireable characteristics and important intangible factors in successful peer review. Also presented is a heuristic protocol for the conduct of successful peer review research evaluations and impact (...) assessments. Problems with peer review are then outlined, followed by examples of peer review of proposed and existing programs in selected federal agencies. Some peer review variants, such as the Science Court, are described, and then research requirements to improve peer review are discussed. (shrink)

Scientific editors’ policies, including peer review, are based mainly on tradition and belief. Do they actually achieve their desired effects, the selection of the best manuscripts and improvement of those published? Editorial decisions have important consequences—to investigators, the scientific community, and all who might benefit from correct information or be harmed by misleading research results. These decisions should be judged not just by intentions of reviewers and editors but also by the actual consequences of their actions. A small but growing (...) number of studies has put editorial policies to a strong scientific test. In a randomized, controlled trial. blinding reviewers to author and institution was usually successful and improved the quality of reviews. Two studies have shown that, contrary to conventional wisdom, reviewers early in their careers give better reviews than senior reviewers. Many studies have shown low agreement between reviewers but there is disagreement about whether this represents a failing of peer review or the expected and valuable effect of choosing reviewers with complementary expertise. In a study of whether manuscripts are improved by peer review and editing, articles published in Annals of Internal Medicine were improved in 33 of 34 dimensions of reporting quality, but published articles still had room for improvement. Because of the central place of peer review in the scientific community and the resources it requires, more studies are needed to define what it does and does not accomplish. (shrink)

The documented low levels of reliability of the peer review process present a serious challenge to editors who must often base their publication decisions on conflicting referee recommendations. The purpose of this article is to discuss this process and examine ways to produce a more reliable and useful peer review system.

The major challenge facing today’s biomedical researchers is the increasing competition for available funds. The competitive review process, through which the National Institutes of Health (NIH) awards grants, is built upon review by a committee of expert scientists. The NIH is firmly committed to ensuring that its peer review system is fair and objective.

This paper reviews the published empirical evidence concerning journal peer review consisting of 68 papers, all but three published since 1975. Peer review improves quality, but its use to screen papers has met with limited success. Current procedures to assure quality and fairness seem to discourage scientific advancement, especially important innovations, because findings that conflict with current beliefs are often judged to have defects. Editors can use procedures to encourage the publication of papers with innovative findings such as invited papers, (...) early-acceptance procedures, author nominations of reviewers, structured rating sheets, open peer review, results-blind review, and, in particular, electronic publication. Some journals are currently using these procedures. The basic principle behind the proposals is to change the decision from whether to publish a paper to how to publish it. (shrink)

Francis Bacon, lawyer, statesman, and philosopher, remains one of the most effectual thinkers in European intellectual history. We can trace his influence from Kant in the 1700s to Darwin a century later. The Advancement of Learning , first published in 1605, contains an unprecedented and thorough systematization of the whole range of human knowledge. Bacon’s argument that the sciences should move away from divine philosophy and embrace empirical observation would forever change the way philosophers and natural scientists interpret their world.

A growing interest in and concern about the adequacy and fairness of modern peer-review practices in publication and funding are apparent across a wide range of scientific disciplines. Although questions about reliability, accountability, reviewer bias, and competence have been raised, there has been very little direct research on these variables.

Why believe in the findings of science? John Ziman argues that scientific knowledge is not uniformly reliable, but rather like a map representing a country we cannot visit. He shows how science has many elements, including alongside its experiments and formulae the language and logic, patterns and preconceptions, facts and fantasies used to illustrate and express its findings. These elements are variously combined by scientists in their explanations of the material world as it lies outside our everyday experience. John Ziman’s (...) book offers at once a valuably clear account and a radically challenging investigation of the credibility of scientific knowledge, searching widely across a range of disciplines for evidence about the perceptions, paradigms and analogies on which all our understanding depends. (shrink)

When the New Organon appeared in 1620, part of a six-part programme of scientific inquiry entitled 'The Great Renewal of Learning', Francis Bacon was at the high point of his political career, and his ambitious work was groundbreaking in its attempt to give formal philosophical shape to a new and rapidly emerging experimentally-based science. Bacon combines theoretical scientific epistemology with examples from applied science, examining phenomena as various as magnetism, gravity, and the ebb and flow of the tides, and anticipating (...) later experimental work by Robert Boyle and others. His work challenges the entire edifice of the philosophy and learning of his time, and has left its mark on all subsequent philosophical discussions of scientific method. This volume presents a new translation of the text into modern English by Michael Silverthorne, and an introduction by Lisa Jardine that sets the work in the context of Bacon's scientific and philosophical activities. (shrink)