Peer review of grant applications, it has been suggested, might be distorted by what is popularly termed old boyism, cronyism, or particularism. We argue that the existing debate emphasizes the more uninteresting aspects of the peer review system and that the operation of old boyism, as currently understood would have little effect on the overall direction of science. We identify a phenomenon of cognitive particularism, which we consider to be more important than the institutional cronyism analyzed in previous studies. We (...) illustrate with material drawn from observation of grant-awarding commit tees of the Science and Engineering Research CounciL In the concluding discussion, we explore some of the possible implications for the peer review system. (shrink)

STS scholarship has produced important insights about relationships between the roles of peer review and the social construction of knowledge. Yet, barriers related to access have been a continual challenge for such work. This article overcomes some past access challenges and explores peer review normativities operating in the new discipline of Engineering Education. In doing so, it contributes new insights about disciplinary development, interdisciplinarity, and peer review as a site of knowledge construction. In particular, it draws attention to an aspect (...) of peer review not previously discussed – how peer review normativities are shaped by disciplinary origins. A content analysis of peer review documentation revealed that a hyperfocus on methods, which can be traced back to disciplinary origins, continues to be a guiding normativity. However, interviews with editors revealed that they do not acknowledge that normativity. Implications of those findings and their misalignment are discussed, as are contrasts with the history of other disciplines. (shrink)

Scientific realists have suggested that changes in our scientific communities over the course of their history have rendered those communities progressively less vulnerable to the problem of unconcieved alternatives over time. I argue in response not only that the most fundamental historical transformations of the scientific enterprise have generated steadily mounting obstacles to revolutionary, transformative, or unorthodox scientific theorizing, but also that we have substantial independent evidence that the institutional apparatus of contemporary scientific inquiry fosters an exceedingly and increasingly theoretically (...) conservative form of that inquiry. I conclude that contemporary scientific communities are actually more vulnerable to the problem of unconceived alternatives than their historical predecessors, and I briefly suggest how we might seek to pursue scientific inquiry in a less theoretically conservative way. (shrink)

Scientific realists have suggested that changes in our scientific communities over the course of their history have rendered those communities progressively less vulnerable to the problem of unconcieved alternatives over time. I argue in response not only that the most fundamental historical transformations of the scientific enterprise have generated steadily mounting obstacles to revolutionary, transformative, or unorthodox scientific theorizing, but also that we have substantial independent evidence that the institutional apparatus of contemporary scientific inquiry fosters an exceedingly and increasingly theoretically (...) conservative form of that inquiry. I conclude that contemporary scientific communities are actually more vulnerable to the problem of unconceived alternatives than their historical predecessors, and I briefly suggest how we might seek to pursue scientific inquiry in a less theoretically conservative way. (shrink)

At present, peer review is the most common method used by funding agencies to make decisions about resource allocation. But how reliable, efficient, and fair is it in practice? The ex ante evaluation of scientific novelty is a fundamentally uncertain endeavor; bias and chance are embedded in the final outcome. In the current study, I will examine some of the most central problems of peer review and highlight the possible benefits of using a lottery as an alternative decision-making mechanism. Lotteries (...) are driven by chance, not reason. The argument made in the study is that the epistemic landscape could benefit in several respects by using a lottery, thus avoiding all types of bias, disagreement, and other limitations associated with the peer review process. Funding agencies could form a pool of funding applicants who have minimal qualification levels and then select randomly within that pool. The benefits of a lottery would not only be that it saves time and resources, but also that it contributes to a more dynamic selection process and increases the epistemic diversity, fairness, and impartiality within academia. (shrink)

The principles and practices of research peer review are described. While the principles are fundamentally generic and apply to peer review across the full spectrum of performing institutions as well as to manuscript/proposal/program peer review, the focus of this paper is peer review of proposed and ongoing programs in federal agencies. The paper describes desireable characteristics and important intangible factors in successful peer review. Also presented is a heuristic protocol for the conduct of successful peer review research evaluations and impact (...) assessments. Problems with peer review are then outlined, followed by examples of peer review of proposed and existing programs in selected federal agencies. Some peer review variants, such as the Science Court, are described, and then research requirements to improve peer review are discussed. (shrink)

The pace of scientific progress may be hindered by the tendency of our academic institutions to reward being popular rather than being right. A market-based alternative, where scientists can more formally 'stake their reputation', is presented here. It offers clear incentives to be careful and honest while contributing to a visible, self-consistent consensus on controversial scientific questions. In addition, it allows patrons to choose questions to be researched without choosing people or methods. The bulk of this paper is spent in (...) examining potential problems with the proposed approach. After this examination, the idea still seems to be plausible and worthy of further study. (shrink)

Epistemological differences fuel continuous and frequently divisive debates in the social sciences and the humanities. Sociologists have yet to consider how such differences affect peer evaluation. The empirical literature has studied distributive fairness, but neglected how epistemological differences affect perception of fairness in decision making. The normative literature suggests that evaluators should overcome their epistemological differences by ‘‘translating’’ their preferred standards into general criteria of evaluation. However, little is known about how procedural fairness actually operates. Drawing on eighty-one interviews with (...) panelists serving on five multidisciplinary fellowship competitions in the social sciences and the humanities, we show that Evaluators generally draw on four epistemological styles to make arguments in favor of and against proposals. These are the constructivist, comprehensive, positivist, and utilitarian styles; and Peer reviewers define a fair decision-making process as one in which panelists engage in ‘‘cognitive contextualization,’’ that is, use epistemological styles most appropriate to the field or discipline of the proposal under review. (shrink)

Although peer review provides the primary mechamism for allocating research funds, very few people have undertaken any serious study of the issue. Most of the literature surrounding peer review comes from individuals who have experienced specific problems with the system. The one detail that does stand out is that though policy makers and researchers seem to view peer review as a specific process, no single form exists. In fact, almost agency and foundation has its own unique process. This presentation will (...) explore the concept of peer review and highlight its strengths and weaknesses. (shrink)