Epistemological differences fuel continuous and frequently divisive debates in the social sciences and the humanities. Sociologists have yet to consider how such differences affect peer evaluation. The empirical literature has studied distributive fairness, but neglected how epistemological differences affect perception of fairness in decision making. The normative literature suggests that evaluators should overcome their epistemological differences by ‘‘translating’’ their preferred standards into general criteria of evaluation. However, little is known about how procedural fairness actually operates. Drawing on eighty-one interviews with (...) panelists serving on five multidisciplinary fellowship competitions in the social sciences and the humanities, we show that Evaluators generally draw on four epistemological styles to make arguments in favor of and against proposals. These are the constructivist, comprehensive, positivist, and utilitarian styles; and Peer reviewers define a fair decision-making process as one in which panelists engage in ‘‘cognitive contextualization,’’ that is, use epistemological styles most appropriate to the field or discipline of the proposal under review. (shrink)

Contemporary debates about scientific institutions and practice feature many proposed reforms. Most of these require increased efforts from scientists. But how do scientists’ incentives for effort interact? How can scientific institutions encourage scientists to invest effort in research? We explore these questions using a game-theoretic model of publication markets. We employ a base game between authors and reviewers, before assessing some of its tendencies by means of analysis and simulations. We compare how the effort expenditures of these groups interact in (...) our model under a variety of settings, such as double-blind and open review systems. We make a number of findings, including that open review can increase the effort of authors in a range of circumstances and that these effects can manifest in a policy-relevant period of time. However, we find that open review’s impact on authors’ efforts is sensitive to the strength of several other influences. (shrink)

Two important aspects of the relationship between peer review and innovation includes the acceptance of articles for publication in journals and the assessment of applications for grants for the funding of research work. While there are well-known examples of the rejection by journals of first choice of many papers that have radically changed the way we think about the world outside ourselves, such papers do get published eventually, however tortuous the process required. With grant applications the situation differs in that (...) the refusal of a grant necessarily curtails the possible research that may be attempted. Here there are many reasons for conservatism and reservation as to the ability of a grant allocation process based on peer review to deliver truly innovative investigations. Other methods are needed; although such methods need not be applied across the board, they should constitute the methods whereby some 10–20% of the grant monies are assigned. The nomination of prizes for specific accomplishments is one way of achieving innovation although this presumes that investigators or institution already have available the money necessary to effect the innovations; otherwise it is a question of the selection and funding of particular individuals or institutions and requiring them to solve particular problems that are set in the broadest of terms. (shrink)

This brief article presents the introduction and draft of the fundamental ideas developed at length in the book of the same title, which gives a challenging point of view about science and its history/philosophy/sociology. Science is in decline. After centuries of great achievements, the exhaustion of new forms and fatigue have reached our culture in all of its manifestations including the pure sciences. Our society is saturated with knowledge which does not offer people any sense in their lives. There is (...) a loss of ideals in the search for great truths and a shift towards an anodyne specialized industry. (shrink)

A relatively high incidence of unsatisfactory review decisions is widely recognised and acknowledged as ‘the peer review problem’. Factors contributing to this problem are identified and examined. Specific examples of unreasonable rejection are considered. It is concluded that weaknesses of the ‘peer review’ system are significant and that they are well known or readily recognisable but that necessary counter-measures are not always enforced. Careful management is necessary to discount hollow opinion or error in review comment. Review and referee functions should (...) be quite separate. (shrink)

A relatively high incidence of unsatisfactory review decisions is widely recognised and acknowledged as ‘the peer review problem’. Factors contributing to this problem are identified and examined. Specific examples of unreasonable rejection are considered. It is concluded that weaknesses of the ‘peer review’ system are significant and that they are well known or readily recognisable but that necessary counter-measures are not always enforced. Careful management is necessary to discount hollow opinion or error in review comment. Review and referee functions should (...) be quite separate. (shrink)

Recently, high media visibility was reached by an experiment that involved “hoaxlike deception” of journals within humanities and social sciences. Its aim was to provide evidence of “inadequate” quality standards especially within gender studies. The article discusses the project in the context of both previous systematic studies of peer reviewing and scientific hoaxes and analyzes its possible empirical outcomes. Despite claims to the contrary, the highly political, both ethically and methodologically flawed “experiment” failed to provide the evidence it sought. The (...) experiences can be summed up as follows: journals with higher impact factors were more likely to reject papers submitted as part of the project; the chances were better, if the manuscript was allegedly based on empirical data; peer reviews can be an important asset in the process of revising a manuscript; and when the project authors, with academic education from neighboring disciplines, closely followed the reviewers’ advice, they were able to learn relatively quickly what is needed for writing an acceptable article. The boundary between a seriously written paper and a “hoax” gradually became blurred. Finally, the way the project ended showed that in the long run, the scientific community will uncover fraudulent practices. (shrink)

The operative nucleus of peer review processes has largely remained a ‘black box’ to analytical empirical research. There is a lack of direct insights into the communicative machinery of peer review, i.e., into ‘gatekeeping in action’. This article attempts to fill a small part of this huge research gap. It is based on an ethnographic case study about peer review communication in a sociological journal. It looks at the final phase of the peer review process: the decisions taken in the (...) oral communication of editors’ meetings. The article describes this meeting as an instrumented talk, supported by written tools and constrained by necessary procedural outcomes. It analyzes examples of interactive negotiations of manuscripts and, in the end, it discusses the procedural rationality of peer review as a public sphere for professional judgment. (shrink)

This paper presents a framework that editors, peer reviewers, and authors can use to identify and resolve efficiently disputes that arise during peer review in scientific journals. The framework is called a scientific dialectical brief. In this framework, differences among authors and reviewers are formatted into specific assertions and the support each party provides for its position. A literature review suggests that scientists use five main types of support; empirical data, reasoning, speculation, feelings, and status. It is suggested that the (...) scientific dialectical brief format can streamline the review process by facilitating rapid differentiation between stronger and weaker support, so that valuable time can be focused on the better-substantiated claims. The paper concludes with some suggestions for implementation. (shrink)

This paper presents a framework that editors, peer reviewers, and authors can use to identify and resolve efficiently disputes that arise during peer review in scientific journals. The framework is called a scientific dialectical brief. In this framework, differences among authors and reviewers are formatted into specific assertions and the support each party provides for its position. A literature review suggests that scientists use five main types of support; empirical data, reasoning, speculation, feelings, and status. It is suggested that the (...) scientific dialectical brief format can streamline the review process by facilitating rapid differentiation between stronger and weaker support, so that valuable time can be focused on the better-substantiated claims. The paper concludes with some suggestions for implementation. (shrink)

In the academic world, peer review is one of the major processes in evaluating a scholars contribution. In this study, we are interested in quantifying the merits of different policies in a peer review process, such as single-blind review, double-blind review, and obtaining authors feedback. Currently, insufficient work has been undertaken to evaluate the benefits of different peer review policies. One of the major reasons for this situation is the inability to conduct any empirical study because data are presently unavailable. (...) In this case, a computer simulation is one of the best ways to conduct a study. We perform a series of simulations to study the effects of different policies on a peer review process. In this study, we focus on the peer review process of a typical computer science conference. Our results point to the crucial role of program chairs in determining the quality and diversity of the articles to be accepted for publication. We demonstrate the importance of discussion among reviewers, suggest circumstances in which the double-blind review policy should be adopted, and question the credibility of the authors feedback mechanism. Finally, we stress that randomness plays an important role in the peer review process, and this role cannot be eliminated. Although our model may not capture every component of a peer review process, it covers some of the most essential elements. Thus, even the simulation results clearly cannot be taken as literal descriptions of an actual peer review process. However, we can at least still use them to identify alternative directions for future study. (shrink)