Journal peer review is an essential part of academic practices.1 But how well does it serve its purpose and which factors have an influence on how close it comes to achieving its aims? Peer review has been widely discussed in empirical literature: it has been studied both qualitatively and quantitatively (e.g., by Cole, who in his 1992 book uses data on how grant applications submitted to National Science Foundation were...

The demand for greater public accountability is changing the nature of ex ante peer review at public science agencies worldwide. Based on a four year research project, this essay examines these changes through an analysis of the process of grant proposal review at two US public science agencies, the National Institutes of Health (NIH) and the National Science Foundation (NSF). Weaving historical and conceptual narratives with analytical accounts, we describe the ways in which these two agencies struggle with the question (...) of incorporating considerations of societal impact into the process of peer review. We use this comparative analysis to draw two main conclusions. First, evaluation of broader societal impacts is not different in kind from evaluation of intellectual merit. Second, the scientific community may actually bolster its autonomy by taking a broader range of considerations into its peer review processes. (shrink)

The aim of this paper is to investigate the consequences of the bibliometrics-based evaluation system of scientific production on the contents and methods of sciences. The research has been conducted by means of in-depth interviews to a multi-disciplinary panel of Italian researchers. We discuss the implications of bibliometrics-based evaluation on the choice of the research topic, on the experimental practices, on the dissemination habits. We observe that the validation of the bibliometrics-based evaluation practices relies on the acceptance and diffusion within (...) the scientific community, and that these practices are self-sustained through their wide application. We discuss possible evolving scenarios, also considering the recent development of digital archives. (shrink)

STS scholarship has produced important insights about relationships between the roles of peer review and the social construction of knowledge. Yet, barriers related to access have been a continual challenge for such work. This article overcomes some past access challenges and explores peer review normativities operating in the new discipline of Engineering Education. In doing so, it contributes new insights about disciplinary development, interdisciplinarity, and peer review as a site of knowledge construction. In particular, it draws attention to an aspect (...) of peer review not previously discussed – how peer review normativities are shaped by disciplinary origins. A content analysis of peer review documentation revealed that a hyperfocus on methods, which can be traced back to disciplinary origins, continues to be a guiding normativity. However, interviews with editors revealed that they do not acknowledge that normativity. Implications of those findings and their misalignment are discussed, as are contrasts with the history of other disciplines. (shrink)

An error in the Fourth Assessment Report of the Intergovernmental Panel on Climate Change, which wrongly predicted the disappearance of Himalayan glaciers by 2035, fueled doubts about the authority, honesty and rigor of the IPCC as a leading institution in climate science and, correspondingly, raised questions about whether global warming is anything more than a hoax put forward by environmentalists. The late and confusing reaction of the IPCC to these allegations only worsened the matter. By comparing assessment reports issued by (...) the IPCC, this paper asks the question: ‘Why, despite extensive peer review, did the Himalayan glacier melting rate error get published?’ I chronicle exactly what happened, consider why it happened as it did, and show how these answers require us to rethink the concept of ‘peer review’ in scientific practice. I also identify several future directions for peer review if it wants to stay ahead of the game, bearing in mind the forthcoming IPCC Assessment Reports. (shrink)

Prediction markets, which trade contracts based on the results of predictions, have been remarkably successful in predicting the results of political events. A number of proposals have been made to extend prediction markets to scientific questions, and some small-scale science prediction markets have been implemented. Advocates for science prediction markets argue that they could alleviate problems in science such as bias in peer review and epistemically unjustified consensus. I argue that bias in peer review and epistemically unjustified consensuses are genuine (...) problems in science, and that current attempts to solve them fail in practical circumstances. Prediction markets do show some promise for answering scientific questions while avoiding the pitfalls of consensus and peer review. However, there are strong reasons to believe that science prediction markets will not perform nearly so well as election prediction markets, and implementing prediction markets on a large scale could have significant unintended consequences for the organization of scientific research. The promise of prediction markets to solve current problems in assessing scientific claims is therefore largely illusory. (shrink)