Scientific realists have suggested that changes in our scientific communities over the course of their history have rendered those communities progressively less vulnerable to the problem of unconcieved alternatives over time. I argue in response not only that the most fundamental historical transformations of the scientific enterprise have generated steadily mounting obstacles to revolutionary, transformative, or unorthodox scientific theorizing, but also that we have substantial independent evidence that the institutional apparatus of contemporary scientific inquiry fosters an exceedingly and increasingly theoretically (...) conservative form of that inquiry. I conclude that contemporary scientific communities are actually more vulnerable to the problem of unconceived alternatives than their historical predecessors, and I briefly suggest how we might seek to pursue scientific inquiry in a less theoretically conservative way. (shrink)

To arrive at their final evaluation of a manuscript or grant proposal, reviewers must convert a submission’s strengths and weaknesses for heterogeneous peer review criteria into a single metric of quality or merit. I identify this process of commensuration as the locus for a new kind of peer review bias. Commensuration bias illuminates how the systematic prioritization of some peer review criteria over others permits and facilitates problematic patterns of publication and funding in science. Commensuration bias also foregrounds a range (...) of structural strategies for realigning peer review practices and institutions with the aims of science. (shrink)

Epistemological differences fuel continuous and frequently divisive debates in the social sciences and the humanities. Sociologists have yet to consider how such differences affect peer evaluation. The empirical literature has studied distributive fairness, but neglected how epistemological differences affect perception of fairness in decision making. The normative literature suggests that evaluators should overcome their epistemological differences by ‘‘translating’’ their preferred standards into general criteria of evaluation. However, little is known about how procedural fairness actually operates. Drawing on eighty-one interviews with (...) panelists serving on five multidisciplinary fellowship competitions in the social sciences and the humanities, we show that Evaluators generally draw on four epistemological styles to make arguments in favor of and against proposals. These are the constructivist, comprehensive, positivist, and utilitarian styles; and Peer reviewers define a fair decision-making process as one in which panelists engage in ‘‘cognitive contextualization,’’ that is, use epistemological styles most appropriate to the field or discipline of the proposal under review. (shrink)

At present, peer review is the most common method used by funding agencies to make decisions about resource allocation. But how reliable, efficient, and fair is it in practice? The ex ante evaluation of scientific novelty is a fundamentally uncertain endeavor; bias and chance are embedded in the final outcome. In the current study, I will examine some of the most central problems of peer review and highlight the possible benefits of using a lottery as an alternative decision-making mechanism. Lotteries (...) are driven by chance, not reason. The argument made in the study is that the epistemic landscape could benefit in several respects by using a lottery, thus avoiding all types of bias, disagreement, and other limitations associated with the peer review process. Funding agencies could form a pool of funding applicants who have minimal qualification levels and then select randomly within that pool. The benefits of a lottery would not only be that it saves time and resources, but also that it contributes to a more dynamic selection process and increases the epistemic diversity, fairness, and impartiality within academia. (shrink)

This article examines the evolution of peer review and the modern editorial processes of scholarly journals by analyzing a novel data set derived from the Royal Society’s archives and covering 1865-1965, that is, the historical period in which refereeing became firmly established. Our analysis reveals how the Royal Society’s editorial processes coped with both an increasing reliance on refereeing and a growth in submissions, while maintaining collective responsibility and minimizing research waste. By engaging more of its fellows in editorial activity, (...) the society was able to establish an equilibrium of number of submissions per reviewer that was relatively stable over time. Nevertheless, our analysis shows that the distribution of editorial work was significantly uneven. Our findings reveal interesting parallels with current concerns about the scale and distribution of peer review work and suggest the strategic importance of the management of the editorial process to achieve a creative mix of community commitment and professional responsibility that is essential in contemporary journals. (shrink)

The way research is, and should be, funded by the public sphere is the subject of renewed interest for sociology, economics, management sciences, and more recently, for the philosophy of science. In this contribution, I propose a qualitative, epistemological criticism of the funding by lottery model, which is advocated by a growing number of scholars as an alternative to peer-review. This lottery scheme draws on the lack of efficiency and of robustness of the peer-review based evaluation to argue that the (...) majority of public resources for basic science should be allocated randomly. I first differentiate between two distinct arguments used to defend this alternative funding scheme based on considerations about the logic of scientific research. To assess their epistemological limits, I then present and develop a conceptual frame, grounded on the notion of “system of practice”, which can be used to understand what precisely it means, for a research project, to be interesting or significant. I use this epistemological analysis to show that the lottery model is not theoretically optimal, since it underestimates the integration of all scientific projects in densely interconnected systems of conceptual, experimental, or technical practices which confer their proper interest to them. I also apply these arguments in order to criticize the classical peer-review process. I finally suggest, as a discussion, that some recently proposed models that bring to the fore a principle of decentralization of the evaluation and selection process may constitute a better alternative, if the practical conditions of their implementation are adequately settled. (shrink)

This paper is intended to be a consideration of the role of multi-level modelling in educational research. It is not a guide on how to design or perform such an analysis. There are several references in the text to sources that teach the practicalities perfectly well, and the technique is anyway similar to other forms of regression and to analysis of variance. Rather, the paper describes what multi-level modelling is, why it is used, and what its limitations are. It does (...) so in the hope that some readers will be enthused sufficiently to become appropriately critical 'consumers' of research using this approach, so building research capacity, and easing pressures on 'specialist' reviewers. Anyone who can read or perform standard multivariate analyses can understand, referee, or conduct a multi-level model. Additionally, the paper makes three key points. The generally small sample size in each cluster at the lowest level of any multi-level model means that there is a danger of a greater bias in the results than in standard analyses that pool the data from all clusters. Even where there are genuine gains through the use of multi-level models these have to be set against a loss in simple intuitive grasp of the results, especially amongst policy-makers and practitioners. Therefore, long term, we are probably better advised to improve our research designs and the quality of the data we collect than to focus on more and more complex forms of analysis to overcome deficiencies in the datasets we already have. (shrink)

The operative nucleus of peer review processes has largely remained a ‘black box’ to analytical empirical research. There is a lack of direct insights into the communicative machinery of peer review, i.e., into ‘gatekeeping in action’. This article attempts to fill a small part of this huge research gap. It is based on an ethnographic case study about peer review communication in a sociological journal. It looks at the final phase of the peer review process: the decisions taken in the (...) oral communication of editors’ meetings. The article describes this meeting as an instrumented talk, supported by written tools and constrained by necessary procedural outcomes. It analyzes examples of interactive negotiations of manuscripts and, in the end, it discusses the procedural rationality of peer review as a public sphere for professional judgment. (shrink)

This paper shows that, for a large range of parameters, the journal editor prefers to delegate the choice to review the manuscript to the biased referee. If the peer review process is informative and the review reports are costly for the reviewers, even biased referees with extreme scientific preferences may choose to become informed about the manuscript’s quality. On the contrary, if the review process is potentially informative but the reviewer reports are not costly for the referees, the biased reviewer (...) has no incentive to become informed about the manuscript. Furthermore, if the reports are costly for referees but the peer review processes are not potentially informative, the biased reviewers will never become informed. In this paper, we also present a web resource that helps editors to experiment with the review process as a device for information transmission. (shrink)

This study examines the tension between the academic evaluation system and the development of unconventional research agendas. While scholars have studied the evaluation of research that crosses disciplinary boundaries from the perspective of peer reviewers, they have paid comparatively little attention to the experiences of the performers of unconventional science. This study asks how researchers develop unconventional research agendas to address a long-standing health problem and, in the process, make sense of the actions of a site visit committee organized to (...) advise a foundation funding the project. This study develops a process narrative on the development of a specific transdisciplinary team proposing to study premature birth. The findings show that when the performers and the evaluators of unconventional science developed competing understandings of the research agenda, transdisciplinary discovery became limited as a particular research topic became taboo. Yet, the study also reveals how the performers of unconventional science challenged the power of the site visit committee by making it seem as though they followed the committee’s decisions. These findings raise questions about the role of private foundations as funders of academic research and the suitability of traditional evaluation procedures for assessing transdisciplinary discovery. (shrink)

While some form of evaluation has always been employed in science (e.g. peer review, hiring), formal systems of evaluation of research and researchers have recently come to play a more prominent role in many countries because of the adoption of new models of governance. According to such models, the quality of the output of both researchers and their institutions is measured, and issues such as eligibility for tenure or the allocation of public funding to research institutions crucially depends on the (...) outcomes of such measures. However, concerns have been raised over the risk that such evaluation may be threatening epistemic pluralism by penalizing the existent heterodox schools of thought and discouraging the pursuit of new ones. It has been proposed that this may happen because of epistemic bias favouring mainstream research programmes. In this paper, I claim that (1) epistemic pluralism is desirable and should be preserved; (2) formal evaluation exercises may threaten epistemic pluralism because they may be affected by some form of epistemic bias; therefore, (3) to preserve epistemic pluralism, we need some strategy to actively dampen epistemic bias. (shrink)

We examine the criticisms and subsequent changes that arise in the course of peer review. Fifty-two scholars who had recently published in Administrative Science Quarterly were surveyed regarding their peer review experience and how their article changed from initial journal submission to eventual publication. Papers that challenged theoretical perspectives faced distinctively high levels of criticism and change, particularly with attention to methodology, while those that offered a new perspective or that extended or combined established perspectives were less criticized and changed. (...) The number of challenge-oriented publications was small as well, suggesting that either few such submissions survive the review process or few are submitted in the first place. Overall, peer review appears open to expansion of the variety of theoretical argument but does little to aid in the winnowing out of established perspectives. (shrink)

The independence of research is a key strategic issue of modern societies. Dealing with it appropriately poses legal, economic, political, social and cultural problems for society, which have been studied by the corresponding disciplines and are increasingly the subject of reflexive discourses of scientific communities. Unfortunately, problems of independence are usually framed in disciplinary contexts without due consideration of other perspectives’ relevance or possible contributions. To overcome these limitations, we review disciplinary perspectives and findings on the independence of research and (...) identify interdisciplinary prospects that could inform a research programme. (shrink)

We reviewed the output of research and innovation cooperation between Russia and the US, including publications and patents, in the four prospective areas of energy efficiency and renewable energy during 2007-2011. Joint US-Russia research groups appear to focus primarily on hydrogen energy (fuel cells), followed by solar photovoltaics. The upcoming areas of smart grid and biofuels were left out entirely both from research and innovation collaboration. Russian patents in green energy technologies registered in the US are very low in comparison (...) to those from Japan, Korea, and China. (shrink)

STS scholarship has produced important insights about relationships between the roles of peer review and the social construction of knowledge. Yet, barriers related to access have been a continual challenge for such work. This article overcomes some past access challenges and explores peer review normativities operating in the new discipline of Engineering Education. In doing so, it contributes new insights about disciplinary development, interdisciplinarity, and peer review as a site of knowledge construction. In particular, it draws attention to an aspect (...) of peer review not previously discussed – how peer review normativities are shaped by disciplinary origins. A content analysis of peer review documentation revealed that a hyperfocus on methods, which can be traced back to disciplinary origins, continues to be a guiding normativity. However, interviews with editors revealed that they do not acknowledge that normativity. Implications of those findings and their misalignment are discussed, as are contrasts with the history of other disciplines. (shrink)

Graphical AbstractFunding agencies (and journals) seem to be discriminating against ideas that are contrary to the mainstream, leading to leading to the preferential funding of predictable and safe research over radically new ideas. To remedy this problem a restructuring of the scientific funding system is needed, e.g. by utilizing laymen - together with scientists - to evaluate grant proposals.

Peer reviews of 84 organic farming grant applications from Sweden were analyzed to determine whether the reviewers’ affiliation to one of two types of agriculture (i.e., organic and conventional) influenced their reviews. Fifteen reviewers were divided into three groups: (1) scientists with experience in organic farming research; (2) scientists with no experience in organic farming research; and (3) users of organic farming research. The two groups of scientists assessed the societal relevance and scientific quality of the grant applications based on (...) three criteria (i.e., presentation, methodology, qualifications), whereas the user group only assessed societal relevance. The analysis showed that the two groups of scientists provided very different reviews. Scientist reviewers with experience in organic farming research agreed more with the user group on research relevance than did scientist reviewers without such experience, and the assessment of relevance was closely correlated to the assessment of scientific quality within both scientific groups. As both scientific groups did not clearly distinguish between societal relevance and scientific quality, the idea of an objective science is challenged. The contextual values associated with the norms of good agriculture were not clearly distinguished from the constitutive values of science associated with the traditional norms of good science. This raises the question of whether organic and conventional grant applications should be mixed for review regardless of the reviewers. (shrink)

The purpose of grant peer review is to identify the most excellent and promising research projects. However, sociologists of science and STS scholars have shown that peer review tends to promote solid low-risk projects at the expense of more original and innovative projects that often come with higher risk. It has also been shown that the review process is affected by significant measures of chance. Against this background, the aim of this study is to theorize the notions of academic judgment (...) and agonistic chance, and to present and analyze situations in which expert reviewers are faced with the challenge of trying to decide which grant proposals to select when there is strong disagreement. The empirical analysis is based on ethnographic observations of ten panel groups at the Swedish Research Council in the areas of natural and engineering sciences. By focusing on disagreement, the study provides a more in-depth understanding of how agonistic chance creeps into the peer review process and becomes part of the consensus that is created. (shrink)