This article examines the evolution of peer review and the modern editorial processes of scholarly journals by analyzing a novel data set derived from the Royal Society’s archives and covering 1865-1965, that is, the historical period in which refereeing became firmly established. Our analysis reveals how the Royal Society’s editorial processes coped with both an increasing reliance on refereeing and a growth in submissions, while maintaining collective responsibility and minimizing research waste. By engaging more of its fellows in editorial activity, (...) the society was able to establish an equilibrium of number of submissions per reviewer that was relatively stable over time. Nevertheless, our analysis shows that the distribution of editorial work was significantly uneven. Our findings reveal interesting parallels with current concerns about the scale and distribution of peer review work and suggest the strategic importance of the management of the editorial process to achieve a creative mix of community commitment and professional responsibility that is essential in contemporary journals. (shrink)

Increasing the “truth per dollar” of money spent on science is one legitimate long-run goal of the economics of science. But before this goal can be achieved, we need to increase our knowledge of the successes and failures of past and current reward structures of science. This essay reviews what economists have learned about the behavior of scientists and the reward structure of science. One important use of such knowledge will be to help policy-makers create a reward structure that is (...) more efficient in the future. (shrink)

This article aims to discuss the contemporary activities and roles that scientific associations play in science and society. It is based on a comprehensive study of scientific associations in Portugal, relying on a multi-method, quantitative and qualitative approach. After a brief review of the literature on associations in the social studies of science, we provide an outline of the expanding field of scientific associations in Portugal. We then proceed to present and discuss the five main roles of associations identified through (...) the research: communication among peers, promotion of research, science dissemination, representation of professional interests and policy advice. We conclude that the external roles of associations have become more important than the internal ones. Whereas the internationalisation of science has moved the communication, collaboration and competition between researchers into the transnational sphere, the links that associations forge between science and other social spheres are still deeply rooted in national settings and much dependant on specific configurations and practices by government, business and other social actors. (shrink)

This article argues that various deviations from the basic principles of the scientific ethos – primarily the appearance of pseudoscience in scientific communities – can be formulated and explained using specific models of game theory, such as the prisoner’s dilemma and the iterated prisoner’s dilemma. The article indirectly tackles the deontology of scientific work as well, in which it is assumed that there is no room for moral skepticism, let alone moral anti-realism, in the ethics of scientific communities. Namely, on (...) the basis of the generally accepted dictum of scientific endeavor as the pursuit of knowledge exclusively for knowledge’s sake, scientifically »right« behavior is seen to be clearly defined and distinguishable from scientifically »wrong« behavior. After elucidating the basic principles of game theory, the article illustrates – by using imaginary and real cases, as well as some views from the philosophyof biology (the units of selection debate) – how this sort of reasoning could be applied in an analysis of the functioning of science. (shrink)

Recently, mainstream philosophy journals have tended to implement more and more stringent forms of peer review, probably in an attempt to prevent editorial decisions that are based on factors other than quality. Against this trend, we propose that journals should relax their standards of acceptance, as well as be less restrictive about whom is to decide what is admitted into the debate. We start by arguing, partly on the basis of the history of peer review in the journal Mind, that (...) past and current peer review practices attest to partisanship with respect to philosophical approach. Then, we explain that such partisanship conflicts with the standard aims of peer review, and that it is both epistemically and morally problematic. This assessment suggests that, if feasible, journals should treat all available and proposed standards of acceptance in philosophy as epistemically equal, and that philosophical work should be evaluated in terms of the novelty and significance of its contribution to developing thought in ways that are of value. Finally, we show, in a programmatic way, that improving the current situation is feasible, and can be done fairly easily. (shrink)

In this paper, we analyze the role of science and technology studies journal editors in organizing and maintaining the peer review economy. We specifically conceptualize peer review as a gift economy running on perpetually renewed experiences of mutual indebtedness among members of an intellectual community. While the peer review system is conventionally presented as self-regulating, we draw attention to its vulnerabilities and to the essential curating function of editors. Aside from inherent complexities, there are various shifts in the broader political–economic (...) and sociotechnical organization of scholarly publishing that have recently made it more difficult for editors to organize robust cycles of gift exchange. This includes the increasing importance of journal metrics and associated changes in authorship practices; the growth and differentiation of the STS journal landscape; and changes in publishing funding models and the structure of the publishing market through which interactions among authors, editors, and reviewers are reconfigured. To maintain a functioning peer review economy in the face of numerous pressures, editors must balance contradictory imperatives: the need to triage intellectual production and rely on established cycles of gift exchange for efficiency, and the need to expand cycles of gift exchange to ensure the sustainability and diversity of the peer review economy. (shrink)

The communist norm requires that scientists widely share the results of their work. Where did this norm come from, and how does it persist? Michael Strevens provides a partial answer to these questions by showing that scientists should be willing to sign a social contract that mandates sharing. However, he also argues that it is not in an individual credit-maximizing scientist's interest to follow this norm. I argue against Strevens that individual scientists can rationally conform to the communist norm, even (...) in the absence of a social contract or other ways of socially enforcing the norm, by proving results to this effect in a game-theoretic model. This shows that the incentives provided to scientists through the priority rule are sufficient to explain both the origins and the persistence of the communist norm, adding to previous results emphasizing the benefits of the incentive structure created by the priority rule. (shrink)

This inaugural handbook documents the distinctive research field that utilizes history and philosophy in investigation of theoretical, curricular and pedagogical issues in the teaching of science and mathematics. It is contributed to by 130 researchers from 30 countries; it provides a logically structured, fully referenced guide to the ways in which science and mathematics education is, informed by the history and philosophy of these disciplines, as well as by the philosophy of education more generally. The first handbook to cover the (...) field, it lays down a much-needed marker of progress to date and provides a platform for informed and coherent future analysis and research of the subject. -/- The publication comes at a time of heightened worldwide concern over the standard of science and mathematics education, attended by fierce debate over how best to reform curricula and enliven student engagement in the subjects There is a growing recognition among educators and policy makers that the learning of science must dovetail with learning about science; this handbook is uniquely positioned as a locus for the discussion. -/- The handbook features sections on pedagogical, theoretical, national, and biographical research, setting the literature of each tradition in its historical context. Each chapter engages in an assessment of the strengths and weakness of the research addressed, and suggests potentially fruitful avenues of future research. A key element of the handbook’s broader analytical framework is its identification and examination of unnoticed philosophical assumptions in science and mathematics research. It reminds readers at a crucial juncture that there has been a long and rich tradition of historical and philosophical engagements with science and mathematics teaching, and that lessons can be learnt from these engagements for the resolution of current theoretical, curricular and pedagogical questions that face teachers and administrators. (shrink)

Under the traditional system of peer-reviewed publication, the degree of prestige conferred to authors by successful publication is tied to the degree of the intellectual rigor of its peer review process: ambitious scientists do well professionally by doing well epistemically. As a result, we should expect journal editors, in their dual role as epistemic evaluators and prestige-allocators, to have the power to motivate improved author behavior through the tightening of publication requirements. Contrary to this expectation, I will argue that the (...) publication bias literature in academic medicine demonstrates that editor interventions have had limited effectiveness in improving the health of the publication and trial registration record, suggesting that much stronger interventions are needed. (shrink)

When the journal Minerva was founded in 1962, science and higher educational issues were high on the agenda, lending impetus to the interdisciplinary field of “Science Studies” qua “Science Policy Studies.” As government expenditures for promoting various branches of science increased dramatically on both sides of the East-West Cold War divide, some common issues regarding research management also emerged and with it an interest in closer academic interaction in the areas of history and policy of science. Through a close reading (...) of many early issues of Minerva but also of its later competitor journal Science Studies (now called Social Studies of Science) the paper traces the initial optimism of an academically based Science Studies dialogue across the Cold War divide and the creation in 1971 of the International Commission for Science Policy Studies as a bridging forum, one that Minerva strangely chose to ignore. In this light, attention is drawn to aspects of the often forgotten history of Science Studies in the former Soviet Union and the Eastern European block. Reviewed also are several early discussions that are still relevant today: e.g., regarding differing concepts of Big Science, science and democracy, autonomy in higher education and what conditions are necessary to sustain academic freedom and scientific integrity (some of Edward Shils’ primary concerns). Finally, it is noted how the question of quantitative methods to measure scientific productivity lay at the heart of a “Science of Science” movement of the 1960s has re-emerged in a new form integral to the notion of a “Science of Science Policy.”. (shrink)

This article analyzes the transformation of Minerva from an intellectual towards a scholarly journal by making use of bibliometric methods. The aim is to provide some empirical insights that help to understand what properties of the journal changed in the course of this transformation process. Minerva was one of the first journals that reflected on science and its role in society and science policy in particular. Analyzing the development of the journal sheds light on the emergence of science (policy) studies (...) and on Minerva’s role as a forerunner in this field. In a first step, the methods will be described. The second section provides some empirical results of the publication output of Minerva and its relations to other journals in the field. The empirical findings are put into a broader perspective in the concluding third section. (shrink)

Psychometrically oriented researchers construe low inter-rater reliability measures for expert peer reviewers as damning for the practice of peer review. I argue that this perspective overlooks different forms of normatively appropriate disagreement among reviewers. Of special interest are Kuhnian questions about the extent to which variance in reviewer ratings can be accounted for by normatively appropriate disagreements about how to interpret and apply evaluative criteria within disciplines during times of normal science. Until these empirical-cum-philosophical analyses are done, it will remain (...) unclear the extent to which low inter-rater reliability measures represent reasonable disagreement rather than arbitrary differences between reviewers. (shrink)

Charles Perrow used the term normal accidents to characterize a type of catastrophic failure that resulted when complex, tightly coupled production systems encountered a certain kind of anomalous event. These were events in which systems failures interacted with one another in a way that could not be anticipated, and could not be easily understood and corrected. Systems of the production of expert knowledge are increasingly becoming tightly coupled. Unlike classical science, which operated with a long time horizon, many current forms (...) of expert knowledge are directed at immediate solutions to complex problems. These are prone to breakdowns like the kind discussed by Perrow. The example of the Homestake mine experiment shows that even in modern physics complex systems can produce knowledge failures that last for decades. The concept of knowledge risk is introduced, and used to characterize the risk of failure in such systems of knowledge production. (shrink)

“Quality” refers nominatively to a standard of performance. Quality is the central idea that differentiates speech protected by academic freedom (the right to worthwhile utterances) from constitutionally protected speech (the right to say anything at all). Extant documents and discussions state that professional peers determine quality based on norms of a field. But professional peers deem utterances and activities as consonant with quality only in reference to criteria that establish meaning of the term. In the absence of articulation, these criteria (...) are ambiguous. Consequently, there exists recurrent confusion about what faculty members have a defensible right to say and do. This article develops an ontology of quality in reference to higher education teaching, a component of academic careers generally not subject to extensive peer review and where instructors thereby exercise considerable autonomy. The ontology identifies three criteria that bound quality: constraint, context, and amplitude. Boundedness exists only insofar as boundaries are controlled. The article examines two types of problems in professional control that affect quality: slippage and overreach. Both are instances of organizational deviance and abrogation of professional ethics. It is argued that the patterns threaten the structural integrity and public confidence of faculty, fields, and higher education institutions. (shrink)

Scholars belong to multiple communities of credit simultaneously. When these communities disagree about a scholarly achievement’s credit assignment, this raises a puzzle for decision and game theor...

An empirically sensitive formulation of the norms of transformative criticism must recognize that even public and shared standards of evaluation can be implemented in ways that unintentionally perpetuate and reproduce forms of social bias that are epistemically detrimental. Helen Longino’s theory can explain and redress such social bias by treating peer evaluations as hypotheses based on data and by requiring a kind of perspectival diversity that bears, not on the content of the community’s knowledge claims, but on the beliefs and (...) norms of the culture of the knowledge community itself. To illustrate how socializing cognition can bias evaluations, we focus on peer-review practices, with some discussion of peer-review practices in philosophy. Data include responses to surveys by editors from general philosophy journals, as well as analyses of reviews and editorial decisions for the 2007 Cognitive Science Society Conference. (shrink)

Two analogies are at the foundation of editions of writings of scientists, technologists and physicians. Both are exemplified in the collection of ‘works’, texts of printed finished versions of contributions. The literary analogy is that of authorship, of the creation of a significant assemblage of words and other symbols. Assemblages of monographs and articles of a scientist are functionally no different than comparable arrays of the writings of theologians, philosophers, poets, novelists and historians.

Select groups and organizations embrace practices that perpetuate their inferiority. The result is the phenomenon we call “mediocrity.” This article examines the conditions under which mediocrity is selected and maintained by groups over time. Mediocrity is maintained by a key social process: the marginalization of the adept, which is a response to the group problem of what to do with the highly able. The problem arises when a majority of a group is comprised of average members who must decide what (...) to do with high performers in the group. To solve this problem, reward systems are subverted to benefit the less able and the adept are cast as deviant. Marginalization is a resolution of two tensions: marginalization of the adept for their behavior, and protection from the adept for the mediocre. The American research university is used as an example to describe the phenomenon and to formulate a theoretic argument. The forms and consequences of marginalization are discussed. Marginalizing the adept illustrates an anti-meritocratic behavioral pattern which serves to sustain social systems on which all people, however able, depend. (shrink)

For over 50 years, Minerva has been one of the leading independent journals in the study of ‘science, learning and policy’. Its pages have much to say about the origins and conduct of the ‘intellectual Cold War’, the defence of academic freedom, the emergence of modernization theory, and pioneering strategies in the social studies of science. This paper revisits Minerva through the life and times of its founding Editor, Edward Shils, and traces his influence on its early years – from (...) its association with the Congress for Cultural Freedom in the 1950s, to the higher reaches of research policy in the 1990s. At the close of his life, Minerva continued to espouse Shils’ commitment to what he saw as the fundamental Enlightenment traditions of consensus, civility, and community. In the 21st century, with the achievements of science producing rapid change in every walk of life, his legacy so far retains an established place in the history of scholarship. Whether that legacy will endure – and if so, what role Minerva will play in its defence – remain key questions for the coming generation. (shrink)

Large differences in scientific productivity between male and female researchers have not yet been explained satisfactorily. This study finds that child care and lack of research collaboration are the two factors that cause significant gender differences in scientific publishing. Women with young children and women who do not collaborate in research with other scientists are clearly less productive than both their male and female colleagues.

This paper presents a framework that editors, peer reviewers, and authors can use to identify and resolve efficiently disputes that arise during peer review in scientific journals. The framework is called a scientific dialectical brief. In this framework, differences among authors and reviewers are formatted into specific assertions and the support each party provides for its position. A literature review suggests that scientists use five main types of support; empirical data, reasoning, speculation, feelings, and status. It is suggested that the (...) scientific dialectical brief format can streamline the review process by facilitating rapid differentiation between stronger and weaker support, so that valuable time can be focused on the better-substantiated claims. The paper concludes with some suggestions for implementation. (shrink)

Epistemological differences fuel continuous and frequently divisive debates in the social sciences and the humanities. Sociologists have yet to consider how such differences affect peer evaluation. The empirical literature has studied distributive fairness, but neglected how epistemological differences affect perception of fairness in decision making. The normative literature suggests that evaluators should overcome their epistemological differences by ‘‘translating’’ their preferred standards into general criteria of evaluation. However, little is known about how procedural fairness actually operates. Drawing on eighty-one interviews with (...) panelists serving on five multidisciplinary fellowship competitions in the social sciences and the humanities, we show that Evaluators generally draw on four epistemological styles to make arguments in favor of and against proposals. These are the constructivist, comprehensive, positivist, and utilitarian styles; and Peer reviewers define a fair decision-making process as one in which panelists engage in ‘‘cognitive contextualization,’’ that is, use epistemological styles most appropriate to the field or discipline of the proposal under review. (shrink)

Born in the 17th century, journal peer review is an extremely diverse technology, constantly torn between two often incompatible goals: the validation of manuscripts conceived as a collective industrial-like reproducible process performed to assert scientific statements, and the dissemination of articles considered as a means to spur scientific discussion, raising controversies, and civically challenging a state of knowledge. Such a situation is particularly conducive to clarifying the processes of valuation and evaluation in journal peer review. In this article, such processes (...) are considered as specific tests in order to emphasize the uncertain properties of pre-tests manuscripts. On the one hand, evaluation tests are examined at the core of the validation of manuscripts, such as defining the coordination of judging instances or controlling the modalities of inter-knowledge between reviewers and authors. They are also studied regarding the dissemination of articles, notably through the contemporary conception of a continuing evaluation test termed “post publication peer review”. On the other hand, valuation tests are both part of the validation of manuscripts, such as the weighting of different judgments of the same manuscript and the tensions that these hierarchies cause, and of the dissemination of articles, such as attention metrics recording the uses of articles. The conclusion sketches out how the articulation of these different tests has recently empowered readers as a new key judging instance for dissemination and for validation, potentially transforming the definition of peers, and thus the whole process of journal peer review. (shrink)