An empirically sensitive formulation of the norms of transformative criticism must recognize that even public and shared standards of evaluation can be implemented in ways that unintentionally perpetuate and reproduce forms of social bias that are epistemically detrimental. Helen Longino’s theory can explain and redress such social bias by treating peer evaluations as hypotheses based on data and by requiring a kind of perspectival diversity that bears, not on the content of the community’s knowledge claims, but on the beliefs and (...) norms of the culture of the knowledge community itself. To illustrate how socializing cognition can bias evaluations, we focus on peer-review practices, with some discussion of peer-review practices in philosophy. Data include responses to surveys by editors from general philosophy journals, as well as analyses of reviews and editorial decisions for the 2007 Cognitive Science Society Conference. (shrink)

Charles Perrow used the term normal accidents to characterize a type of catastrophic failure that resulted when complex, tightly coupled production systems encountered a certain kind of anomalous event. These were events in which systems failures interacted with one another in a way that could not be anticipated, and could not be easily understood and corrected. Systems of the production of expert knowledge are increasingly becoming tightly coupled. Unlike classical science, which operated with a long time horizon, many current forms (...) of expert knowledge are directed at immediate solutions to complex problems. These are prone to breakdowns like the kind discussed by Perrow. The example of the Homestake mine experiment shows that even in modern physics complex systems can produce knowledge failures that last for decades. The concept of knowledge risk is introduced, and used to characterize the risk of failure in such systems of knowledge production. (shrink)

This article argues that various deviations from the basic principles of the scientific ethos – primarily the appearance of pseudoscience in scientific communities – can be formulated and explained using specific models of game theory, such as the prisoner’s dilemma and the iterated prisoner’s dilemma. The article indirectly tackles the deontology of scientific work as well, in which it is assumed that there is no room for moral skepticism, let alone moral anti-realism, in the ethics of scientific communities. Namely, on (...) the basis of the generally accepted dictum of scientific endeavor as the pursuit of knowledge exclusively for knowledge’s sake, scientifically »right« behavior is seen to be clearly defined and distinguishable from scientifically »wrong« behavior. After elucidating the basic principles of game theory, the article illustrates – by using imaginary and real cases, as well as some views from the philosophyof biology (the units of selection debate) – how this sort of reasoning could be applied in an analysis of the functioning of science. (shrink)

“Quality” refers nominatively to a standard of performance. Quality is the central idea that differentiates speech protected by academic freedom (the right to worthwhile utterances) from constitutionally protected speech (the right to say anything at all). Extant documents and discussions state that professional peers determine quality based on norms of a field. But professional peers deem utterances and activities as consonant with quality only in reference to criteria that establish meaning of the term. In the absence of articulation, these criteria (...) are ambiguous. Consequently, there exists recurrent confusion about what faculty members have a defensible right to say and do. This article develops an ontology of quality in reference to higher education teaching, a component of academic careers generally not subject to extensive peer review and where instructors thereby exercise considerable autonomy. The ontology identifies three criteria that bound quality: constraint, context, and amplitude. Boundedness exists only insofar as boundaries are controlled. The article examines two types of problems in professional control that affect quality: slippage and overreach. Both are instances of organizational deviance and abrogation of professional ethics. It is argued that the patterns threaten the structural integrity and public confidence of faculty, fields, and higher education institutions. (shrink)

The main aim of this article is to start a discussion of social pattern, a term that is commonly used in sociology but not specified or defined. The key question can be phrased as follows: Is it possible to transform the notion of social pattern from its current status in sociology as a proto-concept into a fully worked out concept? And if so, how can this be done? To provide material for the discussion we begin by introducing a few different (...) types of patterns that are currently being used (patterns in nature, cultural patterns, statistical patterns, and computationally generated patterns). This is followed by a suggestion for what a strictlysociologicalconcept of social pattern may look like. A useful and theoretically solid concept of social pattern can in our view be constructed by basing it on Weber’s concept of social action. This means thatboththe behavior of the actorsandthe meaning these invest their behavior with must be taken into account. The article ends with a brief discussion of how to use the concept of social patterns in an effective way and what may endanger such a use. (shrink)

Scholars belong to multiple communities of credit simultaneously. When these communities disagree about a scholarly achievement’s credit assignment, this raises a puzzle for decision and game theor...

This study seeks to examine the impact of alumni connections between the evaluators and evaluatees on the results of peer review ratings for the Korean national R&D project and selection success rate. Specifically, this study analyzed the evaluation results of 8,402 research proposal entries submitted between 2007 and 2011 for the “general researchers support project,” all in the Natural Science and Engineering areas and sponsored by the National Research Foundation of Korea. Each proposal entry was evaluated by three evaluators, and (...) approximately 39 percent of the proposals had at least one evaluator from the same university. The results of this study showed that evaluators have a tendency to give relatively high scores to research proposals submitted by the alumni of the same universities as their alma mater. Also, when an evaluator from the same university as an evaluatee was included in the evaluator group, the results of this study showed that the percentage of entry submissions was higher compared to the contrary. Such results show that in the process of peer review–based research proposal evaluations for national R&D projects, alumni connections have significant influence on evaluation results in South Korea. (shrink)

Epistemological differences fuel continuous and frequently divisive debates in the social sciences and the humanities. Sociologists have yet to consider how such differences affect peer evaluation. The empirical literature has studied distributive fairness, but neglected how epistemological differences affect perception of fairness in decision making. The normative literature suggests that evaluators should overcome their epistemological differences by ‘‘translating’’ their preferred standards into general criteria of evaluation. However, little is known about how procedural fairness actually operates. Drawing on eighty-one interviews with (...) panelists serving on five multidisciplinary fellowship competitions in the social sciences and the humanities, we show that Evaluators generally draw on four epistemological styles to make arguments in favor of and against proposals. These are the constructivist, comprehensive, positivist, and utilitarian styles; and Peer reviewers define a fair decision-making process as one in which panelists engage in ‘‘cognitive contextualization,’’ that is, use epistemological styles most appropriate to the field or discipline of the proposal under review. (shrink)

BackgroundTriggered by a series of controversies and diversifying expectations of editorial practices, several innovative peer review procedures and supporting technologies have been proposed. However, adoption of these new initiatives seems slow. This raises questions about the wider conditions for peer review change and about the considerations that inform decisions to innovate. We set out to study the structure of commercial publishers’ editorial process, to reveal how the benefits of peer review innovations are understood, and to describe the considerations that inform (...) the implementation of innovations.MethodsWe carried out field visits to the editorial office of two large academic publishers housing the editorial staff of several hundreds of journals, to study their editorial process, and interviewed editors not affiliated with large publishers. Field notes were transcribed and analysed using coding software.ResultsAt the publishers we analysed, the decision-making structure seems to show both clear patterns of hierarchy and layering of the different editorial practices. While information about new initiatives circulates widely, their implementation depends on assessment of stakeholder’s wishes, impact on reputation, efficiency and implementation costs, with final decisions left to managers at the top of the internal hierarchy. Main tensions arise between commercial and substantial arguments. The editorial process is closely connected to commercial practices of creating business value, and the very specific terms in which business value is understood, such as reputation considerations and the urge to increase efficiency. Journals independent of large commercial publishers tend to have less hierarchically structured processes, report more flexibility to implement innovations, and to a greater extent decouple commercial and editorial perspectives.ConclusionOur study demonstrates that peer review innovations are partly to be understood in light of commercial considerations related to reputation, efficiency and implementations costs. These arguments extend beyond previously studied topics in publishing economics, including publishers’ choice for business or publication models and reach into the very heart of the editorial and peer review process. (shrink)

Peer review of journal submissions has become one of the most important pillars of quality management in academic publishing. Because of growing concerns with the quality and effectiveness of the system, a host of enthusiastic innovators has proposed and experimented with new procedures and technologies. However, little is known about whether these innovations manage to convince other journal editors. This paper will address open questions regarding the implementation of new review procedures, the occurrence rate of various peer review procedures and (...) their distribution over scientific disciplines or academic publishers, as well as the motivations for editors or publishers to engage in novel review procedures. It shows that in spite of enthusiastic innovation, the adoption of new peer review procedures is in fact very slow, with the exception of text similarity scanners. For now, peer review innovations appear to be restricted to specific niches in academic publishing. Analysing these niches, the article concludes with a reflection on the circumstances in which innovations might be more widely implemented. (shrink)

Recently, mainstream philosophy journals have tended to implement more and more stringent forms of peer review, probably in an attempt to prevent editorial decisions that are based on factors other than quality. Against this trend, we propose that journals should relax their standards of acceptance, as well as be less restrictive about whom is to decide what is admitted into the debate. We start by arguing, partly on the basis of the history of peer review in the journal Mind, that (...) past and current peer review practices attest to partisanship with respect to philosophical approach. Then, we explain that such partisanship conflicts with the standard aims of peer review, and that it is both epistemically and morally problematic. This assessment suggests that, if feasible, journals should treat all available and proposed standards of acceptance in philosophy as epistemically equal, and that philosophical work should be evaluated in terms of the novelty and significance of its contribution to developing thought in ways that are of value. Finally, we show, in a programmatic way, that improving the current situation is feasible, and can be done fairly easily. (shrink)

For over 50 years, Minerva has been one of the leading independent journals in the study of ‘science, learning and policy’. Its pages have much to say about the origins and conduct of the ‘intellectual Cold War’, the defence of academic freedom, the emergence of modernization theory, and pioneering strategies in the social studies of science. This paper revisits Minerva through the life and times of its founding Editor, Edward Shils, and traces his influence on its early years – from (...) its association with the Congress for Cultural Freedom in the 1950s, to the higher reaches of research policy in the 1990s. At the close of his life, Minerva continued to espouse Shils’ commitment to what he saw as the fundamental Enlightenment traditions of consensus, civility, and community. In the 21st century, with the achievements of science producing rapid change in every walk of life, his legacy so far retains an established place in the history of scholarship. Whether that legacy will endure – and if so, what role Minerva will play in its defence – remain key questions for the coming generation. (shrink)

The communist norm requires that scientists widely share the results of their work. Where did this norm come from, and how does it persist? Michael Strevens provides a partial answer to these questions by showing that scientists should be willing to sign a social contract that mandates sharing. However, he also argues that it is not in an individual credit-maximizing scientist's interest to follow this norm. I argue against Strevens that individual scientists can rationally conform to the communist norm, even (...) in the absence of a social contract or other ways of socially enforcing the norm, by proving results to this effect in a game-theoretic model. This shows that the incentives provided to scientists through the priority rule are sufficient to explain both the origins and the persistence of the communist norm, adding to previous results emphasizing the benefits of the incentive structure created by the priority rule. (shrink)

This inaugural handbook documents the distinctive research field that utilizes history and philosophy in investigation of theoretical, curricular and pedagogical issues in the teaching of science and mathematics. It is contributed to by 130 researchers from 30 countries; it provides a logically structured, fully referenced guide to the ways in which science and mathematics education is, informed by the history and philosophy of these disciplines, as well as by the philosophy of education more generally. The first handbook to cover the (...) field, it lays down a much-needed marker of progress to date and provides a platform for informed and coherent future analysis and research of the subject. -/- The publication comes at a time of heightened worldwide concern over the standard of science and mathematics education, attended by fierce debate over how best to reform curricula and enliven student engagement in the subjects There is a growing recognition among educators and policy makers that the learning of science must dovetail with learning about science; this handbook is uniquely positioned as a locus for the discussion. -/- The handbook features sections on pedagogical, theoretical, national, and biographical research, setting the literature of each tradition in its historical context. Each chapter engages in an assessment of the strengths and weakness of the research addressed, and suggests potentially fruitful avenues of future research. A key element of the handbook’s broader analytical framework is its identification and examination of unnoticed philosophical assumptions in science and mathematics research. It reminds readers at a crucial juncture that there has been a long and rich tradition of historical and philosophical engagements with science and mathematics teaching, and that lessons can be learnt from these engagements for the resolution of current theoretical, curricular and pedagogical questions that face teachers and administrators. (shrink)

Under the traditional system of peer-reviewed publication, the degree of prestige conferred to authors by successful publication is tied to the degree of the intellectual rigor of its peer review process: ambitious scientists do well professionally by doing well epistemically. As a result, we should expect journal editors, in their dual role as epistemic evaluators and prestige-allocators, to have the power to motivate improved author behavior through the tightening of publication requirements. Contrary to this expectation, I will argue that the (...) publication bias literature in academic medicine demonstrates that editor interventions have had limited effectiveness in improving the health of the publication and trial registration record, suggesting that much stronger interventions are needed. (shrink)

Select groups and organizations embrace practices that perpetuate their inferiority. The result is the phenomenon we call “mediocrity.” This article examines the conditions under which mediocrity is selected and maintained by groups over time. Mediocrity is maintained by a key social process: the marginalization of the adept, which is a response to the group problem of what to do with the highly able. The problem arises when a majority of a group is comprised of average members who must decide what (...) to do with high performers in the group. To solve this problem, reward systems are subverted to benefit the less able and the adept are cast as deviant. Marginalization is a resolution of two tensions: marginalization of the adept for their behavior, and protection from the adept for the mediocre. The American research university is used as an example to describe the phenomenon and to formulate a theoretic argument. The forms and consequences of marginalization are discussed. Marginalizing the adept illustrates an anti-meritocratic behavioral pattern which serves to sustain social systems on which all people, however able, depend. (shrink)

The conventional history of modern scientific peer review begins with the censorship practices of the Royal Society of London in the 1660s. This article traces one strand of the “prehistory” of peer review in the writings of John Amos Comenius and other members of the Hartlib circle, a precursor group to the Royal Society of London. These reformers appear to have first envisioned peer review as a technique for theologians, only later proposing to apply it to philosophy. The importance of (...) peer review was as a technique that would permit a community of theologians or philosophers to resolve disputes internally rather than publicly, since public disputation would (they believed) sow doubt, error, and confusion, and disrupt the social order. (shrink)

The nature of authority is shifting in online scholarly communication. This examination of the history and future of peer review argues that effective online communication requires the development of an open, community?oriented, post?publication system of peer?to?peer review, transforming peer review from a process focused on gatekeeping to one concerned with filtering the wealth of scholarly material made available via the Internet.

This paper presents a framework that editors, peer reviewers, and authors can use to identify and resolve efficiently disputes that arise during peer review in scientific journals. The framework is called a scientific dialectical brief. In this framework, differences among authors and reviewers are formatted into specific assertions and the support each party provides for its position. A literature review suggests that scientists use five main types of support; empirical data, reasoning, speculation, feelings, and status. It is suggested that the (...) scientific dialectical brief format can streamline the review process by facilitating rapid differentiation between stronger and weaker support, so that valuable time can be focused on the better-substantiated claims. The paper concludes with some suggestions for implementation. (shrink)

Despite continued attention, finding adequate criteria for distinguishing “good” from “bad” scholarly journals remains an elusive goal. In this essay, I propose a solution informed by the work of Imre Lakatos and his methodology of scientific research programmes (MSRP). I begin by reviewing several notable attempts at appraising journal quality – focusing primarily on the impact factor and development of journal blacklists and whitelists. In doing so, I note their limitations and link their overarching goals to those found within the (...) philosophy of science. I argue that Lakatos’s MSRP and specifically his classifications of “progressive” and “degenerative” research programmes can be analogized and repurposed for the evaluation of scholarly journals. I argue that this alternative framework resolves some of the limitations discussed above and offers a more considered evaluation of journal quality – one that helps account for the historical evolution of journal-level publication practices and attendant contributions to the growth (or stunting) of scholarly knowledge. By doing so, the seeming problem of journal demarcation is diminished. In the process I utilize two novel tools (the mistake index and scite index) to further illustrate and operationalize aspects of the MSRP. (shrink)

In this paper, we analyze the role of science and technology studies journal editors in organizing and maintaining the peer review economy. We specifically conceptualize peer review as a gift economy running on perpetually renewed experiences of mutual indebtedness among members of an intellectual community. While the peer review system is conventionally presented as self-regulating, we draw attention to its vulnerabilities and to the essential curating function of editors. Aside from inherent complexities, there are various shifts in the broader political–economic (...) and sociotechnical organization of scholarly publishing that have recently made it more difficult for editors to organize robust cycles of gift exchange. This includes the increasing importance of journal metrics and associated changes in authorship practices; the growth and differentiation of the STS journal landscape; and changes in publishing funding models and the structure of the publishing market through which interactions among authors, editors, and reviewers are reconfigured. To maintain a functioning peer review economy in the face of numerous pressures, editors must balance contradictory imperatives: the need to triage intellectual production and rely on established cycles of gift exchange for efficiency, and the need to expand cycles of gift exchange to ensure the sustainability and diversity of the peer review economy. (shrink)

If any nature of science perspective is to be incorporated in science-related curricula, it is hard to imagine a satisfactory didactic toolkit that neglects the social studies of science, the academic field of study of the institutional structures and networks of science. Knowledge production takes place in a world populated by actors, instruments, and ideas, and various epistemic cultures are responsible for providing the concepts, abstractions, and techniques that slowly trickle down the information pathways to become stabilized in university curricula (...) or ossified in high-school science, losing much of the tacit knowledge along the way that enabled the production of knowledge in the first place. The discussion of the sociology of science and the sociology of scientific knowledge is followed by a short overview of recent approaches in science studies and the studies of expertise. (shrink)

Born in the 17th century, journal peer review is an extremely diverse technology, constantly torn between two often incompatible goals: the validation of manuscripts conceived as a collective industrial-like reproducible process performed to assert scientific statements, and the dissemination of articles considered as a means to spur scientific discussion, raising controversies, and civically challenging a state of knowledge. Such a situation is particularly conducive to clarifying the processes of valuation and evaluation in journal peer review. In this article, such processes (...) are considered as specific tests in order to emphasize the uncertain properties of pre-tests manuscripts. On the one hand, evaluation tests are examined at the core of the validation of manuscripts, such as defining the coordination of judging instances or controlling the modalities of inter-knowledge between reviewers and authors. They are also studied regarding the dissemination of articles, notably through the contemporary conception of a continuing evaluation test termed “post publication peer review”. On the other hand, valuation tests are both part of the validation of manuscripts, such as the weighting of different judgments of the same manuscript and the tensions that these hierarchies cause, and of the dissemination of articles, such as attention metrics recording the uses of articles. The conclusion sketches out how the articulation of these different tests has recently empowered readers as a new key judging instance for dissemination and for validation, potentially transforming the definition of peers, and thus the whole process of journal peer review. (shrink)

This article examines the evolution of peer review and the modern editorial processes of scholarly journals by analyzing a novel data set derived from the Royal Society’s archives and covering 1865-1965, that is, the historical period in which refereeing became firmly established. Our analysis reveals how the Royal Society’s editorial processes coped with both an increasing reliance on refereeing and a growth in submissions, while maintaining collective responsibility and minimizing research waste. By engaging more of its fellows in editorial activity, (...) the society was able to establish an equilibrium of number of submissions per reviewer that was relatively stable over time. Nevertheless, our analysis shows that the distribution of editorial work was significantly uneven. Our findings reveal interesting parallels with current concerns about the scale and distribution of peer review work and suggest the strategic importance of the management of the editorial process to achieve a creative mix of community commitment and professional responsibility that is essential in contemporary journals. (shrink)

This paper presents a framework that editors, peer reviewers, and authors can use to identify and resolve efficiently disputes that arise during peer review in scientific journals. The framework is called a scientific dialectical brief. In this framework, differences among authors and reviewers are formatted into specific assertions and the support each party provides for its position. A literature review suggests that scientists use five main types of support; empirical data, reasoning, speculation, feelings, and status. It is suggested that the (...) scientific dialectical brief format can streamline the review process by facilitating rapid differentiation between stronger and weaker support, so that valuable time can be focused on the better-substantiated claims. The paper concludes with some suggestions for implementation. (shrink)

Two analogies are at the foundation of editions of writings of scientists, technologists and physicians. Both are exemplified in the collection of ‘works’, texts of printed finished versions of contributions. The literary analogy is that of authorship, of the creation of a significant assemblage of words and other symbols. Assemblages of monographs and articles of a scientist are functionally no different than comparable arrays of the writings of theologians, philosophers, poets, novelists and historians.

Peer review is a topic of considerable concern to many researchers, and there is a correspondingly large body of research on the topic. This issue of Science and Engineering Ethics presents recent work on peer review that is both grounded in empirical science and is applicable to policy decisions. This research raises two basic questions; (a) how does current peer review operate, and (b) how can it be improved? Topics addressed include descriptions of how peer review is used in Federal (...) agencies. whether peer review leads to better manuscripts, demographic characteristics of authors or reviewers (status or institutional affiliation), blinding of reviewers, authors, or results, reliability and consistency of reviews, accepting a paper before the study is done, simultaneous submission, and use of dispute resolution procedures such as scientific dialectical and pleading protocols. (shrink)

Large differences in scientific productivity between male and female researchers have not yet been explained satisfactorily. This study finds that child care and lack of research collaboration are the two factors that cause significant gender differences in scientific publishing. Women with young children and women who do not collaborate in research with other scientists are clearly less productive than both their male and female colleagues.

This article aims to discuss the contemporary activities and roles that scientific associations play in science and society. It is based on a comprehensive study of scientific associations in Portugal, relying on a multi-method, quantitative and qualitative approach. After a brief review of the literature on associations in the social studies of science, we provide an outline of the expanding field of scientific associations in Portugal. We then proceed to present and discuss the five main roles of associations identified through (...) the research: communication among peers, promotion of research, science dissemination, representation of professional interests and policy advice. We conclude that the external roles of associations have become more important than the internal ones. Whereas the internationalisation of science has moved the communication, collaboration and competition between researchers into the transnational sphere, the links that associations forge between science and other social spheres are still deeply rooted in national settings and much dependant on specific configurations and practices by government, business and other social actors. (shrink)

This paper presents a framework that editors, peer reviewers, and authors can use to identify and resolve efficiently disputes that arise during peer review in scientific journals. The framework is called a scientific dialectical brief. In this framework, differences among authors and reviewers are formatted into specific assertions and the support each party provides for its position. A literature review suggests that scientists use five main types of support; empirical data, reasoning, speculation, feelings, and status. It is suggested that the (...) scientific dialectical brief format can streamline the review process by facilitating rapid differentiation between stronger and weaker support, so that valuable time can be focused on the better-substantiated claims. The paper concludes with some suggestions for implementation. (shrink)