If any nature of science perspective is to be incorporated in science-related curricula, it is hard to imagine a satisfactory didactic toolkit that neglects the social studies of science, the academic field of study of the institutional structures and networks of science. Knowledge production takes place in a world populated by actors, instruments, and ideas, and various epistemic cultures are responsible for providing the concepts, abstractions, and techniques that slowly trickle down the information pathways to become stabilized in university curricula (...) or ossified in high-school science, losing much of the tacit knowledge along the way that enabled the production of knowledge in the first place. The discussion of the sociology of science and the sociology of scientific knowledge is followed by a short overview of recent approaches in science studies and the studies of expertise. (shrink)

This inaugural handbook documents the distinctive research field that utilizes history and philosophy in investigation of theoretical, curricular and pedagogical issues in the teaching of science and mathematics. It is contributed to by 130 researchers from 30 countries; it provides a logically structured, fully referenced guide to the ways in which science and mathematics education is, informed by the history and philosophy of these disciplines, as well as by the philosophy of education more generally. The first handbook to cover the (...) field, it lays down a much-needed marker of progress to date and provides a platform for informed and coherent future analysis and research of the subject. -/- The publication comes at a time of heightened worldwide concern over the standard of science and mathematics education, attended by fierce debate over how best to reform curricula and enliven student engagement in the subjects There is a growing recognition among educators and policy makers that the learning of science must dovetail with learning about science; this handbook is uniquely positioned as a locus for the discussion. -/- The handbook features sections on pedagogical, theoretical, national, and biographical research, setting the literature of each tradition in its historical context. Each chapter engages in an assessment of the strengths and weakness of the research addressed, and suggests potentially fruitful avenues of future research. A key element of the handbook’s broader analytical framework is its identification and examination of unnoticed philosophical assumptions in science and mathematics research. It reminds readers at a crucial juncture that there has been a long and rich tradition of historical and philosophical engagements with science and mathematics teaching, and that lessons can be learnt from these engagements for the resolution of current theoretical, curricular and pedagogical questions that face teachers and administrators. (shrink)

Under the traditional system of peer-reviewed publication, the degree of prestige conferred to authors by successful publication is tied to the degree of the intellectual rigor of its peer review process: ambitious scientists do well professionally by doing well epistemically. As a result, we should expect journal editors, in their dual role as epistemic evaluators and prestige-allocators, to have the power to motivate improved author behavior through the tightening of publication requirements. Contrary to this expectation, I will argue that the (...) publication bias literature in academic medicine demonstrates that editor interventions have had limited effectiveness in improving the health of the publication and trial registration record, suggesting that much stronger interventions are needed. (shrink)

An empirically sensitive formulation of the norms of transformative criticism must recognize that even public and shared standards of evaluation can be implemented in ways that unintentionally perpetuate and reproduce forms of social bias that are epistemically detrimental. Helen Longino’s theory can explain and redress such social bias by treating peer evaluations as hypotheses based on data and by requiring a kind of perspectival diversity that bears, not on the content of the community’s knowledge claims, but on the beliefs and (...) norms of the culture of the knowledge community itself. To illustrate how socializing cognition can bias evaluations, we focus on peer-review practices, with some discussion of peer-review practices in philosophy. Data include responses to surveys by editors from general philosophy journals, as well as analyses of reviews and editorial decisions for the 2007 Cognitive Science Society Conference. (shrink)

Charles Perrow used the term normal accidents to characterize a type of catastrophic failure that resulted when complex, tightly coupled production systems encountered a certain kind of anomalous event. These were events in which systems failures interacted with one another in a way that could not be anticipated, and could not be easily understood and corrected. Systems of the production of expert knowledge are increasingly becoming tightly coupled. Unlike classical science, which operated with a long time horizon, many current forms (...) of expert knowledge are directed at immediate solutions to complex problems. These are prone to breakdowns like the kind discussed by Perrow. The example of the Homestake mine experiment shows that even in modern physics complex systems can produce knowledge failures that last for decades. The concept of knowledge risk is introduced, and used to characterize the risk of failure in such systems of knowledge production. (shrink)

Thesis (Ph. D.)--University of Hawaii at Manoa, 2005.

“Quality” refers nominatively to a standard of performance. Quality is the central idea that differentiates speech protected by academic freedom (the right to worthwhile utterances) from constitutionally protected speech (the right to say anything at all). Extant documents and discussions state that professional peers determine quality based on norms of a field. But professional peers deem utterances and activities as consonant with quality only in reference to criteria that establish meaning of the term. In the absence of articulation, these criteria (...) are ambiguous. Consequently, there exists recurrent confusion about what faculty members have a defensible right to say and do. This article develops an ontology of quality in reference to higher education teaching, a component of academic careers generally not subject to extensive peer review and where instructors thereby exercise considerable autonomy. The ontology identifies three criteria that bound quality: _constraint, context,_ and _amplitude_. Boundedness exists only insofar as boundaries are controlled. The article examines two types of problems in professional control that affect quality: _slippage_ and _overreach_. Both are instances of organizational deviance and abrogation of professional ethics. It is argued that the patterns threaten the structural integrity and public confidence of faculty, fields, and higher education institutions. (shrink)

The main aim of this article is to start a discussion of social pattern, a term that is commonly used in sociology but not specified or defined. The key question can be phrased as follows: Is it possible to transform the notion of social pattern from its current status in sociology as a proto-concept into a fully worked out concept? And if so, how can this be done? To provide material for the discussion we begin by introducing a few different (...) types of patterns that are currently being used (patterns in nature, cultural patterns, statistical patterns, and computationally generated patterns). This is followed by a suggestion for what a strictlysociologicalconcept of social pattern may look like. A useful and theoretically solid concept of social pattern can in our view be constructed by basing it on Weber’s concept of social action. This means thatboththe behavior of the actorsandthe meaning these invest their behavior with must be taken into account. The article ends with a brief discussion of how to use the concept of social patterns in an effective way and what may endanger such a use. (shrink)

In this paper, we analyze the role of science and technology studies journal editors in organizing and maintaining the peer review economy. We specifically conceptualize peer review as a gift economy running on perpetually renewed experiences of mutual indebtedness among members of an intellectual community. While the peer review system is conventionally presented as self-regulating, we draw attention to its vulnerabilities and to the essential curating function of editors. Aside from inherent complexities, there are various shifts in the broader political–economic (...) and sociotechnical organization of scholarly publishing that have recently made it more difficult for editors to organize robust cycles of gift exchange. This includes the increasing importance of journal metrics and associated changes in authorship practices; the growth and differentiation of the STS journal landscape; and changes in publishing funding models and the structure of the publishing market through which interactions among authors, editors, and reviewers are reconfigured. To maintain a functioning peer review economy in the face of numerous pressures, editors must balance contradictory imperatives: the need to triage intellectual production and rely on established cycles of gift exchange for efficiency, and the need to expand cycles of gift exchange to ensure the sustainability and diversity of the peer review economy. (shrink)

Scholars belong to multiple communities of credit simultaneously. When these communities disagree about a scholarly achievement’s credit assignment, this raises a puzzle for decision and game theor...

Peer review of journal submissions has become one of the most important pillars of quality management in academic publishing. Because of growing concerns with the quality and effectiveness of the system, a host of enthusiastic innovators has proposed and experimented with new procedures and technologies. However, little is known about whether these innovations manage to convince other journal editors. This paper will address open questions regarding the implementation of new review procedures, the occurrence rate of various peer review procedures and (...) their distribution over scientific disciplines or academic publishers, as well as the motivations for editors or publishers to engage in novel review procedures. It shows that in spite of enthusiastic innovation, the adoption of new peer review procedures is in fact very slow, with the exception of text similarity scanners. For now, peer review innovations appear to be restricted to specific niches in academic publishing. Analysing these niches, the article concludes with a reflection on the circumstances in which innovations might be more widely implemented. (shrink)

Rooted in altruism theory, the purpose of the double-blind academic journal peer-review process is to: assess the quality of scientific research, minimize the potential for nepotism, and; advance the standards of research through high-quality, constructive feedback. However, considering the limited, if any, public recognition and monetary incentives that referees receive for reviewing manuscripts, academics are often reluctant to squander their limited time toward peer reviewing manuscripts. If they do accept such invitations, referees, at times, do not invest the appropriate time (...) needed and, as a result, scantily review manuscripts, which adversely affects the quality of the review. In addition, given that authors’ identities are not blind to journal editors, there is the potential for bias toward well-established academics from highly-ranked institutions. As a result of these issues, the aims of the academic journal review process are currently not being fulfilled. To rectify these issues, several recommendations, namely: single-blind the editors, pay reviewers, standardize the review process, increase the acceptance standards at academic conferences, and provide constructive feedback, are offered. (shrink)

Born in the 17th century, journal peer review is an extremely diverse technology, constantly torn between two often incompatible goals: the validation of manuscripts conceived as a collective industrial-like reproducible process performed to assert scientific statements, and the dissemination of articles considered as a means to spur scientific discussion, raising controversies, and civically challenging a state of knowledge. Such a situation is particularly conducive to clarifying the processes of valuation and evaluation in journal peer review. In this article, such processes (...) are considered as specific tests in order to emphasize the uncertain properties of pre-tests manuscripts. On the one hand, evaluation tests are examined at the core of the validation of manuscripts, such as defining the coordination of judging instances or controlling the modalities of inter-knowledge between reviewers and authors. They are also studied regarding the dissemination of articles, notably through the contemporary conception of a continuing evaluation test termed “post publication peer review”. On the other hand, valuation tests are both part of the validation of manuscripts, such as the weighting of different judgments of the same manuscript and the tensions that these hierarchies cause, and of the dissemination of articles, such as attention metrics recording the uses of articles. The conclusion sketches out how the articulation of these different tests has recently empowered readers as a new key judging instance for dissemination and for validation, potentially transforming the definition of peers, and thus the whole process of journal peer review. (shrink)

Select groups and organizations embrace practices that perpetuate their inferiority. The result is the phenomenon we call “mediocrity.” This article examines the conditions under which mediocrity is selected and maintained by groups over time. Mediocrity is maintained by a key social process: the marginalization of the adept, which is a response to the group problem of what to do with the highly able. The problem arises when a majority of a group is comprised of average members who must decide what (...) to do with high performers in the group. To solve this problem, reward systems are subverted to benefit the less able and the adept are cast as deviant. Marginalization is a resolution of two tensions: marginalization of the adept for their behavior, and protection from the adept for the mediocre. The American research university is used as an example to describe the phenomenon and to formulate a theoretic argument. The forms and consequences of marginalization are discussed. Marginalizing the adept illustrates an anti-meritocratic behavioral pattern which serves to sustain social systems on which all people, however able, depend. (shrink)

When the journal Minerva was founded in 1962, science and higher educational issues were high on the agenda, lending impetus to the interdisciplinary field of “Science Studies” qua “Science Policy Studies.” As government expenditures for promoting various branches of science increased dramatically on both sides of the East-West Cold War divide, some common issues regarding research management also emerged and with it an interest in closer academic interaction in the areas of history and policy of science. Through a close reading (...) of many early issues of Minerva but also of its later competitor journal Science Studies (now called Social Studies of Science) the paper traces the initial optimism of an academically based Science Studies dialogue across the Cold War divide and the creation in 1971 of the International Commission for Science Policy Studies as a bridging forum, one that Minerva strangely chose to ignore. In this light, attention is drawn to aspects of the often forgotten history of Science Studies in the former Soviet Union and the Eastern European block. Reviewed also are several early discussions that are still relevant today: e.g., regarding differing concepts of Big Science, science and democracy, autonomy in higher education and what conditions are necessary to sustain academic freedom and scientific integrity (some of Edward Shils’ primary concerns). Finally, it is noted how the question of quantitative methods to measure scientific productivity lay at the heart of a “Science of Science” movement of the 1960s has re-emerged in a new form integral to the notion of a “Science of Science Policy.”. (shrink)

This article argues that various deviations from the basic principles of the scientific ethos – primarily the appearance of pseudoscience in scientific communities – can be formulated and explained using specific models of game theory, such as the prisoner’s dilemma and the iterated prisoner’s dilemma. The article indirectly tackles the deontology of scientific work as well, in which it is assumed that there is no room for moral skepticism, let alone moral anti-realism, in the ethics of scientific communities. Namely, on (...) the basis of the generally accepted dictum of scientific endeavor as the pursuit of knowledge exclusively for knowledge’s sake, scientifically »right« behavior is seen to be clearly defined and distinguishable from scientifically »wrong« behavior. After elucidating the basic principles of game theory, the article illustrates – by using imaginary and real cases, as well as some views from the philosophyof biology (the units of selection debate) – how this sort of reasoning could be applied in an analysis of the functioning of science. (shrink)

Large differences in scientific productivity between male and female researchers have not yet been explained satisfactorily. This study finds that child care and lack of research collaboration are the two factors that cause significant gender differences in scientific publishing. Women with young children and women who do not collaborate in research with other scientists are clearly less productive than both their male and female colleagues.

Recently, mainstream philosophy journals have tended to implement more and more stringent forms of peer review, probably in an attempt to prevent editorial decisions that are based on factors other than quality. Against this trend, we propose that journals should relax their standards of acceptance, as well as be less restrictive about whom is to decide what is admitted into the debate. We start by arguing, partly on the basis of the history of peer review in the journal Mind, that (...) past and current peer review practices attest to partisanship with respect to philosophical approach. Then, we explain that such partisanship conflicts with the standard aims of peer review, and that it is both epistemically and morally problematic. This assessment suggests that, if feasible, journals should treat all available and proposed standards of acceptance in philosophy as epistemically equal, and that philosophical work should be evaluated in terms of the novelty and significance of its contribution to developing thought in ways that are of value. Finally, we show, in a programmatic way, that improving the current situation is feasible, and can be done fairly easily. (shrink)

For over 50 years, Minerva has been one of the leading independent journals in the study of ‘science, learning and policy’. Its pages have much to say about the origins and conduct of the ‘intellectual Cold War’, the defence of academic freedom, the emergence of modernization theory, and pioneering strategies in the social studies of science. This paper revisits Minerva through the life and times of its founding Editor, Edward Shils, and traces his influence on its early years – from (...) its association with the Congress for Cultural Freedom in the 1950s, to the higher reaches of research policy in the 1990s. At the close of his life, Minerva continued to espouse Shils’ commitment to what he saw as the fundamental Enlightenment traditions of consensus, civility, and community. In the 21st century, with the achievements of science producing rapid change in every walk of life, his legacy so far retains an established place in the history of scholarship. Whether that legacy will endure – and if so, what role Minerva will play in its defence – remain key questions for the coming generation. (shrink)

The nature of authority is shifting in online scholarly communication. This examination of the history and future of peer review argues that effective online communication requires the development of an open, community?oriented, post?publication system of peer?to?peer review, transforming peer review from a process focused on gatekeeping to one concerned with filtering the wealth of scholarly material made available via the Internet.

This article examines the evolution of peer review and the modern editorial processes of scholarly journals by analyzing a novel data set derived from the Royal Society’s archives and covering 1865-1965, that is, the historical period in which refereeing became firmly established. Our analysis reveals how the Royal Society’s editorial processes coped with both an increasing reliance on refereeing and a growth in submissions, while maintaining collective responsibility and minimizing research waste. By engaging more of its fellows in editorial activity, (...) the society was able to establish an equilibrium of number of submissions per reviewer that was relatively stable over time. Nevertheless, our analysis shows that the distribution of editorial work was significantly uneven. Our findings reveal interesting parallels with current concerns about the scale and distribution of peer review work and suggest the strategic importance of the management of the editorial process to achieve a creative mix of community commitment and professional responsibility that is essential in contemporary journals. (shrink)

BackgroundTriggered by a series of controversies and diversifying expectations of editorial practices, several innovative peer review procedures and supporting technologies have been proposed. However, adoption of these new initiatives seems slow. This raises questions about the wider conditions for peer review change and about the considerations that inform decisions to innovate. We set out to study the structure of commercial publishers’ editorial process, to reveal how the benefits of peer review innovations are understood, and to describe the considerations that inform (...) the implementation of innovations.MethodsWe carried out field visits to the editorial office of two large academic publishers housing the editorial staff of several hundreds of journals, to study their editorial process, and interviewed editors not affiliated with large publishers. Field notes were transcribed and analysed using coding software.ResultsAt the publishers we analysed, the decision-making structure seems to show both clear patterns of hierarchy and layering of the different editorial practices. While information about new initiatives circulates widely, their implementation depends on assessment of stakeholder’s wishes, impact on reputation, efficiency and implementation costs, with final decisions left to managers at the top of the internal hierarchy. Main tensions arise between commercial and substantial arguments. The editorial process is closely connected to commercial practices of creating business value, and the very specific terms in which business value is understood, such as reputation considerations and the urge to increase efficiency. Journals independent of large commercial publishers tend to have less hierarchically structured processes, report more flexibility to implement innovations, and to a greater extent decouple commercial and editorial perspectives.ConclusionOur study demonstrates that peer review innovations are partly to be understood in light of commercial considerations related to reputation, efficiency and implementations costs. These arguments extend beyond previously studied topics in publishing economics, including publishers’ choice for business or publication models and reach into the very heart of the editorial and peer review process. (shrink)

This article analyzes the transformation of Minerva from an intellectual towards a scholarly journal by making use of bibliometric methods. The aim is to provide some empirical insights that help to understand what properties of the journal changed in the course of this transformation process. Minerva was one of the first journals that reflected on science and its role in society and science policy in particular. Analyzing the development of the journal sheds light on the emergence of science (policy) studies (...) and on Minerva’s role as a forerunner in this field. In a first step, the methods will be described. The second section provides some empirical results of the publication output of Minerva and its relations to other journals in the field. The empirical findings are put into a broader perspective in the concluding third section. (shrink)

Prediction markets, which trade contracts based on the results of predictions, have been remarkably successful in predicting the results of political events. A number of proposals have been made to extend prediction markets to scientific questions, and some small-scale science prediction markets have been implemented. Advocates for science prediction markets argue that they could alleviate problems in science such as bias in peer review and epistemically unjustified consensus. I argue that bias in peer review and epistemically unjustified consensuses are genuine (...) problems in science, and that current attempts to solve them fail in practical circumstances. Prediction markets do show some promise for answering scientific questions while avoiding the pitfalls of consensus and peer review. However, there are strong reasons to believe that science prediction markets will not perform nearly so well as election prediction markets, and implementing prediction markets on a large scale could have significant unintended consequences for the organization of scientific research. The promise of prediction markets to solve current problems in assessing scientific claims is therefore largely illusory. (shrink)

This article aims to discuss the contemporary activities and roles that scientific associations play in science and society. It is based on a comprehensive study of scientific associations in Portugal, relying on a multi-method, quantitative and qualitative approach. After a brief review of the literature on associations in the social studies of science, we provide an outline of the expanding field of scientific associations in Portugal. We then proceed to present and discuss the five main roles of associations identified through (...) the research: communication among peers, promotion of research, science dissemination, representation of professional interests and policy advice. We conclude that the external roles of associations have become more important than the internal ones. Whereas the internationalisation of science has moved the communication, collaboration and competition between researchers into the transnational sphere, the links that associations forge between science and other social spheres are still deeply rooted in national settings and much dependant on specific configurations and practices by government, business and other social actors. (shrink)

This paper presents a framework that editors, peer reviewers, and authors can use to identify and resolve efficiently disputes that arise during peer review in scientific journals. The framework is called a scientific dialectical brief. In this framework, differences among authors and reviewers are formatted into specific assertions and the support each party provides for its position. A literature review suggests that scientists use five main types of support; empirical data, reasoning, speculation, feelings, and status. It is suggested that the (...) scientific dialectical brief format can streamline the review process by facilitating rapid differentiation between stronger and weaker support, so that valuable time can be focused on the better-substantiated claims. The paper concludes with some suggestions for implementation. (shrink)

Psychometrically oriented researchers construe low inter-rater reliability measures for expert peer reviewers as damning for the practice of peer review. I argue that this perspective overlooks different forms of normatively appropriate disagreement among reviewers. Of special interest are Kuhnian questions about the extent to which variance in reviewer ratings can be accounted for by normatively appropriate disagreements about how to interpret and apply evaluative criteria within disciplines during times of normal science. Until these empirical-cum-philosophical analyses are done, it will remain (...) unclear the extent to which low inter-rater reliability measures represent reasonable disagreement rather than arbitrary differences between reviewers. (shrink)

Despite continued attention, finding adequate criteria for distinguishing “good” from “bad” scholarly journals remains an elusive goal. In this essay, I propose a solution informed by the work of Imre Lakatos and his methodology of scientific research programmes (MSRP). I begin by reviewing several notable attempts at appraising journal quality – focusing primarily on the impact factor and development of journal blacklists and whitelists. In doing so, I note their limitations and link their overarching goals to those found within the (...) philosophy of science. I argue that Lakatos’s MSRP and specifically his classifications of “progressive” and “degenerative” research programmes can be analogized and repurposed for the evaluation of scholarly journals. I argue that this alternative framework resolves some of the limitations discussed above and offers a more considered evaluation of journal quality – one that helps account for the historical evolution of journal-level publication practices and attendant contributions to the growth (or stunting) of scholarly knowledge. By doing so, the seeming problem of journal demarcation is diminished. In the process I utilize two novel tools (the mistake index and scite index) to further illustrate and operationalize aspects of the MSRP. (shrink)

This study seeks to examine the impact of alumni connections between the evaluators and evaluatees on the results of peer review ratings for the Korean national R&D project and selection success rate. Specifically, this study analyzed the evaluation results of 8,402 research proposal entries submitted between 2007 and 2011 for the “general researchers support project,” all in the Natural Science and Engineering areas and sponsored by the National Research Foundation of Korea. Each proposal entry was evaluated by three evaluators, and (...) approximately 39 percent of the proposals had at least one evaluator from the same university. The results of this study showed that evaluators have a tendency to give relatively high scores to research proposals submitted by the alumni of the same universities as their alma mater. Also, when an evaluator from the same university as an evaluatee was included in the evaluator group, the results of this study showed that the percentage of entry submissions was higher compared to the contrary. Such results show that in the process of peer review–based research proposal evaluations for national R&D projects, alumni connections have significant influence on evaluation results in South Korea. (shrink)

Increasing the “truth per dollar” of money spent on science is one legitimate long-run goal of the economics of science. But before this goal can be achieved, we need to increase our knowledge of the successes and failures of past and current reward structures of science. This essay reviews what economists have learned about the behavior of scientists and the reward structure of science. One important use of such knowledge will be to help policy-makers create a reward structure that is (...) more efficient in the future. (shrink)