Research on bias in peer review examines scholarly communication and funding processes to assess the epistemic and social legitimacy of the mechanisms by which knowledge communities vet and self-regulate their work. Despite vocal concerns, a closer look at the empirical and methodological limitations of research on bias raises questions about the existence and extent of many hypothesized forms of bias. In addition, the notion of bias is predicated on an implicit ideal that, once articulated, raises questions about the normative implications (...) of research on bias in peer review. This review provides a brief description of the function, history, and scope of peer review; articulates and critiques the conception of bias unifying research on bias in peer review; characterizes and examines the empirical, methodological, and normative claims of bias in peer review research; and assesses possible alternatives to the status quo. We close by identifying ways to expand conceptions and studies of bias to countenance the complexity of social interactions among actors involved directly and indirectly in peer review. (shrink)

An empirically sensitive formulation of the norms of transformative criticism must recognize that even public and shared standards of evaluation can be implemented in ways that unintentionally perpetuate and reproduce forms of social bias that are epistemically detrimental. Helen Longino’s theory can explain and redress such social bias by treating peer evaluations as hypotheses based on data and by requiring a kind of perspectival diversity that bears, not on the content of the community’s knowledge claims, but on the beliefs and (...) norms of the culture of the knowledge community itself. To illustrate how socializing cognition can bias evaluations, we focus on peer-review practices, with some discussion of peer-review practices in philosophy. Data include responses to surveys by editors from general philosophy journals, as well as analyses of reviews and editorial decisions for the 2007 Cognitive Science Society Conference. (shrink)

In a recent article in the British Journal for the Philosophy of Science, Heesen and Bright argue that prepublication peer review should be abolished and replaced with postpublication peer review (provided the matter is judged purely on epistemic grounds). In this article, I show that there are three problems with their argument. First, it fails to consider the epistemic cost of implementing the change to postpublication peer review. Second, it fails to consider some potential epistemic benefits of prepublication peer review, (...) which involve avoiding bias. Third, it fails to consider some potential epistemic disadvantages of postpublication peer review, which stem from the greater number of papers that would be published under that system. (shrink)

This study seeks to examine the impact of alumni connections between the evaluators and evaluatees on the results of peer review ratings for the Korean national R&D project and selection success rate. Specifically, this study analyzed the evaluation results of 8,402 research proposal entries submitted between 2007 and 2011 for the “general researchers support project,” all in the Natural Science and Engineering areas and sponsored by the National Research Foundation of Korea. Each proposal entry was evaluated by three evaluators, and (...) approximately 39 percent of the proposals had at least one evaluator from the same university. The results of this study showed that evaluators have a tendency to give relatively high scores to research proposals submitted by the alumni of the same universities as their alma mater. Also, when an evaluator from the same university as an evaluatee was included in the evaluator group, the results of this study showed that the percentage of entry submissions was higher compared to the contrary. Such results show that in the process of peer review–based research proposal evaluations for national R&D projects, alumni connections have significant influence on evaluation results in South Korea. (shrink)

BackgroundTriggered by a series of controversies and diversifying expectations of editorial practices, several innovative peer review procedures and supporting technologies have been proposed. However, adoption of these new initiatives seems slow. This raises questions about the wider conditions for peer review change and about the considerations that inform decisions to innovate. We set out to study the structure of commercial publishers’ editorial process, to reveal how the benefits of peer review innovations are understood, and to describe the considerations that inform (...) the implementation of innovations.MethodsWe carried out field visits to the editorial office of two large academic publishers housing the editorial staff of several hundreds of journals, to study their editorial process, and interviewed editors not affiliated with large publishers. Field notes were transcribed and analysed using coding software.ResultsAt the publishers we analysed, the decision-making structure seems to show both clear patterns of hierarchy and layering of the different editorial practices. While information about new initiatives circulates widely, their implementation depends on assessment of stakeholder’s wishes, impact on reputation, efficiency and implementation costs, with final decisions left to managers at the top of the internal hierarchy. Main tensions arise between commercial and substantial arguments. The editorial process is closely connected to commercial practices of creating business value, and the very specific terms in which business value is understood, such as reputation considerations and the urge to increase efficiency. Journals independent of large commercial publishers tend to have less hierarchically structured processes, report more flexibility to implement innovations, and to a greater extent decouple commercial and editorial perspectives.ConclusionOur study demonstrates that peer review innovations are partly to be understood in light of commercial considerations related to reputation, efficiency and implementations costs. These arguments extend beyond previously studied topics in publishing economics, including publishers’ choice for business or publication models and reach into the very heart of the editorial and peer review process. (shrink)

Peer review of journal submissions has become one of the most important pillars of quality management in academic publishing. Because of growing concerns with the quality and effectiveness of the system, a host of enthusiastic innovators has proposed and experimented with new procedures and technologies. However, little is known about whether these innovations manage to convince other journal editors. This paper will address open questions regarding the implementation of new review procedures, the occurrence rate of various peer review procedures and (...) their distribution over scientific disciplines or academic publishers, as well as the motivations for editors or publishers to engage in novel review procedures. It shows that in spite of enthusiastic innovation, the adoption of new peer review procedures is in fact very slow, with the exception of text similarity scanners. For now, peer review innovations appear to be restricted to specific niches in academic publishing. Analysing these niches, the article concludes with a reflection on the circumstances in which innovations might be more widely implemented. (shrink)

Peer review is often taken to be the main form of quality control on academic research. Usually journals carry this out. However, parts of maths and physics appear to have a parallel, crowd-sourced model of peer review, where papers are posted on the arXiv to be publicly discussed. In this paper we argue that crowd-sourced peer review is likely to do better than journal-solicited peer review at sorting papers by quality. Our argument rests on two key claims. First, crowd-sourced peer (...) review will lead on average to more reviewers per paper than journal-solicited peer review. Second, due to the wisdom of the crowds, more reviewers will tend to make better judgments than fewer. We make the second claim precise by looking at the Condorcet Jury Theorem as well as two related jury theorems developed specifically to apply to peer review. (shrink)

Business ethics journals have appeared on a few ranked lists that are specific to this niche discipline. As with more traditional academic disciplines, these rankings are used for academic rewards such as faculty tenure and promotion, along with department and school ratings. Journal ranking has been subject to considerable criticism even as its administrative use persists. Among the criticisms are that journal quality is a poor proxy for article quality, citation rate is an imperfect reflection of article influence, and bias (...) may be introduced into rankings by visibility characteristics such as journal age, size, circulation, and experience of the rater with a journal. This research note studies the effect of journal age and size on the rankings of business ethics journals compiled by Beets, Lewis, and Brower, by Albrecht, Thompson, Hoopes, and Rodrigo, and by Serenko and Bontis. Significant correlation was found for journal age with the administratively derived Beets et al. ranking. No significant correlation was found for size in any ranking study. Results were not significant for the Albrecht et al. and the Serenko and Bontis rankings representing the perspectives from surveys of active researchers or citation analysis. Perhaps sometimes a journal’s reputation precedes it, as perception of journal quality may be biased by journal visibility, either because it has been published and available for a number of years, or because it is well known and likely to be cited. (shrink)

It has been argued that implicit biases are operative in philosophy and lead to significant epistemic costs in the field. Philosophers working on this issue have focussed mainly on implicit gender and race biases. They have overlooked ideological bias, which targets political orientations. Psychologists have found ideological bias in their field and have argued that it has negative epistemic effects on scientific research. I relate this debate to the field of philosophy and argue that if, as some studies suggest, the (...) same bias also exists in philosophy then it will lead to hitherto unrecognised epistemic hazards in the field. Furthermore, the bias is epistemically different from the more familiar biases in respects that are important for epistemology, ethics, and metaphilosophy. (shrink)

ABSTRACT:This study examines whether the empirical evidence on the relationship between corporate social performance (CSP) and corporate financial performance (CFP) differs depending on the publication outlet in which that evidence appears. This moderator meta-analysis, based on a total sample size of 33,878 observations, suggests that published CSP-CFP findings have been shaped by differences in institutional logics in different subdisciplines of organization studies. In economics, finance, and accounting journals, the average correlations were only about half the magnitude of the findings published (...) in Social Issues in Management, Business Ethics, or Business and Society journals (mean corrected correlation coefficientof.22 vs..49, respectively). Specifically, economists did not find null or negative CSP-CFP correlations, and average findings published in general management outlets (=.41) were closer to Social Issues in Management, Business Ethics, and Business and Society results than to findings reported in economics, finance, and accounting journals. (shrink)

Implicit bias results from living in a society structured by race. Tamar Gendler has drawn attention to several epistemic costs of implicit bias and concludes that paying some costs is unavoidable. In this paper, we reconstruct Gendler’s argument and argue that the epistemic costs she highlights can be avoided. Though epistemic agents encode discriminatory information from the environment, not all encoded information is activated. Agents can construct local epistemic environments that do not activate biasing representations, effectively avoiding the consequences of (...) activation. We conclude that changing our local environments provides a way to avoid paying implicit bias’s epistemic costs. (shrink)

A growing body of literature has identified potential problems that can compromise the quality, fairness, and integrity of journal peer review, including inadequate review, inconsistent reviewer reports, reviewer biases, and ethical transgressions by reviewers. We examine the evidence concerning these problems and discuss proposed reforms, including double-blind and open review. Regardless of the outcome of additional research or attempts at reforming the system, it is clear that editors are the linchpin of peer review, since they make decisions that have a (...) significant impact on the process and its outcome. We consider some of the steps editors should take to promote quality, fairness and integrity in different stages of the peer review process and make some recommendations for editorial conduct and decision-making. (shrink)

On the surface, developing a social psychology of science seems compelling as a way to understand how individual social cognition – in aggregate – contributes towards individual and group behavior within scientific communities (Kitcher, 2002). However, in cases where the functional input-output profile of psychological processes cannot be mapped directly onto the observed behavior of working scientists, it becomes clear that the relationship between psychological claims and normative philosophy of science should be refined. For example, a robust body of social (...) psychological research demonstrates implicit gender bias in the evaluation of others (e.g., Steinpreis, Anders, & Ritzke, 1999). Many expected implicit bias to be a major cause of women’s underrepresentation in math intensive fields of science; however, quantitative sociological research of hiring and manuscript and grant evaluation has discovered no gender disparity in outcomes (Ceci & Williams, 2011). Why might this be so? This paper will discuss methodological challenges that go beyond classic problems of external validity in extrapolating psychological effects and explanations to scientific communities. These problems include more complex external validity issues raised by the introduction of multi-process models of cognition (e.g., implicit versus explicit social cognition) as well as the reflexive role that folk and experimental theories of social psychology play in guiding the behavior of scientists at the individual and community level. (shrink)

The Gendered Conference Campaign seeks to reduce the prevalence of conferences at which the keynote speakers are all male. Such conferences, according to proponents of the campaign, stereotype philosophy as male, contribute to implicit bias against women and perpetuate stereotype threat. I argue, first, that if a more diverse list of keynote speakers were the correct way to counter harms such as implicit bias and stereotype threat, then a Gendered Conference Campaign would not be the solution. The campaign would need (...) to include other groups that are the victims of implicit bias and stereotype within philosophy. Other challenges to proponents of the Gendered Conference Campaign are then presented. (shrink)

This paper summarizes recent and ongoing experimental work regarding the reality, nature, effects, and causes of the underrepresentation of women in academic philosophy. We first present empirical data on several aspects of underrepresentation, and then consider various reasons why this gender imbalance is problematic. We then turn to the published and preliminary results of empirical work aimed at identifying factors that might explain it.

Saul_Jennifer, Scepticism and Implicit Bias.

This essay offers a defense of ‘intellectual authority’, primarily by pointing out the untoward implications of its conceptual opposite, ‘institutional authority’, in a wide variety of contexts. An opening discussion explores conditions for the possibility of intellectual authority in legal, humanistic, and aesthetic disciplines. Social science literature documenting and describing the biasing influence of institutional authority is then canvassed and analyzed in some detail. A final section assays the theoretical significance of various efforts to eliminate non‐intellectual bias and influence, with (...) special reference to the example of “blind reviewing’ of scholarly manuscripts. (shrink)

This paper presents a framework that editors, peer reviewers, and authors can use to identify and resolve efficiently disputes that arise during peer review in scientific journals. The framework is called a scientific dialectical brief. In this framework, differences among authors and reviewers are formatted into specific assertions and the support each party provides for its position. A literature review suggests that scientists use five main types of support; empirical data, reasoning, speculation, feelings, and status. It is suggested that the (...) scientific dialectical brief format can streamline the review process by facilitating rapid differentiation between stronger and weaker support, so that valuable time can be focused on the better-substantiated claims. The paper concludes with some suggestions for implementation. (shrink)

A relatively high incidence of unsatisfactory review decisions is widely recognised and acknowledged as ‘the peer review problem’. Factors contributing to this problem are identified and examined. Specific examples of unreasonable rejection are considered. It is concluded that weaknesses of the ‘peer review’ system are significant and that they are well known or readily recognisable but that necessary counter-measures are not always enforced. Careful management is necessary to discount hollow opinion or error in review comment. Review and referee functions should (...) be quite separate. (shrink)

This paper reviews the published empirical evidence concerning journal peer review consisting of 68 papers, all but three published since 1975. Peer review improves quality, but its use to screen papers has met with limited success. Current procedures to assure quality and fairness seem to discourage scientific advancement, especially important innovations, because findings that conflict with current beliefs are often judged to have defects. Editors can use procedures to encourage the publication of papers with innovative findings such as invited papers, (...) early-acceptance procedures, author nominations of reviewers, structured rating sheets, open peer review, results-blind review, and, in particular, electronic publication. Some journals are currently using these procedures. The basic principle behind the proposals is to change the decision from whether to publish a paper to how to publish it. (shrink)

Lane, K. A., Banaji, M. R., Nosek, B. A., & Greenwald, A. G. (2007). Understanding and using the Implicit Association Test: IV. What we know (so far) (Pp. 59–102). In B. Wittenbrink & N. S. Schwarz (Eds.). Implicit measures of attitudes: Procedures and controversies . New York: Guilford Press. PDF - 652KB ].

The pressure for publication is ever present in academe. Rules for submission are elucidated by conferences, proceedings and journals for the benefit of authors; however, the rules for reviewers and editors are not so well established or consistent. This treatise examines examples of abuse of the editorial process and points to a need for formal recognition of rules for review. The manuscript culminates with proposed Codes of Ethics for researchers, referees and editors and suggestions for improvement of the peer review (...) process. (shrink)

Over the past hundred years, a number of scientific investigators claim to have adduced experimental evidence for phenomena information” seems to behave like a weak signal that has to compete for the information-processing resources of the organism, a reduction of ongoing sensorimotor activity may facilitate ESP detection. Such a meaningful convergence of results suggests that psi phenomena may represent a unitary, coherent process whose nature and compatibility with current physical theory have yet to be determined. The theoretical implications and potential (...) practical applications of psi could be significant, irrespective of the small magnitude of psi effects in laboratory settings. (shrink)

The failure to consider the Sokal affair in light of other, related episodes has contributed to a wholesale misreading of its significance. The episode has often been offered as evidence for the bankruptcy of a broad and diverse collection offields, variously referred to as cultural studies of science, sociology of science, history of science, and science and technology studies. However, when viewed in context, the Sokal affair illustrates pre cisely why social scientific and humanistic studies of science are necessary. To (...) develop this argument, the author explicitly compares Alan Sokal's experiment with a similar experiment, performed by William M. Epstein and published in this very journal. Comparing the research questions, methods, ethics, and reception of these two experi ments not only reveals the limitations of Sokal's critique but also shows that Sokal has unwittingly endorsed one of the central lines of research in science and technology studies. (shrink)

One of the most dramatic techniques for constructing accounts of "undiscovery" or incompetence in science involves the manipulative deception—in some accounts, the "hoaxing"—of the putatively incompetent researcher, ostensibly as an experiment to evaluate his or her methodology and the soundness of his or her knowledge claims. In this article, the author examines five cases in which such deceptions have been employed, noting the patterns of argument that typically follow these deceptions and the factors that seem to determine the power of (...) these arguments. The author suggests that the "hoaxer" in such cases is most likely to be successful when his or her views about the targeted researcher's methodology and knowledge claims are orthodox with respect to his or her intended audience. (shrink)

This article reports the results of a study of confirmational response bias among social work journals. A contrived research paper with positive findings and its negative mirror image were submitted to two different groups of social work journals and to two comparison groups of journals outside social work. The quantitative results, suggesting bias, are tentative; but the qualitative findings based upon an analysis of the referee comments are clear and consistent. Few referees from prestigious or nonprestcgrous social work journals prepared (...) reviews that were knowledgeable, scientifically astute, or objective. The best reviews came from journals outside of soccal work or from journals that are accepted as social work journals but originate with other disciplines. (shrink)

The fallacies, as traditionally conceived, are wrong ways of reasoning that nevertheless appear attractive to us. Recently, however, Woods (2013) has argued that they don’t merit such a title, and that what we take to be fallacies are instead largely virtuous forms of reasoning. This reformation of the fallacies forms part of Woods’ larger project to naturalize logic. In this paper I will look to his analysis of the argumentum ad hominem as a case study for the prospects of this (...) project. I will argue that the empirical literature on implicit bias presents a difficulty for the reformation of the ad hominem as cognitively virtuous. Cases where implicit bias influences our assessment of the truth or claim or argument are instances of ad hominem reasoning, and these qualify as fallacious on Woods’ own definition. (shrink)

Born in the 17th century, journal peer review is an extremely diverse technology, constantly torn between two often incompatible goals: the validation of manuscripts conceived as a collective industrial-like reproducible process performed to assert scientific statements, and the dissemination of articles considered as a means to spur scientific discussion, raising controversies, and civically challenging a state of knowledge. Such a situation is particularly conducive to clarifying the processes of valuation and evaluation in journal peer review. In this article, such processes (...) are considered as specific tests in order to emphasize the uncertain properties of pre-tests manuscripts. On the one hand, evaluation tests are examined at the core of the validation of manuscripts, such as defining the coordination of judging instances or controlling the modalities of inter-knowledge between reviewers and authors. They are also studied regarding the dissemination of articles, notably through the contemporary conception of a continuing evaluation test termed “post publication peer review”. On the other hand, valuation tests are both part of the validation of manuscripts, such as the weighting of different judgments of the same manuscript and the tensions that these hierarchies cause, and of the dissemination of articles, such as attention metrics recording the uses of articles. The conclusion sketches out how the articulation of these different tests has recently empowered readers as a new key judging instance for dissemination and for validation, potentially transforming the definition of peers, and thus the whole process of journal peer review. (shrink)