An empirically sensitive formulation of the norms of transformative criticism must recognize that even public and shared standards of evaluation can be implemented in ways that unintentionally perpetuate and reproduce forms of social bias that are epistemically detrimental. Helen Longino’s theory can explain and redress such social bias by treating peer evaluations as hypotheses based on data and by requiring a kind of perspectival diversity that bears, not on the content of the community’s knowledge claims, but on the beliefs and (...) norms of the culture of the knowledge community itself. To illustrate how socializing cognition can bias evaluations, we focus on peer-review practices, with some discussion of peer-review practices in philosophy. Data include responses to surveys by editors from general philosophy journals, as well as analyses of reviews and editorial decisions for the 2007 Cognitive Science Society Conference. (shrink)

is normative in the sense that it aims to make recommendations for improving human judgment; it aims to have a practical impact on morally and politically significant human decisions and actions; and it studies normative, rational judgment qua rational judgment. These nonstandard ways of understanding ACP as normative collectively suggest a new interpretation of the strong replacement thesis that does not call for replacing normative epistemic concepts, relations, and inquiries with descriptive, causal ones. Rather, it calls for recognizing that the (...) aims and normative inquiries of epistemology and normative psychology have become intermutual in nature. Key Words: Heuristics and biases • applied cognitive psychology • normative psychology • rationality • naturalized epistemology • Epistemics • Applied Naturalized Epistemology • strong replacement • strategic reliabilism • ameliorative psychology. (shrink)

Scientific editors’ policies, including peer review, are based mainly on tradition and belief. Do they actually achieve their desired effects, the selection of the best manuscripts and improvement of those published? Editorial decisions have important consequences—to investigators, the scientific community, and all who might benefit from correct information or be harmed by misleading research results. These decisions should be judged not just by intentions of reviewers and editors but also by the actual consequences of their actions. A small but growing (...) number of studies has put editorial policies to a strong scientific test. In a randomized, controlled trial. blinding reviewers to author and institution was usually successful and improved the quality of reviews. Two studies have shown that, contrary to conventional wisdom, reviewers early in their careers give better reviews than senior reviewers. Many studies have shown low agreement between reviewers but there is disagreement about whether this represents a failing of peer review or the expected and valuable effect of choosing reviewers with complementary expertise. In a study of whether manuscripts are improved by peer review and editing, articles published in Annals of Internal Medicine were improved in 33 of 34 dimensions of reporting quality, but published articles still had room for improvement. Because of the central place of peer review in the scientific community and the resources it requires, more studies are needed to define what it does and does not accomplish. (shrink)

Scientific editors’ policies, including peer review, are based mainly on tradition and belief. Do they actually achieve their desired effects, the selection of the best manuscripts and improvement of those published? Editorial decisions have important consequences—to investigators, the scientific community, and all who might benefit from correct information or be harmed by misleading research results. These decisions should be judged not just by intentions of reviewers and editors but also by the actual consequences of their actions. A small but growing (...) number of studies has put editorial policies to a strong scientific test. In a randomized, controlled trial. blinding reviewers to author and institution was usually successful and improved the quality of reviews. Two studies have shown that, contrary to conventional wisdom, reviewers early in their careers give better reviews than senior reviewers. Many studies have shown low agreement between reviewers but there is disagreement about whether this represents a failing of peer review or the expected and valuable effect of choosing reviewers with complementary expertise. In a study of whether manuscripts are improved by peer review and editing, articles published in Annals of Internal Medicine were improved in 33 of 34 dimensions of reporting quality, but published articles still had room for improvement. Because of the central place of peer review in the scientific community and the resources it requires, more studies are needed to define what it does and does not accomplish. (shrink)

Research on bias in peer review examines scholarly communication and funding processes to assess the epistemic and social legitimacy of the mechanisms by which knowledge communities vet and self-regulate their work. Despite vocal concerns, a closer look at the empirical and methodological limitations of research on bias raises questions about the existence and extent of many hypothesized forms of bias. In addition, the notion of bias is predicated on an implicit ideal that, once articulated, raises questions about the normative implications (...) of research on bias in peer review. This review provides a brief description of the function, history, and scope of peer review; articulates and critiques the conception of bias unifying research on bias in peer review; characterizes and examines the empirical, methodological, and normative claims of bias in peer review research; and assesses possible alternatives to the status quo. We close by identifying ways to expand conceptions and studies of bias to countenance the complexity of social interactions among actors involved directly and indirectly in peer review. (shrink)

Manuscript reviews are intended to be objective, empirical assessments of the scientific worth of papers submitted for publication. However, critics have charged that manuscript reviews are unreliable, unconstructive, and biased in a number of ways . A review of the empirical literature in this area indicates: that inter-reviewer reliability in manscript assessments is clearly inadequate, that reviewer bias can sometimes influence manuscript assessments, and that there is a dearth of empirical data supporting the predictive and discriminant validity of manuscript assessment (...) procedures. Based on the available evidence it seems that manuscript reviews are more strongly influenced by chance factors than by systematic reviewer or editorial bias. Nonetheless, our desire to conceptualize manuscript reviews in psychology as objective, empirical assessments has produced a number of undesired results. An alternative approach to manuscript review based on an adversery model rather than a scientific model is presented. Advantages of an adversary model as a method for identifying sound research are discussed. Changes in current publication policies that would allow research findings to be disseminated more efficiently are also described. (shrink)

Rychlak and Rychlak make a number of interesting points regarding my proposal for a model of manuscript review that is based on an adversary approach rather than on the traditional "empirical assessment" approach . In this paper, I address three issues raised by Rychlak and Rychlak: how the "burden of proof" should be allocated in the adversary model; what definition of "proof" would work best within the framework of the model; and how the adversary model changes the relationship of author, (...) reviewer and editor during the manuscript review process. Continued discussion of the strengths and weaknesses of the adversary model-and of the traditional manuscript review system-is clearly warranted. (shrink)