Peer reviewers at many funding agencies and scientific journals are asked to score submissions both on individual criteria and overall. The overall scores should be some kind of aggregate of the criteria scores. Carole Lee identifies this as a potential locus for bias to enter the peer review process, which she calls commensuration bias. Here I view the aggregation of scores through the lens of social choice theory. I argue that, when reviewing grant proposals, it is in many cases impossible (...) to avoid commensuration bias. (shrink)

Prepublication peer review should be abolished. We consider the effects that such a change will have on the social structure of science, paying particular attention to the changed incentive structure and the likely effects on the behaviour of individual scientists. We evaluate these changes from the perspective of epistemic consequentialism. We find that where the effects of abolishing prepublication peer review can be evaluated with a reasonable level of confidence based on presently available evidence, they are either positive or neutral. (...) We conclude that on present evidence abolishing peer review weakly dominates the status quo. (shrink)

Formal models of scientific inquiry, aimed at capturing socio-epistemic aspects underlying the process of scientific research, have become an important method in formal social epistemology and philosophy of science. In this introduction to the special issue we provide a historical overview of the development of formal models of this kind and analyze their methodological contributions to discussions in philosophy of science. In particular, we show that their significance consists in different forms of ‘methodological iteration’ whereby the models initiate new lines (...) of inquiry, isolate and clarify problems with existing knowledge claims, and stimulate further research. (shrink)

This paper offers a new angle on the common idea that the process of science does not support epistemic diversity. Under minimal assumptions on the nature of journal editing, we prove that editorial procedures, even when impartial in themselves, disadvantage less prominent research programs. This purely statistical bias in article selection further skews existing differences in the success rate and hence attractiveness of research programs, and exacerbates the reputation difference between the programs. After a discussion of the modeling assumptions, the (...) paper ends with a number of recommendations that may help promote scientific diversity through editorial decision making. (shrink)

Remco Heesen has recently argued in favor of the editorial practice of triple-anonymous review on the grounds that ``an injustice is committed against certain authors'' under non-anonymous review. On the other hand, he concedes that the information waste of triple-anonymous review does handicap editors, in particular sacrificing a boost in the average quality of accepted papers that would otherwise be conferred by non-anonymous review. In this paper it is observed that by devoting comparatively greater reviewing resources to the papers of (...) unfamiliar authors, editors practicing non-anonymous review can, without loss of information, avoid subjecting authors to the sorts of injustices observed by Heesen. Thus they can reap the efficiency gains of non-anonymous review without sacrificing fairness. (shrink)