How should one react when one has a belief, but knows that other people—who have roughly the same evidence as one has, and seem roughly as likely to react to it correctly—disagree? This paper argues that the disagreement of other competent inquirers often requires one to be much less confident in one’s opinions than one would otherwise be.

Peer reviewers at many funding agencies and scientific journals are asked to score submissions both on individual criteria and overall. The overall scores should be some kind of aggregate of the criteria scores. Carole Lee identifies this as a potential locus for bias to enter the peer review process, which she calls commensuration bias. Here I view the aggregation of scores through the lens of social choice theory. I argue that, when reviewing grant proposals, it is in many cases impossible (...) to avoid commensuration bias. (shrink)

A number of formal models, including a highly influential model from Hong and Page, purport to show that functionally diverse groups often beat groups of individually high-performing agents in solving problems. Thompson argues that in Hong and Page’s model, that the diverse groups are created by a random process explains their success, not the diversity. Here, I defend the diversity interpretation of the Hong and Page result. The failure of Thompson’s argument shows that to understand the value of functional diversity, (...) we should be clearer about how we conceive of and measure that diversity. (shrink)

Recent philosophical work has praised the reward structure of science, while recent empirical work has shown that many scientific results may not be reproducible. I argue that the reward structure of science incentivizes scientists to focus on speed and impact at the expense of the reproducibility of their work, thus contributing to the so-called reproducibility crisis. I use a rational choice model to identify a set of sufficient conditions for this problem to arise, and I argue that these conditions plausibly (...) apply to a wide range of research situations. Currently proposed solutions will not fully address this problem. Philosophical commentators should temper their optimism about the reward structure of science. (shrink)

There is growing interest in understanding and eliciting division of labor within groups of scientists. This paper illustrates the need for this division of labor through a historical example, and a formal model is presented to better analyze situations of this type. Analysis of this model reveals that a division of labor can be maintained in two different ways: by limiting information or by endowing the scientists with extreme beliefs. If both features are present however, cognitive diversity is maintained indefinitely, (...) and as a result agents fail to converge to the truth. Beyond the mechanisms for creating diversity suggested here, this shows that the real epistemic goal is not diversity but transient diversity. (shrink)

Social scientists use many different methods, and there are often substantial disagreements about which method is appropriate for a given research question. In response to this uncertainty about the relative merits of different methods, W. E. B. Du Bois advocated for and applied “methodological triangulation”. This is to use multiple methods simultaneously in the belief that, where one is uncertain about the reliability of any given method, if multiple methods yield the same answer that answer is confirmed more strongly than (...) it could have been by any single method. Against this, methodological purists believe that one should choose a single appropriate method and stick with it. Using tools from voting theory, we show Du Boisian methodological triangulation to be more likely to yield the correct answer than purism, assuming the scientist is subject to some degree of diffidence about the relative merits of the various methods. This holds even when in fact only one of the methods is appropriate for the given research question. (shrink)

This article examines two questions about scientists’ search for knowledge. First, which search strategies generate discoveries effectively? Second, is it advantageous to diversify search strategies? We argue pace Weisberg and Muldoon, “Epistemic Landscapes and the Division of Cognitive Labor”, that, on the first question, a search strategy that deliberately seeks novel research approaches need not be optimal. On the second question, we argue they have not shown epistemic reasons exist for the division of cognitive labor, identifying the errors that led (...) to their conclusions. Furthermore, we generalize the epistemic landscape model, showing that one should be skeptical about the benefits of social learning in epistemically complex environments. (shrink)

This is an important book precisely because there is none other quite like it.

In the latter half of the twentieth century, philosophers of science have argued (implicitly and explicitly) that epistemically rational individuals might compose epistemically irrational groups and that, conversely, epistemically rational groups might be composed of epistemically irrational individuals. We call the conjunction of these two claims the Independence Thesis, as they together imply that methodological prescriptions for scientific communities and those for individual scientists might be logically independent of one another. We develop a formal model of scientific inquiry, define four (...) criteria for individual and group epistemic rationality, and then prove that the four definitions diverge, in the sense that individuals will be judged rational when groups are not and vice versa. We conclude by explaining implications of the inconsistency thesis for (i) descriptive history and sociology of science and (ii) normative prescriptions for scientific communities. (shrink)

During the last three decades, reflections on the growth of scientific knowledge have inspired historians, sociologists, and some philosophers to contend that scientific objectivity is a myth. In this book, Kitcher attempts to resurrect the notions of objectivity and progress in science by identifying both the limitations of idealized treatments of growth of knowledge and the overreactions to philosophical idealizations. Recognizing that science is done not by logically omniscient subjects working in isolation, but by people with a variety of personal (...) and social interests, who cooperate and compete with one another, he argues that, nonetheless, we may conceive the growth of science as a process in which both our vision of nature and our ways of learning more about nature improve. Offering a detailed picture of the advancement of science, he sets a new agenda for the philosophy of science and for other "science studies" disciplines. (shrink)

We often ask for the opinion of a group of individuals. How strongly does the scientific community believe that the rate at which sea levels are rising has increased over the last 200 years? How likely does the UK Treasury think it is that there will be a recession if the country leaves the European Union? What are these group credences that such questions request? And how do they relate to the individual credences assigned by the members of the particular (...) group in question? According to the credal judgement aggregation principle, linear pooling, the credence function of a group should be a weighted average or linear pool of the credence functions of the individuals in the group. In this chapter, I give an argument for linear pooling based on considerations of accuracy. And I respond to two standard objections to the aggregation principle. (shrink)

Prepublication peer review should be abolished. We consider the effects that such a change will have on the social structure of science, paying particular attention to the changed incentive structure and the likely effects on the behaviour of individual scientists. We evaluate these changes from the perspective of epistemic consequentialism. We find that where the effects of abolishing prepublication peer review can be evaluated with a reasonable level of confidence based on presently available evidence, they are either positive or neutral. (...) We conclude that on present evidence abolishing peer review weakly dominates the status quo. (shrink)

Some scientists are happy to follow in the footsteps of others; some like to explore novel approaches. It is tempting to think that herein lies an epistemic division of labor conducive to overall scientific progress: the latter point the way to fruitful areas of research, and the former more fully explore those areas. Weisberg and Muldoon’s model, however, suggests that it would be best if all scientists explored novel approaches. I argue that this is due to implausible modeling choices, and (...) I present an alternative ‘epistemic landscape’ model that demonstrates the alleged benefits from division of labor, with one restriction. (shrink)

A growing interest in and concern about the adequacy and fairness of modern peer-review practices in publication and funding are apparent across a wide range of scientific disciplines. Although questions about reliability, accountability, reviewer bias, and competence have been raised, there has been very little direct research on these variables.

My topic is two-fold: a reductive account of expertise as an epistemic phenomenon, and applying the reductive account to the question of whether or not philosophers enjoy expertise. I conclude, on the basis of the reductive account, that even though philosophers enjoy something akin to second-order expertise (i.e. they are often experts on the positions of other philosophers, current trends in the philosophical literature, the history of philosophy, conceptual analysis and so on), they nevertheless lack first-order philosophical expertise (i.e. expertise (...) on philosophical positions themselves such as the nature of mind, causality, normativity and so forth). Throughout the paper, I respond to potential objections. (shrink)

In this paper, I defend my original objection to Hales’ suicide machine argument against Hales’ response. I argue Hales’ criticisms are either misplaced or underestimate the strength of my objection; if the constraints of the original objection are respected, my original objection blocks Hales’ reply. To be thorough, I restate an improved version of the objection to the suicide machine argument. I conclude that Hales fails to motivate a reasonable worry as to the supposed suicidal nature of presentist time travel.

Kuhn’s famous thesis that there is ‘no unique algorithm’ for choosing between rival scientific theories is analysed using the machinery of social choice theory. It is shown that the problem of theory choice as posed by Kuhn is formally identical to a standard social choice problem. This suggests that analogues of well-known results from the social choice literature, such as Arrow’s impossibility theorem, may apply to theory choice. If an analogue of Arrow’s theorem does hold for theory choice this would (...) refute Kuhn’s thesis, but it would also pose a threat to the rationality of science, a threat that is if anything more worrying than that posed by Kuhn. Various possible ‘escape routes’ from Arrow’s impossibility result are examined, in particular Amartya Sen’s idea of ‘enriching the informational basis’. It is shown that Sen’s idea can be applied to the problem of theory choice in science. This in turn sheds light on two well-known approaches to inductive inference in philosophy of science: Bayesianism and statistical model selection. (shrink)

The aggregation of consistent individual judgments on logically interconnected propositions into a collective judgment on those propositions has recently drawn much attention. Seemingly reasonable aggregation procedures, such as propositionwise majority voting, cannot ensure an equally consistent collective conclusion. The literature on judgment aggregation refers to that problem as the discursive dilemma. In this paper, we motivate that many groups do not only want to reach a factually right conclusion, but also want to correctly evaluate the reasons for that conclusion. In (...) other words, we address the problem of tracking the true situation instead of merely selecting the right outcome. We set up a probabilistic model analogous to Bovens and Rabinowicz (2006) and compare several aggregation procedures by means of theoretical results, numerical simulations and practical considerations. Among them are the premise-based, the situation-based and the distance-based procedure. Our findings confirm the conjecture in Hartmann, Pigozzi and Sprenger (2008) that the premise-based procedure is a crude, but reliable and sometimes even optimal form of judgment aggregation. (shrink)

Advocates of the self-corrective thesis argue that scientific method will refute false theories and find closer approximations to the truth in the long run. I discuss a contemporary interpretation of this thesis in terms of frequentist statistics in the context of the behavioral sciences. First, I identify experimental replications and systematic aggregation of evidence (meta-analysis) as the self-corrective mechanism. Then, I present a computer simulation study of scientific communities that implement this mechanism to argue that frequentist statistics may converge upon (...) a correct estimate or not depending on the social structure of the community that uses it. Based on this study, I argue that methodological explanations of the “replicability crisis” in psychology are limited and propose an alternative explanation in terms of biases. Finally, I conclude suggesting that scientific self-correction should be understood as an interaction effect between inference methods and social structures. (shrink)

Computer simulation of an epistemic landscape model, modified to include explicit representation of a centralized funding body, show the method of funding allocation has significant effects on communal trade-off between exploration and exploitation, with consequences for the community’s ability to generate significant truths. The results show this effect is contextual, and depends on the size of the landscape being explored, with funding that includes explicit random allocation performing significantly better than peer-review on large landscapes. The paper proposes a way of (...) incorporating external institutional factors in formal social epistemology, and offers a way of bringing such investigations to bear on current research policy questions. (shrink)

Bruner shows that in cultural interactions, members of minority groups will learn to interact with members of majority groups more quickly—minorities tend to meet majorities more often as a brute fact of their respective numbers—and, as a result, may come to be disadvantaged in situations where they divide resources. In this paper, we discuss the implications of this effect for epistemic communities. We use evolutionary game theoretic methods to show that minority groups can end up disadvantaged in academic interactions like (...) bargaining and collaboration as a result of this effect. These outcomes are more likely, in our models, the smaller the minority group. They occur despite assumptions that majority and minority groups do not differ with respect to skill level, personality, preference, or competence of any sort. Furthermore, as we will argue, these disadvantaged outcomes for minority groups may negatively impact the progress of epistemic communities. (shrink)

Group judgements are often – implicitly or explicitly – influenced by their members’ individual expertise. However, given that expertise is seldom recognized fully and that some distortions may occur (bias, correlation, etc.), it is not clear that differential weighting is an epistemically advantageous strategy with respect to straight averaging. Our paper characterizes a wide set of conditions under which differential weighting outperforms straight averaging and embeds the results into the multidisciplinary group decision-making literature.

Research on bias in peer review examines scholarly communication and funding processes to assess the epistemic and social legitimacy of the mechanisms by which knowledge communities vet and self-regulate their work. Despite vocal concerns, a closer look at the empirical and methodological limitations of research on bias raises questions about the existence and extent of many hypothesized forms of bias. In addition, the notion of bias is predicated on an implicit ideal that, once articulated, raises questions about the normative implications (...) of research on bias in peer review. This review provides a brief description of the function, history, and scope of peer review; articulates and critiques the conception of bias unifying research on bias in peer review; characterizes and examines the empirical, methodological, and normative claims of bias in peer review research; and assesses possible alternatives to the status quo. We close by identifying ways to expand conceptions and studies of bias to countenance the complexity of social interactions among actors involved directly and indirectly in peer review. (shrink)

A growing interest in and concern about the adequacy and fairness of modern peer-review practices in publication and funding are apparent across a wide range of scientific disciplines. Although questions about reliability, accountability, reviewer bias, and competence have been raised, there has been very little direct research on these variables.The present investigation was an attempt to study the peer-review process directly, in the natural setting of actual journal referee evaluations of submitted manuscripts. As test materials we selected 12 already published (...) research articles by investigators from prestigious and highly productive American psychology departments, one article from each of 12 highly regarded and widely read American psychology journals with high rejection rates (80%) and nonblind refereeing practices.With fictitious names and institutions substituted for the original ones (e.g., Tri-Valley Center for Human Potential), the altered manuscripts were formally resubmitted to the journals that had originally refereed and published them 18 to 32 months earlier. Of the sample of 38 editors and reviewers, only three (8%) detected the resubmissions. This result allowed nine of the 12 articles to continue through the review process to receive an actual evaluation: eight of the nine were rejected. Sixteen of the 18 referees (89%) recommended against publication and the editors concurred. The grounds for rejection were in many cases described as “serious methodological flaws.” A number of possible interpretations of these data are reviewed and evaluated. (shrink)

Computer simulation of an epistemic landscape model, modified to include explicit representation of a centralized funding body, show the method of funding allocation has significant effects on communal trade-off between exploration and exploitation, with consequences for the community’s ability to generate significant truths. The results show this effect is contextual, and depends on the size of the landscape being explored, with funding that includes explicit random allocation performing significantly better than peer review on large landscapes. The article proposes a way (...) of incorporating external institutional factors in formal social epistemology, and offers a way of bringing such investigations to bear on current research policy questions. 1Introduction2Theoretical Background3Model Description4Simulation Details 4.1Simulating the epistemic landscape4.2Simulating agents4.3Simulating communal knowledge4.4Simulating funding strategies4.5Simulating merit dynamics5Results and Discussion 5.1Experiment 1: The winner-takes-it-all mechanism only5.2Experiment 2: All dynamic mechanisms5.3Experiment 3: Adding a new funding mechanism 5.4Experiment 4: Varying the degree of myopia5.5Experiment 5: Variability of individual epistemic gain5.6Experiment 6: Likelihood of renewal6Discussion7Conclusion. (shrink)