Koehler's summary and critique of research on the base rate fallacy is cogent and persuasive. However, he may have overstated the case, and his suggestions for future research may be too restrictive. We agree that methodological approaches to this topic should be broadened, but we argue that experimental laboratory research and the Bayesian normative standard are useful and should not be abandoned.

Base rates have no necessary relation to judgments that are not themselves probabilities. There is no logical imperative, for instance, that behavioral base rates must affect causal attributions or that base rate information should affect judgments of legal liability. Decision theorists should be cautious in arguing that base rates place normative constraints on judgments of anything other than posterior probabilities.

A recent study showed physicians' reasoning about a realistic case to be ignorant of base rate. It also showed physicians interpreting information pertinent to base rate differently, depending on whether it was presented early or late in the case. Although these adult reasoners might do better if given hints through talk of relative frequencies, this would not prove that they had no problem of base rate neglect.

We have been oversold on the base rate fallacy in probabilistic judgment from an empirical, normative, and methodological standpoint. At the empirical level, a thorough examination of the base rate literature (including the famous lawyer–engineer problem) does not support the conventional wisdom that people routinely ignore base rates. Quite the contrary, the literature shows that base rates are almost always used and that their degree of use depends on task structure and representation. Specifically, base rates play a relatively larger role (...) in tasks where base rates are implicitly learned or can be represented in frequentist terms. Base rates are also used more when they are reliable and relatively more diagnostic than available individuating information. At the normative level, the base rate fallacy should be rejected because few tasks map unambiguously into the narrow framework that is held up as the standard of good decision making. Mechanical applications of Bayes's theorem to identify performance errors are inappropriate when (1) key assumptions of the model are either unchecked or grossly violated, and (2) no attempt is made to identify the decision maker's goals, values, and task assumptions. Methodologically, the current approach is criticized for its failure to consider how the ambiguous, unreliable, and unstable base rates of the real world are and should be used. Where decision makers' assumptions and goals vary, and where performance criteria are complex, the traditional Bayesian standard is insufficient. Even where predictive accuracy is the goal in commonly defined problems, there may be situations (e.g., informationally redundant environments) in which base rates can be ignored with impunity. A more ecologically valid research program is called for. This program should emphasize the development of prescriptive theory in rich, realistic decision environments. (shrink)

‘Bayesian epistemology’ became an epistemological movement in the 20th century, though its two main features can be traced back to the eponymous Reverend Thomas Bayes (c. 1701-61). Those two features are: (1) the introduction of a formal apparatus for inductive logic; (2) the introduction of a pragmatic self-defeat test (as illustrated by Dutch Book Arguments) for epistemic rationality as a way of extending the justification of the laws of deductive logic to include a justification for the laws of inductive logic. (...) The formal apparatus itself has two main elements: the use of the laws of probability as coherence constraints on rational degrees of belief (or degrees of confidence) and the introduction of a rule of probabilistic inference, a rule or principle of conditionalization. (shrink)

This is surely a bit of Socrates' famous irony. He draws the analogy to explain how his friends should regard poetry as they regretfully banish it from the ideal state. But lovers were no more sensible then than they are now. The advice to banish poetry, undermined already by Plato's own delight and skill in drama, is perhaps undermined still further by this evocation of a 'sensible' lover who counts love so well lost. Yet Socrates' image is one of avowed (...) rationality and prudence. The sensible lover imitates the older literary example of Ulysses' tying himself to the mast. (The example belongs therefore to the class of problems treated in Elster (1979)). Both this lover and Ulysses foresee that under certain possible future conditions, their opinions, values and preferences will or would differ from what they are now, in a very definite fashion. To what extent is such foresight possible? Correspondingly (when we do not claim foreknowledge) to what extent is such opinion reasonable, rational, coherent, or consistent in some suitably broad sense? It is not easy to understand exactly what is possible or even logically permissible in this respect. In an earlier paper, "Belief and the Will", I argued for a principle ("Reflection") to govern such deliberation. Here I will both generalize the treatment of opinion in "Belief and the Will" and respond to criticism. Critical examples mainly resembled the story of Ulysses who foresaw a period of dysfunction (at the sound 2 of the sirens) in his epistemic and/or doxastic future. Other criticism focused on the model of opinion used (precise numerical subjective probability) and on the merits of Dutch Book arguments. The present argument will not rely on Dutch Book arguments and strategies, and the Reflection principle will be formulated so as to apply also to vague opinion. (shrink)

A vexing question in Bayesian epistemology is how an agent should update on evidence which she assigned zero prior credence. Some theorists have suggested that, in such cases, the agent should update by Kolmogorov conditionalization, a norm based on Kolmogorov’s theory of regular conditional distributions. However, it turns out that in some situations, a Kolmogorov conditionalizer will plan to always assign a posterior credence of zero to the evidence she learns. Intuitively, such a plan is irrational and easily Dutch bookable. (...) In this paper, we propose a revised norm, Kolmogorov-Blackwell conditionalization, which avoids this problem. We prove a Dutch book theorem and converse Dutch book theorem for this revised norm, and relate our results to those of Rescorla (2018). (shrink)

I follow Hájek (Synthese 137:273–323, 2003c) by taking objective probability to be a function of two propositional arguments—that is, I take conditional probability as primitive. Writing the objective probability of q given r as P(q, r), I argue that r may be chosen to provide less than a complete and exact description of the world’s history or of its state at any time. It follows that nontrivial objective probabilities are possible in deterministic worlds and about the past. A very simple (...) chance–credence relation is also then natural, namely that reasonable credence equals objective probability. In other words, we should set our actual credence in a proposition equal to the proposition’s objective probability conditional on available background information. One advantage of that approach is that the background information is not subject to an admissibility requirement, as it is in standard formulations of the Principal Principle. Another advantage is that the “undermining” usually thought to follow from Humean supervenience can be avoided. Taking objective probability to be a two-argument function is not merely a technical matter, but provides us with vital flexibility in addressing significant philosophical issues. (shrink)

Any instance (i.e., event, behavior, trait) belongs to infinitely many reference classes, hence there are infinitely many base rates from which to choose. People clearly do not entertain all possible reference classes, however, so something must be limiting the search space. We suggest some possible mechanisms that determine which reference class is evoked for the purpose of judgment and decision.

Norms—that is, specifications of what we ought to do—play a critical role in the study of informal argumentation, as they do in studies of judgment, decision-making and reasoning more generally. Specifically, they guide a recurring theme: are people rational? Though rules and standards have been central to the study of reasoning, and behavior more generally, there has been little discussion within psychology about why (or indeed if) they should be considered normative despite the considerable philosophical literature that bears on this (...) topic. In the current paper, we ask what makes something a norm, with consideration both of norms in general and a specific example: norms for informal argumentation. We conclude that it is both possible and desirable to invoke norms for rational argument, and that a Bayesian approach provides solid normative principles with which to do so. (shrink)

Koehler is right to argue for more nuanced interpretation of base rate anomalies. These anomalies are best understood in relation to a broader class of cognitive anomalies, which are important for theory and practice. Recognizing a need for more nuanced analysis should not be taken as a license for treating the effects as “explained away.”.

Two distinct issues are sometimes confused in the base rate literature: Why do people make logical mistakes in the assessment of probabilities? and why do subjects not use base rates the way experimenters do? The latter problem may often reflect differences in an implicit reference class rather than a disinclination to update a base rate by Bayes' theorem. Also important are considerations concerning the interaction of several potentially relevant base rates.

Most philosophers agree that abduction (in the sense of Inference to the Best Explanation) is a type of inference that is frequently employed, in some form or other, both in everyday and in scientific reasoning. However, the exact form as well as the normative status of abduction are still matters of controversy. This entry contrasts abduction with other types of inference; points at prominent uses of it, both in and outside philosophy; considers various more or less precise statements of it; (...) discusses its normative status; and highlights possible connections between abduction and Bayesian confirmation theory. (shrink)

The base rate literature has an opposite twin in the social psychological literature on stereotypes, which concludes that people use their preexisting beliefs about probabilistic category attributes too much, rather than not enough. This ironic discrepancy arises because beliefs about category attributes enhance accuracy when the beliefs are accurate and diminish accuracy when they are not. To determine the accuracy of base rate/stereotype beliefs requires research that addresses specific content.

The thesis that probabilities of conditionals are conditional probabilities has putatively been refuted many times by so-called ‘triviality results’, although it has also enjoyed a number of resurrections. In this paper I assault it yet again with a new such result. I begin by motivating the thesis and discussing some of the philosophical ramifications of its fluctuating fortunes. I will canvas various reasons, old and new, why the thesis seems plausible, and why we should care about its fate. I will (...) look at some objections to Lewis’s famous triviality results, and thus some reasons for the pursuit of further triviality results. I will generalize Lewis’s results in ways that meet the objections. I will conclude with some reflections on the demise of the thesis—or otherwise. (shrink)

In Belief and the Will, van Fraassen employed a diachronic Dutch Book argument to support a counterintuitive principle called Reflection. There and subsequently van Fraassen has put forth Reflection as a linchpin for his views in epistemology and the philosophy of science, and for the voluntarism (first-person reports of subjective probability are undertakings of commitments) that he espouses as an alternative to descriptivism (first-person reports of subjective probability are merely self-descriptions). Christensen and others have attacked Reflection, taking it to have (...) unpalatable consequences. We prescind from the question of the cogency of diachronic Dutch Book arguments, and focus on Reflection's proper interpretation. We argue that Reflection is not as counterintuitive as it appears — that once interpreted properly the status of the counterexamples given by Christensen and others is left open. We show also that descriptivism can make sense of Reflection, while voluntarism is not especially well suited to do so. (shrink)

The debates between Bayesian, frequentist, and other methodologies of statistics have tended to focus on conceptual justifications, sociological arguments, or mathematical proofs of their long run properties. Both Bayesian statistics and frequentist (“classical”) statistics have strong cases on these grounds. In this article, we instead approach the debates in the “Statistics Wars” from a largely unexplored angle: simulations of different methodologies’ performance in the short to medium run. We conducted a large number of simulations using a straightforward decision problem based (...) around tossing a coin with unknown bias and then placing bets. In this simulation, we programmed four players, inspired by Bayesian statistics, frequentist statistics, Jon Williamson’s version of Objective Bayesianism, and a player who simply extrapolates from observed frequencies to general frequencies. The last player functions as a benchmark: a statistical methodology should at least outperform a crude form of induction. We focused on the performance of these methodologies in guiding the players towards good decisions. Unlike an earlier simulation study of this type, we found no systematic difference in performance between the Bayesian and frequentist players, provided the Bayesian used a flat prior and the frequentist used a low confidence level. Unlike that study, we were able to use Big Data methods to mitigate problems of random error in the simulation results. The Williamsonian player, who is a novel element of our study, also had no systematic differences in their performance, provided that they use a low confidence level. These players performed similarly even in the very short run, when players were making different decisions. Our study indicates that all three methodologies should be taken seriously by philosophers and practitioners of statistics. However, the frequentist and Williamsonian performed poorly when their confidence levels were high, and the Bayesian was surprisingly harmed by biased priors, providing some unexpected lessons for these methodologies when facing this type of decision problem. (shrink)

Kyburg’s opposition to the subjective Bayesian theory, and in particular to its advocates’ indiscriminate and often questionable use of Dutch Book arguments, is documented and much of it strongly endorsed. However, it is argued that an alternative version, proposed by both de Finetti at various times during his long career, and by Ramsey, is less vulnerable to Kyburg’s misgivings. This is a logical interpretation of the formalism, one which, it is argued, is both more natural and also avoids other, widely-made (...) objections to Bayesianism. (shrink)

There is a familiar risk of antinomy if fromxisEand p(xisH/xisE) =rit is permissible to infer p(xisH) =r, and what Carnap (1950) called “The requirement of total evidence” will not prevent such antinomies satisfactorily. What is needed instead is a properly developed theory of evidential weight.

Abstract(1) There is enough contradictory evidence regarding the role of base rates in category learning to confirm the nonexistence of biases in such learning. (2) It is not always possible to activate statistical reasoning through frequentist representation. (3) It is necessary to use the concept of systematic processing in reconsidering the published work on biases.

Consistent with Koehler's position, we propose a generalization of the base rate fallacy and earlier conservatism literatures. In studies using both traditional tasks and new tasks based on ecologically valid base rates, our subjects typically underweight individuating information at least as much as they underweight base rates. The implications of cue consistency for averaging heuristics are discussed.

Cognitive algebra strongly disproved the representativeness heuristic almost before it was published; and therewith it also disproved the base rate fallacy. Cognitive algebra provides a theoretical foundation for judgment-decision theory through its joint solution to the two fundamental problems – true measurement of subjective values, and cognitive rules for integration of multiple determinants.

The environment in which humans evolved is strongly and positively autocorrelated in space and time. Probabilistic judgments based on the assumption of independence may not yield evolutionarily adaptive behavior. A number of “faults” of human reasoning are not faulty under fuzzy arithmetic, a nonprobabilistic calculus of reasoning under uncertainty that may be closer to that underlying human decision making.

Koehler's call for a reanalysis of the base rate fallacy is particularly important in the applied domain of accounting, since base rate data appear to be an important input for many accounting tasks. In this commentary I discuss the use of base rates in accounting and explain why more flexible standards of performance are important when judging the use of base rates.

I discuss two ways of justifying reflection principles. First, I propose that an undogmatic reading of dynamic Dutch book arguments provides them with a sound foundation. Second, I show also that minimizing expected inaccuracy leads to a novel argument for reflection principles. The required inaccuracy measures comprise a natural class of functions that can be derived from a generalization of a condition known as propriety or immodesty. This shows that reflection principles are an essential feature not just of consistent degrees (...) of belief but also of degrees of belief that approximate truth. (shrink)

Formal epistemologists criticise the Conciliatory View of peer disagreement for being non-commutative with conditionalisation, path dependent and does not preserve the independence between propositions. Failing to commute with conditionalisation, one may switch the order between conciliating and conditionalising and obtain different outcomes. Failing to be path independent, the outcome of conciliation varies with the order of the acquisition of new testimonies. Failing to preserve the independence between propositions, one may suffer from a sure-loss and hence be deemed irrational. The three (...) formal deficiencies urge people to abandon the Conciliatory View. This paper aims to show that one may save the Conciliatory View by conciliating with nonlinear functions. Research in the study of opinion pooling shows that the three deficiencies are not problems of the Conciliatory View, but problems of linear averaging. Hence, one can get rid of these formal deficiencies by making conciliation with nonlinear averaging functions. After showing how the three deficiencies can be avoided, I will explore the features of nonlinear averaging functions and argue that they have properties that correctly capture people’s intuition concerning disagreement. The conclusion, therefore, is to suggest epistemologists develop a more fine-grained taxonomy for cases of disagreement. With a deliberate categorisation of different kinds of disagreement, epistemologists can pick the proper averaging rule to apply in each specific case, and get rid of possible formal deficiencies. (shrink)

Objective Bayesianism has been criticised on the grounds that objective Bayesian updating, which on a finite outcome space appeals to the maximum entropy principle, differs from Bayesian conditionalisation. The main task of this paper is to show that this objection backfires: the difference between the two forms of updating reflects negatively on Bayesian conditionalisation rather than on objective Bayesian updating. The paper also reviews some existing criticisms and justifications of conditionalisation, arguing in particular that the diachronic Dutch book justification fails (...) because diachronic Dutch book arguments are subject to a reductio: in certain circumstances one can Dutch book an agent however she changes her degrees of belief . One may also criticise objective Bayesianism on the grounds that its norms are not compulsory but voluntary, the result of a stance. It is argued that this second objection also misses the mark, since objective Bayesian norms are tied up in the very notion of degrees of belief. (shrink)

The fallacy beneath the base rate fallacy is that we know what a base rate is. We talk as if base rates and individuating information were two different kinds of information. From a Bayesian perspective, however, the only difference between base rate and individuating information is – which comes first.

Commentators agree that simple conclusions about a general base rate fallacy are not appropriate. It is more constructive to identify conditions under which base rates are differentially weighted. Commentators also agree that improving the ecological validity of the research is desirable, although this is less important to those interested exclusively in psychological processes. The philosophers and ecologists among the commentators offer a kinder perspective on base rate reasoning than the psychologists. My own perspective is that the interesting questions (both psychological (...) and normative/prescriptive) are best answered through empirical studies that use real-world performance standards. (shrink)

This commentary discusses three points: (1) The implications of the fact that it is rational to ignore base rates if probabilities are estimated by frequencies from samples without missing data (natural sampling); (2) second order probabilities distributions are a plausible way to model imprecise probabilities; and (3) Bayesian networks represent a normative reference for multi-cue models of probabilistic inference.

Base rates are vital in predicting violent criminal recidivism. However, both lay people given simulated prediction tasks and professionals milking real life predictions appear insensitive to variations in the base rate of violent recidivism. Although there are techniques to help decision makers attend to base rates, increased decision accuracy is better sought in improved actuarial models as opposed to improved clinicians.

In contrast to traditional research on base-rate neglect, an ecologically-oriented research program would analyze the correspondence between cognitive algorithms and the nature of information in the environment. Bayesian computations turn out to be simpler when information is represented in frequency formats as opposed to the probability formats used in previous research. Frequency formats often enable even uninstructed subjects to perform Bayesian reasoning.

Abstract(1) The miscitations of seminal experiments in the base rate literature adds to the existing database of systematic miscitations of wellknown psychological experiments. These miscitations may be caused by a process of reconstructive remembering. (2) Representative design should be the methodological core of Koehler's call for ecologically valid research. This approach can benefit both basic and applied research.

This commentary presents a self-assessment inventory that will allow readers to determine their own attitude toward the base rate fallacy and its literature. The inventory is scientifically valid but not Medicare/Medicaid reimbursable.

When base rates are learned and used in an experiential manner subjects show better base rate use, perhaps because the implicit learning system is engaged. A causal framework in which base rates are relevant might also be necessary. Humans might thus perform better on more ecologically valid tasks, which are likely to contain those three components.

Koehler is right that base rate information is used, to various degrees, both in laboratory tasks and in everyday life. However, it is not time to turn our backs on laboratory tasks and focus solely on ecologically valid decision making. Tightly controlled experimental data are still needed to understandhowbase rate information is used, and how this varies among groups.

Koehler's target article attempts a balanced view of the relevance of knowledge of base rates to judgments of subjective or credal probability, but he is not sensitive enough to the difference between requiring and permitting the equation of probability judgments with base rates, the interaction between precision of base rate and reference class information, and the possibility of indeterminate probability judgment.

Failure to consider base rate is regarded as potentially hazardous, mainly because its consideration is assumed to be determined solely by P(H/D), the probability of the individuating data if the hypothesis is true, and not at all by P(D/˜H), the probability if the hypothesis is false. However, when P(D/˜H) is unconfounded from P(D/H), it turns out to be the stronger determinant of base rate consideration.

Setting the two hypotheses of complete neglect and full use of base rates against each other is inappropriate. The proper question concerns the degree to which base rates are used (or neglected), and under what conditions. We outline alternative approaches and recommend regression analysis. Koehler's conclusion that we have been oversold on the base rate fallacy seems to be premature.

The important question is how people process probabilistic information, not whether they process it in accordance with a normative model that we never should have expected them to be capable of following. Experience is not the cure, as widely thought, to problems with utilizing base rate information.

As illustrated by research Koehler himself cites (Dawes et al. 1993), the purpose of experiments is to choose between contrasting explanations of past observations – rather than to seek statistical generalizations about the prevalence of effects. True external validity results not from sampling various problems that are representative of “real world” decision making, but from reproducing an effect in the laboratory with minimal contamination (including from real world factors).

The base rate fallacy is directly dependent on a particular judgment paradigm in which information may be unambiguously designated as either “base rate” or “individuating,” and in which subjects make two-stage sequential judgments. The paradigm may be a poor match for real world settings, and the fallacy may thus be undefined for natural ecologies of judgment.

Koehler's work will assist the effort to understand legal fact finding. It leaves two questions somewhat open: (i) the extent to which empirical research can measure correctness of fact-finding, a function that involves the resolution of normative questions and (ii) the standards judges should use in the absence of the research he advocates.