As illustrated by research Koehler himself cites (Dawes et al. 1993), the purpose of experiments is to choose between contrasting explanations of past observations – rather than to seek statistical generalizations about the prevalence of effects. True external validity results not from sampling various problems that are representative of “real world” decision making, but from reproducing an effect in the laboratory with minimal contamination (including from real world factors).

The base rate fallacy is directly dependent on a particular judgment paradigm in which information may be unambiguously designated as either “base rate” or “individuating,” and in which subjects make two-stage sequential judgments. The paradigm may be a poor match for real world settings, and the fallacy may thus be undefined for natural ecologies of judgment.

There is a familiar risk of antinomy if fromxisEand p(xisH/xisE) =rit is permissible to infer p(xisH) =r, and what Carnap (1950) called “The requirement of total evidence” will not prevent such antinomies satisfactorily. What is needed instead is a properly developed theory of evidential weight.

The object of this paper is to show why recent research in the psychology of deductive and probabilistic reasoning does not have.

Koehler's work will assist the effort to understand legal fact finding. It leaves two questions somewhat open: (i) the extent to which empirical research can measure correctness of fact-finding, a function that involves the resolution of normative questions and (ii) the standards judges should use in the absence of the research he advocates.

There are currently two robust traditions in philosophy dealing with doxastic attitudes: the tradition that is concerned primarily with all-or-nothing belief, and the tradition that is concerned primarily with degree of belief or credence. This paper concerns the relationship between belief and credence for a rational agent, and is directed at those who may have hoped that the notion of belief can either be reduced to credence or eliminated altogether when characterizing the norms governing ideally rational agents. It presents a (...) puzzle which lends support to two theses. First, that there is no formal reduction of a rational agent’s beliefs to her credences, because belief and credence are each responsive to different features of a body of evidence. Second, that if our traditional understanding of our practices of holding each other responsible is correct, then belief has a distinctive role to play, even for ideally rational agents, that cannot be played by credence. The question of which avenues remain for the credence-only theorist is considered. (shrink)

Corroborating Testimony, Probability and Surprise’, Erik J. Olsson ascribes to L. Jonathan Cohen the claims that if two witnesses provide us with the same information, then the less probable the information is, the more confident we may be that the information is true (C), and the stronger the information is corroborated (C*). We question whether Cohen intends anything like claims (C) and (C*). Furthermore, he discusses the concurrence of witness reports within a context of independent witnesses, whereas the witnesses in (...) Olsson's model are not independent in the standard sense. We argue that there is much more than, in Olsson's words, ‘a grain of truth’ to claim (C), both on his own characterization as well as on Cohen's characterization of the witnesses. We present an analysis for independent witnesses in the contexts of decision-making under risk and decision-making under uncertainty and generalize the model for n witnesses. As to claim (C*), Olsson's argument is contingent on the choice of a particular measure of corroboration and is not robust in the face of alternative measures. Finally, we delimit the set of cases to which Olsson's model is applicable. 1 Claim (C) examined for Olsson's characterization of the relationship between the witnesses 2 Claim (C) examined for two or more independent witnesses 3 Robustness and multiple measures of corroboration 4 Discussion. (shrink)

In this paper, we look at reasoning with evidence and facts in criminal cases. We show how this reasoning may be analysed in a dialectical way by means of critical questions that point to typical sources of doubt. We discuss critical questions about the evidential arguments adduced, about the narrative accounts of the facts considered, and about the way in which the arguments and narratives are connected in an analysis. Our treatment shows how two different types of knowledge, represented as (...) schemes, play a role in reasoning with evidence: argumentation schemes and story schemes. (shrink)

This paper discusses the ability of explanationist theories of epistemic justification to account for the justification we have for holding beliefs about the future. McCain’s explanationist account of the relation of evidential support is supposedly in a better position than other theories of this type to correctly handle cases involving beliefs about the future. However, the results delivered by this account have been questioned by Byerly and Martin. This paper argues that McCain’s account is, in fact, able to deliver plausible (...) results in cases involving such beliefs and that explanationism, if properly articulated, is illuminating with respect to the justification we have for holding such beliefs, as it manages to correctly distinguish evidence that only supports believing probabilistic claims about the future from evidence that is sufficient to believe that a particular event will happen. (shrink)

This article examines Smith’s recent treatment of the problem of statistical evidence and the conception of epistemic justification that he puts forward. Two possible solutions to the problem of statistical evidence that result from his analysis of cases involving a contrast between statistical and individual evidence are considered. The solution resulting from Smith’s conception of epistemic justification is shown to be inferior to the solution calling for an explanationist conception of epistemic justification. As a result, Smith’s analysis of cases illustrating (...) the problem of statistical evidence, given his own standards of theory choice, is shown to support an explanationist conception of epistemic justification as an alternative to the probabilistic conception that he urges us to reject. (shrink)

Cognitive algebra strongly disproved the representativeness heuristic almost before it was published; and therewith it also disproved the base rate fallacy. Cognitive algebra provides a theoretical foundation for judgment-decision theory through its joint solution to the two fundamental problems – true measurement of subjective values, and cognitive rules for integration of multiple determinants.

"A Survey of Ranking Theory": The paper gives an up-to-date survey of ranking theory. It carefully explains the basics. It elaborates on the ranking theoretic explication of reasons and their balance. It explains the dynamics of belief statable in ranking terms and indicates how the ranks can thereby be measured. It suggests how the theory of Bayesian nets can be carried over to ranking theory. It indicates what it might mean to objectify ranks. It discusses the formal and the philosophical (...) aspects of the tight relation and the complementarity of ranks and probabilities. It closes with comparative remarks on predecessors and other philosophical proposals as well as formal models developed in AI. (shrink)

Commentators agree that simple conclusions about a general base rate fallacy are not appropriate. It is more constructive to identify conditions under which base rates are differentially weighted. Commentators also agree that improving the ecological validity of the research is desirable, although this is less important to those interested exclusively in psychological processes. The philosophers and ecologists among the commentators offer a kinder perspective on base rate reasoning than the psychologists. My own perspective is that the interesting questions (both psychological (...) and normative/prescriptive) are best answered through empirical studies that use real-world performance standards. (shrink)

We have been oversold on the base rate fallacy in probabilistic judgment from an empirical, normative, and methodological standpoint. At the empirical level, a thorough examination of the base rate literature (including the famous lawyer–engineer problem) does not support the conventional wisdom that people routinely ignore base rates. Quite the contrary, the literature shows that base rates are almost always used and that their degree of use depends on task structure and representation. Specifically, base rates play a relatively larger role (...) in tasks where base rates are implicitly learned or can be represented in frequentist terms. Base rates are also used more when they are reliable and relatively more diagnostic than available individuating information. At the normative level, the base rate fallacy should be rejected because few tasks map unambiguously into the narrow framework that is held up as the standard of good decision making. Mechanical applications of Bayes's theorem to identify performance errors are inappropriate when (1) key assumptions of the model are either unchecked or grossly violated, and (2) no attempt is made to identify the decision maker's goals, values, and task assumptions. Methodologically, the current approach is criticized for its failure to consider how the ambiguous, unreliable, and unstable base rates of the real world are and should be used. Where decision makers' assumptions and goals vary, and where performance criteria are complex, the traditional Bayesian standard is insufficient. Even where predictive accuracy is the goal in commonly defined problems, there may be situations (e.g., informationally redundant environments) in which base rates can be ignored with impunity. A more ecologically valid research program is called for. This program should emphasize the development of prescriptive theory in rich, realistic decision environments. (shrink)

Base rates have no necessary relation to judgments that are not themselves probabilities. There is no logical imperative, for instance, that behavioral base rates must affect causal attributions or that base rate information should affect judgments of legal liability. Decision theorists should be cautious in arguing that base rates place normative constraints on judgments of anything other than posterior probabilities.

Abstract(1) The miscitations of seminal experiments in the base rate literature adds to the existing database of systematic miscitations of wellknown psychological experiments. These miscitations may be caused by a process of reconstructive remembering. (2) Representative design should be the methodological core of Koehler's call for ecologically valid research. This approach can benefit both basic and applied research.

Koehler's summary and critique of research on the base rate fallacy is cogent and persuasive. However, he may have overstated the case, and his suggestions for future research may be too restrictive. We agree that methodological approaches to this topic should be broadened, but we argue that experimental laboratory research and the Bayesian normative standard are useful and should not be abandoned.

This commentary presents a self-assessment inventory that will allow readers to determine their own attitude toward the base rate fallacy and its literature. The inventory is scientifically valid but not Medicare/Medicaid reimbursable.

When base rates are learned and used in an experiential manner subjects show better base rate use, perhaps because the implicit learning system is engaged. A causal framework in which base rates are relevant might also be necessary. Humans might thus perform better on more ecologically valid tasks, which are likely to contain those three components.

Base rates are vital in predicting violent criminal recidivism. However, both lay people given simulated prediction tasks and professionals milking real life predictions appear insensitive to variations in the base rate of violent recidivism. Although there are techniques to help decision makers attend to base rates, increased decision accuracy is better sought in improved actuarial models as opposed to improved clinicians.

Consistent with Koehler's position, we propose a generalization of the base rate fallacy and earlier conservatism literatures. In studies using both traditional tasks and new tasks based on ecologically valid base rates, our subjects typically underweight individuating information at least as much as they underweight base rates. The implications of cue consistency for averaging heuristics are discussed.

Koehler is right that base rate information is used, to various degrees, both in laboratory tasks and in everyday life. However, it is not time to turn our backs on laboratory tasks and focus solely on ecologically valid decision making. Tightly controlled experimental data are still needed to understandhowbase rate information is used, and how this varies among groups.

Any instance (i.e., event, behavior, trait) belongs to infinitely many reference classes, hence there are infinitely many base rates from which to choose. People clearly do not entertain all possible reference classes, however, so something must be limiting the search space. We suggest some possible mechanisms that determine which reference class is evoked for the purpose of judgment and decision.

The fallacy beneath the base rate fallacy is that we know what a base rate is. We talk as if base rates and individuating information were two different kinds of information. From a Bayesian perspective, however, the only difference between base rate and individuating information is – which comes first.

Koehler is right to argue for more nuanced interpretation of base rate anomalies. These anomalies are best understood in relation to a broader class of cognitive anomalies, which are important for theory and practice. Recognizing a need for more nuanced analysis should not be taken as a license for treating the effects as “explained away.”.

Koehler's target article attempts a balanced view of the relevance of knowledge of base rates to judgments of subjective or credal probability, but he is not sensitive enough to the difference between requiring and permitting the equation of probability judgments with base rates, the interaction between precision of base rate and reference class information, and the possibility of indeterminate probability judgment.

Two distinct issues are sometimes confused in the base rate literature: Why do people make logical mistakes in the assessment of probabilities? and why do subjects not use base rates the way experimenters do? The latter problem may often reflect differences in an implicit reference class rather than a disinclination to update a base rate by Bayes' theorem. Also important are considerations concerning the interaction of several potentially relevant base rates.

The underutilization of base rates is a consistent finding. The strong claim that base rates are ignored has been rejected and this needs no further emphasis. Following the path of “normal science,” research examines the conditions predicting changes in the degree of underutilization. A scientific revolution that might dethrone the heuristics and biases paradigm is not in sight.

Koehler's call for a reanalysis of the base rate fallacy is particularly important in the applied domain of accounting, since base rate data appear to be an important input for many accounting tasks. In this commentary I discuss the use of base rates in accounting and explain why more flexible standards of performance are important when judging the use of base rates.

This commentary discusses three points: (1) The implications of the fact that it is rational to ignore base rates if probabilities are estimated by frequencies from samples without missing data (natural sampling); (2) second order probabilities distributions are a plausible way to model imprecise probabilities; and (3) Bayesian networks represent a normative reference for multi-cue models of probabilistic inference.