Do people tend to be overconfident? Many think so. They’ve run studies on whether people are calibrated: whether their average confidence in their opinions matches the proportion of those opinions that are true. Under certain conditions, people are systematically ‘over-calibrated’—for example, of the opinions they’re 80% confident in, only 60% are true. From this empirical over-calibration, it’s inferred that people are irrationally overconfident. My question: When and why is this inference warranted? Answering it requires articulating a general connection between being (...) rational and being right—something extant studies have not done. I show how to do so using the notion of deference. This provides a theoretical foundation for calibration research, but also reveals a flaw: the connection between being rational and being right is much weaker than is standardly assumed, since rational people can often be expected to be miscalibrated. Thus we can’t test whether people are overconfident by simply testing whether they are over-calibrated; instead, we must try to predict the rational deviations from calibration, and then compare those predictions to people’s performance. I show how this can be done—and that doing so complicates the interpretation of robust empirical effects. (shrink)

Searching for information is critical in many situations. In medicine, for instance, careful choice of a diagnostic test can help narrow down the range of plausible diseases that the patient might have. In a probabilistic framework, test selection is often modeled by assuming that people's goal is to reduce uncertainty about possible states of the world. In cognitive science, psychology, and medical decision making, Shannon entropy is the most prominent and most widely used model to formalize probabilistic uncertainty and the (...) reduction thereof. However, a variety of alternative entropy metrics (Hartley, Quadratic, Tsallis, Rényi, and more) are popular in the social and the natural sciences, computer science, and philosophy of science. Particular entropy measures have been predominant in particular research areas, and it is often an open issue whether these divergences emerge from different theoretical and practical goals or are merely due to historical accident. Cutting across disciplinary boundaries, we show that several entropy and entropy reduction measures arise as special cases in a unified formalism, the Sharma–Mittal framework. Using mathematical results, computer simulations, and analyses of published behavioral data, we discuss four key questions: How do various entropy models relate to each other? What insights can be obtained by considering diverse entropy models within a unified framework? What is the psychological plausibility of different entropy models? What new questions and insights for research on human information acquisition follow? Our work provides several new pathways for theoretical and empirical research, reconciling apparently conflicting approaches and empirical findings within a comprehensive and unified information‐theoretic formalism. (shrink)

The canonical Bayesian solution to the ravens paradox faces a problem: it entails that black non-ravens disconfirm the hypothesis that all ravens are black. I provide a new solution that avoids this problem. On my solution, black ravens confirm that all ravens are black, while non-black non-ravens and black non-ravens are neutral. My approach is grounded in certain relations of epistemic dependence, which, in turn, are grounded in the fact that the kind raven is more natural than the kind black. (...) The solution applies to any generalization “All F’s are G” in which F is more natural than G. (shrink)

It is customary to distinguish experimental from purely observational sciences. The former include physics and molecular biology, the latter astronomy and palaeontology. Experiments involve actively intervening in the course of nature, as opposed to observing events that would have happened anyway. When a molecular biologist inserts viral DNA into a bacterium in his laboratory, this is an experiment; but when an astronomer points his telescope at the heavens, this is an observation. Without the biologist’s handiwork the bacterium would never have (...) contained foreign DNA; but the planets would have continued orbiting the sun whether or not the astronomer had directed his telescope skyward. The observational/experimental distinction would probably be difficult to make precise 1, as the notion of an ‘intervention’ is not easily defined, but it is intuitively fairly clear, and is frequently invoked by scientists and historians of science. Experimentation, or ‘putting questions to nature’, is often cited as a hallmark of the modern scientific method, something that permitted the enormous advances of the last 350 years. And it is sometimes said that the social sciences lag behind the natural because controlled experiments cannot be done so readily in the former. Moreover in certain sciences, e.g. epidemiology, students are explicitly taught that experimental data is preferable to observational data, particularly for doing causal inference. So the distinction between observational and experimental science has quite wide currency, and is often regarded as methodologically significant. Surprisingly, mainstream philosophy of science has had rather little to say about the observational/experimental distinction. 2 For example, discussions of confirmation usually invoke a notion of ‘evidence’, to be contrasted with ‘theory’ or ‘hypothesis’; the aim is to understand how the evidence bears on the hypothesis. But whether this ‘evidence’ comes from observation or experiment generally plays no role in the discussion; this is true …. (shrink)

Wason’s selection task is a paramount experimental problem in the study of human reasoning, often connected with the celebrated ravens paradox in the philosophical literature. Various normative accounts of the selection task rely on a Bayesian approach. Some claim vindication of participants’ rationality. Others don’t, thus following Wason’s original intuition that observed responses are mistaken. In this article we argue that despite claims to the contrary, all these accounts actually speak to the same effect: Wason was right. First, we provide (...) a new accuracy-based analysis of the selection task that includes the existing proposals as special cases. We then show on this basis that none can actually vindicate participants’ rationality. We conclude that all normative renditions considered eventually concur: all in all, Bayesians should follow Wason in the selection task. (shrink)

In the present paper we empirically investigate the psychometric properties of some of the most famous statistical and logical cognitive illusions from the “heuristics and biases” research program by Daniel Kahneman and Amos Tversky, who nearly 50 years ago introduced fascinating brain teasers such as the famous Linda problem, the Wason card selection task, and so-called Bayesian reasoning problems (e.g., the mammography task). In the meantime, a great number of articles has been published that empirically examine single cognitive illusions, theoretically (...) explaining people’s faulty thinking, or proposing and experimentally implementing measures to foster insight and to make these problems accessible to the human mind. Yet these problems have thus far usually been empirically analyzed on an individual-item level only (e.g., by experimentally comparing participants’ performance on various versions of one of these problems). In this paper, by contrast, we examine these illusions as a group and look at the ability to solve them as a psychological construct. Based on an sample ofN= 2,643 Luxembourgian school students of age 16–18 we investigate the internal psychometric structure of these illusions (i.e., Are they substantially correlated? Do they form a reflexive or a formative construct?), their connection to related constructs (e.g., Are they distinguishable from intelligence or mathematical competence in a confirmatory factor analysis?), and the question of which of a person’s abilities can predict the correct solution of these brain teasers (by means of a regression analysis). (shrink)

Biased research occurs frequently in the sciences. In this paper, I will focus on one particular kind of biased research: research that is subject to sponsorship bias. I will address the following epistemological question: what precisely is epistemically wrong with biased research of this kind? I will defend the evidential account of epistemic wrongness: that is, research affected by sponsorship bias is epistemically wrong if and only if the researchers in question make false claims about the evidential support of some (...) hypothesis H by data E. I will argue that the evidential account captures the epistemic wrongness of three paradigmatic types of sponsorship bias. (shrink)

Bayesian confirmation theories might be the best standing theories of confirmation to date, but they are certainly not paradox-free. Here I recognize that BCTs’ appeal mainly comes from the fact that they capture some of our intuitions about confirmation better than those the- ories that came before them and that the superiority of BCTs is suffi- ciently justified by those advantages. Instead, I will focus on Sylvan and Nola’s claim that it is desirable that our best theory of confirmation be (...) as paradox-free as possible. For this reason, I will show that, as they respond to different interests, the project of the BCTs is not incompatible with Sylvan and Nola’s project of a paradox-free confirmation logic. In fact, it will turn out that, provided we are ready to embrace some degree of non-classicality, both projects complement each other nicely. (shrink)

ABSTRACTIn this paper, I review some recent treatments of Simpson's Paradox, and I propose a new rationalizing explanation of its paradoxicality.