Policymakers are responsible for key decisions about political governance. Usually, they are selected or elected based on experience and then supported in their decision-making by the additional counsel of subject experts. Those satisfied with this system believe these individuals – generally speaking – will have the right intuitions about the best types of action. This is important because political decisions have ethical implications; they affect how we all live in society. Nevertheless, there is a wealth of research that cautions against (...) trusting human judgment as it can be severely flawed. This paper will look at the root causes of the most common errors of human judgment before arguing – contra the instincts of many – that future AI systems could take a range of political decisions more reliably. I will argue that, if/when engineers establish ethically robust systems, governments will have a moral obligation to refer to them as a part of decision-making. (shrink)

Previous studies have shown that people often use heuristics in making inferences and that subjective memory experiences, such as recognition or familiarity of objects, can be valid cues for inferences. So far, many researchers have used the binary choice task in which two objects are presented as alternatives (e.g., “Which city has the larger population, city A or city B?”). However, objects can be presented not only as alternatives but also in a question (e.g., “Which country is city X in, (...) country A or country B?”). In such a situation, people can make inferences based on the relationship between the object in the question and each object given as an alternative. In the present study, we call this type of task a “relationships‐comparison task.” We modeled the three inference strategies that people could apply to solve it (familiarity‐matching [FM; a new heuristic we propose in this study], familiarity heuristic [FH], and knowledge‐based inference [KI]) to examine people's inference processes. Through Studies 1, 2, and 3, we found that (a) people tended to rely on heuristics, and that FM (inferences based on similarity in familiarity between objects) well explained participants' inference patterns; (b) FM could work as an ecologically rational strategy for the relationships–comparison task since it could effectively reflect environmental structures, and that the use of FM could be highly replicable and robust; and (c) people could sometimes use a decision strategy like FM, even in their daily lives (consumer behaviors). The nature of the relationships–comparison task and human cognitive processes is discussed. (shrink)

It is commonly thought that moral rules and principles, such as ‘Keep your promises,’ ‘Respect autonomy,’ and ‘Distribute goods according to need ,’ should play an essential role in our moral deliberation. Particularists have challenged this view by arguing that principled guidance leads us to engage in worse decision making because principled guidance is too rigid and it leads individuals to neglect or distort relevant details. However, when we examine empirical literature on the use of rules and principles in other (...) domains, we find that people can learn to use rules discriminately and that rule-based models tend to outperform even expert judgment. I argue that this evidence poses a problem for the moral particularist. If the particularist claims that we should not rely on decision-making rules when making practical decisions and it turns out that these rules help us make better decisions, then the particularists’ prescriptive account is deficient. However, if the particularist claims that we should rely on practical decision-making rules but not on moral rules, she needs to explain how practical rules are different from moral rules and why we should rely on the former but not the latter. (shrink)

PROBEX (PROBabilities from EXemplars), a model of probabilistic inference and probability judgment based on generic knowledge is presented. Its properties are that: (a) it provides an exemplar model satisfying bounded rationality; (b) it is a “lazy” algorithm that presumes no pre‐computed abstractions; (c) it implements a hybrid‐representation, similarity‐graded probability. We investigate the ecological rationality of PROBEX and find that it compares favorably with Take‐The‐Best and multiple regression (Gigerenzer, Todd, & the ABC Research Group, 1999). PROBEX is fitted to the point (...) estimates, decisions, and probability assessments by human participants. The best fit is obtained for a version that weights frequency heavily and retrieves only two exemplars. It is proposed that PROBEX implements speed and frugality in a psychologically plausible way. (shrink)

This paper motivates using explicit methods in linguistics by attempting to estimate the size of a linguistic data set. Such estimations are difficult because redundant data can easily pad the data set. To address this, I offer some explicit operationalizations of the data and their features. But for linguistic data, negative associations don’t indicate true redundancy, and yet for many measures they can be mathematically impossible to ignore. It is proven that this troublesome phenomenon has positive Lebesgue measure, is monotonically (...) increasing, and that these two features hold robustly in four different ways. (shrink)

Many important decisions depend on unknown states of the world. Society is increasingly relying on statistical predictive models to make decisions in these cases. While predictive models are useful, previous research has documented that (a) individual decision makers distrust models and (b) people’s predictions are often worse than those of models. These findings indicate a lack of awareness of how to evaluate predictions generally. This includes concepts like the loss function used to aggregate errors or whether error is training error (...) or generalisation error. To address this gap, we present three studies testing how lay people visually evaluate the predictive accuracy of models. We found that (a) participant judgements of prediction errors were more similar to absolute error than squared error (Study 1), (b) we did not detect a difference in participant reactions to training error versus generalisation error (Study 2), and (c) participants rated complex models as more accurate when comparing two models, but rated simple models as more accurate when shown single models in isolation (Study 3). When communicating about models, researchers should be aware that the public’s visual evaluation of models may disagree with their method of measuring errors and that many may fail to recognise overfitting. (shrink)

Since its introduction, multivariate pattern analysis, or ‘neural decoding’, has transformed the field of cognitive neuroscience. Underlying its influence is a crucial inference, which we call the decoder’s dictum: if information can be decoded from patterns of neural activity, then this provides strong evidence about what information those patterns represent. Although the dictum is a widely held and well-motivated principle in decoding research, it has received scant philosophical attention. We critically evaluate the dictum, arguing that it is false: decodability is (...) a poor guide for revealing the content of neural representations. However, we also suggest how the dictum can be improved on, in order to better justify inferences about neural representation using MVPA. 1Introduction 2A Brief Primer on Neural Decoding: Methods, Application, and Interpretation 2.1What is multivariate pattern analysis? 2.2The informational benefits of multivariate pattern analysis 3Why the Decoder’s Dictum Is False 3.1We don’t know what information is decoded 3.2The theoretical basis for the dictum 3.3Undermining the theoretical basis 4Objections and Replies 4.1Does anyone really believe the dictum? 4.2Good decoding is not enough 4.3Predicting behaviour is not enough 5Moving beyond the Dictum 6Conclusion. (shrink)

Intractability and optimality are two sides of one coin: Optimal models are often intractable, that is, they tend to be excessively complex, or NP-hard. We explain the meaning of NP-hardness in detail and discuss how modem computer science circumvents intractability by introducing heuristics and shortcuts to optimality, often replacing optimality by means of sufficient sub-optimality. Since the principles of decision theory dictate balancing the cost of computation against gain in accuracy, statistical inference is currently being reshaped by a vigorous new (...) trend: the science of simplicity. Simple models, as we show for specific cases, are not just tractable, they also tend to be robust. Robustness is the ability of a model to extract relevant information from data, disregarding noise.Recently, Gigerenzer, Todd and the ABC Research Group (1999) have put forward a collection of fast and frugal heuristics as simple, boundedly rational inference strategies used by the unaided mind in real world inference problems. This collection of heuristics has suggestively been called the adaptive toolbox. In this paper we will focus on a comparison task in order to illustrate the simplicity and robustness of some of the heuristics in the adaptive toolbox in contrast to the intractability and the fragility of optimal solutions. We will concentrate on three important classes of models for comparison-based inference and, in each of these classes, search for that to be used as benchmarks to evaluate the performance of fast and frugal heuristics: lexicographic trees, linear modes and Bayesian networks. Lexicographic trees are interesting because they are particularly simple models that have been used by humans throughout the centuries. Linear models have been traditionally used by cognitive psychologists as models for human inference, while Bayesian networks have only recently been introduced in statistics and computer science. Yet it is the Bayesian networks that are the best possible benchmarks for evaluating the fast and frugal heuristics, as we will show in this paper. (shrink)

This article provides an overview of recent results on lexicographic, linear, and Bayesian models for paired comparison from a cognitive psychology perspective. Within each class, we distinguish subclasses according to the computational complexity required for parameter setting. We identify the optimal model in each class, where optimality is defined with respect to performance when fitting known data. Although not optimal when fitting data, simple models can be astonishingly accurate when generalizing to new data. A simple heuristic belonging to the class (...) of lexicographic models is Take The Best (Gigerenzer & Goldstein (1996) Psychol. Rev. 102: 684). It is more robust than other lexicographic strategies which use complex procedures to establish a cue hierarchy. In fact, it is robust due to its simplicity, not despite it. Similarly, Take The Best looks up only a fraction of the information that linear and Bayesian models require; yet it achieves performance comparable to that of models which integrate information. Due to its simplicity, frugality, and accuracy, Take The Best is a plausible candidate for a psychological model in the tradition of bounded rationality. We review empirical evidence showing the descriptive validity of fast and frugal heuristics. (shrink)

Methodological dualism in linguistics occurs when its theories are subjected to standards that are inappropriate for them qua scientific theories. Despite much opposition, methodological dualism abounds in contemporary thinking. In this paper, I treat linguistics as a scientific activity and explore some instances of dualism. By extracting some ubiquitous aspects of scientific methodology from its typically quantitative expression, I show that two recent instances of methodologically dualistic critiques of linguistics are ill-founded. I then show that there are nonetheless some divergences (...) between linguistic and other ordinary scientific methods, reflecting yet a third instance of methodological dualism. (shrink)

Structural equation modelling (SEM) is outlined and compared with two non-linear alternatives, artificial neural networks and ''fast and frugal'' models. One particular non-linear decision-making situation is discussed, that exemplified by a lexicographic semi-order. We illustrate the use of SEM on a dataset derived from 539 volunteers' responses to questions about food-related risks. Our conclusion is that SEM is a useful member of the armoury of techniques available to the student of human judgement: it subsumes several multivariate statistical techniques and permits (...) their flexible combination, and it provides robust goodness-of-fit statistics and is available in (generally) easy-to-use computer packages. Although the number of tasks for which SEM provides a persuasive psychological model is small, it is very useful in identifying the important variables and their inter-relations that contribute to task performance, and thus can constitute a valuable intermediate staging point between raw data and a fully fledged psychological theory. (shrink)

Statistical procedures can be applied to episodes in the history of science in order to weight attributes to predict short-term survival of theories; an asymptotic method is used to show that short-term survival is a valid proxy for ultimate survival; and a theoretical argument is made that ultimate survival is a valid proxy for objective truth. While realists will appreciate this last step, instrumentalists do not need it to benefit from the actuarial procedures of cliometric metatheory. Introduction A plausible proxy (...) for Peircean consensus Assessing the validity of theory attributes as predictors of theory survival 3.1 Linear discriminant function 3.2 Factor analysis 3.3 Taxometric analysis Verisimilitude index Satisfying both instrumentalists and realists Recapitulation Implementation of cliometric metatheory * Correspondence about this article may be addressed to Leslie Yonce at pemeehle{at}umn.edu' + u + '@' + d + ''//--> This article had been completed by Paul Meehl at the time of his death on 14 February 2003. His wife, Leslie J. Yonce, is grateful to Keith Gunderson (University of Minnesota, Center for Philosophy of Science) and Niels G. Waller (Psychology Department, Vanderbilt University) for advice with some final editing details. (shrink)

Assuming it rational to treat other persons' probabilities as epistemically significant, how shall their judgements be weighted (Barnes [1998])? Several plausible methods exist, but theorems in classical psychometrics greatly reduce the importance of the problem. If scientists' judgements tend to be positively correlated, the difference between two randomly weighted composites shrinks as the number of judges rises. Since, for reasons such as representative coverage, minimizing bias, and avoiding elitism, we would rarely employ small numbers of judges (e.g. less than 10), (...) the difference between two weighting systems becomes negligible. Suggestions are made for quantifying verisimilitude, identifying 'types' of scientists or theories (taxometrics), inferring latent factors, and estimating reliability of pooled judgements. (shrink)