We argue that behavioural public policies (BPP) should be categorized by the kind of mechanism through which they operate, not by the kind of treatment they implement. Reviewing the energy consumption BPP literature, we argue (i) that BPPs are currently categorized by treatment; (ii) that treatment-based categories are subject to mechanistic heterogeneity: there is substantial variation of mechanisms within each treatment type; and (iii) that they also display mechanistic overlap: there is substantial overlap between mechanisms across treatment types. Consequently, current (...) categorizations of BPPs do not reveal the conditions of their efficacy and should be revised to better reflect mechanistic information. (shrink)

Public policymakers and institutional decision-makers routinely face questions about whether interventions “work”: does universal basic income improve people’s welfare and stimulate entrepreneurial activity? Do gated alleyways reduce burglaries or merely shift the crime burden to neighbouring communities? What is the most cost-effective way to improve students’ reading abilities? These are empirical questions that seem best answered by looking at the world, rather than trusting speculations about what will be effective. Evidence-based policy (EBP) is a movement that concretizes this intuition. It (...) maintains that policy should be based on evidence of “what works.” Not any evidence will do, however. Following on the heels of its intellectual progenitor, evidence-based medicine (EBM), EBP insists on the use of high-quality evidence produced in accordance with rigorous methodological standards. Though intuitively compelling, EBP has attracted significant criticism from methodologists, political scientists, and philosophers. This chapter provides a critical overview of EBP through a philosophical lens, reviewing and discussing some of the most pressing challenges that EBP faces and outlining some proposals for improving it. I first provide a brief overview of EBP and distinguish a broader and narrower understanding of it. I then review existing lines of criticism of EBP, two of which are considered in detail. The first elaborates how EBP struggles with extrapolation, that is, the use of evidence from study populations to make inferences about the effects of policies on novel target populations. The second maintains that EBP’s methodological tenets are deeply entwined with moral and political values, which can threaten EBP’s promise to promote objectivity in policymaking. Finally, I make some proposals for how EBP could be improved going forward. (shrink)

Sustainability science seeks to extend scientific investigation into domains characterized by a distinct problem-solving agenda, physical and social complexity, and complex moral and ethical landscapes. In this endeavor it arguably pushes scientific investigation beyond its usual comfort zones, raising fundamental issues about how best to structure such investigation. Philosophers of science have long scrutinized the structure of science and scientific practices, and the conditions under which they operate effectively. We propose a critical engagement between sustainability scientists and philosophers of science (...) with respect to how to engage in scientific activity in these complex domains. We identify specific issues philosophers of science raise concerning current sustainability science and the contributions philosophers can make to resolving them. In conclusion we reflect on the steps philosophers of science could take to advance sustainability science. (shrink)

In the human sciences, experimental research is used to establish causal relationships. However, the extrapolation of these results to the target population can be problematic. To facilitate extrapolation, we propose to use the statistical technique Latent Class Regression Analysis in combination with the analogical reasoning theory for extrapolation. This statistical technique can identify latent classes that differ in the effect of X on Y. In order to extrapolate by means of analogical reasoning, one can characterize the latent classes by a (...) combination of features, and then compare these features to features of the target. (shrink)

This article introduces comparative process tracing as a two-step methodological approach that combines theory, chronology, and comparison. For each studied case, the processes leading “from A to B” are reconstructed and analyzed in terms of ideal-type social mechanisms and then compared by making use of the identified mechanisms and ideal-type periodization. Central elements of CPT are path dependence, critical junctures and focal points, social mechanisms, context, periodization, and counterfactual analysis. The CPT approach is described, discussed, and compared with more formal (...) and deterministic forms of process tracing, which are found to be less fruitful for systematic comparison. (shrink)

Although researchers agree on the importance of validity, they have not yet reached a consensus on what validity consists of. This article traces the historic trajectories of validity theory development in three research traditions: psychometrics, experiment in social settings, and animal models of psychiatric disorders, showing that the multiplicity in the concept of validity is shaped by its multiple historic trajectories and reflects the diversity of practices and concerns in different research traditions. I argue that specifying validity of what target (...) practice and for what purpose in discussions helps to connect validity to its rich context that gives rise to its specific meaning and relevance. (shrink)

Proponents of evidence-based medicine and some philosophers of science seem to agree that knowledge of mechanisms can help solve the problem of applying results of controlled studies to target populations (‘the problem of extrapolation’). We describe the problem of extrapolation, characterize mechanisms, and outline how mechanistic knowledge might be used to solve the problem. Our main thesis is that there are four often overlooked problems with using mechanistic knowledge to solve the problem of extrapolation. First, our understanding of mechanisms is (...) often (and arguably, likely to remain) incomplete. Secondly, knowledge of mechanisms is not always applicable outside the tightly controlled laboratory conditions in which it is gained. Thirdly, mechanisms can behave paradoxically. Fourthly, as Daniel Steel points out, using mechanistic knowledge faces the problem of the ‘extrapolator’s circle’. At the same time, when the problems with mechanistic knowledge have been addressed, such knowledge can and should be used to mitigate (nothing can entirely solve) the problem of extrapolation. (shrink)

Résumé Les questions d’échelle ont donné lieu à une large réflexion s’agissant de la modélisation et de l’expérimentation des systèmes physiques, mais elles n’ont pas été discutées dans le cadre des expériences économiques. Dans cet article, on distingue deux sortes d’expériences, les expériences « génériques » et « spécifiques ». Développant une comparaison entre deux études expérimentales en laboratoire portant sur l’« effet des prix postés », on montre que les questions d’échelle deviennent importantes dans les expériences spécifiques en raison (...) de la réduction aux dimensions du laboratoire de l’échelle de temps propre au marché cible. Cela impose des choix dans la configuration matérielle de l’expérience comme des changements de rôle pour les sujets expérimentaux. Notre discussion, donc, s’inscrit au sein de la littérature récente sur la validité externe et la matérialité des expériences. JEL Classification : B41, C90, L92, L98. (shrink)

This paper argues that an important type of experiment-target inference, extrapolating causal effects, requires models to be successful. Focusing on extrapolation in Evidence-Based Policy, it is ar...

How are middle-range theories about causal mechanisms built from case studies in the social sciences? My aim is to answer this question by analyzing and improving Derek Beach and Rasmus Brun Pedersen’s account of the method of theory-building process-tracing. After having introduced the basic issues and concepts, I move on to analyze their descriptions of theory-building process-tracing. I identify some ambiguities and problems in their notions of middle-range theory and causal mechanism. In the constructive part of the paper, I provide (...) accounts of causal mechanisms and middle-range theories that aim to resolve the problems and ambiguities in Beach and Pedersen’s views. I close the paper by showing how the method of theory-building process-tracing can be improved by utilizing these views. (shrink)

The experimental revolution in the social sciences is one of the most significant methodological shifts undergone by the field since the ‘quantitative revolution’ in the nineteenth century. One of the often valued features of social science experimentation is precisely the fact that there are clear methodological rules regarding hypothesis testing that come from the methods of the natural sciences and from the methodology of RCTs in the biomedical sciences, and that allow for the adjudication among contentious causal claims. We examine (...) critically this claim and argue that some current understandings of the practices that surround social science experimentation overestimate the degree to which experiments can actually fulfil this role as “objective” adjudicators, by neglecting the importance of shared background knowledge or assumptions and of consensus regarding the validity of the constructs involved in an experiment. We take issue with the way the distinction between internal and external validity is often used to comment on the inferential import of experiments, used both among practitioners and among philosophers of science. We describe the ways in which the more common use of the internal/external distinction differs from Cook and Campbell’s original methodological project, in which construct validity and the four-fold validity typology were all important in assessing the inferential import of experiments. We argue that the current uses of the labels internal and external, as applied to experimental validity, help to encroach a simplistic view on the inferential import of experiments that, in turn, misrepresents their capacity to provide objective knowledge about the causal relations between variables. (shrink)

Martin Jones has criticized my account of the methodology of experimental economics on three points: the impossibility of testing external validity claims in the laboratory, my reconstruction of external validity inferences as analogical arguments, and the distinction between laboratory and non-laboratory sciences. I defend my account here and try to eliminate some misunderstandings that may have prompted Jones’s criticism.

Evidential pluralists, like Federica Russo and Jon Williamson, argue that causal claims should be corroborated by establishing both the existence of a suitable correlation and a suitable mechanism complex. At first glance, this fits well with mixed method research in the social sciences, which often involves a pluralist combination of statistical and mechanistic evidence. However, statistical evidence concerns a population of cases, while mechanistic evidence is found in individual case studies. How should researchers combine such general statistical evidence and specific (...) mechanistic evidence? This article discusses a very recent answer to this question, ‘multi-method large-N qualitative analysis’ or multi-method LNQA, popular in political science and international relations studies of rare events like democratic transitions and cease-fire agreements. Multi-method LNQA combines a comprehensive study of all relevant event cases with statistical analysis, in an attempt to solve the issues of generalization faced by other types of qualitative research, such as selection bias and lack of representativeness. I will argue that the kind of general causal claim that multi-method LNQA is after, however, is crucially different from the average treatment effect found in statistical analysis and can in fact only be supported with mechanistic evidence. I conclude from this that mixed method research, and thereby evidential pluralism, may be inappropriate in this context. (shrink)

Jeffrey Friedman’s Power Without Knowledge argues forcefully that there are inherent limitations to the predictability of human action, due to a circumstance he calls “ideational heterogeneity.” However, our resources for predicting human action somewhat reliably in the light of ideational heterogeneity have not been exhausted yet, and there are no in-principle barriers to progress in tackling the problem. There are, however, other strong reasons to think that disagreement among epistocrats is bound to persist, such that it will be difficult to (...) decide who has “the right answer” to a given technocratic problem. These reasons have to do with competing visions of the good society, fact/value entanglement, and the fragility of the facts of the social sciences. (shrink)

Francesco Guala once wrote that ‘The problem of extrapolation is a minor scandal in the philosophy of science’. This paper agrees with the statement, but for reasons different from Guala’s. The scandal is not, or not any longer, that the problem has been ignored in the philosophy of science. The scandal is that framing the problem as one of external validity encourages poor evidential reasoning. The aim of this paper is to propose an alternative—an alternative which constitutes much better evidential (...) reasoning about target systems of interest, and which makes do without consideration of external validity. (shrink)

Russo and Williamson put forward the following thesis: in order to establish a causal claim in medicine, one normally needs to establish both that the putative cause and putative effect are appropriately correlated and that there is some underlying mechanism that can account for this correlation. I argue that, although the Russo-Williamson thesis conflicts with the tenets of present-day evidence-based medicine, it offers a better causal epistemology than that provided by present-day EBM because it better explains two key aspects of (...) causal discovery. First, the thesis better explains the role of clinical studies in establishing causal claims. Second, it yields a better account of extrapolation. (shrink)

This paper addresses the problem of how scientific knowledge, which is always locally generated, becomes accepted in other sites. The analysis suggests that there are a small number of strategies that enable scientists to resituate knowledge and that these strategies are generic: they are not restricted to specific disciplines or modes of doing science but rather are found in a variety of different forms across the sciences.

While the history and methodology of laboratory experiments in economics have been extensively studied by philosophers, those of field experiments have not attracted much attention until recently. What is the historical context in which field experiments have been advocated? And what are the methodological rationales for conducting experiments in the field as opposed to in the lab? This article addresses these questions by combining historical and methodological perspectives. In terms of history, we show that the movement toward field experiments in (...) economics has two distinct roots. One is the general orientation of medical and social sciences to evidence-based policy evaluation, which gave rise to randomized field experiments in economics (e.g., behavioral public policy, poverty alleviation policy). The other is an awareness of several methodological limitations of lab experiments in economics, which required practitioners to get out of the lab and into the field. In these senses, the movement is a consequence of influences from both outside and inside economics: the general evidence-based trend in policy science and an internal methodological development of experimental economics. In terms of methodology, we show that these two roots resulted in two somewhat different notions of “external validity” as methodological rationales of field experiment. Finally, we suggest that analysis of experiments as exhibits highlights a methodological strategy in which both strands complement each other. (shrink)

This article discusses an epistemological problem faced by causal explanations of action and a proposed solution. The problem is to justify why one particular reason rather than another is specified as causally efficacious. It is argued that the problem arises independently of one’s preferred conception of singular causal claims, psychological and psychophysical generalizations, and our folk-psychological competence. The proposed fallibilist solution involves the supplementation of the reason given by narratives that contextualize it and provide additional criteria for justifying the causal (...) claim. It is argued that narratives have a distinctive structure that can afford the justification of causal attributions without sui generis powers of narrative explanation having to be invoked. (shrink)

Amalgamating evidence from heterogeneous sources and across levels of inquiry is becoming increasingly important in many pure and applied sciences. This special issue provides a forum for researchers from diverse scientific and philosophical perspectives to discuss evidence amalgamation, its methodologies, its history, its pitfalls, and its potential. We situate the contributions therein within six themes from the broad literature on this subject: the variety-of-evidence thesis, the philosophy of meta-analysis, the role of robustness/sensitivity analysis for evidence amalgamation, its bearing on questions (...) of extrapolation and external validity of experiments, its connection with theory development, and its interface with causal inference, especially regarding causal theories of cancer. (shrink)

External validity is typically regarded as the downside of case study research by methodologists and social scientists; case studies, however, are often aimed at drawing lessons that are generalizable to new contexts. The gap between the generalizability potential of case studies and the research goals demands closer scrutiny. I suggest that the conclusion that case study research is weak in external validity follows from a set of assumptions that I term the "traditional view," which are disputable at best. In this (...) view, external validity is treated as a matter of mere representativeness. I argue that it is best understood instead as a problem of inference and that the emphasis should be placed on the comparability of the study rather than on the typicality of the case. By making case studies highly comparable, their external validity can be reliably and efficiently assessed and, in this way, their generalizability potential enhanced. (shrink)