The experimental approach in economics is a driving force behind some of the most exciting developments in the field. The 'experimental revolution' was based on a series of bold philosophical premises which have remained until now mostly unexplored. This book provides the first comprehensive analysis and critical discussion of the methodology of experimental economics, written by a philosopher of science with expertise in the field. It outlines the fundamental principles of experimental inference in order to investigate their power, scope and (...) limitations. The author demonstrates that experimental economists have a lot to gain by discussing openly the philosophical principles that guide their work, and that philosophers of science have a lot to learn from their ingenious techniques devised by experimenters in order to tackle difficult scientific problems. (shrink)

Within public health, and increasingly other areas of social policy, there are widespread calls to increase or improve the use of evidence for policy-making. Often these calls rest on an assumption that increased evidence utilisation will be a more efficient or effective means of achieving social goals. Yet a clear elucidation of what can be considered “good evidence” for policy is rarely articulated. Many of the current discussions of best practise in the health policy sector derive from the evidence-based medicine (...) movement, embracing the “hierarchy of evidence” that places experimental trials as pre-eminent in terms of methodological quality. However, a number of problems arise if these hierarchies are used to rank or prioritise policy relevance. Challenges in applying evidence hierarchies to policy questions arise from the fact that the EBM hierarchies rank evidence of intervention effect on a specified and limited number of outcomes. Previous authors have noted that evidence forms at the top of... (shrink)

It is often supposed that the spectacular successes of our modern mathematical sciences support a lofty vision of a world completely ordered by one single elegant theory. In this book Nancy Cartwright argues to the contrary. When we draw our image of the world from the way modern science works - as empiricism teaches us we should - we end up with a world where some features are precisely ordered, others are given to rough regularity and still others behave in (...) their own diverse ways. This patchwork makes sense when we realise that laws are very special productions of nature, requiring very special arrangements for their generation. Combining classic and newly written essays on physics and economics, The Dappled World carries important philosophical consequences and offers serious lessons for both the natural and the social sciences. (shrink)

According to current hierarchies of evidence for EBM, evidence of correlation (e.g., from RCTs) is always more important than evidence of mechanisms when evaluating and establishing causal claims. We argue that evidence of mechanisms needs to be treated alongside evidence of correlation. This is for three reasons. First, correlation is always a fallible indicator of causation, subject in particular to the problem of confounding; evidence of mechanisms can in some cases be more important than evidence of correlation when assessing a (...) causal claim. Second, evidence of mechanisms is often required in order to obtain evidence of correlation (for example, in order to set up and evaluate RCTs). Third, evidence of mechanisms is often required in order to generalise and apply causal claims. While the EBM movement has been enormously successful in making explicit and critically examining one aspect of our evidential practice, i.e., evidence of correlation, we wish to extend this line of work to make explicit and critically examine a second aspect of our evidential practices: evidence of mechanisms. (shrink)

Inferences like these are known as extrapolations.

Sections include: experiments and generalised causal inference; statistical conclusion validity and internal validity; construct validity and external validity; quasi-experimental designs that either lack a control group or lack pretest observations on the outcome; quasi-experimental designs that use both control groups and pretests; quasi-experiments: interrupted time-series designs; regresssion discontinuity designs; randomised experiments: rationale, designs, and conditions conducive to doing them; practical problems 1: ethics, participation recruitment and random assignment; practical problems 2: treatment implementation and attrition; generalised causal inference: a grounded theory; (...) generalised causal inference: methods for single studies; generalised causal inference: methods for multiple studies; a critical assessment of our assumptions. (shrink)

Experimental “localism” stresses the importance of context‐specific knowledge, and the limitations of universal theories in science. I illustrate Latour's radical approach to localism and show that it has some unpalatable consequences, in particular the suggestion that problems of external validity (or how to generalize experimental results to nonlaboratory circumstances) cannot be solved. In the last part of the paper I try to sketch a solution to the problem of external validity by extending Mayo's error‐probabilistic approach.

Francesco Guala once wrote that ‘The problem of extrapolation is a minor scandal in the philosophy of science’. This paper agrees with the statement, but for reasons different from Guala’s. The scandal is not, or not any longer, that the problem has been ignored in the philosophy of science. The scandal is that framing the problem as one of external validity encourages poor evidential reasoning. The aim of this paper is to propose an alternative—an alternative which constitutes much better evidential (...) reasoning about target systems of interest, and which makes do without consideration of external validity. (shrink)

Hunting Causes and Using Them argues that causation is not one thing, as commonly assumed, but many. There is a huge variety of causal relations, each with different characterizing features, different methods for discovery and different uses to which it can be put. In this collection of new and previously published essays, Nancy Cartwright provides a critical survey of philosophical and economic literature on causality, with a special focus on the currently fashionable Bayes-nets and invariance methods - and it exposes (...) a huge gap in that literature. Almost every account treats either exclusively how to hunt causes or how to use them. But where is the bridge between? It's no good knowing how to warrant a causal claim if we don't know what we can do with that claim once we have it. This book will interest philosophers, economists and social scientists. (shrink)

In order to make scientific results relevant to practical decision making, it is often necessary to transfer a result obtained in one set of circumstances—an animal model, a computer simulation, an economic experiment—to another that may differ in relevant respects—for example, to humans, the global climate, or an auction. Such inferences, which we can call extrapolations, are a type of argument by analogy. This essay sketches a new approach to analogical inference that utilizes chain graphs, which resemble directed acyclic graphs (...) (DAGs) except in allowing that nodes may be connected by lines as well as arrows. This chain graph approach generalizes the account of extrapolation I provided in my (2008) book and leads to new insights that integrate the contributions of the other participants of this symposium. More specifically, this approach explicates the role of “fingerprints,” or distinctive markers, as a strategy for avoiding an underdetermination problem having to do with spurious analogies. Moreover, it shows how the extrapolator’s circle, one of the central challenges for extrapolation highlighted in my book, is closely tied to distinctive markers and the Markov condition as it applies to chain graphs. Finally, the approach suggests additional ways in which investigations of a model can provide information about a target that are illustrated by examples concerning nanomaterials in sunscreens and Wendy Parker’s discussion of fingerprints in climate science. (shrink)

It has recently been argued that successful evidence-based policy should rely on two kinds of evidence: statistical and mechanistic. The former is held to be evidence that a policy brings about the desired outcome, and the latter concerns how it does so. Although agreeing with the spirit of this proposal, we argue that the underlying conception of mechanistic evidence as evidence that is different in kind from correlational, difference-making or statistical evidence, does not correctly capture the role that information about (...) mechanisms should play in evidence-based policy. We offer an alternative account of mechanistic evidence as information concerning the causal pathway connecting the policy intervention to its outcome. Not only can this be analyzed as evidence of difference-making, it is also to be found at any level and is obtainable by a broad range of methods, both experimental and observational. Using behavioral policy as an illustration, we draw the implications of this revised understanding of mechanistic evidence for debates concerning policy extrapolation, evidence hierarchies, and evidence integration. (shrink)

Comparative process tracing is the best analysis of extrapolation inferences in the philosophical and scientific literature so far. In this essay I examine some similarities and differences between comparative process tracing and former attempts to capture the logic of extrapolation, such as the analogical approach. I show that these accounts are not different in spirit, although comparative process tracing supersedes previous proposals in terms of analytical detail. I also examine some qualms about the possibility of drawing extrapolation inferences in the (...) social sciences and conclude by suggesting that there may be cases of extrapolation without process tracing. (shrink)

Proponents of evidence-based medicine and some philosophers of science seem to agree that knowledge of mechanisms can help solve the problem of applying results of controlled studies to target populations (‘the problem of extrapolation’). We describe the problem of extrapolation, characterize mechanisms, and outline how mechanistic knowledge might be used to solve the problem. Our main thesis is that there are four often overlooked problems with using mechanistic knowledge to solve the problem of extrapolation. First, our understanding of mechanisms is (...) often (and arguably, likely to remain) incomplete. Secondly, knowledge of mechanisms is not always applicable outside the tightly controlled laboratory conditions in which it is gained. Thirdly, mechanisms can behave paradoxically. Fourthly, as Daniel Steel points out, using mechanistic knowledge faces the problem of the ‘extrapolator’s circle’. At the same time, when the problems with mechanistic knowledge have been addressed, such knowledge can and should be used to mitigate (nothing can entirely solve) the problem of extrapolation. (shrink)

While the history and methodology of laboratory experiments in economics have been extensively studied by philosophers, those of field experiments have not attracted much attention until recently. What is the historical context in which field experiments have been advocated? And what are the methodological rationales for conducting experiments in the field as opposed to in the lab? This article addresses these questions by combining historical and methodological perspectives. In terms of history, we show that the movement toward field experiments in (...) economics has two distinct roots. One is the general orientation of medical and social sciences to evidence-based policy evaluation, which gave rise to randomized field experiments in economics (e.g., behavioral public policy, poverty alleviation policy). The other is an awareness of several methodological limitations of lab experiments in economics, which required practitioners to get out of the lab and into the field. In these senses, the movement is a consequence of influences from both outside and inside economics: the general evidence-based trend in policy science and an internal methodological development of experimental economics. In terms of methodology, we show that these two roots resulted in two somewhat different notions of “external validity” as methodological rationales of field experiment. Finally, we suggest that analysis of experiments as exhibits highlights a methodological strategy in which both strands complement each other. (shrink)