The claims of RCTs to be the gold standard rest on the fact that the ideal RCT is a deductive method: if the assumptions of the test are met, a positive result implies the appropriate causal conclusion. This is a feature that RCTs share with a variety of other methods, which thus have equal claim to being a gold standard. This paper describes some of these other deductive methods and also some useful non-deductive methods, including the hypothetico-deductive method. It argues (...) that with all deductive methods, the benefit that the conclusions follow deductively in the ideal case comes with a great cost: narrowness of scope. This is an instance of the familiar trade-off between internal and external validity. RCTs have high internal validity but the formal methodology puts severe constraints on the assumptions a target population must meet to justify exporting a conclusion from the test population to the target. The paper reviews one such set of assumptions to show the kind of knowledge required. The overall conclusion is that to draw causal inferences about a target population, which method is best depends case-by-case on what background knowledge we have or can come to obtain. There is no gold standard. (shrink)

Sections include: experiments and generalised causal inference; statistical conclusion validity and internal validity; construct validity and external validity; quasi-experimental designs that either lack a control group or lack pretest observations on the outcome; quasi-experimental designs that use both control groups and pretests; quasi-experiments: interrupted time-series designs; regresssion discontinuity designs; randomised experiments: rationale, designs, and conditions conducive to doing them; practical problems 1: ethics, participation recruitment and random assignment; practical problems 2: treatment implementation and attrition; generalised causal inference: a grounded theory; (...) generalised causal inference: methods for single studies; generalised causal inference: methods for multiple studies; a critical assessment of our assumptions. (shrink)

Randomized controlled trials (RCTs) are widely taken as the gold standard for establishing causal conclusions. Ideally conducted they ensure that the treatment ‘causes’ the outcome—in the experiment. But where else? This is the venerable question of external validity. I point out that the question comes in two importantly different forms: Is the specific causal conclusion warranted by the experiment true in a target situation? What will be the result of implementing the treatment there? This paper explains how the probabilistic theory (...) of causality implies that RCTs can establish causal conclusions and thereby provides an account of what exactly that causal conclusion is. Clarifying the exact form of the conclusion shows just what is necessary for it to hold in a new setting and also how much more is needed to see what the actual outcome would be there were the treatment implemented. (shrink)

The claims of randomized controlled trials to be the gold standard rest on the fact that the ideal RCT is a deductive method: if the assumptions of the test are met, a positive result implies the appropriate causal conclusion. This is a feature that RCTs share with a variety of other methods, which thus have equal claim to being a gold standard. This article describes some of these other deductive methods and also some useful non-deductive methods, including the hypothetico-deductive method. (...) It argues that with all deductive methods, the benefit that the conclusions follow deductively in the ideal case comes with a great cost: narrowness of scope. This is an instance of the familiar trade-off between internal and external validity. RCTs have high internal validity but the formal methodology puts severe constraints on the assumptions a target population must meet to justify exporting a conclusion from the test population to the target. The article reviews one such set of assumptions to show the kind of knowledge required. The overall conclusion is that to draw causal inferences about a target population, which method is best depends case-by-case on what background knowledge we have or can come to obtain. There is no gold standard. (shrink)

Many weaknesses of game theory are cured by new models that embody simple cognitive principles, while maintaining the formalism and generality that makes game theory useful. Social preference models can generate team reasoning by combining reciprocation and correlated equilibrium. Models of limited iterated thinking explain data better than equilibrium models do; and they self-repair problems of implausibility and multiplicity of equilibria.

This paper discusses the so-called non-interference assumption (NIA) grounding causal inference in trials in both medicine and the social sciences. It states that for each participant in the experiment, the value of the potential outcome depends only upon whether she or he gets the treatment. Drawing on methodological discussion in clinical trials and laboratory experiments in economics, I defend the necessity of partial forms of blinding as a warrant of the NIA, to control the participants’ expectations and their strategic interactions (...) with the experimenter. (shrink)

This paper discusses the so-called non-interference assumption (NIA) grounding causal inference in trials in both medicine and the social sciences. It states that for each participant in the experiment, the value of the potential outcome depends only upon whether she or he gets the treatment. Drawing on methodological discussion in clinical trials and laboratory experiments in economics, I defend the necessity of partial forms of blinding as a warrant of the NIA, to control the participants’ expectations and their strategic interactions (...) with the experimenter. (shrink)

Until recently, economics was generally understood to be a nonexperimental science with a hypothetico‐deductive methodology. This article considers how the methodology of economics has changed with the spread of experimental methods. Initially, most experimental economists saw their work as testing pre‐existing theories. However, a method of systematic inductive enquiry in which theory plays a less central role is now evolving. This method is structured around the discovery and progressive refinement of regularities. “Exhibits”—experimental designs that generate significant regularities—are taking over some (...) of the functions formerly performed by theoretical models. †To contact the author, please write to: School of Economics, University of East Anglia, Norwich NR4 7TJ, UK; e‐mail: [email protected]. (shrink)

In order to make scientific results relevant to practical decision making, it is often necessary to transfer a result obtained in one set of circumstances—an animal model, a computer simulation, an economic experiment—to another that may differ in relevant respects—for example, to humans, the global climate, or an auction. Such inferences, which we can call extrapolations, are a type of argument by analogy. This essay sketches a new approach to analogical inference that utilizes chain graphs, which resemble directed acyclic graphs (...) (DAGs) except in allowing that nodes may be connected by lines as well as arrows. This chain graph approach generalizes the account of extrapolation I provided in my (2008) book and leads to new insights that integrate the contributions of the other participants of this symposium. More specifically, this approach explicates the role of “fingerprints,” or distinctive markers, as a strategy for avoiding an underdetermination problem having to do with spurious analogies. Moreover, it shows how the extrapolator’s circle, one of the central challenges for extrapolation highlighted in my book, is closely tied to distinctive markers and the Markov condition as it applies to chain graphs. Finally, the approach suggests additional ways in which investigations of a model can provide information about a target that are illustrated by examples concerning nanomaterials in sunscreens and Wendy Parker’s discussion of fingerprints in climate science. (shrink)

An article by Alexandra Kirsch accepted for publication in Cognitive Processing occasioned debate among reviewers about broad methodological issues in cognitive science. One of these issues is the proper place of Popperian falsificationism in the interdisciplinary cluster. Another is the tension between abstract models and theories that apply to wide classes of cognitive systems, and models of more restricted scope intended to predict specifically human patterns of thought and behavior. The lead editorial in a Commentary debate invited by the journal’s (...) editors considers these issues from the perspective of a pragmatist philosophy of science inspired by Herbert Simon’s classic, MIT Press, Cambridge, 1969) reflections on the blurring of the distinction between science and engineering in cognitive science. (shrink)

Francesco Guala once wrote that ‘The problem of extrapolation is a minor scandal in the philosophy of science’. This paper agrees with the statement, but for reasons different from Guala’s. The scandal is not, or not any longer, that the problem has been ignored in the philosophy of science. The scandal is that framing the problem as one of external validity encourages poor evidential reasoning. The aim of this paper is to propose an alternative—an alternative which constitutes much better evidential (...) reasoning about target systems of interest, and which makes do without consideration of external validity. (shrink)

The experimental approach in economics is a driving force behind some of the most exciting developments in the field. The 'experimental revolution' was based on a series of bold philosophical premises which have remained until now mostly unexplored. This book provides the first comprehensive analysis and critical discussion of the methodology of experimental economics, written by a philosopher of science with expertise in the field. It outlines the fundamental principles of experimental inference in order to investigate their power, scope and (...) limitations. The author demonstrates that experimental economists have a lot to gain by discussing openly the philosophical principles that guide their work, and that philosophers of science have a lot to learn from their ingenious techniques devised by experimenters in order to tackle difficult scientific problems. (shrink)

I consider recent strategies proposed by econometricians for extrapolating causal effects from experimental to target populations. I argue that these strategies fall prey to the extrapolator’s circle: they require so much knowledge about the target population that the causal effects to be extrapolated can be identified from information about the target alone. I then consider comparative process tracing as a potential remedy. Although specifically designed to evade the extrapolator’s circle, I argue that CPT is unlikely to facilitate extrapolation in typical (...) econometrics and evidence-based policy applications. To argue this, I offer a distinction between two kinds of extrapolation, attributive and predictive, the latter being prevalent in econometrics and evidence-based policy. I argue that CPT is not helpful for predictive extrapolation when using the kinds of evidence that econometricians and evidence-based policy researchers prefer. I suggest that econometricians may need to consider qualitative evidence to overcome this problem. (shrink)

A series of recent debates in experimental economics have associated demand effects with the artificiality of the experimental setting and have linked it to the problem of external validity. In this paper, we argue that these associations can be misleading, partly because of the ambiguity with which “artificiality” has been defined, but also because demand effects and external validity are related in complex ways. We argue that artificiality may be directly as well as inversely correlated with demand effects. We also (...) distinguish between the demand effects of experimentation and the reactions that they may trigger and that might endanger experimental validity. We conclude that economists should pay more attention to the way in which subjects construe the experimental task and learn to exploit subjects’ reactivity to expectations in their experiments. (shrink)

This target article is concerned with the implications of the surprisingly different experimental practices in economics and in areas of psychology relevant to both economists and psychologists, such as behavioral decision making. We consider four features of experimentation in economics, namely, script enactment, repeated trials, performance-based monetary payments, and the proscription against deception, and compare them to experimental practices in psychology, primarily in the area of behavioral decision making. Whereas economists bring a precisely defined “script” to experiments for participants to (...) enact, psychologists often do not provide such a script, leaving participants to infer what choices the situation affords. By often using repeated experimental trials, economists allow participants to learn about the task and the environment; psychologists typically do not. Economists generally pay participants on the basis of clearly defined performance criteria; psychologists usually pay a flat fee or grant a fixed amount of course credit. Economists virtually never deceive participants; psychologists, especially in some areas of inquiry, often do. We argue that experimental standards in economics are regulatory in that they allow for little variation between the experimental practices of individual researchers. The experimental standards in psychology, by contrast, are comparatively laissez-faire. We believe that the wider range of experimental practices in psychology reflects a lack of procedural regularity that may contribute to the variability of empirical findings in the research fields under consideration. We conclude with a call for more research on the consequences of methodological preferences, such as the use on monetary payments, and propose a “do-it-both-ways” rule regarding the enactment of scripts, repetition of trials, and performance-based monetary payments. We also argue, on pragmatic grounds, that the default practice should be not to deceive participants. Key Words: behavioral decision making; cognitive illusions; deception; experimental design; experimental economics; experimental practices; financial incentives; learning; role playing. (shrink)

The Ultimatum Game is one of the most successful experimental designs in the history of the social sciences. In this article I try to explain this success—what makes it a “paradigmatic experiment”—stressing in particular its versatility. Despite the intentions of its inventors, the Ultimatum Game was never a good design to test economic theory, and it is now mostly used as a heuristic tool for the observation of nonstandard preferences or as a “social thermometer” for the observation of culture‐specific norms. (...) †To contact the author, please write to: Department of Sociology and Philosophy, University of Exeter, Amory Building, Exeter EX4 4DT, UK; e‐mail: [email protected]. (shrink)

Comparative process tracing is the best analysis of extrapolation inferences in the philosophical and scientific literature so far. In this essay I examine some similarities and differences between comparative process tracing and former attempts to capture the logic of extrapolation, such as the analogical approach. I show that these accounts are not different in spirit, although comparative process tracing supersedes previous proposals in terms of analytical detail. I also examine some qualms about the possibility of drawing extrapolation inferences in the (...) social sciences and conclude by suggesting that there may be cases of extrapolation without process tracing. (shrink)

Experimental “localism” stresses the importance of context‐specific knowledge, and the limitations of universal theories in science. I illustrate Latour's radical approach to localism and show that it has some unpalatable consequences, in particular the suggestion that problems of external validity (or how to generalize experimental results to nonlaboratory circumstances) cannot be solved. In the last part of the paper I try to sketch a solution to the problem of external validity by extending Mayo's error‐probabilistic approach.

This book presents a history of behavioral economics. The recurring theme is that behavioral economics reflects and contributes to a fundamental reorientation of the epistemological foundations upon which economics had been based since the days of Smith, Ricardo, and Mill. With behavioral economics, the discipline has shifted from grounding its theories in generalized characterizations to building theories from behavioral assumptions directly amenable to empirical validation and refutation. The book proceeds chronologically and takes the reader from von Neumann and Morgenstern's axioms (...) of rational behavior, through the incorporation of rational decision theory in psychology in the 1950s–70s, to the creation and rise of behavioral economics in the 1980s and 1990s at the Sloan and Russell Sage Foundations. (shrink)

What motives underlie the ways humans interact socially? Are these the same for all societies? Are these part of our nature, or influenced by our environments?Over the last decade, research in experimental economics has emphatically falsified the textbook representation of Homo economicus. Literally hundreds of experiments suggest that people care not only about their own material payoffs, but also about such things as fairness, equity and reciprocity. However, this research left fundamental questions unanswered: Are such social preferences stable components of (...) human nature; or, are they modulated by economic, social and cultural environments? Until now, experimental research could not address this question because virtually all subjects had been university students, and while there are cultural differences among student populations throughout the world, these differences are small compared to the full range of human social and cultural environments. A vast amount of ethnographic and historical research suggests that people's motives are influenced by economic, social, and cultural environments, yet such methods can only yield circumstantial evidence about human motives. Combining ethnographic and experimental approaches to fill this gap, this book breaks new ground in reporting the results of a large cross-cultural study aimed at determining the sources of social preferences that underlie the diversity of human sociality. The same experiments which provided evidence for social preferences among university students were performed in fifteen small-scale societies exhibiting a wide variety of social, economic and cultural conditions by experienced field researchers who had also done long-term ethnographic field work in these societies. The findings of these experiments demonstrated that no society in which experimental behaviour is consistent with the canonical model of self-interest. Indeed, results showed that the variation in behaviour is far greater than previously thought, and that the differences between societies in market integration and the importance of cooperation explain a substantial portion of this variation, which individual-level economic and demographic variables could not. Finally, the extent to which experimental play mirrors patterns of interaction found in everyday life is traced.The book starts with a succinct but substantive introduction to the use of game theory as an analytical tool and its use in the social sciences for the rigorous testing of hypotheses about fundamental aspects of social behaviour outside artificially constructed laboratories. The results of the fifteen case studies are summarized in a suggestive chapter about the scope of the project. (shrink)

The authors explore the history of experiments in economics, provide examples of different types of experiments and show that the growing use of experimental methods is transforming economics into an empirical science.

Analyzes the nature of international disagreements and conflict resolution in terms of game theory and non-zero-sum games.

Inferences like these are known as extrapolations.

This is a slightly longer version of an entry prepared for the 2nd edition of The New Palgrave Dictionary of Economics, edited by Steven Durlauf and Lawrence Blume (Palgrave-Macmillan, forthcoming). Since the New Palgrave does not include acknowledgments, I should use this chance to thank Roger Backhouse, Philippe Fontaine, Daniel Kahneman, Kyu Sang Lee, Ivan Moscati, and Vernon Smith for their help and suggestions in preparing this paper.

It is widely believed that cost-sharing - charging a subsidized, positive price - for a health product is necessary to avoid wasting resources on those who will not use or do not need the product. We explore this argument in the context of a field experiment in Kenya, in which we randomized the price at which pregnant women could buy long lasting anti-malarial insecticide-treated nets (ITNs) at prenatal clinics. We find no evidence that cost-sharing reduces wastage on those that will (...) not use the product: women who received free ITNs are not less likely to use them than those who paid subsidized positive prices. We also find no evidence that cost-sharing induces selection of women who need the net more: those who pay higher prices appear no sicker than the prenatal clients in the control group in terms of measured anemia (an important indicator of malaria). Cost-sharing does, however, considerably dampen demand. We find that uptake drops by 75 percent when the price of ITNs increases from 0 to $0.75, the price at which ITNs are currently sold to pregnant women in Kenya. We combine our estimates in a cost-effectiveness analysis of ITN prices on infant mortality that incorporates both private and social returns to ITN usage. Overall, given the large positive externality associated with widespread usage of insecticide-treated nets, our results suggest that free distribution to pregnant women is both more effective and more cost-effective than cost-sharing. (shrink)