In this paper we consider the problem of how to measure the strength of statistical evidence from the perspective of evidence amalgamation operations. We begin with a fundamental measurement amalgamation principle : for any measurement, the inputs and outputs of an amalgamation procedure must be on the same scale, and this scale must have a meaningful interpretation vis a vis the object of measurement. Using the p value as a candidate evidence measure, we examine various commonly used approaches to amalgamation (...) of evidence across similar studies, including standard forms of meta-analysis. We show that none of these methods satisfies MAP. Thus an underlying measurement problem remains. We argue that a successful approach to evidence amalgamation necessitates a solution to the problem of evidence measurement, and we suggest some lines of reasoning that might guide further work towards this end. (shrink)

Establishing the effectiveness of medical treatments is one of the most important aspects of medical practice. Bradford Hill's viewpoints play an important role in inferring causality in medicine,...

The explanatory/pragmatic-trial distinction enjoys a burgeoning philosophical and medical literature and a significant contingent of support among philosophers and healthcare stakeholders as an important way to assess the design and results of randomized controlled trials. A major motivation has been the need to provide relevant, generalizable data to drive healthcare decisions. While talk of pragmatic and explanatory trials could be seen as convenient shorthand, the distinction can also be seen as harboring deeper issues related to inferential strategies used to evaluate (...) causal claims regarding medical treatments. A comprehensive, critical analysis of the distinction and underlying epistemological framework upon which the distinction is based, particularly with respect to treatment effectiveness, has yet to be forthcoming. I provide this, analyzing the distinction’s relationship to generalizability and cognate distinctions between ideal conditions and real-world practice, internal and external validity, and efficacy and effectiveness. I also analyze recent philosophical work that relies on the explanatory/pragmatic-trial distinction and that advocates for more pragmatic trials. I conclude that as an organizing principle for trial-design decisions and trial evaluation, the explanatory/pragmatic-trial distinction is conceptually problematic and not as useful as its proponents seem to think. Since some pragmatic-trial features can be inimical to establishing treatment effectiveness, pragmatic-trial features should not be conflated with pragmatic trials’ avowed goals. If the distinction is to be useful, it and some associated concepts, including generalizability, should be reformulated, lest they continue to underlie a medical epistemology that could contribute to methodologically flawed and potentially unethical advice for the design and interpretation of trials. (shrink)

An astonishing volume and diversity of evidence is available for many hypotheses in the biomedical and social sciences. Some of this evidence—usually from randomized controlled trials (RCTs)—is amalgamated by meta-analysis. Despite the ongoing debate regarding whether or not RCTs are the ‘gold-standard’ of evidence, it is usually meta-analysis which is considered the best source of evidence: meta-analysis is thought by many to be the platinum standard of evidence. However, I argue that meta-analysis falls far short of that standard. Different meta-analyses (...) of the same evidence can reach contradictory conclusions. Meta-analysis fails to provide objective grounds for intersubjective assessments of hypotheses because numerous decisions must be made when performing a meta-analysis which allow wide latitude for subjective idiosyncrasies to influence its outcome. I end by suggesting that an older tradition of evidence in medicine—the plurality of reasoning strategies appealed to by the epidemiologist Sir Bradford Hill—is a superior strategy for assessing a large volume and diversity of evidence. (shrink)

If well-designed, the results of a Randomised Clinical Trial can justify a causal claim between treatment and effect in the study population; however, additional information might be needed to carry over this result to another population. RCTs have been criticized exactly on grounds of failing to provide this sort of information Evidence, inference and enquiry. Oxford University Press, New York, 2011), as well as to black-box important details regarding the mechanisms underpinning the causal law instantiated by the RCT result. On (...) the other side, so-called In Silico Clinical Trials face the same criticisms addressed against standard modelling and simulation techniques, and cannot be equated to experiments Philosophy of molecular medicine: foundational issues in research and practice, Routledge, New York, 2017; Parker in Synthese 169:483–496, 2009; Parke in Philos Sci 81:516–536, 2014; Diez Roux in Am J Epidemiol 181:100–102, 2015 and related discussions in Frigg and Reiss in Synthese 169:593–613, 2009; Winsberg in Synthese 169:575–592, 2009; Beisbart and Norton in Int Stud Philos Sci 26:403–422, 2012). We undertake a formal analysis of both methods in order to identify their distinct contribution to causal inference in the clinical setting. Britton et al.’s study :E2098–E2105, 2013) on the impact of ion current variability on cardiac electrophysiology is used for illustrative purposes. We deduce that, by predicting variability through interpolation, ISCTs aid with problems regarding extrapolation of RCTs results, and therefore in assessing their external validity. Furthermore, ISCTs can be said to encode “thick” causal knowledge —as opposed to “thin” difference-making information inferred from RCTs. Hence, ISCTs and RCTs cannot replace one another but rather, they are complementary in that the former provide information about the determinants of variability of causal effects, while the latter can, under certain conditions, establish causality in the first place. (shrink)

The aim of this paper is to reconstruct and critically assess the evidential relationship between neuroscience and educational practice. To do this, I reconstruct a standard way in which evidence from neuroscience is used to support recommendations about educational practice, that is, testing pedagogical interventions using neuroimaging methods, and discuss and critically assess the inference behind this approach. I argue that this inference rests on problematic assumptions, and, therefore, that neuroimaging intervention studies have no special evidential status for basing educational (...) practice. I conclude arguing that these limitations could be resolved by integrating evidence from neurocognitive and educational science. (shrink)

It has recently been argued that successful evidence-based policy should rely on two kinds of evidence: statistical and mechanistic. The former is held to be evidence that a policy brings about the desired outcome, and the latter concerns how it does so. Although agreeing with the spirit of this proposal, we argue that the underlying conception of mechanistic evidence as evidence that is different in kind from correlational, difference-making or statistical evidence, does not correctly capture the role that information about (...) mechanisms should play in evidence-based policy. We offer an alternative account of mechanistic evidence as information concerning the causal pathway connecting the policy intervention to its outcome. Not only can this be analyzed as evidence of difference-making, it is also to be found at any level and is obtainable by a broad range of methods, both experimental and observational. Using behavioral policy as an illustration, we draw the implications of this revised understanding of mechanistic evidence for debates concerning policy extrapolation, evidence hierarchies, and evidence integration. (shrink)

In this paper we argue for a naturalistic solution to some of the methodological controversies in regulatory science, on the basis of two case studies: toxicology and health claim regulation. We analyze the debates related to the scientific evidence that is considered necessary for regulatory decision making in each of those two fields, with a particular attention to the interactions between scientific and regulatory aspects. This analysis allows us to identify two general stances in the debate: a) one that argues (...) for more permissive standards of evidence and for methodological pluralism, and b) an opposing one that not only defends strict evidence requirements but also stipulates the use of one particular scientific methodologies for data generation. We argue that the real-world outcomes produced by alternative regulatory options are a vital piece of information that allows for the empirical assessment of these two stances. In particular, this information on outcomes makes it possible to analyze which standards of evidence and scientific methods generate the most useful knowledge as input for regulatory decision making. Our conclusion is that instead of an a priori selection of methodologies and standards, such decisions ought to be based on empirical evidence related to real-world outcomes. (shrink)

Philosophical discussions on causal inference in medicine are stuck in dyadic camps, each defending one kind of evidence or method rather than another as best support for causal hypotheses. Whereas Evidence Based Medicine advocates the use of Randomised Controlled Trials and systematic reviews of RCTs as gold standard, philosophers of science emphasise the importance of mechanisms and their distinctive informational contribution to causal inference and assessment. Some have suggested the adoption of a pluralistic approach to causal inference, and an inductive (...) rather than hypothetico-deductive inferential paradigm. However, these proposals deliver no clear guidelines about how such plurality of evidence sources should jointly justify hypotheses of causal associations. We here develop such guidelines by first giving a philosophical analysis of the underpinnings of Hill’s viewpoints on causality. We then put forward an evidence-amalgamation framework adopting a Bayesian net approach to model causal inference in pharmacology for the assessment of harms. Our framework accommodates a number of intuitions already expressed in the literature concerning the EBM vs. pluralist debate on causal inference, evidence hierarchies, causal holism, relevance, and reliability. (shrink)

Extrapolating causal effects from experiments to novel populations is a common practice in evidence-based-policy, development economics and other social science areas. Drawing on experimental evidence of policy effectiveness, analysts aim to predict the effects of policies in new populations, which might differ importantly from experimental populations. Existing approaches made progress in articulating the sorts of similarities one needs to assume to enable such inferences. It is also recognized, however, that many of these assumptions will remain surrounded by significant uncertainty in (...) practice. Unfortunately, the existing literature says little on how analysts may articulate and manage these uncertainties. This paper aims to make progress on these issues. First, it considers several existing ideas that bear on issues of uncertainty, elaborates the challenges they face, and extracts some useful rationales. Second, it outlines a novel approach, called the support graph approach, that builds on these rationales and allows analysts to articulate and manage uncertainty in extrapolation in a systematic and unified way. (shrink)

Science-based methods for assessing the practical rationality of a proposed public policy typically represent assumed future outcomes of policies and values attributed to these outcomes in an idealized, that is, intentionally distorted way and abstracted from aspects that are deemed irrelevant. Different types of methods do so in different ways. As a consequence, they instantiate the properties that result from abstraction and idealization such as conceptual simplicity versus complexity, or comprehensiveness versus selectivity of the values under consideration to different degrees. (...) I hold that none of these methods is best in general. Instead, I opt for the valuation method that is useful for the policy issue in question both in terms of its relevance and in terms of its practicability. Relevance requires that the method can represent and account for what is at stake in the policy issue. Practicability refers to aspects such as easy versus difficult handling of the method. To argue for the claim, I evaluate three types of valuation methods: cost–benefit analysis that rests on unidimensional measurement and ranking, multi-criteria decision analysis that applies multi-dimensional measurement but unidimensional ranking, and non-aggregate indicator systems that operate with multi-dimensional measurement and sometimes also multi-dimensional ranking. Second-order justification indicating whether and how the valuation method chosen is capable of accounting for the substantive value considerations that constitute the real-world policy issue in question renders the conditions on which the results of a proposed policy evaluation rest transparent. (shrink)

The term “effective,” on its own, is honorific but vague. Interventions against serious mental illness may be “effective” at goals as diverse as reducing “apparent sadness” or providing housing. Underexamined use of “effective” and other success terms often obfuscates differences and incompatibilities in interventions, degrees of effectiveness, key omissions in effectiveness standards, and values involved in determining what counts as “effective.” Yet vague use of such success terms is common in the research, clinical, and policy realms, with consequences that negatively (...) affect the care offered to individuals experiencing serious mental illness. A pragmatist-oriented solution to these problems suggests that when people use success terms, they need to explain and defend the goals and supporting values embedded in the terms, asking and answering the questions, “Effective at what? For whom? How effective? And why that goal?” Practical and epistemic standards for effectiveness will likely remain plural for good reasons, but each standard should be well explained and well justified. (shrink)

Farmers all over the world perform experiments, and have done so since long before modern experimental science and its recognized forerunners. There is a rich anthropological literature on these experiments, but the philosophical issues that they give rise to have not received much attention. Based on the anthropological literature, this study investigates methodological and philosophical issues pertaining to farmers’ experiments, including the choice of interventions to be tested, the planning of experiments, and the use of control fields and other means (...) to deal with confounding factors. Farmers’ experiments have some advantages over the field trials of agricultural scientists, but also some comparative disadvantages. The two experimental traditions are complementary, and neither of them can replace the other. Several aspects of farmers’ experiments are shown to have a direct bearing on central topics in the philosophy of science. (shrink)

Natural experiments are an increasingly popular research design in political science. This popularity raises a number of questions. First, what are natural experiments and why are they appealing? Second, what makes a good natural experiment? And finally, are natural experiments able to provide resources for knowledge production that other methodologies cannot or do not provide? Using Mary Morgan’s and Thad Dunning’s recent work on natural experiments, I offer answers to the first two questions and use the analysis to argue that (...) natural experiments highlight features of knowledge production that support methodological pluralism and the multiple aims of research. (shrink)

Philosophers of science and medicine now aspire to provide useful, socially relevant accounts of mechanism. Existing accounts have forged the path by attending to mechanisms in historical context, scientific practice, the special sciences, and policy. Yet, their primary focus has been on more proximate issues related to therapeutic effectiveness. To take the next step toward social relevance, we must investigate the challenges facing researchers, clinicians, and policy makers involving values and social context. Accordingly, we learn valuable lessons about the connections (...) between mechanistic processes and more fundamental reasons for medical interventions, particularly moral, ethical, religious, and political concerns about health, agency, and power. This paper uses debates over the controversial morning-after pill to gain insight into the deeper reasons for the production and use of mechanistic knowledge throughout biomedical research, clinical practice, and governmental regulation. To practice socially relevant philosophy of science, I argue that we need to account for mechanistic knowledge beyond immediate effectiveness, such as how it can also provide moral guidance, aid ethical categorization in the clinic, and function as a political instrument. Such insights have implications for medical epistemology, including the value-laden dimensions of mechanistic reasoning and the “epistemic friction” of values. Furthermore, there are broader impacts for teaching research ethics and understanding the role of science advisors as political advocates. (shrink)

Public policymakers and institutional decision-makers routinely face questions about whether interventions “work”: does universal basic income improve people’s welfare and stimulate entrepreneurial activity? Do gated alleyways reduce burglaries or merely shift the crime burden to neighbouring communities? What is the most cost-effective way to improve students’ reading abilities? These are empirical questions that seem best answered by looking at the world, rather than trusting speculations about what will be effective. Evidence-based policy (EBP) is a movement that concretizes this intuition. It (...) maintains that policy should be based on evidence of “what works.” Not any evidence will do, however. Following on the heels of its intellectual progenitor, evidence-based medicine (EBM), EBP insists on the use of high-quality evidence produced in accordance with rigorous methodological standards. Though intuitively compelling, EBP has attracted significant criticism from methodologists, political scientists, and philosophers. This chapter provides a critical overview of EBP through a philosophical lens, reviewing and discussing some of the most pressing challenges that EBP faces and outlining some proposals for improving it. I first provide a brief overview of EBP and distinguish a broader and narrower understanding of it. I then review existing lines of criticism of EBP, two of which are considered in detail. The first elaborates how EBP struggles with extrapolation, that is, the use of evidence from study populations to make inferences about the effects of policies on novel target populations. The second maintains that EBP’s methodological tenets are deeply entwined with moral and political values, which can threaten EBP’s promise to promote objectivity in policymaking. Finally, I make some proposals for how EBP could be improved going forward. (shrink)

Experimental design is one aspect of a scientific method. A well-designed, properly conducted experiment aims to control variables in order to isolate and manipulate causal effects and thereby maximize internal validity, support causal inferences, and guarantee reliable results. Traditionally employed in the natural sciences, experimental design has become an important part of research in the social and behavioral sciences. Experimental methods are also endorsed as the most reliable guides to policy effectiveness. Through a discussion of some of the central concepts (...) associated with experimental design, including controlled variation and randomization, this chapter will provide a summary of key ethical issues that tend to arise in experimental contexts. In addition, by exploring assumptions about the nature of causation and by analyzing features of causal relationships, systems, and inferences in social contexts, this chapter will summarize the ways in which experimental design can undermine the integrity of not only social and behavioral research but policies implemented on the basis of such research. (shrink)

The US Food and Drug Administration (FDA) is fraught with controversies over the role of values and politics in regulatory science, especially with drugs in the realm of reproductive health. Philosophers and science studies scholars have investigated the ways in which social context shapes medical knowledge through value judgments, and feminist scholars and activists have criticized sexism and injustice in reproductive medicine. Nonetheless, there has been no systematic study of values and gender norms in FDA drug regulation. I focus on (...) three questions about values in regulatory science. First, how have societal values and gender norms shaped the way that the FDA regulates drugs in the realm of reproductive health, specifically with drug labels? Second, what are the ethical, epistemic, and social consequences of these influences on regulation for women and other marginalized groups? Third, which societal values and gender norms ought to influence drug regulation about reproductive health, and how ought this happen? Integrating philosophical analysis with historical archival research and in-depth interviews, I conduct three case studies of drug labeling about reproductive health: (1) the “drug fact” about the mechanism of the morning-after pill; (2) the package inserts for patients about the health risks of oral contraceptives; and (3) the special physician labels made for prescribing drugs to pregnant women. I identify three challenges for the FDA and suggest ways to reduce the influence of sexist values and facilitate feminist alternatives. First, across these cases, I have found that there are many ways in which other concerns in reproductive medicine (such as zygotic life, fetal health, and population control) have devalued women’s health. Second, both knowledge and ignorance xvi about their reproductive health have contributed to women’s oppression, especially poor women and women of color. Finally, by avoiding the epistemic dimensions of ethics, powerful, mostly male parties in medicine (such as doctors, pharmaceutical companies, and religious institutions) have misused “informed consent,” “religious freedom,” and “paternalism” for unethical purposes. For improvement, I suggest extracting sexist values and gender norms from regulatory science that cause epistemic injustices, and I point to success stories for reforming sexism with feminism at the FDA. (shrink)