How can philosophy of science be of more practical use? One thing we can do is provide practicable advice about how to determine when one empirical claim is relevant to the truth of another; i.e., about evidential relevance. This matters especially for evidence-based policy, where advice is thin—and misleading—about how to tell what counts as evidence for policy effectiveness. This paper argues that good efficacy results (as in randomized controlled trials), which are all the rage now, are only a very (...) small part of the story. To tell what facts are relevant for judging policy effectiveness, we need to construct causal scenarios about will happen when the policy is implemented. (shrink)

Sections include: experiments and generalised causal inference; statistical conclusion validity and internal validity; construct validity and external validity; quasi-experimental designs that either lack a control group or lack pretest observations on the outcome; quasi-experimental designs that use both control groups and pretests; quasi-experiments: interrupted time-series designs; regresssion discontinuity designs; randomised experiments: rationale, designs, and conditions conducive to doing them; practical problems 1: ethics, participation recruitment and random assignment; practical problems 2: treatment implementation and attrition; generalised causal inference: a grounded theory; (...) generalised causal inference: methods for single studies; generalised causal inference: methods for multiple studies; a critical assessment of our assumptions. (shrink)

Peter Urbach has argued, on Bayesian grounds, that experimental randomization serves no useful purpose in testing causal hypothesis. I maintain that he fails to distinguish general issues of statistical inference from specific problems involved in identifying causes. I concede the general Bayesian thesis that random sampling is inessential to sound statistical inference. But experimental randomization is a different matter, and often plays an essential role in our route to causal conclusions.

The proposed practice of "evidence-based medicine," which calls for careful clinical judgment in evaluating the "best available evidence," should be differentiated from the special collection of data regarded as suitable evidence. Although the proposed practice does not seem new, the new collection of "best available" information has major constraints for the care of individual patients. Derived almost exclusively from randomized trials and meta-analyses, the data do not include many types of treatments or patients seen in clinical practice; and the results (...) show comparative efficacy of treatment for an "average" randomized patient, not for pertinent subgroups formed by such cogent clinical features as severity of symptoms, illness, co-morbidity, and other clinical nuances. The intention-to-treat analyses do not reflect important post-randomization events leading to altered treatment; and the results seldom provide suitable background data when therapy is given prophylactically rather than remedially, or when therapeutic advantages are equivocal. Randomized trial information is also seldom available for issues in etiology, diagnosis, and prognosis, and for clinical decisions that depend on pathophysiologic changes, psychosocial factors and support, personal preferences of patients, and strategies for giving comfort and reassurance. The laudable goal of making clinical decisions based on evidence can be impaired by the restricted quality and scope of what is collected as "best available evidence." The authoritative aura given to the collection, however, may lead to major abuses that produce inappropriate guidelines or doctrinaire dogmas for clinical practice. (shrink)