What kinds of evidence reliably support predictions of effectiveness for health and social care interventions? There is increasing reliance, not only for health care policy and practice but also for more general social and economic policy deliberation, on evidence that comes from studies whose basic logic is that of JS Mill's method of difference. These include randomized controlled trials, case–control studies, cohort studies, and some uses of causal Bayes nets and counterfactual-licensing models like ones commonly developed in econometrics. The topic (...) of this paper is the 'external validity' of causal conclusions from these kinds of studies. We shall argue two claims. Claim, negative: external validity is the wrong idea; claim, positive: 'capacities' are almost always the right idea, if there is a right idea to be had. If we are right about these claims, it makes big problems for policy decisions. Many advice guides for grading policy predictions give top grades to a proposed policy if it has two good Mill's-method-of difference studies that support it. But if capacities are to serve as the conduit for support from a method-of-difference study to an effectiveness prediction, much more evidence, and much different in kind, is required. We will illustrate the complexities involved with the case of multisystemic therapy, an internationally adopted intervention to try to diminish antisocial behaviour in young people. (shrink)

Cartwright and Munro argued that extrapolation of findings from randomized controlled trials to other settings can be difficult because information about the underlying causal structure and subgroups is often not available. They advocated the use of ‘capacities’ – that is fixed causal contributions – in predicting effects of interventions. In psychiatry, it is often not possible to determine what the fixed causal contributions are and one can only establish ‘approximate capacities’. However, using ‘approximate capacities’ does imply a different way of (...) evaluating health services, especially combined interventions. In health service research, if different studies, randomized controlled trials or other designs, have given different outcomes, the best way to investigate the effectiveness of a particular way of service organization is not to conduct more randomized controlled trials. It is preferable to study the effects of certain elements of the complex intervention, which have been tested before in other settings, that is investigating ‘approximate capacities’. One should check whether the separate elements do form a part of the complex intervention in practice and whether they have the same effect as in other studies and if not, why not. This enhances knowledge about the underlying causal structure and increases the possibility of extrapolation of the findings. (shrink)