This book offers a philosophy for error-prone humans trying to understand messy systems in the real world.

To what use can causal claims established in good studies be put? We give examples of studies from which inaccurate inferences were made about target policy situations. The usual diagnosis is that the studies in question lack external validity, which means that the same results do not hold in the target as in study. That’s a label that just repeats what we already knew. We offer a deeper analysis. Our analysis points to the need for interdisciplinarity and to the demand (...) to focus not on the study – as the expression ‘external validity’ invites you to do – but on the target. The call for interdisciplinary approaches to real life problems is common since it is widely acknowledged that what happens in the real world seldom falls under the auspices of any single research domain. Our focus is on one specific real life problem: how to use causal claims from good studies to help predict whether the policies tested will work in a new situation. Our analysis of what it takes to back up these predictions points up very specific stages in the process of prediction where we are bound to get it wrong if we do not diversify our concepts, our knowledge and our methods. We isolate two reasons inferences from study to target fail. First, policy variables do not produce results on their own; they need helping factors. The distribution of helping factors is likely to be unique or local for each study, so one cannot expect external validity to be all that common. Second, researchers often give too concrete a description of the cause in the study for it to carry over to the target.ion is necessary to get causes that travel. There is no sure-fire way to guard against these problems. But the unavailability of one perfect tool does not imply there are no second best contrivances. Two general pointers for Good Practice in policy advice follow from our diagnosis: focus on the concrete details in the target and use cross discipline heuristics that diversify background knowledge. (shrink)

To what use can causal claims established in good studies be put? We give examples of studies from which inaccurate inferences were made about target policy situations. The usual diagnosis is that the studies in question lack external validity, which means that the same results do not hold in the target as in study. That’s a label that just repeats what we already knew. We offer a deeper analysis. Our analysis points to the need for interdisciplinarity and to the demand (...) to focus not on the study – as the expression ‘external validity’ invites you to do – but on the target. The call for interdisciplinary approaches to real life problems is common since it is widely acknowledged that what happens in the real world seldom falls under the auspices of any single research domain. Our focus is on one specific real life problem: how to use causal claims from good studies to help predict whether the policies tested will work in a new situation. Our analysis of what it takes to back up these predictions points up very specific stages in the process of prediction where we are bound to get it wrong if we do not diversify our concepts, our knowledge and our methods. We isolate two reasons inferences from study to target fail. First, policy variables do not produce results on their own; they need helping factors. The distribution of helping factors is likely to be unique or local for each study, so one cannot expect external validity to be all that common. Second, researchers often give too concrete a description of the cause in the study for it to carry over to the target.ion is necessary to get causes that travel. There is no sure-fire way to guard against these problems. But the unavailability of one perfect tool does not imply there are no second best contrivances. Two general pointers for Good Practice in policy advice follow from our diagnosis: focus on the concrete details in the target and use cross discipline heuristics that diversify background knowledge. (shrink)