Measuring the effectiveness of medical interventions faces three epistemological challenges: the choice of good measuring instruments, the use of appropriate analytic measures, and the use of a reliable method of extrapolating measures from an experimental context to a more general context. In practice each of these challenges contributes to overestimating the effectiveness of medical interventions. These challenges suggest the need for corrective normative principles. The instruments employed in clinical research should measure patient-relevant and disease-specific parameters, and should not be sensitive (...) to parameters that are only indirectly relevant. Effectiveness always should be measured and reported in absolute terms (using measures such as 'absolute risk reduction'), and only sometimes should effectiveness also be measured and reported in relative terms (using measures such as 'relative risk reduction')-employment of relative measures promotes an informal fallacy akin to the base-rate fallacy, which can be exploited to exaggerate claims of effectiveness. Finally, extrapolating from research settings to clinical settings should more rigorously take into account possible ways in which the intervention in question can fail to be effective in a target population. (shrink)

Over the past several decades, we devoted much energy to generating, reviewing and summarizing evidence. We have given far less attention to the issue of how to thoughtfully apply the evidence once we have it. That’s fine if all we care about is that our clinical decisions are evidence-based, but not so good if we also want them to be well-reasoned. Let us not forget that evidence based medicine (EBM) grew out of an interest in making medicine ‘rational’, with the (...) idea that rational clinical evaluations should be evidence-based. I agree with the uncontroversial statement that the best decision is supported, at least in part, by the best available evidence. Rationality, however, is constituted by reasoning, not evidence. Complete arguments are necessary for rational evaluations, arguments that begin with general evidence and end in a conclusion about a particular patient. In order to traverse these inferential gaps, medicine must address the issue of how to establish, as an intermediate premise, what the evidence has to say about the efficacy of an intervention for particular patients in a particular practice setting. (shrink)

Policy-makers face an uncertain world. One way of getting a handle on decision-making in such an environment is to rely on evidence. Despite the recent increase in post-fact figures in politics, evidence-based policymaking takes centre stage in policy-setting institutions. Often, however, policy-makers face large volumes of evidence from different sources. Robustness analysis can, prima facie, handle this evidential diversity. Roughly, a hypothesis is supported by robust evidence if the different evidential sources are in agreement. In this thesis, I strengthen the (...) case for the use of robustness analysis in evidence-based policymaking by answering open research questions about this inference technique. First, I argue that existing taxonomies miss a fruitful category of robustness reasoning, that is predictive stability. Second, I claim that derivational robustness analysis – the investigation of whether the results of different models are in agreement – can yield interesting insights even if not the entire relevant model space is covered by available models or if the model results are only partially in agreement. Third, I claim that expert knowledge is necessary to address questions that arise when one applies measurement robustness analysis – the investigation into whether multiple means of measurement yield the same result. Finally, I argue that, in situations where evidence from different measurements is not in agreement, it can be advisable to no longer take all of the evidence into account. This can be done in a rationally defensible way by choosing the most adequate theory or model underlying parts of the evidence set. I discuss examples from climate, medical, and economic policy-making to establish my claims. (shrink)

When appraising research papers, how much understanding is enough? More specifically, in deciding whether research results can inform practice, do appraisers need to substantively understand how findings are derived or is it sufficient simply to grasp that suitable analytic techniques were chosen and used by researchers? The degree or depth of understanding that research appraisers need to attain before findings can legitimately/sensibly inform practice is underexplored. In this paper it is argued that, where knowledge/justified beliefs derived from research evidence prompt (...) actions that materially affect patient care, appraisers have an epistemic duty to demand high (maximal) rather than low (minimal) levels of understanding regards finding derivation (i.e. appraisers have a duty to seek a superior epistemic situation). If this argument holds assumptions about appraiser competence/ability and the feasibility of currentUKconceptions of evidence based practice are destabilized. (shrink)

Simple extrapolation is the orthodox approach to extrapolating from clinical trials in evidence-based medicine: extrapolate the relative effect size from the trial unless there is a compelling reason not to do so. I argue that this method relies on a myth and a fallacy. The myth of simple extrapolation is the idea that the relative risk is a ‘golden ratio’ that is usually transportable due to some special mathematical or theoretical property. The fallacy of simple extrapolation is an unjustified argument (...) from ignorance: we conclude that the relative effect size is transportable in the absence of evidence to the contrary. In short, simple extrapolation is a deeply problematic solution to the problem of extrapolation. (shrink)

Medical necessity (claiming that a medical intervention or care is – at minimum – reasonable, appropriate and acceptable) depends on empirical evidence and on the interpretation of that evidence. Evidence and its interpretation define the standard of care. This commentary argues that both the evidence base and its interpretation are currently weak gatekeepers. Empirical meta-research suggests that very few medical interventions have high quality evidence in support of their effectiveness and very few of them also have relatively thorough assessments of (...) their potential harms. Therefore, evidence on the risk-benefit ratios carries almost always very large uncertainty. Arbitration about medical necessity is thus left to the interpretation process. Professional guidelines are notoriously unreliable and biased in this regard. Regulatory approval ends up being the key arbitrator, but over the years the regulatory process has been subverted. Regulatory approval currently does not mean that an intervention has a favourable risk-benefit ratio, but simply that it can be marketed, sold and made profit from. The process leads to a tragedy of commons where the final victim is society at large: medical necessity is invoked as an alibi for medicine to absorb societal resources. (shrink)