In the biomedical context, policy makers face a large amount of potentially discordant evidence from different sources. This prompts the question of how this evidence should be aggregated in the interests of best-informed policy recommendations. The starting point of our discussion is Hunter and Williams’ recent work on an automated aggregation method for medical evidence. Our negative claim is that it is far from clear what the relevant criteria for evaluating an evidence aggregator of this sort are. What is the (...) appropriate balance between explicitly coded algorithms and implicit reasoning involved, for instance, in the packaging of input evidence? In short: What is the optimal degree of ‘automation’? On the positive side: We propose the ability to perform an adequate robustness analysis as the focal criterion, primarily because it directs efforts to what is most important, namely, the structure of the algorithm and the appropriate extent of automation. Moreover, where there are resource constraints on the aggregation process, one must also consider what balance between volume of evidence and accuracy in the treatment of individual evidence best facilitates inference. There is no prerogative to aggregate the total evidence available if this would in fact reduce overall accuracy. (shrink)

Background: Evidence suggesting adverse drug reactions often emerges unsystematically and unpredictably in form of anecdotal reports, case series and survey data. Safety trials and observational studies also provide crucial information regarding the (un-)safety of drugs. Hence, integrating multiple types of pharmacovigilance evidence is key to minimising the risks of harm. Methods: In previous work, we began the development of a Bayesian framework for aggregating multiple types of evidence to assess the probability of a putative causal link between drugs and side (...) effects. This framework arose out of a philosophical analysis of the Bradford Hill Guidelines. In this article, we expand the Bayesian framework and add “evidential modulators,” which bear on the assessment of the reliability of incoming study results. The overall framework for evidence synthesis, “E-Synthesis”, is then applied to a case study. Results: Theoretically and computationally, E-Synthesis exploits coherence of partly or fully independent evidence converging towards the hypothesis of interest (or of conflicting evidence with respect to it), in order to update its posterior probability. With respect to other frameworks for evidence synthesis, our Bayesian model has the unique feature of grounding its inferential machinery on a consolidated theory of hypothesis confirmation (Bayesian epistemology), and in allowing any data from heterogeneous sources (cell-data, clinical trials, epidemiological studies), and methods (e.g., frequentist hypothesis testing, Bayesian adaptive trials, etc.) to be quantitatively integrated into the same inferential framework. Conclusions: E-Synthesis is highly flexible concerning the allowed input, while at the same time relying on a consistent computational system, that is philosophically and statistically grounded. Furthermore, by introducing evidential modulators, and thereby breaking up the different dimensions of evidence (strength, relevance, reliability), E-Synthesis allows them to be explicitly tracked in updating causal hypotheses. (shrink)

Policy-makers face an uncertain world. One way of getting a handle on decision-making in such an environment is to rely on evidence. Despite the recent increase in post-fact figures in politics, evidence-based policymaking takes centre stage in policy-setting institutions. Often, however, policy-makers face large volumes of evidence from different sources. Robustness analysis can, prima facie, handle this evidential diversity. Roughly, a hypothesis is supported by robust evidence if the different evidential sources are in agreement. In this thesis, I strengthen the (...) case for the use of robustness analysis in evidence-based policymaking by answering open research questions about this inference technique. First, I argue that existing taxonomies miss a fruitful category of robustness reasoning, that is predictive stability. Second, I claim that derivational robustness analysis – the investigation of whether the results of different models are in agreement – can yield interesting insights even if not the entire relevant model space is covered by available models or if the model results are only partially in agreement. Third, I claim that expert knowledge is necessary to address questions that arise when one applies measurement robustness analysis – the investigation into whether multiple means of measurement yield the same result. Finally, I argue that, in situations where evidence from different measurements is not in agreement, it can be advisable to no longer take all of the evidence into account. This can be done in a rationally defensible way by choosing the most adequate theory or model underlying parts of the evidence set. I discuss examples from climate, medical, and economic policy-making to establish my claims. (shrink)

Hierarchies of evidence are an important and influential tool for appraising evidence in medicine. In recent years, hierarchies have been formally adopted by organizations including the Cochrane Collaboration [1], NICE [2,3], the WHO [4], the US Preventive Services Task Force [5], and the Australian NHMRC [6,7]. The development of such hierarchies has been regarded as a central part of Evidence-Based Medicine, a movement within healthcare which prioritises the use of epidemiological evidence such as that provided by Randomised Controlled Trials. Philosophical (...) work on the methodology of medicine has so far mostly focused on claims about the superiority of RCTs, and hence has largely neglected the questions of what hierarchies are, what assumptions they require, and how they affect clinical practice. This thesis shows that there is great variation in the hierarchies defended and in the interpretations they are, and can be, given. The interpretative assumptions made in using hierarchies are crucial to the content and defensibility of the underlying philosophical commitments concerning evidence and medical practice. Once this variation is been identified, it becomes clear that the little philosophical work that has been done so far affects only some hierarchies, under some interpretations. Modest interpretations offered by La Caze [11], conditional hierarchies like GRADE [12-14], and heuristic approaches such as that defended by Howick et al. [15,16] all survive previous philosophical criticism. This thesis extends previous criticisms by arguing that modest interpretations are so weak as to be unhelpful for clinical practice; that GRADE and similar conditional models omit clinically relevant information, such as information about variation in treatments’ effects and the causes of different responses to therapy; and that heuristic approaches lack the necessary empirical support. The conclusion is that hierarchies in general embed untenable philosophical assumptions: principally that information about average treatment effects backed by high-quality evidence can justify strong recommendations, and that the impact of evidence from individual studies can and should be appraised in isolation. Hierarchies are a poor basis for the application of evidence in clinical practice. The Evidence-Based Medicine movement should move beyond them and explore alternative tools for appraising the overall evidence for therapeutic claims. (shrink)