The evidence based medicine movement has championed the need for objective and transparent methods of clinical guideline development. The Grades of Recommendation, Assessment, Development, and Evaluation (GRADE) framework was developed for that purpose. Central to this framework is criteria for assessing the quality of evidence from clinical studies and the impact that body of evidence should have on our confidence in the clinical effectiveness of a therapy under examination. Grades of Recommendation, Assessment, Development, and Evaluation has been adopted by a (...) number of professional medical societies and organizations as a means for orienting the development of clinical guidelines. As a result, the method of GRADE has implications on how health care is delivered and patient outcomes. In this paper, we reveal several issues with the underlying logic of GRADE that warrant further discussion. First, the definitions of the “grades of evidence” provided by GRADE, while explicit, are functionally vague. Second, the “criteria for assigning grade of evidence” is seemingly arbitrary and arguably logically incoherent. Finally, the GRADE method is unclear on how to integrate evidence grades with other important factors, such as patient preferences, and trade‐offs between costs, benefits, and harms when proposing a clinical practice recommendation. Much of the GRADE method requires judgement on the part of the user, making it unclear as to how the framework reduces bias in recommendations or makes them more transparent—both goals of the programme. It is our view that the issues presented in this paper undermine GRADE's justificatory scheme, thereby limiting the usefulness of GRADE as a tool for developing clinical recommendations. (shrink)

Translating risk estimates derived from epidemiologic study into evidence of causality for a particular patient is problematic. The difficulty of this process is not unique to the medical context; rather, courts are also challenged with the task of using risk estimates to infer evidence of cause in particular cases. Thus, an examination of how this is done in a legal context might provide insight into when and how it is appropriate to use risk information as evidence of cause in a (...) medical context. A careful study of the case of Goodman v. Viljoen, a medical malpractice suit litigated in the Ontario Superior Court of Justice in 2011, reveals different approaches to how risk information is used as or might be considered a substitute for evidence of causation, and the pitfalls associated with these approaches. Achieving statistical thresholds, specifically minimizing the probability of falsely rejecting the null hypothesis, and exceeding a relative risk of 2, plays a significant role in establishing causality of the particular in the legal setting. However, providing a reasonable explanation or establishing “biological plausibility” of the causal association also seems important, and (to some) may even take precedent over statistical thresholds for a given context. (shrink)

Rationale, Aims, and Objectives: Confidence (or belief) that a therapy is effective is essential to practicing clinical medicine. GRADE, a popular framework for developing clinical recommendations, provides a means for assigning how much confidence one should have in a therapy's effect estimate. One's level of confidence (or “degree of belief”) can also be modelled using Bayes theorem. In this paper, we look through both a GRADE and Bayesian lens to examine how one determines confidence in the effect estimate. Methods: Philosophical (...) examination. Results: The GRADE framework uses a criteria‐based method to assign a quality of evidence level. The criteria pertain mostly to considerations of methodological rigour, derived from a modified evidence‐based medicine evidence hierarchy. The four levels of quality relate to the level of confidence one should have in the effect estimate. The Bayesian framework is not bound by a predetermined set of criteria. Bayes theorem shows how a rational agent adjusts confidence (ie, degree of belief) in the effect estimate on the basis of the available evidence. Such adjustments relate to the principles of incremental confirmation and evidence proportionism. Use of the Bayesian framework reveals some potential pitfalls in GRADE's criteria‐based thinking on confidence that are out of step with our intuitions on evidence. Conclusions: A rational thinker uses all available evidence to formulate beliefs. The GRADE criteria seem to suggest that we discard some of that information when other, more favoured information (eg, derived from clinical trials) is available. The GRADE framework should strive to ensure that the whole evidence base is considered when determining confidence in the effect estimate. The incremental value of such evidence on determining confidence in the effect estimate should be assigned in a manner that is theoretically or empirically justified, such that confidence is proportional to the evidence, both for and against it. (shrink)

Rationale, aims, and objectives: One of the sectors challenged by the COVID-19 pandemic is medical research. COVID-19 originates from a novel coronavirus (SARSCoV- 2) and the scientific community is faced with the daunting task of creating a novel model for this pandemic or, in other words, creating novel science. This paper is the first part of a series of two papers that explore the intricate relationship between the different challenges that have hindered biomedical research and the generation of scientific knowledge (...) during the COVID-19 pandemic. Methods: During the early stages of the pandemic, research conducted on hydroxychloroquine (HCQ) was chaotic and sparked several heated debates with respect to the scientific methods used and the quality of knowledge generated. Research on HCQ is used as a case study in both papers. The authors explored biomedical databases, peer-reviewed journals, pre-print servers, and media articles to identify relevant literature on HCQ and COVID-19, and examined philosophical perspectives on medical research in the context of this pandemic and previous global health challenges. Results: This paper demonstrates that a lack of prioritization among research questions and therapeutics was responsible for the duplication of clinical trials and the dispersion of precious resources. Study designs, aimed at minimising biases and increasing objectivity, were, instead, the subject of fruitless oppositions. The duplication of research works, combined with poor-quality research, has greatly contributed to slowing down the creation of novel scientific knowledge. Conclusions: The COVID-19 pandemic presented challenges in terms of (1) finding and prioritising relevant research questions and (2) choosing study designs that are appropriate for a time of emergency. (shrink)

Rationale, aims and objectives: The COVID-19 pandemic has impacted every facet of society, including medical research. This paper is the second part of a series of articles that explore the intricate relationship between the different challenges that have hindered biomedical research and the generation of novel scientific knowledge during the COVID-19 pandemic. In the first part of this series, we demonstrated that, in the context of COVID-19, the scientific community has been faced with numerous challenges with respect to (1) finding (...) and prioritizing relevant research questions and (2) choosing study designs that are appropriate for a time of emergency. Methods: During the early stages of the pandemic, research conducted on hydroxychloroquine (HCQ) sparked several heated debates with respect to the scientific methods used and the quality of knowledge generated. Research on HCQ is used as a case study in both papers. The authors explored biomedical databases, peerreviewed journals, pre-print servers and media articles to identify relevant literature on HCQ and COVID-19, and examined philosophical perspectives on medical research in the context of this pandemic and previous global health challenges. Results: This second paper demonstrates that a lack of research prioritization and methodological rigour resulted in the generation of fleeting and inconsistent evidence that complicated the development of public health guidelines. The reporting of scientific findings to the scientific community and general public highlighted the difficulty of finding a balance between accuracy and speed. Conclusions: The COVID-19 pandemic presented challenges in terms of (3) evaluating evidence for the purpose of making evidence-based decisions and (4) sharing scientific findings with the rest of the scientific community. This second paper demonstrates that the four challenges outlined in the first and second papers have often compounded each other and have contributed to slowing down the creation of novel scientific knowledge during the COVID-19 pandemic. (shrink)