Bioethics, Volume 36, Issue 2, Page 162-169, February 2022.

The use of black box algorithms in medicine has raised scholarly concerns due to their opaqueness and lack of trustworthiness. Concerns about potential bias, accountability and responsibility, patient autonomy and compromised trust transpire with black box algorithms. These worries connect epistemic concerns with normative issues. In this paper, we outline that black box algorithms are less problematic for epistemic reasons than many scholars seem to believe. By outlining that more transparency in algorithms is not always necessary, and by explaining that (...) computational processes are indeed methodologically opaque to humans, we argue that the reliability of algorithms provides reasons for trusting the outcomes of medical artificial intelligence. To this end, we explain how computational reliabilism, which does not require transparency and supports the reliability of algorithms, justifies the belief that results of medical AI are to be trusted. We also argue that several ethical concerns remain with black box algorithms, even when the results are trustworthy. Having justified knowledge from reliable indicators is, therefore, necessary but not sufficient for normatively justifying physicians to act. This means that deliberation about the results of reliable algorithms is required to find out what is a desirable action. Thus understood, we argue that such challenges should not dismiss the use of black box algorithms altogether but should inform the way in which these algorithms are designed and implemented. When physicians are trained to acquire the necessary skills and expertise, and collaborate with medical informatics and data scientists, black box algorithms can contribute to improving medical care. (shrink)

There is much to learn from Durán and Jongsma’s paper.1 One particularly important insight concerns the relationship between epistemology and ethics in medical artificial intelligence. In clinical environments, the task of AI systems is to provide risk estimates or diagnostic decisions, which then need to be weighed by physicians. Hence, while the implementation of AI systems might give rise to ethical issues—for example, overtreatment, defensive medicine or paternalism2—the issue that lies at the heart is an epistemic problem: how can physicians (...) know whether to trust decisions made by AI systems? In this manner, various studies examining the interaction of AI systems and physicians have shown that without being able to evaluate their trustworthiness, especially novice physicians become over-reliant on algorithmic support—and ultimately are led astray by incorrect decisions.3–5 This leads to a second insight from the paper, namely that even if some AI system happens to be opaque, it is still not built on the moon. To assess its trustworthiness, AI developers or physicians have different sorts of higher order evidence at hand. Most importantly, …. (shrink)

The authors put forward an interesting response to detractors of black box algorithms. According to the authors, what is of ethical relevance for medical artificial intelligence is not so much their transparency, but rather their reliability as a process capable of producing accurate and trustworthy results. The implications of this view are twofold. First, it is permissible to implement a black box algorithm in clinical settings, provided the algorithm’s epistemic authority is tempered by physician expertise and consideration of patient autonomy. (...) Second, physicians are not expected to possess exhaustive knowledge or understanding of the algorithmic computation by which they verify or augment their medical opinions. The potential of these algorithms to improve diagnostic and procedural accuracy alongside the quality of patient decision-making is undoubtedly a boon to modern medicine, but blind deference to them is neither feasible nor responsible, as several logistical and ethical quagmires noted by the authors still remain inherent in algorithmic software. I concur …. (shrink)