The use of black box algorithms in medicine has raised scholarly concerns due to their opaqueness and lack of trustworthiness. Concerns about potential bias, accountability and responsibility, patient autonomy and compromised trust transpire with black box algorithms. These worries connect epistemic concerns with normative issues. In this paper, we outline that black box algorithms are less problematic for epistemic reasons than many scholars seem to believe. By outlining that more transparency in algorithms is not always necessary, and by explaining that (...) computational processes are indeed methodologically opaque to humans, we argue that the reliability of algorithms provides reasons for trusting the outcomes of medical artificial intelligence. To this end, we explain how computational reliabilism, which does not require transparency and supports the reliability of algorithms, justifies the belief that results of medical AI are to be trusted. We also argue that several ethical concerns remain with black box algorithms, even when the results are trustworthy. Having justified knowledge from reliable indicators is, therefore, necessary but not sufficient for normatively justifying physicians to act. This means that deliberation about the results of reliable algorithms is required to find out what is a desirable action. Thus understood, we argue that such challenges should not dismiss the use of black box algorithms altogether but should inform the way in which these algorithms are designed and implemented. When physicians are trained to acquire the necessary skills and expertise, and collaborate with medical informatics and data scientists, black box algorithms can contribute to improving medical care. (shrink)

In this paper we argue that transparency of machine learning algorithms, just as explanation, can be defined at different levels of abstraction. We criticize recent attempts to identify the explanation of black box algorithms with making their decisions (post-hoc) interpretable, focusing our discussion on counterfactual explanations. These approaches to explanation simplify the real nature of the black boxes and risk misleading the public about the normative features of a model. We propose a new form of algorithmic transparency, that consists in (...) explaining algorithms as an intentional product, that serves a particular goal, or multiple goals (Daniel Dennet’s design stance) in a given domain of applicability, and that provides a measure of the extent to which such a goal is achieved, and evidence about the way that measure has been reached. We call such idea of algorithmic transparency “design publicity.” We argue that design publicity can be more easily linked with the justification of the use and of the design of the algorithm, and of each individual decision following from it. In comparison to post-hoc explanations of individual algorithmic decisions, design publicity meets a different demand (the demand for impersonal justification) of the explainee. Finally, we argue that when models that pursue justifiable goals (which may include fairness as avoidance of bias towards specific groups) to a justifiable degree are used consistently, the resulting decisions are all justified even if some of them are (unavoidably) based on incorrect predictions. For this argument, we rely on John Rawls’s idea of procedural justice applied to algorithms conceived as institutions. (shrink)

Organizations such as the EU High-Level Expert Group on AI and the IEEE have recently formulated ethical principles and (moral) values that should be adhered to in the design and deployment of artificial intelligence (AI). These include respect for autonomy, non-maleficence, fairness, transparency, explainability, and accountability. But how can we ensure and verify that an AI system actually respects these values? To help answer this question, I propose an account for determining when an AI system can be said to embody (...) certain values. This account understands embodied values as the result of design activities intended to embed those values in such systems. AI systems are here understood as a special kind of sociotechnical system that, like traditional sociotechnical systems, are composed of technical artifacts, human agents, and institutions but—in addition—contain artificial agents and certain technical norms that regulate interactions between artificial agents and other elements of the system. The specific challenges and opportunities of embedding values in AI systems are discussed, and some lessons for better embedding values in AI systems are drawn. (shrink)

In recent years, a plethora of high-profile scientific publications has been reporting about machine learning algorithms outperforming clinicians in medical diagnosis or treatment recommendations. This has spiked interest in deploying relevant algorithms with the aim of enhancing decision-making in healthcare. In this paper, we argue that instead of straightforwardly enhancing the decision-making capabilities of clinicians and healthcare institutions, deploying machines learning algorithms entails trade-offs at the epistemic and the normative level. Whereas involving machine learning might improve the accuracy of medical (...) diagnosis, it comes at the expense of opacity when trying to assess the reliability of given diagnosis. Drawing on literature in social epistemology and moral responsibility, we argue that the uncertainty in question potentially undermines the epistemic authority of clinicians. Furthermore, we elucidate potential pitfalls of involving machine learning in healthcare with respect to paternalism, moral responsibility and fairness. At last, we discuss how the deployment of machine learning algorithms might shift the evidentiary norms of medical diagnosis. In this regard, we hope to lay the grounds for further ethical reflection of the opportunities and pitfalls of machine learning for enhancing decision-making in healthcare. (shrink)

It has recently been proposed to incorporate the use of a “Patient Preference Predictor” (PPP) into the process of making treatment decisions for incapacitated patients. A PPP would predict which treatment option a given incapacitated patient would most likely prefer, based on the individual’s characteristics and information on what treatment preferences are correlated with these characteristics. Including a PPP in the shared decision-making process between clinicians and surrogates has the potential to better realize important ethical goals for making treatment decisions (...) for incapacitated patients. However, developing and implementing a PPP poses significant practical challenges. The present paper discusses these practical challenges and considers ways to address them. (shrink)

The standard approach to treatment decision making for incapacitated patients often fails to provide treatment consistent with the patient’s preferences and values and places significant stress on surrogate decision makers. These shortcomings provide compelling reason to search for methods to improve current practice. Shared decision making between surrogates and clinicians has important advantages, but it does not provide a way to determine patients’ treatment preferences. Hence, shared decision making leaves families with the stressful challenge of identifying the patient’s preferred treatment (...) option. To address this concern, the present paper proposes to incorporate the use of a “Patient Preference Predictor” (PPP) into the shared decision-making process between surrogates and clinicians. A PPP would predict which treatment option a given incapacitated patient would most likely prefer, based on the individual’s characteristics and information on what treatment preferences are correlated with these characteristics. Use of a PPP is likely to increase the chances that incapacitated patients are treated consistent with their preferences and values and might reduce the stress and burden on their surrogates. Including a PPP in the shared decision-making process therefore has the potential to realize important ethical goals for making treatment decisions for incapacitated patients. The present paper justifies this approach on conceptual and normative grounds. (shrink)

When patients cannot make their own treatment decisions, surrogates typically step in to do it for them. Surrogate decision‐making is far from ideal, of course, as the surrogate may not know what the patient prefers or what best promotes her interests. One way to improve it would be to arm surrogates with information about what patients in similar circumstances tend to prefer, allowing them to make empirically grounded predictions about what their patient would want.

Like many, I find the idea of relying on patient preference predictors in life-or-death cases ethically troubling. As part of his stimulating discussion, Sharadin1 diagnoses such unease as a worry that using PPPs disrespects patients’ autonomy, by treating their most intimate and significant desires as if they were caused by their demographic traits. I agree entirely with Sharadin’s ‘debunking’ response to this concern: we can use statistical correlations to predict others’ preferences without thereby assuming any causal claim. However, I suspect (...) that, for at least some of us, our unease about PPPs stems from a different kind of ‘autonomy’ concern. In this commentary, then, I will explore this concern, and show how it relates to Sharadin’s work. Very many of our preferences are caused, ultimately, by facts which are outside our control, such as our demographic features. However, I suggest that we can still act autonomously on the basis of such preferences, when they are preferences which we endorse. Imagine, for example, that Jane has grown up in a church-growing environment, which has shaped many of her preferences, …. (shrink)

The patient preference predictor is a proposed computer-based algorithm that would predict the treatment preferences of decisionally incapacitated patients. Incorporation of a PPP into the decision-making process has the potential to improve implementation of the substituted judgement standard by providing more accurate predictions of patients’ treatment preferences than reliance on surrogates alone. Yet, critics argue that methods for making treatment decisions for incapacitated patients should be judged on a number of factors beyond simply providing them with the treatments they would (...) have chosen for themselves. These factors include the extent to which the decision-making process recognises patients’ freedom to choose and relies on evidence the patient themselves would take into account when making treatment decisions. These critics conclude that use of a PPP should be rejected on the grounds that it is inconsistent with these factors, especially as they relate to proper respect for patient autonomy. In this paper, we review and evaluate these criticisms. We argue that they do not provide reason to reject use of a PPP, thus supporting efforts to develop a full-scale PPP and to evaluate it in practice. There are no data in this work. (shrink)

Artificial intelligence (AI) systems are increasingly being used in healthcare, thanks to the high level of performance that these systems have proven to deliver. So far, clinical applications have focused on diagnosis and on prediction of outcomes. It is less clear in what way AI can or should support complex clinical decisions that crucially depend on patient preferences. In this paper, we focus on the ethical questions arising from the design, development and deployment of AI systems to support decision-making around (...) cardiopulmonary resuscitation and the determination of a patient’s Do Not Attempt to Resuscitate status (also known as code status). The COVID-19 pandemic has made us keenly aware of the difficulties physicians encounter when they have to act quickly in stressful situations without knowing what their patient would have wanted. We discuss the results of an interview study conducted with healthcare professionals in a university hospital aimed at understanding the status quo of resuscitation decision processes while exploring a potential role for AI systems in decision-making around code status. Our data suggest that (1) current practices are fraught with challenges such as insufficient knowledge regarding patient preferences, time pressure and personal bias guiding care considerations and (2) there is considerable openness among clinicians to consider the use of AI-based decision support. We suggest a model for how AI can contribute to improve decision-making around resuscitation and propose a set of ethically relevant preconditions—conceptual, methodological and procedural—that need to be considered in further development and implementation efforts. (shrink)

Patient preference predictors (PPPs) promise to provide medical professionals with a new solution to the problem of making treatment decisions on behalf of incapacitated patients. I show that the use of PPPs faces a version of a normative problem familiar from legal scholarship: the problem of naked statistical evidence. I sketch two sorts of possible reply, vindicating and debunking, and suggest that our reply to the problem in the one domain ought to mirror our reply in the other. The conclusion (...) is thus conditional: if we think the problem of naked statistical evidence is a serious problem in the legal domain, then we should be concerned about the symmetrical problem for PPPs. (shrink)

Purpose Although surrogate decision-making is prevalent in intensive care units and concerns with decision quality are well documented, little is known about how clinicians help family members understand the surrogate role. We investigated whether and how clinicians provide normative guidance to families regarding how to function as a surrogate. Subjects and methods We audiorecorded and transcribed 73 ICU family conferences in which clinicians anticipated discussing goals of care for incapacitated patients at high risk of death. We developed and applied a (...) coding framework to identify normative statements by clinicians regarding what considerations should guide surrogates’ decisions, including whether clinicians explained one or more of Buchanan and Brock’s three standard principles of SDM to family members. Results Clinicians made at least one statement about how to perform the surrogate role in 24 conferences ). We observed three general types of normative guidance provided to surrogates, with some conferences containing more than one type of guidance: counselling about one or more standard principles of SDM ; counselling surrogates to make decisions centred on the patient as a person, without specifying how to accomplish that ; and counselling surrogates to make decisions based on the family’s values. Conclusions Clinicians did not provide normative guidance about the surrogate role in two-thirds of family conferences for incapacitated patients at high risk for death. When they did, clinicians’ guidance was often incomplete and sometimes conflicted with standard principles of SDM. Future work is needed to understand whether providing explicit guidance on how to perform the surrogate role improves decision-making or mitigates surrogates’ psychological distress. (shrink)

Purpose Although surrogate decision-making is prevalent in intensive care units and concerns with decision quality are well documented, little is known about how clinicians help family members understand the surrogate role. We investigated whether and how clinicians provide normative guidance to families regarding how to function as a surrogate. Subjects and methods We audiorecorded and transcribed 73 ICU family conferences in which clinicians anticipated discussing goals of care for incapacitated patients at high risk of death. We developed and applied a (...) coding framework to identify normative statements by clinicians regarding what considerations should guide surrogates’ decisions, including whether clinicians explained one or more of Buchanan and Brock’s three standard principles of SDM to family members. Results Clinicians made at least one statement about how to perform the surrogate role in 24 conferences ). We observed three general types of normative guidance provided to surrogates, with some conferences containing more than one type of guidance: counselling about one or more standard principles of SDM ; counselling surrogates to make decisions centred on the patient as a person, without specifying how to accomplish that ; and counselling surrogates to make decisions based on the family’s values. Conclusions Clinicians did not provide normative guidance about the surrogate role in two-thirds of family conferences for incapacitated patients at high risk for death. When they did, clinicians’ guidance was often incomplete and sometimes conflicted with standard principles of SDM. Future work is needed to understand whether providing explicit guidance on how to perform the surrogate role improves decision-making or mitigates surrogates’ psychological distress. (shrink)