Large language models (LLMs) structure the linguistic landscape by reflecting certain beliefs and assumptions. In this paper, we address the risk of people unthinkingly adopting and being determined by the values or worldviews embedded in LLMs. We provide a Nietzschean critique of LLMs and, based on the concept of will to power, consider LLMs as will-to-power organisations. This allows us to conceptualise the interaction between self and LLMs as power struggles, which we understand as negotiation. Currently, the invisibility and incomprehensibility (...) of LLMs make it difficult, if not impossible, to engage in such negotiations. This bears the danger that LLMs make reality increasingly homogeneous by recycling beliefs and creating feedback loops that ultimately freeze power struggles and thus consolidate the status quo. In view of this, LLMs constrain self-formation. Based on our critique, we provide some recommendations on how to develop interactions with LLMs that enable negotiations that allow for different ways of being. (shrink)

Within the ethical debate on Machine Learning-driven decision support systems (ML_CDSS), notions such as “human in the loop” or “meaningful human control” are often cited as being necessary for ethical legitimacy. In addition, ethical principles usually serve as the major point of reference in ethical guidance documents, stating that conflicts between principles need to be weighed and balanced against each other. Starting from a neo-Kantian viewpoint inspired by Onora O'Neill, this article makes a concrete suggestion of how to interpret the (...) role of the “human in the loop” and to overcome the perspective of rivaling ethical principles in the evaluation of AI in health care. We argue that patients should be perceived as “fellow workers” and epistemic partners in the interpretation of ML_CDSS outputs. We further highlight that a meaningful process of integrating (rather than weighing and balancing) ethical principles is most appropriate in the evaluation of medical AI. (shrink)