As systems based on opaque Artificial Intelligence (AI) continue to flourish in diverse real-world applications, understanding these black box models has become paramount. In response, Explainable AI (XAI) has emerged as a field of research with practical and ethical benefits across various domains. This paper not only highlights the advancements in XAI and its application in real-world scenarios but also addresses the ongoing challenges within XAI, emphasizing the need for broader perspectives and collaborative efforts. We bring together experts from diverse (...) fields to identify open problems, striving to synchronize research agendas and accelerate XAI in practical applications. By fostering collaborative discussion and interdisciplinary cooperation, we aim to propel XAI forward, contributing to its continued success. Our goal is to put forward a comprehensive proposal for advancing XAI. To achieve this goal, we present a manifesto of 27 open problems categorized into nine categories. These challenges encapsulate the complexities and nuances of XAI and offer a road map for future research. For each problem, we provide promising research directions in the hope of harnessing the collective intelligence of interested stakeholders. (shrink)

This research reviews explanation and interpretation for Explainable Artificial Intelligence (XAI) methods in order to boost complex machine learning model interpretability. The study shows the influence and belief of XAI in users that trust an Artificial Intelligence system and investigates ethical concerns, particularly fairness and biasedness of all the nontransparent models. It discusses the shortfalls related to XAI techniques, putting crucial emphasis on extended scope, enhancement and scalability potential. A number of outstanding issuesespecially in need of further work can involve (...) standardization, user-centered design and interdisciplinary in strategies for improving the practical utility of XAI. (shrink)

Abstract: As artificial intelligence (AI) systems become increasingly complex and pervasive, the need for transparency and interpretability has never been more critical. Explainable AI (XAI) addresses this need by providing methods and techniques to make AI decisions more understandable to humans. This paper explores the core principles of XAI, highlighting its importance for trust, accountability, and ethical AI deployment. We examine various XAI techniques, including interpretable models and post-hoc explanation methods, and discuss their strengths and limitations. Additionally, we present case (...) studies demonstrating the practical applications of XAI across diverse domains such as healthcare, finance, and autonomous systems. The paper also addresses the ongoing challenges and outlines future research directions aimed at enhancing the effectiveness and applicability of XAI. By bridging the gap between complex AI systems and human understanding, XAI plays a pivotal role in fostering more reliable and responsible AI technologies. (shrink)

In a recent paper, Erasmus et al. (2021) defend the idea that the ambiguity of the term “explanation” in explainable AI (XAI) can be solved by adopting any of four different extant accounts of explanation in the philosophy of science: the Deductive Nomological, Inductive Statistical, Causal Mechanical, and New Mechanist models. In this chapter, I show that the authors’ claim that these accounts can be applied to deep neural networks as they would to any natural phenomenon is mistaken. I also (...) provide a more general argument as to why the notion of explainability as it is currently used in the XAI literature bears little resemblance to the traditional concept of scientific explanation. It would be more fruitful to use the label “understandable AI” to avoid the confusion that surrounds the goal and purposes of XAI. In the second half of the chapter, I argue for a pragmatic conception of understanding that is better suited to play the central role attributed to explanation in XAI. Following De Regt (2017) and Kuorikoski and Ylikoski (2015), the conditions of satisfaction for understanding an ML system are fleshed out in terms of an agent’s success in using the system. (shrink)

Explainable AI (xAI) methods are important for establishing trust in using black-box models. However, recent criticism has mounted against current xAI methods that they disagree, are necessarily false, and can be manipulated, which has started to undermine the deployment of black-box models. Rudin (2019) goes so far as to say that we should stop using black-box models altogether in high-stakes cases because xAI explanations ‘must be wrong’. However, strict fidelity to the truth is historically not a desideratum in science. Idealizations (...) – the intentional distortions introduced to scientific theories and models – are commonplace in the natural sciences and are seen as a successful scientific tool. Thus, it is not falsehood qua falsehood that is the issue. In this paper, I outline the need for xAI research to engage in idealization evaluation. Drawing on the use of idealizations in the natural sciences and philosophy of science, I introduce a novel framework for evaluating whether xAI methods engage in successful idealizations or deceptive explanations (SIDEs). SIDEs evaluates whether the limitations of xAI methods, and the distortions that they introduce, can be part of a successful idealization or are indeed deceptive distortions as critics suggest. I discuss the role that existing research can play in idealization evaluation and where innovation is necessary. Through a qualitative analysis we find that leading feature importance methods and counterfactual explanations are subject to idealization failure and suggest remedies for ameliorating idealization failure. (shrink)

Previous research in Explainable Artificial Intelligence (XAI) suggests that a main aim of explainability approaches is to satisfy specific interests, goals, expectations, needs, and demands regarding artificial systems (we call these “stakeholders' desiderata”) in a variety of contexts. However, the literature on XAI is vast, spreads out across multiple largely disconnected disciplines, and it often remains unclear how explainability approaches are supposed to achieve the goal of satisfying stakeholders' desiderata. This paper discusses the main classes of stakeholders calling for explainability (...) of artificial systems and reviews their desiderata. We provide a model that explicitly spells out the main concepts and relations necessary to consider and investigate when evaluating, adjusting, choosing, and developing explainability approaches that aim to satisfy stakeholders' desiderata. This model can serve researchers from the variety of different disciplines involved in XAI as a common ground. It emphasizes where there is interdisciplinary potential in the evaluation and the development of explainability approaches. (shrink)

The growing adoption of deep learning models in critical domains, such as healthcare, finance, and autonomous systems, has highlighted the need for interpretability and transparency. Explainable AI (XAI) aims to provide insights into the decision-making processes of complex models, improving their trustworthiness and enabling accountability. However, one of the key challenges is balancing the trade-off between model transparency and performance. While explainability can sometimes compromise the predictive power of models, deep learning, with its inherent complexity, exacerbates this issue. This paper (...) explores the current landscape of XAI in deep learning, examining methods such as LIME, SHAP, and attention mechanisms. We also investigate the impact of explainability on model accuracy, fairness, and user trust, and propose strategies to achieve a balance between transparency and performance. Finally, we present case studies where XAI has been successfully integrated into deep learning applications, demonstrating its potential in various real-world scenarios. (shrink)

The industrial proliferation of AI necessitates robust contextual transparency, often lacking in current reporting. This paper introduces a framework for comprehensive reporting of AI, GenAI, and Agentic AI, moving beyond performance metrics. It prioritizes data provenance, algorithmic clarity, operational context, and decision rationale, crucial for trust and accountability. Data provenance ensures integrity, algorithmic clarity demystifies operations, operational context situates performance, and XAI elucidates outputs. For GenAI, transparency on model architecture, training data, and ethics is paramount. Agentic AI requires insights into (...) reinforcement learning, multiagent interactions, and safety. This framework, encompassing data, algorithmic, operational, and ethical aspects, advocates for standardized reporting, stakeholder engagement, and continuous monitoring. Addressing data privacy, system complexity, and lack of standards requires collaboration. Future research includes automated tools and international standards, aligned with IEEE standards. Contextual reporting is crucial for responsible AI, trust, and regulatory navigation, bridging AI sophistication and societal comprehension, promoting ethical standards. (shrink)

With the increased complexity of machine learning models and their widespread use in cloud applications, interpretability and transparency of decision-making are the highest priority. Explainable AI (XAI) methods seek to shed light on the inner workings of machine learning models, hence making them more interpretable and enabling users to rely on them. In this article, we explain the importance of XAI in cloud-computer environments, specifically with regards to having interpretable models and explainable decision-making. [1] XAI is the essence of a (...) paradigm shift in cloud-based ML, promoting transparency, accountability, and ethical decision-making. As cloud- based ML keeps becoming mainstream, the need for XAI increases, highlighting the need for continued innovation and cooperation for realizing the full potential of interpretable AI systems. We speak about current techniques for realizing explainability in AI systems and their feasibility and issues in cloud environments. Additionally, we discuss the implications of XAI among different stakeholders such as developers, end-users, and regulatory authorities and identify future research directions in this fast-growing area. (shrink)

Necessity and sufficiency are the building blocks of all successful explanations. Yet despite their importance, these notions have been conceptually underdeveloped and inconsistently applied in explainable artificial intelligence (XAI), a fast-growing research area that is so far lacking in firm theoretical foundations. Building on work in logic, probability, and causality, we establish the central role of necessity and sufficiency in XAI, unifying seemingly disparate methods in a single formal framework. We provide a sound and complete algorithm for computing explanatory factors (...) with respect to a given context, and demonstrate its flexibility and competitive performance against state of the art alternatives on various tasks. (shrink)

The opacity of some recent Machine Learning (ML) techniques have raised fundamental questions on their explainability, and created a whole domain dedicated to Explainable Artificial Intelligence (XAI). However, most of the literature has been dedicated to explainability as a scientific problem dealt with typical methods of computer science, from statistics to UX. In this paper, we focus on explainability as a pedagogical problem emerging from the interaction between lay users and complex technological systems. We defend an empirical methodology based on (...) field work, which should go beyond the in-vitro analysis of UX to examine in-vivo problems emerging in the field. Our methodology is also comparative, as it chooses to steer away from the almost exclusive focus on ML to compare its challenges with those faced by more vintage algorithms. Finally, it is also philosophical, as we defend the relevance of the philosophical literature to define the epistemic desiderata of a good explanation. This study was conducted in collaboration with Etalab, a Task Force of the French Prime Minister in charge of Open Data & Open Government Policies, dealing in particular with the enforcement of the right to an explanation. In order to illustrate and refine our methodology before going up to scale, we conduct a preliminary work of case studies on the main different types of algorithms used by the French administration: computation, matching algorithms and ML. We study the merits and drawbacks of a recent approach to explanation, which we baptize input-output black box reasoning or BBR for short. We begin by presenting a conceptual framework including the distinctions necessary to a study of pedagogical explainability. We proceed to algorithmic case studies, and draw model-specific and model-agnostic lessons and conjectures. (shrink)

Artificial intelligence (AI) systems are increasingly adopted to make decisions in domains such as business, education, health care, and criminal justice. However, such algorithmic decision systems can have prevalent biases against marginalized social groups and undermine social justice. Explainable artificial intelligence (XAI) is a recent development aiming to make an AI system’s decision processes less opaque and to expose its problematic biases. This paper argues against technical XAI, according to which the detection and interpretation of algorithmic bias can be handled (...) more or less independently by technical experts who specialize in XAI methods. Drawing on resources from feminist epistemology, we show why technical XAI is mistaken. Specifically, we demonstrate that the proper detection of algorithmic bias requires relevant interpretive resources, which can only be made available, in practice, by actively involving a diverse group of stakeholders. Finally, we suggest how feminist theories can help shape integrated XAI: an inclusive social-epistemic process that facilitates the amelioration of algorithmic bias. (shrink)

In this paper I argue that the search for explainable models and interpretable decisions in AI must be reformulated in terms of the broader project of offering a pragmatic and naturalistic account of understanding in AI. Intuitively, the purpose of providing an explanation of a model or a decision is to make it understandable to its stakeholders. But without a previous grasp of what it means to say that an agent understands a model or a decision, the explanatory strategies will (...) lack a well-defined goal. Aside from providing a clearer objective for XAI, focusing on understanding also allows us to relax the factivity condition on explanation, which is impossible to fulfill in many machine learning models, and to focus instead on the pragmatic conditions that determine the best fit between a model and the methods and devices deployed to understand it. After an examination of the different types of understanding discussed in the philosophical and psychological literature, I conclude that interpretative or approximation models not only provide the best way to achieve the objectual understanding of a machine learning model, but are also a necessary condition to achieve post hoc interpretability. This conclusion is partly based on the shortcomings of the purely functionalist approach to post hoc interpretability that seems to be predominant in most recent literature. (shrink)

For synergistic interactions between humans and artificial intelligence (AI) systems, AI outputs often need to be explainable to people. Explainable AI (XAI) systems are commonly tested in human user studies. However, whether XAI researchers consider potential cultural differences in human explanatory needs remains unexplored. We highlight psychological research that found significant differences in human explanations between many people from Western, commonly individualist countries and people from non-Western, often collectivist countries. We argue that XAI research currently overlooks these variations and that (...) many popular XAI designs implicitly and problematically assume that Western explanatory needs are shared cross-culturally. Additionally, we systematically reviewed over 200 XAI user studies and found that most studies did not consider relevant cultural variations, sampled only Western populations, but drew conclusions about human-XAI interactions more generally. We also analyzed over 30 literature reviews of XAI studies. Most reviews did not mention cultural differences in explanatory needs or flag overly broad cross-cultural extrapolations of XAI user study results. Combined, our analyses provide evidence of a cultural bias toward Western populations in XAI research, highlighting an important knowledge gap regarding how culturally diverse users may respond to widely used XAI systems that future work can and should address. (shrink)

The growing use of artificial intelligence (AI) systems in decision-making across various domains has raised critical concerns about bias, fairness, and transparency. AI algorithms can inadvertently perpetuate biases based on the data they are trained on, resulting in outcomes that disproportionately affect certain groups. This paper proposes new methods for detecting and mitigating bias in AI systems while ensuring greater algorithmic transparency. The focus is on developing innovative approaches to identify bias at multiple stages of AI development, from data collection (...) to model deployment. Additionally, the paper emphasizes the need for transparent AI models that allow for explainability and accountability in decision-making. The proposed methods include novel fairness metrics, new tools for detecting biases in datasets, and frameworks for ensuring transparency through explainable AI (XAI) techniques. (shrink)

Many ethical frameworks require artificial intelligence (AI) systems to be explainable. Explainable AI (XAI) models are frequently tested for their adequacy in user studies. Since different people may have different explanatory needs, it is important that participant samples in user studies are large enough to represent the target population to enable generalizations. However, it is unclear to what extent XAI researchers reflect on and justify their sample sizes or avoid broad generalizations across people. We analyzed XAI user studies (N = (...) 220) published between 2012 and 2022. Most studies did not offer rationales for their sample sizes. Moreover, most papers generalized their conclusions beyond their target population, and there was no evidence that broader conclusions in quantitative studies were correlated with larger samples. These methodological problems can impede evaluations of whether XAI systems implement the explainability called for in ethical frameworks. We outline principles for more inclusive XAI user studies. (shrink)

Implicit stochastic models, including both ‘deep neural networks’ (dNNs) and the more recent unsupervised foundational models, cannot be explained. That is, it cannot be determined how they work, because the interactions of the millions or billions of terms that are contained in their equations cannot be captured in the form of a causal model. Because users of stochastic AI systems would like to understand how they operate in order to be able to use them safely and reliably, there has emerged (...) a new field called ‘explainable AI’ (XAI). When we examine the XAI literature, however, it becomes apparent that its protagonists have redefined the term ‘explanation’ to mean something else, namely: ‘interpretation’. Interpretations are indeed sometimes possible, but we show that they give at best only a subjective understanding of how a model works. We propose an alternative to XAI, namely certified AI (CAI), and describe how an AI can be specified, realized, and tested in order to become certified. The resulting approach combines ontologies and formal logic with statistical learning to obtain reliable AI systems which can be safely used in technical applications. (shrink)

The increasing prevalence of mental health issues, particularly stress, has necessitated the development of data-driven, interpretable machine learning models for early detection and intervention. This study leverages multimodal data, including activity levels, perceived stress scores (PSS), and event counts, to predict stress levels among individuals. A series of models, including Logistic Regression, Random Forest, Gradient Boosting, and Neural Networks, were evaluated for their predictive performance. Results demonstrated that ensemble models, particularly Random Forest and Gradient Boosting, performed significantly better compared to (...) Logistic Regression. Random Forest achieved an accuracy of 73%, while Gradient Boosting delivered a balanced precision-recall tradeoff with an accuracy of 72%. Gradient Boosting outperformed in identifying high-stress instances, achieving a recall of 70%, making it the most reliable model for stress prediction. Explainable AI (XAI), which refers to the ability of machine learning models to provide transparent and understandable explanations for their predictions, was employed in this study using SHAP (SHapley Additive exPlanations). SHAP values revealed that Perceived Stress Score (PSS) had the most significant impact on predictions, followed by event count and activity inference. Higher PSS scores strongly correlated with high-stress predictions, while increased event counts and activity levels were associated with lower stress. These findings underscore the importance of incorporating behavioral patterns in stress diagnostics and highlight the utility of explainable models in improving transparency, trust, and usability for clinical decision-making. This research establishes a robust foundation for deploying interpretable machine learning systems to support mental health diagnostics and enhance clinician decision support. (shrink)

There has been a push in recent years to provide better explanations for how AIs make their decisions. Most of this push has come from the ethical concerns that go hand in hand with AIs making decisions that affect humans. Outside of the strictly ethical concerns that have prompted the study of explainable AIs (XAIs), there has been research interest in the mere possibility of creating XAIs in various domains. In general, the more accurate we make our models the harder (...) they are to explain. Go playing AIs like AlphaGo and KataGo provide fantastic examples of this phenomenon. In this paper, I discuss a non-exhaustive list of the leading theories of explanation and what each of these theories would say about the explainability of AI-played moves of Go. Finally, I consider the possibility of ever explaining AI-played Go moves in a way that meets the four principles of XAI. I conclude, somewhat pessimistically, that Go is not as imminently explainable as other domains. As such, the probability of having an XAI for Go that meets the four principles is low. (shrink)

Can AI developers be held epistemically responsible for the processing of their AI systems when these systems are epistemically opaque? And can explainable AI (XAI) provide public justificatory reasons for opaque AI systems’ outputs? Koskinen (2024) gives negative answers to both questions. Here, I respond to her and argue for affirmative answers. More generally, I suggest that when considering people’s uncertainty about the factors causally determining an opaque AI’s output, it might be worth keeping in mind that a degree of (...) uncertainty about conclusions is inevitable even in entirely human-based empirical science because in induction there’s always a risk of getting it wrong. Keeping this in mind may help appreciate that requiring full transparency from AI systems before epistemically trusting their outputs might be unusually (and potentially overly) demanding. (shrink)

In a recent article, Erasmus, Brunet, and Fisher (2021) argue that Artificial Neural Networks (ANNs) are explainable. They survey four influential accounts of explanation: the Deductive-Nomological model, the Inductive-Statistical model, the Causal-Mechanical model, and the New-Mechanist model. They argue that, on each of these accounts, the features that make something an explanation is invariant with regard to the complexity of the explanans and the explanandum. Therefore, they conclude, the complexity of ANNs (and other Machine Learning models) does not make them (...) less explainable. In this reply, it is argued that Erasmus et al. left out one influential account of explanation from their discussion: the Unificationist model. It is argued that, on the Unificationist model, the features that makes something an explanation is sensitive to complexity. Therefore, on the Unificationist model, ANNs (and other Machine Learning models) are not explainable. It is emphasized that Erasmus et al.’s general strategy is correct. The literature on explainable Artificial Intelligence can benefit by drawing from philosophical accounts of explanation. However, philosophical accounts of explanation do not settle the problem of whether ANNs are explainable because they do not unanimously declare that explanation is invariant with regard to complexity. (shrink)

Distributed Constraint Optimization Problems (DCOPs) provide a framework for solving multi-agent coordination tasks efficiently. However, their black-box nature often limits transparency and trust in decision-making processes. This paper explores methods to enhance interpretability in DCOPs, leveraging explainable AI (XAI) techniques. We introduce a novel approach incorporating heuristic explanations, constraint visualization, and modelagnostic methods to provide insights into DCOP solutions. Experimental results demonstrate that our method improves human understanding and debugging of DCOP solutions while maintaining solution quality.

Hybrid cloud infrastructure is increasingly becoming essential to enable scalable artificial intelligence (AI) as well as data processing, and it offers organizations greater flexibility, computational capabilities, and cost efficiency. This paper discusses the strategic use of hybrid cloud environments to enhance AI-based data workflows while addressing key challenges such as latency, integration complexity, infrastructure management, and security. In-depth discussions of solutions like federated multi-cloud models, cloud-native workload automation, quantum computing, and blockchaindriven data governance are presented. Examples of real-world implementation case (...) studies in industries including healthcare, retail, finance, and manufacturing are provided to prove the real benefit of hybrid cloud adoption. New trends like explainable AI (XAI), automated machine learning (AutoML) and federated learning are also discussed here as key enablers of future hybrid cloud expansion. (shrink)

This article addresses computational procedures that are no longer constrained by human modes of representation and considers how these procedures could be philosophically understood in terms of ‘algorithmic thought’. Research in deep learning is its case study. This artificial intelligence (AI) technique operates in computational ways that are often opaque. Such a black-box character demands rethinking the abstractive operations of deep learning. The article does so by entering debates about explainability in AI and assessing how technoscience and technoculture tackle the (...) possibility to ‘re-present’ the algorithmic procedures of feature extraction and feature learning to the human mind. The article thus mobilises the notion of incommensurability (originally developed in the philosophy of science) to address explainability as a communicational and representational issue, which challenges phenomenological and existential modes of comparison between human and algorithmic ‘thinking’ operations. (shrink)

Advancements in machine learning have fuelled the popularity of using AI decision algorithms in procedures such as bail hearings, medical diagnoses and recruitment. Academic articles, policy texts, and popularizing books alike warn that such algorithms tend to be opaque: they do not provide explanations for their outcomes. Building on a causal account of transparency and opacity as well as recent work on the value of causal explanation, I formulate a moral concern for opaque algorithms that is yet to receive a (...) systematic treatment in the literature: when such algorithms are used in life-changing decisions, they can obstruct us from effectively shaping our lives according to our goals and preferences, thus undermining our autonomy. I argue that this concern deserves closer attention as it furnishes the call for transparency in algorithmic decision-making with both new tools and new challenges. (shrink)

In a recent reply to our article, “What is Interpretability?,” Prasetya argues against our position that artificial neural networks are explainable. It is claimed that our indefeasibility thesis—that adding complexity to an explanation of a phenomenon does not make the phenomenon any less explainable—is false. More precisely, Prasetya argues that unificationist explanations are defeasible to increasing complexity, and thus, we may not be able to provide such explanations of highly complex AI models. The reply highlights an important lacuna in our (...) original paper, the omission of the unificationist account of explanation, and affords us the opportunity to respond. Here, we argue that artificial neural networks are explainable in a way that should satisfy unificationists and that interpretability methods present ways in which ML theories can achieve unification. (shrink)

This paper examines how predictive analytics enhances risk management in financial institutions. Advanced tools like machine learning and statistical modeling help predict risks, identify trends, and implement strategies to prevent losses by analyzing historical and real-time data. It covers the use of predictive analytics for credit risk, market risk, operational risk, and fraud detection, with practical case studies. Additionally, it discusses challenges, ethical issues, and prospects in this field.

The banking sector faces growing challenges in identifying and managing risks due to the complexity of financial transactions and increasing fraud. This research presents a framework that combines multiple agents with deep learning to improve risk prediction in banking. Each agent focuses on specific tasks like cleaning data, selecting important features, and detecting unusual activities, ensuring a detailed risk assessment. A deep learning model is used to analyze large amounts of transaction data and identify patterns that may signal potential risks. (...) Tests on real-world banking data show that this approach is more accurate, faster, and effective than traditional methods. By combining the strengths of multi-agent systems and deep learning, this study offers a reliable and flexible solution to help banks manage and adapt to changing risks more effectively. (shrink)

This study presents a novel transformer-based model specifically designed for financial forecasting, integrating explainability mechanisms such as SHAP (SHapley Additive exPlanations) values and attention visualizations to enhance interpretability. Unlike previous models, which often compromise between accuracy and transparency, our approach balances predictive accuracy with interpretability, allowing stakeholders to gain deeper insights into the factors driving market changes. By revealing critical market influences through feature importance and attention maps, this model provides both robustness and transparency, catering to the needs of high-stakes (...) financial environments. (shrink)

We introduce and discuss a knowledge-driven distillation approach to explaining black-box models by means of two kinds of interpretable models. The first is perceptron (or threshold) connectives, which enrich knowledge representation languages such as Description Logics with linear operators that serve as a bridge between statistical learning and logical reasoning. The second is Trepan Reloaded, an ap- proach that builds post-hoc explanations of black-box classifiers in the form of decision trees enhanced by domain knowledge. Our aim is, firstly, to target (...) a model-agnostic distillation approach exemplified with these two frameworks, secondly, to study how these two frameworks interact on a theoretical level, and, thirdly, to investigate use-cases in ML and AI in a comparative manner. Specifically, we envision that user-studies will help determine human understandability of explanations generated using these two frameworks. (shrink)