In response to calls for research to improve human-machine teaming, we present a “perspective” paper that explores techniques from computer science that can enhance machine agents for human-machine teams. As part of this paper, we summarize the state of the science on critical team competencies identified for effective HMT, discuss technological gaps preventing machines from fully realizing these competencies, and identify ways that emerging artificial intelligence capabilities may address these gaps and enhance performance in HMT. We extend beyond extant literature (...) by incorporating recent technologies and techniques and describing their potential for contributing to the advancement of HMT. (shrink)

Due to the ongoing advancements in technology, socio-technical collaboration has become increasingly prevalent. This poses challenges in terms of governance and accountability, as well as issues in various other fields. Therefore, it is crucial to familiarize decision-makers and researchers with the core of human–machine collaboration. This study introduces a taxonomy that enables identification of the very nature of human–machine interaction. A literature review has revealed that automation and technical autonomy are main parameters for describing and understanding such interaction. Both aspects (...) must be carefully evaluated, as their increase has potentially far-reaching consequences. Hence, these two concepts comprise the taxonomy’s axes. Five levels of automation and five levels of technical autonomy are introduced below, based on the assumption that both automation and autonomy are gradual. The levels of automation were developed from existing approaches; those of autonomy were carefully derived from a review of the literature. The taxonomy’s use is also explained, as are its limitations and avenues for further research. (shrink)

This article describes a new and innovative methodology for calibrating trust in ethical actions by Lethal Autonomous Weapon Systems (LAWS). For the foreseeable future, LAWS will require human operators for mission planning, decision-making, and supervisory control; yet humans lack the cognitive bandwidth and processing speed to make prompt, real-time ethical decisions. As a result, trustworthy Artificial Intelligence (AI) will be required to support ethical decision-making. We use a Bayesian ethical decision model for: (1) human setting of ethical preferences and thresholds (...) in accordance with Laws of War and tactical criteria; and (2) highlighting the factors that contribute to strike/no-strike recommendations for human evaluation. The model can perform an ethical analysis, provide a quantitative ethical strike/no-strike score, and recommend actions to reduce decision uncertainty. In this article, we describe successful initial evaluation trials of the Bayesian model and of a human interface for interaction with the model. Our Bayesian ethical decision model has an immediate application in wargames; the model can also be used to train operators in understanding the principles and key factors relevant to ethical decision-making; and it may eventually be used in actual military operations employing LAWS. (shrink)

The use of automation in cars is increasing. In future vehicles, drivers will no longer be in charge of the main driving task and may be allowed to perform a secondary task. However, they might be requested to regain control of the car if a hazardous situation occurs. Performing a secondary task might increase drivers' mental workload and consequently decrease the takeover performance if the workload level exceeds a certain threshold. Knowledge about the driver's mental state might hence be useful (...) for increasing safety in conditionally automated vehicles. Measuring drivers' workload continuously is essential to support the driver and hence limit the number of accidents in takeover situations. This goal can be achieved using machine learning techniques to evaluate and classify the drivers' workload in real-time. To evaluate the usefulness of physiological data as an indicator for workload in conditionally automated driving, three physiological signals from 90 subjects were collected during 25 min of automated driving in a fixed-base simulator. Half of the participants performed a verbal cognitive task to induce mental workload while the other half only had to monitor the environment of the car. Three classifiers, sensor fusion and levels of data segmentation were compared. Results show that the best model was able to successfully classify the condition of the driver with an accuracy of 95%. In some cases, the model benefited from sensors' fusion. Increasing the segmentation level increased the performance of the model for windows smaller than 4 min, but decreased for windows larger than 4 min. In conclusion, the study showed that a high level of drivers' mental workload can be accurately detected while driving in conditional automation based on 4-min recordings of respiration and skin conductance. (shrink)

Increasingly, our cities are confronted with crises. Fuelled by climate change and a loss of biodiversity, increasing inequalities and fragmentation, challenges range from social unrest and outbursts of violence to heatwaves, torrential rainfall, or epidemics. As crises require rapid interventions that overwhelm human decision-making capacity, AI has been portrayed as a potential avenue to support or even automate decision-making. In this paper, I analyse the specific challenges of AI in urban crisis management as an example and test case for many (...) super wicked decision problems. These super wicked problems are characterised by a coincidence of great complexity and urgency. I will argue that from this combination, specific challenges arise that are only partially covered in the current guidelines and standards around trustworthy or human-centered AI. By following a decision-centric perspective, I argue that to solve urgent crisis problems, the context, capacities, and networks need to be addressed. AI for crisis response needs to follow dedicated design principles that ensure (i) human control in complex social networks, where many humans interact with AI; (ii) principled design that considers core principles of crisis response such as solidarity and humanity; (iii) designing for the most vulnerable. As such this paper is meant to inspire researchers, AI developers and practitioners in the space of AI for (urban) crisis response – and other urgent and complex problems that urban planners are confronted with. (shrink)

Neuroadaptive technology (NAT) is a closed-loop neurotechnology designed to enhance human–computer interaction. NAT works by collecting neurophysiological data, which are analysed via autonomous algorithms to create actions and adaptations at the user interface. This paper concerns how interaction with NAT can mediate self-related processing (SRP), such as self-awareness, self-knowledge, and agency. We begin with a postphenomenological analysis of the NAT closed loop to highlight the built-in selectivities of machine hermeneutics, i.e., autonomous chains of algorithms that convert data into an assessment (...) of psychological states/intentions. We argue that these algorithms produce an assessment of lived experience that is quantitative, reductive, and highly simplistic. This reductive assessment of lived experience is presented to the user via feedback at the NAT interface and subsequently mediates SRP. It is argued that congruence between system feedback and SRP determines the precise character of the alterity relation between human user and system. If feedback confirms SRP, the technology is regarded as a quasi-self. If there is a disagreement between SRP and feedback from the system, NAT is perceived to be a quasi-other. We argue that the design of the user interface shapes the precise ways in which NAT can mediate SRP. (shrink)

Prior research has demonstrated that trust in robots and performance of robots are two important factors that influence human–autonomy teaming. However, other factors may influence users’ perceptions and use of autonomous systems, such as perceived intent of robots and decision authority of the robots. The current study experimentally examined participants’ trust in an autonomous security robot (ASR), perceived trustworthiness of the ASR, and desire to use an ASR that varied in levels of decision authority and benevolence. Participants (N = 340) (...) were recruited from Amazon Mechanical Turk. Results revealed the participants had increased trust in the ASR when the robot was described as having benevolent intent compared to self-protective intent. There were several interactions between decision authority and intent when predicting the trust process, showing that intent may matter the most when the robot has discretion on executing that intent. Participants stated a desire to use the ASR in a military context compared to a public context. Implications for this research demonstrate that as robots become more prevalent in jobs paired with humans, factors such as transparency provided for the robot's intent and its decision authority will influence users’ trust and trustworthiness. (shrink)

Artificial intelligence plays an increasingly important role in legal disputes, influencing not only the reality outside the court but also the judicial decision-making process itself. While it is clear why judges may generally benefit from technology as a tool for reducing effort costs or increasing accuracy, the presence of technology in the judicial process may also affect the public perception of the courts. In particular, if individuals are averse to adjudication that involves a high degree of automation, particularly given fairness (...) concerns, then judicial technology may yield lower benefits than expected. However, the degree of aversion may well depend on how technology is used, i.e., on the timing and strength of judicial reliance on algorithms. Using an exploratory survey, we investigate whether the stage in which judges turn to algorithms for assistance matters for individual beliefs about the fairness of case outcomes. Specifically, we elicit beliefs about the use of algorithms in four different stages of adjudication: (i) information acquisition, (ii) information analysis, (iii) decision selection, and (iv) decision implementation. Our analysis indicates that individuals generally perceive the use of algorithms as fairer in the information acquisition stage than in other stages. However, individuals with a legal profession also perceive automation in the decision implementation stage as less fair compared to other individuals. Our findings, hence, suggest that individuals do care about how and when algorithms are used in the courts. (shrink)

Maintenance decision errors can result in very costly problems. The 4th industrial revolution has given new opportunities for the development of and use of intelligent decision support systems. With these technological advancements, key concerns focus on gaining a better understanding of the linkage between the technicians’ knowledge and the intelligent decision support systems. The research reported in this study has two primary objectives. To propose a theoretical model that links technicians’ knowledge and intelligent decision support systems, and to present a (...) use case how to apply the theoretical model. The foundation of the new model builds upon two main streams of study in the decision support literature: “distribution” of knowledge among different agents, and “collaboration” of knowledge for reaching a shared goal. This study resulted in the identification of two main gaps: firstly, there must be a greater focus upon the technicians’ knowledge; secondly, technicians need assistance to maintain their focus on the big picture. We used the cognitive fit theory, and the theory of distributed situation awareness to propose the new theoretical model called “distributed collaborative awareness model.” The model considers both explicit and implicit knowledge and accommodates the dynamic challenges involved in operational level maintenance. As an application of this model, we identify and recommend some technological developments required in augmented reality based maintenance decision support. (shrink)

Airline pilots are attributed ultimate responsibility and final authority over their aircraft to ensure the safety and well-being of all its occupants. Yet, with the advent of automation technologies, a dissonance has emerged in that pilots have lost their actual decision-making authority as well as their ability to act in an adequate fashion towards meeting their responsibilities when unexpected circumstances or emergencies occur. Across the literature in human factor studies, we show how automated algorithmic technologies have wrestled control away from (...) airline pilots. The inter-related ethical and pragmatic consequences resulting from such a dissonance are presented. This includes pilots serving as ‘legal sponges’, being caught in ‘moral crumple zones’, as well as real impacts on safety and well-being of passengers and crew. One potential avenue presented in the literature involves a redistribution of responsibility towards all agential entities within the cockpit system. We discuss the possible pitfalls to this approach, including the reinforcement of pilot skills erosion as well as technology’s own limitations in adaptively dealing with unexpected circumstances. We justify our position by showing how algorithmic technology’s representational frames mask or severely limit the pilots’ natural reflex to ‘practically cope’ and gain an ‘optimal grip’ with a given context/situation at hand. Here, we present the concept of relevant or meaningful control, which pilots, when allowed to develop their natural brain-body reflexes, gain across gradual mastery of the aircraft. Finally, we propose a socio-technical design approach in which pilots can re-appropriate full human meaningful control of the aircraft, thus allowing them to meet their attributed responsibilities. (shrink)

This article discusses mechanisms and principles for assignment of moral responsibility to intelligent robots, with special focus on military robots. We introduce the concept autonomous power as a new concept, and use it to identify the type of robots that call for moral considerations. It is furthermore argued that autonomous power, and in particular the ability to learn, is decisive for assignment of moral responsibility to robots. As technological development will lead to robots with increasing autonomous power, we should be (...) prepared for a future when people blame robots for their actions. It is important to, already today, investigate the mechanisms that control human behavior in this respect. The results may be used when designing future military robots, to control unwanted tendencies to assign responsibility to the robots. Independent of the responsibility issue, the moral quality of robots’ behavior should be seen as one of many performance measures by which we evaluate robots. How to design ethics based control systems should be carefully investigated already now. From a consequentialist view, it would indeed be highly immoral to develop robots capable of performing acts involving life and death, without including some kind of moral framework. (shrink)

The paper presents a framework to realise “meaningful human control” over Automated Driving Systems. The framework is based on an original synthesis of the results of the multidisciplinary research project “Meaningful Human Control over Automated Driving Systems” lead by a team of engineers, philosophers, and psychologists at Delft University of the Technology from 2017 to 2021. Meaningful human control aims at protecting safety and reducing responsibility gaps. The framework is based on the core assumption that human persons and institutions, not (...) hardware and software and their algorithms, should remain ultimately—though not necessarily directly—in control of, and thus morally responsible for, the potentially dangerous operation of driving in mixed traffic. We propose an Automated Driving System to be under meaningful human control if it behaves according to the relevant reasons of the relevant human actors (tracking), and that any potentially dangerous event can be related to a human actor (tracing). We operationalise the requirements for meaningful human control through multidisciplinary work in philosophy, behavioural psychology and traffic engineering. The tracking condition is operationalised via a proximal scale of reasons and the tracing condition via an evaluation cascade table. We review the implications and requirements for the behaviour and skills of human actors, in particular related to supervisory control and driver education. We show how the evaluation cascade table can be applied in concrete engineering use cases in combination with the definition of core components to expose deficiencies in traceability, thereby avoiding so-called responsibility gaps. Future research directions are proposed to expand the philosophical framework and use cases, supervisory control and driver education, real-world pilots and institutional embedding. (shrink)

In recent years, autonomous systems have become an important research area and application domain, with a significant impact on modern society. Such systems are characterized by different levels of autonomy and complex communication infrastructures that allow for collective decision-making strategies. There exist several publications that tackle ethical aspects in such systems, but mostly from the perspective of a single agent. In this paper we go one step further and discuss these ethical challenges from the perspective of an aggregate of autonomous (...) systems capable of collective decision-making. In particular, in this paper, we propose the Caesar approach through which we model the collective ethical decision-making process of a group of actors—agents and humans, as well as define the building blocks for the agents participating in such a process, namely Caesar agents. Factors such as trust, security, safety, and privacy, which affect the degree to which a collective decision is ethical, are explicitly captured in Caesar. Finally, we argue that modeling the collective decision-making in Caesar provides support for accountability. (shrink)

The relationship between a human being and an AI system has to be considered as a collaborative process between two agents during the performance of an activity. When there is a collaboration between two people, a fundamental characteristic of that collaboration is that there is co-supervision, with each agent supervising the actions of the other. Such supervision ensures that the activity achieves its objectives, but it also means that responsibility for the consequences of the activity is shared. If there is (...) no co-supervision, neither collaborator can be held co-responsible for the actions of the other. When the collaboration is between a person and an AI system, co-supervision is also necessary to ensure that the objectives of the activity are achieved, but this also means that there is co-responsibility for the consequences of the activities. Therefore, if each agent's responsibility for the consequences of the activity depends on the effectiveness and efficiency of the supervision that that agent performs over the other agent's actions, it will be necessary to take into account the way in which that supervision is carried out and the factors on which it depends. In the case of the human supervision of the actions of an AI system, there is a wealth of psychological research that can help us to establish cognitive and non-cognitive boundaries and their relationship to the responsibility of humans collaborating with AI systems. There is also psychological research on how an external observer supervises and evaluates human actions. This research can be used to programme AI systems in such a way that the boundaries of responsibility for AI systems can be established. In this article, we will describe some examples of how such research on the task of supervising the actions of another agent can be used to establish lines of shared responsibility between a human being and an AI system. The article will conclude by proposing that we should develop a methodology for assessing responsibility based on the results of the collaboration between a human being and an AI agent during the performance of one common activity. (shrink)