Prior research has demonstrated that trust in robots and performance of robots are two important factors that influence human–autonomy teaming. However, other factors may influence users’ perceptions and use of autonomous systems, such as perceived intent of robots and decision authority of the robots. The current study experimentally examined participants’ trust in an autonomous security robot (ASR), perceived trustworthiness of the ASR, and desire to use an ASR that varied in levels of decision authority and benevolence. Participants (N = 340) (...) were recruited from Amazon Mechanical Turk. Results revealed the participants had increased trust in the ASR when the robot was described as having benevolent intent compared to self-protective intent. There were several interactions between decision authority and intent when predicting the trust process, showing that intent may matter the most when the robot has discretion on executing that intent. Participants stated a desire to use the ASR in a military context compared to a public context. Implications for this research demonstrate that as robots become more prevalent in jobs paired with humans, factors such as transparency provided for the robot's intent and its decision authority will influence users’ trust and trustworthiness. (shrink)

This article describes a new and innovative methodology for calibrating trust in ethical actions by Lethal Autonomous Weapon Systems (LAWS). For the foreseeable future, LAWS will require human operators for mission planning, decision-making, and supervisory control; yet humans lack the cognitive bandwidth and processing speed to make prompt, real-time ethical decisions. As a result, trustworthy Artificial Intelligence (AI) will be required to support ethical decision-making. We use a Bayesian ethical decision model for: (1) human setting of ethical preferences and thresholds (...) in accordance with Laws of War and tactical criteria; and (2) highlighting the factors that contribute to strike/no-strike recommendations for human evaluation. The model can perform an ethical analysis, provide a quantitative ethical strike/no-strike score, and recommend actions to reduce decision uncertainty. In this article, we describe successful initial evaluation trials of the Bayesian model and of a human interface for interaction with the model. Our Bayesian ethical decision model has an immediate application in wargames; the model can also be used to train operators in understanding the principles and key factors relevant to ethical decision-making; and it may eventually be used in actual military operations employing LAWS. (shrink)

Legislation and ethical guidelines around the globe call for effective human oversight of AI-based systems in high-risk contexts – that is oversight that reliably reduces the risks otherwise associated with the use of AI-based systems. Such risks may relate to the imperfect accuracy of systems (e.g., inaccurate classifications) or to ethical concerns (e.g., unfairness of outputs). Given the significant role that human oversight is expected to play in the operation of AI-based systems, it is crucial to better understand the conditions (...) for effective human oversight. We argue that the reliable detection of errors (as an umbrella term for inaccuracies and unfairness) is crucial for effective human oversight. We then propose that Signal Detection Theory (SDT) offers a promising framework for better understanding what affects people’s sensitivity (i.e., how well they are able to detect errors) and response bias (i.e., the tendency to report errors given a perceived evidence of an error) in detecting errors. Whereas an SDT perspective on the detection of inaccuracies is straightforward, we demonstrate its broader applicability by detailing the specifics for an SDT perspective on unfairness detection, including the need to choose a standard for (un)fairness. Additionally, we illustrate that an SDT perspective helps to better understand the conditions for effective error detection by showing examples of task-, system-, and person-related factors that may affect the sensitivity and response bias of humans tasked with detecting unfairness associated with the use of AI-based systems. Finally, we discuss future research directions for an SDT perspective on error detection. (shrink)

This study reviews the current information concerning the measurement of the situation awareness (SA) of the teleoperated drivers of remotely controlled cars. The teleoperated drivers who drive these cars are in a remote location, and they control the cars through a communication interface. The objective methods with probes are beneficial in measuring SA on a closed circuit without real traffic. Questions specifically should address the information provided on the road by haptic sensations, such as the slope of the road and (...) the vehicle's speed. Methods for measuring SA that involve probes and interruptions obviously are not suitable for use on public roads. A stable environment for the display and control of the communication interface is suitable for an eye tracker in measuring SA. These features also facilitate the use of subjective observer-rating methods. Both of these methods are suitable for driving on real roads because they are not intrusive. SA research in a real-road environment also should demonstrate how the SA of other drivers is affected by seeing a car without a driver. Given the remote character of driving, cultural differences in cognition may have a significant influence on the SA of the teleoperated driver. (shrink)

The automobile is a tool like no other. There is much excitement and enthusiasm for new and emerging transportation tools such as vehicle automation and driver assistance systems. Although there are high hopes for these technologies, there are many unknowns including the extent to which these new transportation tools can realistically and reliably improve driver safety and affect the subjective driving experience. This paper explores these questions in the context of evaluating collision warning systems on the behavior and perceptions of (...) drivers with different amounts of experience and places the results in the greater context of tool use and skilled learning focusing on the adoption and utilization of mental models. (shrink)