How ought you to evaluate your options if you’re uncertain about which axiology is true? One prominent response is Expected Moral Value Maximisation, the view that under axiological uncertainty, an option is better than another if and only if it has the greater expected moral value across axiologies. EMVM raises two fundamental questions. First, there’s a question about what it should even mean. In particular, it presupposes that we can compare moral value across axiologies. So to even understand EMVM, we (...) need to explain what it is for such comparisons to hold. Second, assuming that we understand it, there’s a question about whether EMVM is true. Since there are many plausible rivals, we need an argument to defend it. In this paper, I’ll introduce a representation theorem for axiological uncertainty to answer these two questions. Roughly, the theorem shows that if all our axiologies satisfy the von Neumann–Morgenstern axioms, and if the facts about which options are better than which in light of your uncertainty also satisfy these axioms as well as a Pareto condition, then these facts have a relevantly unique expected utility representation. If I’m right, this theorem at once affords us a compelling way to understand EMVM—and specifically intertheoretic comparisons—and a systematic argument for its truth. (shrink)

Nick Bostrom and others have suggested treating decision-making under moral uncertainty as analogous to parliamentary decision-making. The core suggestion of this “parliamentary approach” is that the competing moral theories function like delegates to the parliament, and that these delegates then make decisions by some combination of bargaining and voting. There seems some reason to hope that such an approach might avoid standard objections to existing approaches (for example, the “maximise expected choiceworthiness” (MEC) and “my favourite theory” approaches). However, the parliamentary (...) approach is so far extremely underspecified, making it largely indeterminate how such a model will in fact behave in the respects that those concerned with moral uncertainty care about. This paper explores one way of making it precise. We treat predicaments of moral uncertainty as analogous to bargaining situations alone (setting aside voting), and apply a version of the Nash solution that is standard in bargaining theory. The resulting model does indeed perform in many of the hoped-for ways. However, so also does a version of MEC that employs a structural approach to intertheoretic comparisons. It seems to us an open question which, regarding this version of MEC and the bargaining-theoretic approach, is superior to the other. We identify the key points on which the two differ. (shrink)

People can be uncertain in their moral judgments. Philosophers have argued that such uncertainty can either refer to the underlying empirical facts (empirical uncertainty) or to the normative evaluation of these facts itself (normative uncertainty). Psychological investigations of this distinction, however, are rare. In this paper, we combined factor-analytical and experimental approaches to show that empirical and normative uncertainty describe two related but different psychological states. In Study 1, we asked N = 265 participants to describe a case of moral (...) uncertainty and to rate different aspects of their uncertainty about this case. Across this wide range of moral scenarios, our items loaded onto three reliable factors: lack of information, unclear consequences, and normative uncertainty. In Study 2, we confirmed this factor structure using predefined stimulus material. N = 402 participants each rated eight scenarios that systematically varied in their degree of uncertainty regarding the consequences of the described actions and in the value conflict that was inherent to them. The empirical uncertainty factors were mainly affected by the introduction of uncertainty regarding consequences, and the normative uncertainty factor was mainly affected by the introduction of value conflict. Our studies provide evidence that the distinction between empirical and normative uncertainty accurately describes a psychological reality. We discuss the relevance of our findings for research on moral judgments and decision-making, and folk metaethics. (shrink)

How ought you to evaluate your options if you're uncertain about what's fundamentally valuable? A prominent response is Expected Value Maximisation (EVM)—the view that under axiological uncertainty, an option is better than another if and only if it has the greater expected value across axiologies. But the expected value of an option depends on quantitative probability and value facts, and in particular on value comparisons across axiologies. We need to explain what it is for such facts to hold. Also, EVM (...) is by no means self-evident. We need an argument to defend that it’s true. This book introduces an axiomatic approach to answer these worries. It provides an explication of what EVM means by use of representation theorems: intertheoretic comparisons can be understood in terms of facts about which options are better than which, and mutatis mutandis for intratheoretic comparisons and axiological probabilities. And it provides a systematic argument to the effect that EVM is true: the theory can be vindicated through simple axioms. The result is a formally cogent and philosophically compelling extension of standard decision theory, and original take on the problem of axiological or normative uncertainty. (shrink)

Even if we know all relevant descriptive facts about an act, we can still be uncertain about its moral acceptability. Most literature on how to act under such normative uncertainty operates on moral realism, the metaethical view that there are objective moral facts. Lay people largely report anti-realist intuitions, which poses the question of how these intuitions affect their interpretation and handling of normative uncertainty. Results from two quasi-experimental studies (total N = 365) revealed that most people did not interpret (...) normative uncertainty as referring to objective moral facts but rather as uncertainty regarding one’s own view, uncertainty regarding the culturally accepted view or as the result of ambivalence. Especially the anti-realist majority of participants interpreted normative uncertainty different to how it is described in the literature on choice under normative uncertainty. Metaethical views were also associated with lay peoples’ choice of uncertainty reduction strategies and with assumptions about the intended aim of such strategies. The current findings suggest that empirical investigations of normative uncertainty might benefit from considering folk metaethical pluralism, as the lay public largely disagrees with the metaethical assumptions underlying the current discourse on choice under normative uncertainty. (shrink)

In ‘Normative Uncertainty as a Voting Problem’, William MacAskill argues that positive credence in ordinal-structured or intertheoretically incomparable normative theories does not prevent an agent from rationally accounting for her normative uncertainties in practical deliberation. Rather, such an agent can aggregate the theories in which she has positive credence by methods borrowed from voting theory—specifically, MacAskill suggests, by a kind of weighted Borda count. The appeal to voting methods opens up a promising new avenue for theories of rational choice under (...) normative uncertainty. The Borda rule, however, is open to at least two serious objections. First, it seems implicitly to ‘cardinalize’ ordinal theories, and so does not fully face up to the problem of merely ordinal theories. Second, the Borda rule faces a problem of option individuation. MacAskill attempts to solve this problem by invoking a measure on the set of practical options. But it is unclear that there is any natural way of defining such a measure that will not make the output of the Borda rule implausibly sensitive to irrelevant empirical features of decision-situations. After developing these objections, I suggest an alternative: the McKelvey uncovered set, a Condorcet method that selects all and only the maximal options under a strong pairwise defeat relation. This decision rule has several advantages over Borda and mostly avoids the force of MacAskill’s objection to Condorcet methods in general. (shrink)

In the growing literature on decision-making under moral uncertainty, a number of skeptics have argued that there is an insuperable barrier to rational "hedging" for the risk of moral error, namely the apparent incomparability of moral reasons given by rival theories like Kantianism and utilitarianism. Various general theories of intertheoretic value comparison have been proposed to meet this objection, but each suffers from apparently fatal flaws. In this paper, I propose a more modest approach that aims to identify classes of (...) moral theories that share common principles strong enough to establish bases for intertheoretic comparison. I show that, contra the claims of skeptics, there are often rationally perspicuous grounds for precise, quantitative value comparisons within such classes. In light of this fact, I argue, the existence of some apparent incomparabilities between widely divergent moral theories cannot serve as a general argument against hedging for one's moral uncertainties. (shrink)

According to ‘Excluders’, descriptive uncertainty – but not normative uncertainty – matters to what we ought to do. Recently, several authors have argued that those wishing to treat normative uncertainty differently from descriptive uncertainty face a dependence problem because one's descriptive uncertainty can depend on one's normative uncertainty. The aim of this paper is to determine whether the phenomenon of dependence poses a decisive problem for Excluders. I argue that existing arguments fail to show this, and that, while stronger ones (...) can be found, Excluders can escape them. (shrink)

In this paper, I enter the debate between those who hold that our normative uncertainty matters for what we ought to do, and those who hold that only our descriptive uncertainty matters. I argue that existing views in both camps have unacceptable implications in cases where our descriptive beliefs depend on our normative beliefs. I go on to propose a fix which is available only to those who hold that normative uncertainty matters, ultimately leaving the challenge as a threat to (...) recent skepticism about such views. (shrink)

As AI Systems become increasingly autonomous, they are expected to engage in decision-making processes that have moral implications. In this research we integrate theoretical and empirical lines of thought to address the matters of moral reasoning and moral uncertainty in AI Systems. We reconceptualize the metanormative framework for decision-making under moral uncertainty and we operationalize it through a latent class choice model. The core idea being that moral heterogeneity in society can be codified in terms of a small number of (...) classes with distinct moral preferences and that this codification can be used to express moral uncertainty of an AI. Choice analysis allows for the identification of classes and their moral preferences based on observed choice data. Our reformulation of the metanormative framework is theory-rooted and practical in the sense that it avoids runtime issues in real time applications. To illustrate our approach we conceptualize a society in which AI Systems are in charge of making policy choices. While one of the systems uses a baseline morally certain model, the other uses a morally uncertain model. We highlight cases in which the AI Systems disagree about the policy to be chosen, thus illustrating the need to capture moral uncertainty in AI systems. (shrink)

A popular model of normative decision-making under uncertainty suggests choosing the option with the maximum expected moral choice-worthiness, where the choice-worthiness values from each moral theory, which are assumed commensurable, are weighted by credence and combined. This study adds descriptive uncertainty about the non-moral facts of a situation into the model by treating choice-worthiness as a random variable. When agents face greater descriptive uncertainty, the choice-worthiness random variable will have a greater spread and a larger standard deviation. MEC, as a (...) decision rule, is sensitive only to the expected value of the random variable and not to its standard deviation. For example, MEC is insensitive to cases where an option with a large degree of descriptive uncertainty may have a higher probability of being below some threshold of impermissibility than does an option with less dispersion, even though the latter has a higher expected choice-worthiness. When applied to the same situation, similar moral theories will have statistically correlated choice-worthiness values. This correlation affects the dispersion of the credence-weighted sum of the random variables but not its expected value. Thus, MEC is insensitive to aspects of the normative situation to which a good decision-rule should be sensitive. (shrink)

This article is a critical survey of the debate over the value of the social discount rate, with a particular focus on climate change. The ma- jority of the material surveyed is from the economics rather than from the philosophy literature, but the emphasis of the survey itself is on founda- tions in ethical and other normative theory rather than highly technical details. I begin by locating the standard approach to discounting within the overall landscape of ethical theory, and explaining (...) the assumptions and simplifications that are needed in order to arrive at the model that is standard in the discounting literature. The article then covers the general theory of the Ramsey equation and its relationship to observed interest rates, arguments for and against a positive rate of pure time preference, the consumption elasticity of utility, and the effect of various sorts of uncertainty on the discount rate. Finally, it turns specifically to the ap- plication of this debate to the case of climate change, focussing on the recent controversy over the low discount rate used in the Stern Review of the Economics of Climate Change. (shrink)

While ordinary decision theory focuses on empirical uncertainty, real decision-makers also face normative uncertainty: uncertainty about value itself. From a purely formal perspective, normative uncertainty is comparable to (Harsanyian or Rawlsian) identity uncertainty in the 'original position', where one's future values are unknown. A comprehensive decision theory must address twofold uncertainty -- normative and empirical. We present a simple model of twofold uncertainty, and show that the most popular decision principle -- maximising expected value (`Expectationalism') -- has different formulations, namely (...) Ex-Ante Expectationalism, Ex-Post Expectationalism, and hybrid theories. These alternative theories recommend different decisions, reasoning modes, and attitudes to risk. But they converge under an interesting (necessary and sufficient) condition. (shrink)

This paper presents some impossibility results for certain views about what you should do when you are uncertain about which moral theory is true. I show that under reasonable and extremely minimal ways of defining what a moral theory is, it follows that the concept of expected moral choiceworthiness is undefined, and more generally that any theory of decision-making under moral uncertainty must generate pathological results.

The development of artificial intelligence will require systems of ethical decision making to be adapted for automatic computation. However, projects to implement moral reasoning in artificial moral agents so far have failed to satisfactorily address the widespread disagreement between competing approaches to moral philosophy. In this paper I argue that the proper response to this situation is to design machines to be fundamentally uncertain about morality. I describe a computational framework for doing so and show that it efficiently resolves common (...) obstacles to the implementation of moral philosophy in intelligent machines. (shrink)

This paper discusses a novel problem at the intersection of ethics and epistemology: there can be cases in which moral considerations seem to "encroach'' upon belief from multiple directions at once, and possibly to varying degrees, thereby leaving their overall effect on belief unclear. We introduce these cases -- cases of moral encroachment under moral uncertainty -- and show that they pose a problem for all predominant accounts of moral encroachment. We then address the problem, by developing a modular Bayesian (...) framework that, we argue, is sufficiently flexible and scaleable to accommodate the multifaceted uncertainty we describe while still generating clear recommendations for an agent's beliefs. Our framework has several practical upshots, and we close by articulating them: we derive insights about the relationship between moral character and doxastic behavior and make suggestions for how to encourage people to revise their doxastic states in morally laudable ways, without deviating from core Bayesian norms. (shrink)

This paper presents a unifying diagnosis of a number of important problems facing existing models of rational choice under moral uncertainty and proposes a remedy. I argue that the problems of (i) severely limited scope, (ii) intertheoretic comparisons, and (iii) 'swamping’ all stem from the way in which values are assigned to options in decision rules such as Maximisation of Expected Choiceworthiness. By assigning values to options under a given moral theory by asking something like ‘how much do I desire (...) this option, supposing this theory is true?’ rather than ‘how much value does this theory assign to this option?’ these problems can be avoided, while the appealing features of these accounts can be preserved. This amendment provides a role for the preferences, desires, or goals of rational agents that is curiously absent from the existing discussion of what individuals rationally ought to do when they are uncertain about what they morally ought to do. (shrink)

This thesis is about making decisions when we are uncertain about what will happen, how valuable it will be, and even how to make decisions. Even the most sure-footed amongst us are sometimes uncertain about all three, but surprisingly little attention has been given to the latter two. The three essays that constitute my thesis hope to do a small part in rectifying this problem. The first essay is about the value of finding out how to make decisions. Society spends (...) considerable resources funding people (like me) to research decision-making, so it is natural to wonder whether society is getting a good deal. This question is so shockingly underresearched that bedrock facts are readily discoverable, such as when this kind of information is valuable. My second essay concerns whether we can compare value when we are uncertain about value. Many people are in fact uncertain about value, and how we deal with this uncertainty hinges on these comparisons. I argue that value comparisons are only sometimes possible; I call this weak comparability. This essay is largely a synthesis of the literature, but I also present an argument which begins with a peculiar view of the self: it is as if each of us is a crowd of different people separated by time (but connected by continuity of experience). I’m not the first to endorse this peculiar view of the self, but I am the first to show how it supports the benign view that value is sometimes comparable. We may be uncertain of any decision rules, even those that would tell us how to act when we face uncertainty in decision rules. We may be uncertain of how to decide how to decide how to... And so on. If so, we might have to accept infinitely many decision rules just to make any mundane decision, such as whether to pick up a five-cent piece from the gutter. My third essay addresses this problem of regress. I think all of our decisions are forced: we must decide now or continue to deliberate. Surprisingly, this allows us to avoid the original problem. I call this solution “when forced, do your best”. (shrink)

How should an agent decide what to do when she is uncertain not just about morally relevant empirical matters, like the consequences of some course of action, but about the basic principles of morality itself? This question has only recently been taken up in a systematic way by philosophers. Advocates of moral hedging claim that an agent should weigh the reasons put forward by each moral theory in which she has positive credence, considering both the likelihood that that theory is (...) true and the strength of the reasons it posits. The view that it is sometimes rational to hedge for one's moral uncertainties, however, has recently come under attack both from those who believe that an agent should always be guided by the dictates of the single moral theory she deems most probable and from those who believe that an agent's moral beliefs are simply irrelevant to what she ought to do. Among the many objections to hedging that have been pressed in the recent literature is the worry that there is no non-arbitrary way of making the intertheoretic comparisons of moral value necessary to aggregate the value assignments of rival moral theories into a single ranking of an agent's options. -/- This dissertation has two principal objectives: First, I argue that, contra these recent objections, an agent's moral beliefs and uncertainties are relevant to what she rationally ought to do, and more particularly, that agents are at least sometimes rationally required to hedge for their moral uncertainties. My principal argument for these claims appeals to the enkratic conception of rationality, according to which the requirements of practical rationality derive from an agent's beliefs about the objective, desire-independent value or choiceworthiness of her options. Second, I outline a new general theory of rational choice under moral uncertainty. Central to this theory is the idea of content-based aggregation, that the principles according to which an agent should compare and aggregate rival moral theories are grounded in the content of those theories themselves, including not only their value assignments but also the metaethical and other non-surface-level propositions that underlie, justify, or explain those value assignments. (shrink)

I analyze recent discussions about making moral decisions under normative uncertainty. I discuss whether this kind of uncertainty should have practical consequences for decisions and whether there are reliable methods of reasoning that deal with the possibility that we are wrong about some moral issues. I defend a limited use of the decision theory model of reasoning in cases of normative uncertainty.