How should you decide what to do when you're uncertain about basic normative principles (e.g., Kantianism vs. utilitarianism)? A natural suggestion is to follow some "second-order" norm: e.g., "comply with the first-order norm you regard as most probable" or "maximize expected choiceworthiness". But what if you're uncertain about second-order norms too -- must you then invoke some third-order norm? If so, it seems that any norm-guided response to normative uncertainty is doomed to a vicious regress. In this paper, I aim (...) to rescue second-order norms from this threat of regress. I first elaborate and defend the suggestion some philosophers have entertained that the regress problem forces us to accept normative externalism, the view that at least one norm is incumbent on agents regardless of their beliefs or evidence concerning that norm. But, I then argue, we need not accept externalism about first-order (e.g., moral) norms, thus closing off any question of what an agent should do in light of her normative beliefs. Rather, it is more plausible to ascribe external force to a single, second-order rational norm: the enkratic principle, correctly formulated. This modest form of externalism, I argue, is both intrinsically well-motivated and sufficient to head off the threat of regress. (shrink)

Effective Altruism is a rapidly growing and influential contemporary philosophical movement committed to updating utilitarianism in both theory and practice. The movement focuses on identifying urgent but neglected causes and inspiring supererogatory giving to meet the need. It also tries to build a broader coalition by adopting a more ecumenical approach to ethics which recognizes a wide range of values and moral constraints. These interesting developments distinguish Effective Altruism from the utilitarianism of the past in ways that invite cooperation and (...) warrant a fresh look from Thomists. Nonetheless Effective Altruism’s fundamentally consequentialist and aggregative model for ethics precludes more foundational agreement with Thomistic ethics in ways that limit the extent of practical cooperation. (shrink)

Moral latitude is only ever a matter of coincidence on the most popular decision procedure in the literature on moral uncertainty. In all possible choice situations other than those in which two or more options happen to be tied for maximal expected choiceworthiness, Maximize Expected Choiceworthiness implies that only one possible option is uniquely appropriate. A better theory of appropriateness would be more sensitive to the decision maker’s credence in theories that endorse agent-centred prerogatives. In this paper, we will develop (...) and defend a modified version of this view that allows for agent-centred prerogatives and supererogation as part of its verdicts about appropriateness. (shrink)

Nick Bostrom and others have suggested treating decision-making under moral uncertainty as analogous to parliamentary decision-making. The core suggestion of this “parliamentary approach” is that the competing moral theories function like delegates to the parliament, and that these delegates then make decisions by some combination of bargaining and voting. There seems some reason to hope that such an approach might avoid standard objections to existing approaches (for example, the “maximise expected choiceworthiness” (MEC) and “my favourite theory” approaches). However, the parliamentary (...) approach is so far extremely underspecified, making it largely indeterminate how such a model will in fact behave in the respects that those concerned with moral uncertainty care about. This paper explores one way of making it precise. We treat predicaments of moral uncertainty as analogous to bargaining situations alone (setting aside voting), and apply a version of the Nash solution that is standard in bargaining theory. The resulting model does indeed perform in many of the hoped-for ways. However, so also does a version of MEC that employs a structural approach to intertheoretic comparisons. It seems to us an open question which, regarding this version of MEC and the bargaining-theoretic approach, is superior to the other. We identify the key points on which the two differ. (shrink)

People can be uncertain in their moral judgments. Philosophers have argued that such uncertainty can either refer to the underlying empirical facts (empirical uncertainty) or to the normative evaluation of these facts itself (normative uncertainty). Psychological investigations of this distinction, however, are rare. In this paper, we combined factor-analytical and experimental approaches to show that empirical and normative uncertainty describe two related but different psychological states. In Study 1, we asked N = 265 participants to describe a case of moral (...) uncertainty and to rate different aspects of their uncertainty about this case. Across this wide range of moral scenarios, our items loaded onto three reliable factors: lack of information, unclear consequences, and normative uncertainty. In Study 2, we confirmed this factor structure using predefined stimulus material. N = 402 participants each rated eight scenarios that systematically varied in their degree of uncertainty regarding the consequences of the described actions and in the value conflict that was inherent to them. The empirical uncertainty factors were mainly affected by the introduction of uncertainty regarding consequences, and the normative uncertainty factor was mainly affected by the introduction of value conflict. Our studies provide evidence that the distinction between empirical and normative uncertainty accurately describes a psychological reality. We discuss the relevance of our findings for research on moral judgments and decision-making, and folk metaethics. (shrink)

How ought you to evaluate your options if you're uncertain about what's fundamentally valuable? A prominent response is Expected Value Maximisation (EVM)—the view that under axiological uncertainty, an option is better than another if and only if it has the greater expected value across axiologies. But the expected value of an option depends on quantitative probability and value facts, and in particular on value comparisons across axiologies. We need to explain what it is for such facts to hold. Also, EVM (...) is by no means self-evident. We need an argument to defend that it’s true. This book introduces an axiomatic approach to answer these worries. It provides an explication of what EVM means by use of representation theorems: intertheoretic comparisons can be understood in terms of facts about which options are better than which, and mutatis mutandis for intratheoretic comparisons and axiological probabilities. And it provides a systematic argument to the effect that EVM is true: the theory can be vindicated through simple axioms. The result is a formally cogent and philosophically compelling extension of standard decision theory, and original take on the problem of axiological or normative uncertainty. (shrink)

Presenting the first comprehensive, in-depth study of hyperintensionality, this book equips readers with the basic tools needed to appreciate some of current and future debates in the philosophy of language, semantics, and metaphysics. After introducing and explaining the major approaches to hyperintensionality found in the literature, the book tackles its systematic connections to normativity and offers some contributions to the current debates. The book offers undergraduate and graduate students an essential introduction to the topic, while also helping professionals in related (...) fields get up to speed on open research-level problems. (shrink)

When we are faced with a choice among acts, but are uncertain about the true state of the world, we may be uncertain about the acts’ “choiceworthiness”. Decision theories guide our choice by making normative claims about how we should respond to this uncertainty. If we are unsure which decision theory is correct, however, we may remain unsure of what we ought to do. Given this decision-theoretic uncertainty, meta-theories attempt to resolve the conflicts between our decision theories...but we may be (...) unsure which meta-theory is correct as well. This reasoning can launch a regress of ever-higher-order uncertainty, which may leave one forever uncertain about what one ought to do. There is, fortunately, a class of circumstances under which this regress is not a problem. If one holds a cardinal understanding of subjective choiceworthiness, and accepts certain other criteria, one’s hierarchy of metanormative uncertainty ultimately converges to precise definitions of “subjective choiceworthiness” for any finite set of acts. If one allows the metanormative regress to extend to the transfinite ordinals, the convergence criteria can be weakened further. Finally, the structure of these results applies straightforwardly not just to decision-theoretic uncertainty, but also to other varieties of normative uncertainty, such as moral uncertainty. (shrink)

Integrated assessment models play a major role in the science and policy of climate change. Similarly to other widely used computational tools for addressing socially relevant problems, IAMs need to account for the key uncertainties characterizing processes and socio-economic responses. In the case of climate change, these are particularly complex given the very long-term nature of climate and the deep uncertainty characterizing technological and human systems. Here we draw from philosophical discussion of mathematical modeling of social problems and review the (...) role of uncertainty in climate-economic modeling. In agreement with the literature, we highlight the crucial role of epistemic uncertainty in IAMs. We posit that the normative components of models, more than the physical and socio-techno-economic ones, are the most fraught by uncertainty and yet the least understood. We suggest a research agenda to explore uncertainties of evaluation frameworks, transcending the current implicit normativity of IAMs. (shrink)

Rohana faces a choice where she can produce either a better outcome by lying or a worse outcome by telling the truth. She justifiably, but falsely, believes in.

In ‘Normative Uncertainty as a Voting Problem’, William MacAskill argues that positive credence in ordinal-structured or intertheoretically incomparable normative theories does not prevent an agent from rationally accounting for her normative uncertainties in practical deliberation. Rather, such an agent can aggregate the theories in which she has positive credence by methods borrowed from voting theory—specifically, MacAskill suggests, by a kind of weighted Borda count. The appeal to voting methods opens up a promising new avenue for theories of rational choice under (...) normative uncertainty. The Borda rule, however, is open to at least two serious objections. First, it seems implicitly to ‘cardinalize’ ordinal theories, and so does not fully face up to the problem of merely ordinal theories. Second, the Borda rule faces a problem of option individuation. MacAskill attempts to solve this problem by invoking a measure on the set of practical options. But it is unclear that there is any natural way of defining such a measure that will not make the output of the Borda rule implausibly sensitive to irrelevant empirical features of decision-situations. After developing these objections, I suggest an alternative: the McKelvey uncovered set, a Condorcet method that selects all and only the maximal options under a strong pairwise defeat relation. This decision rule has several advantages over Borda and mostly avoids the force of MacAskill’s objection to Condorcet methods in general. (shrink)

The aim of this paper is to examine whether it would be advantageous to introduce knowledge norms instead of the currently assumed rational credence norms into the debate about decision making under normative uncertainty. There is reason to think that this could help us better accommodate cases in which agents are rationally highly confident in false moral views. I show how Moss’ view of probabilistic knowledge can be fruitfully employed to develop a decision theory that delivers plausible verdicts in these (...) cases. I also argue that, for this new view to be better than existing alternatives, it must adopt a particular solution to the new evil demon problem, which asks whether agents and their BIV-counterparts are equally justified. In order to get an attractive decision theory for cases of moral uncertainty, we must reject the claim that agents and their BIV-counterparts are equally justified. Moreover, the resulting view must be supplemented with a moral epistemology that explains how it is possible to be rationally morally uncertain. This is especially challenging if we assume that moral truths are knowable a priori. (shrink)

:One view of practical reasoning is that it involves the weighing of reasons. It is not clear, however, how the weights of reasons combine, especially given the logical and substantive relations among different reasons. Nor is it clear how the weighing of reasons relates to decision theoretic maximization of expected value. This paper constructs a formal model of reasons and their weight in order to shed light on these issues. The model informs philosophical debates about reasons, such as the question (...) of whether reasons or values are more fundamental and the opposing doctrines of atomism and holism about reasons. (shrink)

Artificially intelligent systems will be used to make increasingly important decisions about us. Many of these decisions will have to be made without universal agreement about the relevant moral facts. For other kinds of disagreement, it is at least usually obvious what kind of solution is called for. What makes moral disagreement especially challenging is that there are three different ways of handling it. Moral solutions apply a moral theory or related principles and largely ignore the details of the disagreement. (...) Compromise solutions apply a method of finding a compromise and taking information about the disagreement as input. Epistemic solutions apply an evidential rule that treats the details of the disagreement as evidence of moral truth. Proposals for all three kinds of solutions can be found in the AI ethics and value alignment literature, but little has been said to justify choosing one over the other. I argue that the choice is best framed in terms of moral risk. (shrink)

In this paper, I enter the debate between those who hold that our normative uncertainty matters for what we ought to do, and those who hold that only our descriptive uncertainty matters. I argue that existing views in both camps have unacceptable implications in cases where our descriptive beliefs depend on our normative beliefs. I go on to propose a fix which is available only to those who hold that normative uncertainty matters, ultimately leaving the challenge as a threat to (...) recent skepticism about such views. (shrink)

This paper develops a philosophical account of moral disruption. According to Robert Baker, moral disruption is a process in which technological innovations undermine established moral norms without clearly leading to a new set of norms. Here I analyze this process in terms of moral uncertainty, formulating a philosophical account with two variants. On the harm account, such uncertainty is always harmful because it blocks our knowledge of our own and others’ moral obligations. On the qualified harm account, there is no (...) harm in cases where moral uncertainty is related to innovation that is “for the best” in historical perspective or where uncertainty is the expression of a deliberative virtue. The two accounts are compared by applying them to Baker’s historical case of the introduction of mechanical ventilation and organ transplantation technologies, as well as the present-day case of mass data practices in the health domain. (shrink)

This paper surveys the state-of-the-art in machine ethics, that is, considerations of how to implement ethical behaviour in robots, unmanned autonomous vehicles, or software systems. The emphasis is on covering the breadth of ethical theories being considered by implementors, as well as the implementation techniques being used. There is no consensus on which ethical theory is best suited for any particular domain, nor is there any agreement on which technique is best placed to implement a particular theory. Another unresolved problem (...) in these implementations of ethical theories is how to objectively validate the implementations. The paper discusses the dilemmas being used as validating ‘whetstones’ and whether any alternative validation mechanism exists. Finally, it speculates that an intermediate step of creating domain-specific ethics might be a possible stepping stone towards creating machines that exhibit ethical behaviour. (shrink)

As AI Systems become increasingly autonomous, they are expected to engage in decision-making processes that have moral implications. In this research we integrate theoretical and empirical lines of thought to address the matters of moral reasoning and moral uncertainty in AI Systems. We reconceptualize the metanormative framework for decision-making under moral uncertainty and we operationalize it through a latent class choice model. The core idea being that moral heterogeneity in society can be codified in terms of a small number of (...) classes with distinct moral preferences and that this codification can be used to express moral uncertainty of an AI. Choice analysis allows for the identification of classes and their moral preferences based on observed choice data. Our reformulation of the metanormative framework is theory-rooted and practical in the sense that it avoids runtime issues in real time applications. To illustrate our approach we conceptualize a society in which AI Systems are in charge of making policy choices. While one of the systems uses a baseline morally certain model, the other uses a morally uncertain model. We highlight cases in which the AI Systems disagree about the policy to be chosen, thus illustrating the need to capture moral uncertainty in AI systems. (shrink)

This paper argues in favor of a particular account of decision‐making under normative uncertainty: that, when it is possible to do so, one should maximize expected choice‐worthiness. Though this position has been often suggested in the literature and is often taken to be the ‘default’ view, it has so far received little in the way of positive argument in its favor. After dealing with some preliminaries and giving the basic motivation for taking normative uncertainty into account in our decision‐making, we (...) consider and provide new arguments against two rival accounts that have been offered—the accounts that we call ‘My Favorite Theory’ and ‘My Favorite Option’. We then give a novel argument for comparativism—the view that, under normative uncertainty, one should take into account both probabilities of different theories and magnitudes of choice‐worthiness. Finally, we further argue in favor of maximizing expected choice‐worthiness and consider and respond to five objections. (shrink)

A number of philosophers have claimed that we should take not just empirical uncertainty but also fundamental moral uncertainty into account in our decision-making, and that, despite widespread moral disagreement, doing so would allow us to draw robust lessons for some issues in practical ethics. In this article, I argue that, so far, the implications for practical ethics have been drawn too simplistically. First, the implications of moral uncertainty for normative ethics are far more wide-ranging than has been noted so (...) far. Second, one can't straightforwardly argue from moral uncertainty to particular conclusions in practical ethics, both because of ‘interaction’ effects between moral issues, and because of the variety of different possible intertheoretic comparisons that one can reasonably endorse. (shrink)

Many prominent ethicists, including Shelly Kagan, John Rawls, and Thomas Scanlon, accept a kind of epistemic modesty thesis concerning our capacity to carry out the project of ethical theorizing. But it is a thesis that has received surprisingly little explicit and focused attention, despite its widespread acceptance. After explaining why the thesis is true, I argue that it has several implications in metaethics, including, especially, implications that should lead us to rethink our understanding of Reductive Realism. In particular, the thesis (...) of epistemic modesty in ethics implies a kind of epistemic modesty about the metaphysical nature of ethics, if Reductive Realism about the metaphysical nature of ethics is true, and it implies that normative concepts are indispensable to practical deliberation in a way that answers an influential objection to Reductive Realism from Jonathan Dancy, David Enoch, William FitzPatrick, and Derek Parfit. (shrink)

ABSTRACTCitizens in John Rawls's well-ordered society face an assurance dilemma. They wish to act justly only if they are reasonably sure their fellow citizens will also act justly. According to Rawls, this assurance problem is solved through public reasoning. This paper argues that public reason cannot serve this function. It begins by arguing that one kind of incompleteness public reason faces that most Rawlsians grant is ubiquitous but unproblematic from a normative standpoint is problematic from an assurance perspective: it makes (...) it possible for citizens to argue for policy conclusions that are favored by their private interests, rather than justice. In response, perhaps the thing to do is structure deliberative democratic institutions such that citizens will always be incentivized to use public reasons to only argue for conclusions they believe are favored by justice. The paper proves that this is impossible by extending the Gibbard–Satterthwaite theorem. (shrink)

A popular model of normative decision-making under uncertainty suggests choosing the option with the maximum expected moral choice-worthiness, where the choice-worthiness values from each moral theory, which are assumed commensurable, are weighted by credence and combined. This study adds descriptive uncertainty about the non-moral facts of a situation into the model by treating choice-worthiness as a random variable. When agents face greater descriptive uncertainty, the choice-worthiness random variable will have a greater spread and a larger standard deviation. MEC, as a (...) decision rule, is sensitive only to the expected value of the random variable and not to its standard deviation. For example, MEC is insensitive to cases where an option with a large degree of descriptive uncertainty may have a higher probability of being below some threshold of impermissibility than does an option with less dispersion, even though the latter has a higher expected choice-worthiness. When applied to the same situation, similar moral theories will have statistically correlated choice-worthiness values. This correlation affects the dispersion of the credence-weighted sum of the random variables but not its expected value. Thus, MEC is insensitive to aspects of the normative situation to which a good decision-rule should be sensitive. (shrink)

This paper explores what kind of uncertainty a research program governing solar climate engineering through Stratospheric Aerosol Injection (SAI) needs to account for. Specifically, it tries to answer two central issues with regards to SAI research and it’s ethical evaluation: One, what irreducible uncertainties remain throughout the decision-process, and, two, how do these remaining uncertainties affect the ethical evaluation of SAI research. The main assumption is that decisions on SAI research governance will be made under normative uncertainty, i.e. situations under (...) irreducible knowledge-constraints that arise in concrete, practical decision-contexts. These decision-contexts are multi-lateral and empirically ambiguous, and the decision-makers need to reconcile a plurality of values. While normative uncertainty complicates the ethical evaluation of policy decisions, I argue that moral considerations can be accommodated through the inclusion of recognitional, participatory justice approaches, as well as adaptive and anticipatory governance methods. (shrink)

In this paper, I argue that the fetishism objection to moral hedging fails. The objection rests on a reasons-responsiveness account of moral worth, according to which an action has moral worth only if the agent is responsive to moral reasons. However, by adopting a plausible theory of non-ideal moral reasons, one can endorse a reasons-responsiveness account of moral worth while maintaining that moral hedging is sometimes an appropriate response to moral uncertainty. Thus, the theory of moral worth upon which the (...) fetishism objection relies does not, in fact, support that objection. (shrink)

This paper looks at philosophical questions that arise in the context of AI alignment. It defends three propositions. First, normative and technical aspects of the AI alignment problem are interrelated, creating space for productive engagement between people working in both domains. Second, it is important to be clear about the goal of alignment. There are significant differences between AI that aligns with instructions, intentions, revealed preferences, ideal preferences, interests and values. A principle-based approach to AI alignment, which combines these elements (...) in a systematic way, has considerable advantages in this context. Third, the central challenge for theorists is not to identify ‘true’ moral principles for AI; rather, it is to identify fair principles for alignment that receive reflective endorsement despite widespread variation in people’s moral beliefs. The final part of the paper explores three ways in which fair principles for AI alignment could potentially be identified. (shrink)

A straightforward way to handle moral uncertainty is simply to follow the moral theory in which you have most credence. This approach is known as My Favourite Theory. In this paper, I argue that, in some cases, My Favourite Theory prescribes choices that are, sequentially, worse in expected moral value than the opposite choices according to each moral theory you have any credence in. In addition this, problem generalizes to other approaches that avoid intertheoretic comparisons of value, such as My (...) Favourite Option, the Borda Rule, Variance Normalization, and the Principle of Maximizing Expected Normalized Moral Value. (shrink)

While ordinary decision theory focuses on empirical uncertainty, real decision-makers also face normative uncertainty: uncertainty about value itself. From a purely formal perspective, normative uncertainty is comparable to (Harsanyian or Rawlsian) identity uncertainty in the 'original position', where one's future values are unknown. A comprehensive decision theory must address twofold uncertainty -- normative and empirical. We present a simple model of twofold uncertainty, and show that the most popular decision principle -- maximising expected value (`Expectationalism') -- has different formulations, namely (...) Ex-Ante Expectationalism, Ex-Post Expectationalism, and hybrid theories. These alternative theories recommend different decisions, reasoning modes, and attitudes to risk. But they converge under an interesting (necessary and sufficient) condition. (shrink)

Multidimensional adjectives are ubiquitous in natural language. An adjective F is multidimensional just in case whether F applies to an object or pair of objects depends on how those objects stand with respect to multiple underlying dimensions of F-ness. Developing a semantics for multidimensional adjectives requires us to address the problem of dimensional aggregation: how do the application conditions of an adjective F in its positive and comparative forms depend on its underlying dimensions? Here we develop a semantics for multidimensional (...) adjectives that incorporates aggregation functions. We then explore an analogy between dimensional aggregation and preference aggregation, bringing results from social choice theory to bear on the number and kind of aggregation functions that are admissible in a context. These results suggest that, for any given adjective, there will often be multiple aggregation functions admissible, meaning that multidimensional comparatives are often vague. (shrink)

Metanormativists hold that moral uncertainty can affect how we ought, in some morally authoritative sense, to act. Many metanormativists aim to generalize expected utility theory for normative uncertainty. Such accounts face the “easy problem of intertheoretic comparisons”: the worry that distinct theories’ assessments of choiceworthiness are incomparable. The easy problem may well be resolvable, but another problem looms: while some moral theories assign cardinal degrees of choiceworthiness, other theories’ choiceworthiness assignments are merely ordinal. Expected choiceworthiness over such theories is undefined. (...) Call this the “hard problem of intertheoretic comparisons.” This paper argues that to solve the hard problem, we should model moral theories with imprecise choiceworthiness. Imprecise choiceworthiness assignments can model incomplete cardinal information about choiceworthiness, with precise cardinal choiceworthiness and merely ordinal choiceworthiness as limiting cases. Generalizing familiar decision theories for imprecise choiceworthiness to the case of moral uncertainty generates puzzles, however: natural generalizations seem to require reifying parts of the model that don’t correspond to anything in normative reality. I discuss three ways of addressing this problem: by demystifying the reified elements by using them as promiscuously as possible; by constructing alternative decision theories that don’t require the troublesome elements; and by employing an alternative model of metanormative decision problems, and of moral uncertainty generally. (shrink)

What should we do when we are not certain about what we morally should do? There is a long history of theorizing about decision-making under empirical uncertainty, but surprisingly little has been written about the moral uncertainty expressed by this question. Only very recently have philosophers started to systematically address the nature of such uncertainty and its impacts on decision-making. This paper addresses the main problems raised by moral uncertainty and critically examines some proposed solutions.

The development of artificial intelligence will require systems of ethical decision making to be adapted for automatic computation. However, projects to implement moral reasoning in artificial moral agents so far have failed to satisfactorily address the widespread disagreement between competing approaches to moral philosophy. In this paper I argue that the proper response to this situation is to design machines to be fundamentally uncertain about morality. I describe a computational framework for doing so and show that it efficiently resolves common (...) obstacles to the implementation of moral philosophy in intelligent machines. (shrink)

As well as disagreeing about how much one should donate to charity, moral theories also disagree about where one should donate. In light of this disagreement, how should the morally uncertain philanthropist allocate her donations? In many cases, one intuitively attractive option is for the philanthropist to split her donations across all of the charities that are recommended by moral views in which she has positive credence, with each charity’s share being proportional to her credence in the moral theories that (...) recommend that particular charity. Despite the fact that something like this approach is already being used by real-world philanthropists to distribute billions of dollars of donations, it is not supported by any of the approaches to moral uncertainty that have been proposed thus-far in the philosophical literature. In this paper, we will develop a bargaining-based approach to moral uncertainty that honors the proportionality intuition in favor of splitting one’s donations. We also show how this bargaining-based approach has several further advantages over the best alternative proposals. (shrink)

This thesis is about making decisions when we are uncertain about what will happen, how valuable it will be, and even how to make decisions. Even the most sure-footed amongst us are sometimes uncertain about all three, but surprisingly little attention has been given to the latter two. The three essays that constitute my thesis hope to do a small part in rectifying this problem. The first essay is about the value of finding out how to make decisions. Society spends (...) considerable resources funding people (like me) to research decision-making, so it is natural to wonder whether society is getting a good deal. This question is so shockingly underresearched that bedrock facts are readily discoverable, such as when this kind of information is valuable. My second essay concerns whether we can compare value when we are uncertain about value. Many people are in fact uncertain about value, and how we deal with this uncertainty hinges on these comparisons. I argue that value comparisons are only sometimes possible; I call this weak comparability. This essay is largely a synthesis of the literature, but I also present an argument which begins with a peculiar view of the self: it is as if each of us is a crowd of different people separated by time (but connected by continuity of experience). I’m not the first to endorse this peculiar view of the self, but I am the first to show how it supports the benign view that value is sometimes comparable. We may be uncertain of any decision rules, even those that would tell us how to act when we face uncertainty in decision rules. We may be uncertain of how to decide how to decide how to... And so on. If so, we might have to accept infinitely many decision rules just to make any mundane decision, such as whether to pick up a five-cent piece from the gutter. My third essay addresses this problem of regress. I think all of our decisions are forced: we must decide now or continue to deliberate. Surprisingly, this allows us to avoid the original problem. I call this solution “when forced, do your best”. (shrink)

Social choice theory is the study of collective decision processes and procedures. It is not a single theory, but a cluster of models and results concerning the aggregation of individual inputs (e.g., votes, preferences, judgments, welfare) into collective outputs (e.g., collective decisions, preferences, judgments, welfare). Central questions are: How can a group of individuals choose a winning outcome (e.g., policy, electoral candidate) from a given set of options? What are the properties of different voting systems? When is a voting system (...) democratic? How can a collective (e.g., electorate, legislature, collegial court, expert panel, or committee) arrive at coherent collective preferences or judgments on some issues, on the basis of its members' individual preferences or judgments? How can we rank different social alternatives in an order of social welfare? Social choice theorists study these questions not just by looking at examples, but by developing general models and proving theorems. (shrink)

This paper explores the idea that it is instrumentally valuable to learn normative truths. We consider an argument for "normative hedging" based on this principle, and examine the structure of decision-making under moral uncertainty that arises from it. But it also turns out that the value of normative information is inconsistent with the principle that learning *empirical* truths is instrumentally valuable. We conclude with a brief comment on "metanormative regress.".

The ethics of emerging forms of artificial intelligence has become a prolific subject in both academic and public spheres. A great deal of these concerns flow from the need to ensure that these technologies do not cause harm—physical, emotional or otherwise—to the human agents with which they will interact. In the literature, this challenge has been met with the creation of artificial moral agents: embodied or virtual forms of artificial intelligence whose decision procedures are constrained by explicit normative principles, requiring (...) the implementation of what is commonly called artificial morality into these agents. To date, the types of reasoning structures and principles which inform artificial morality have been of two kinds: first, an ethically maximal vision of artificial morality which relies on the strict implementation of traditional moral theories such as Kantian deontology or Utilitarianism, and second, a more minimalist vision which applies stochastic AI techniques to large data sets of human moral preferences so as to illicit or intuit general principles and preferences for the design of artificial morality. Taken individually, each approach is unable to fully answer the challenge of producing inoffensive behavior in artificial moral agents, most especially since both forms are unable to strike a balance between the ideal set of constraints which morality imposes on one hand, and the types of constraints public acceptability imposes, on the other. We provide an alternative approach to the design of artificial morality, the Ethical Valence Theory, whose purpose is to accommodate this balance, and apply this approach to the case of autonomous vehicles. (shrink)

Lake et al. propose three criteria which, they argue, will bring artificial intelligence (AI) systems closer to human cognitive abilities. In this paper, we explore the application of these criteria to a particular domain of human cognition: our capacity for moral reasoning. In doing so, we explore a set of considerations relevant to the development of AI moral decision-making. Our main focus is on the relation between dual-process accounts of moral reasoning and model-free/model-based forms of machine learning. We also discuss (...) how work in embodied and situated cognition could provide a valu- able perspective on future research. (shrink)

According to the traditional Bayesian view of credence, its structure is that of precise probability, its objects are descriptive propositions about the empirical world, and its dynamics are given by conditionalization. Each of the three essays that make up this thesis deals with a different variation on this traditional picture. The first variation replaces precise probability with sets of probabilities. The resulting imprecise Bayesianism is sometimes motivated on the grounds that our beliefs should not be more precise than the evidence (...) calls for. One known problem for this evidentially motivated imprecise view is that in certain cases, our imprecise credence in a particular proposition will remain the same no matter how much evidence we receive. In the first essay I argue that the problem is much more general than has been appreciated so far, and that it’s difficult to avoid without compromising the initial evidentialist motivation. The second variation replaces descriptive claims with moral claims as the objects of credence. I consider three standard arguments for probabilism with respect to descriptive uncertainty—representation theorem arguments, Dutch book arguments, and accuracy arguments—in order to examine whether such arguments can also be used to establish probabilism with respect to moral uncertainty. In the second essay, I argue that by and large they can, with some caveats. First, I don’t examine whether these arguments can be given sound non-cognitivist readings, and any conclusions therefore only hold conditional on cognitivism. Second, decision-theoretic representation theorems are found to be less convincing in the moral case, because there they implausibly commit us to thinking that intertheoretic comparisons of value are always possible. Third and finally, certain considerations may lead one to think that imprecise probabilism provides a more plausible model of moral epistemology. The third variation considers whether, in addition to conditionalization, agents may also change their minds by becoming aware of propositions they had not previously entertained, and therefore not previously assigned any probability. More specifically, I argue that if we wish to make room for reflective equilibrium in a probabilistic moral epistemology, we must allow for awareness growth. In the third essay, I sketch the outline of such a Bayesian account of reflective equilibrium. Given that this account gives a central place to awareness growth, and that the rationality constraints on belief change by awareness growth are much weaker than those on belief change by conditionalization, it follows that the rationality constraints on the credences of agents who are seeking reflective equilibrium are correspondingly weaker. (shrink)

The principle that rational agents should maximize expected utility or choiceworthiness is intuitively plausible in many ordinary cases of decision-making under uncertainty. But it is less plausible in cases of extreme, low-probability risk (like Pascal's Mugging), and intolerably paradoxical in cases like the St. Petersburg and Pasadena games. In this paper I show that, under certain conditions, stochastic dominance reasoning can capture most of the plausible implications of expectational reasoning while avoiding most of its pitfalls. Specifically, given sufficient background uncertainty (...) about the choiceworthiness of one's options, many expectation-maximizing gambles that do not stochastically dominate their alternatives "in a vacuum" become stochastically dominant in virtue of that background uncertainty. But, even under these conditions, stochastic dominance will not require agents to accept options whose expectational superiority depends on sufficiently small probabilities of extreme payoffs. The sort of background uncertainty on which these results depend looks unavoidable for any agent who measures the choiceworthiness of her options in part by the total amount of value in the resulting world. At least for such agents, then, stochastic dominance offers a plausible general principle of choice under uncertainty that can explain more of the apparent rational constraints on such choices than has previously been recognized. (shrink)