Risk

Many philosophers accept that, as well as having a right that others not harm us, we also have a right that others not subject us to a risk of harm. And yet, when we attempt to spell out precisely what this ‘right against risk imposition’ involves, we encounter a series of notorious puzzles. Existing attempts to deal with these puzzles have tended to focus on the nature of rights – but I propose an approach that focusses instead on the nature (...) of risk. The key move is to distinguish two different ways in which to conceptualise the risk that a given action presents – one of which is linked to the notion of probability and the other to the notion of normalcy. (shrink)

In situations where we ignore everything but the space of possibilities, we ought to suspend judgment—that is, remain agnostic—about which of these possibilities is the case. This means that we cannot sum our degrees of belief in different possibilities, something that has been formalized as an axiom of non-additivity. Consistent with this way of representing our ignorance, I defend a doxastic norm that recommends that we should nevertheless follow a certain additivity of possibilities: even if we cannot sum degrees of (...) belief in different possibilities, we should be more confident in larger groups of possibilities. It is thus shown that, in the type of situation considered (in so-called ‘classical ignorance’, i.e. “behind a thin veil of ignorance”), it is epistemically rational for advocates of suspending judgment to endorse this comparative confidence; while on the other hand it is shown that, even in classical ignorance, no stronger belief—such as a precise uniform probability distribution—is warranted. (shrink)

Rowe has recently argued that pure risk of harm cannot itself be a harm. I respond to Rowe and argue that given an appropriate understanding of objective probabilities, pure objective risk of harm can itself be a harm.

Many people are worried about the harmful effects of climate change but nevertheless enjoy some activities that contribute to the emission of greenhouse gas (driving, flying, eating meat, etc.), the main cause of climate change. How should such people make choices between engaging in and refraining from enjoyable greenhouse-gas-emitting activities? In this chapter, we look at the answer provided by decision theory. Some scholars think that the right answer is given by interactive decision theory, or game theory; and moreover think (...) that since private climate decisions are instances of the prisoner’s dilemma, one rationally should engage in these activities provided that one enjoys them. Others think that the right answer is given by expected utility theory, the best-known version of individual decision theory under risk and uncertainty. In this chapter, we review these different answers, with a special focus on the latter answer and the debate it has generated. (shrink)

I defend a weak version of the Pigou-Dalton principle for chances. The principle says that it is better to increase the survival chance of a person who is more likely to die rather than a person who is less likely to die, assuming that the two people do not differ in any other morally relevant respect. The principle justifies plausible moral judgements that standard ex post views, such as prioritarianism and rank-dependent egalitarianism, cannot accommodate. However, the principle can be justified (...) by the same reasoning that has recently been used to defend the core axiom of ex post prioritarianism and egalitarianism, namely, Pigou-Dalton for well-being. The arguably biggest challenge for proponents of Pigou-Dalton for chances is that it violates State Dominance for social prospects. However, I argue that we have independent reason for rejecting State Dominance for social prospects, since it prevents a social planner from properly respecting people's preferences. (shrink)

In a recent paper in this journal Duncan Pritchard responds to an objection to the modal account of risk pressed by Ebert, Smith and Durbach ( 2020 ). In this paper, I expand upon the objection and argue that it still stands. I go on to consider a more general question raised by this exchange – whether risk is ‘objective’, or whether it is something that varies from one perspective to another.

Many people are worried about the harmful effects of climate change but nevertheless enjoy some activities that contribute to the emission of greenhouse gas (driving, flying, eating meat, etc.), the main cause of climate change. How should such people make choices between engaging in and refraining from enjoyable greenhouse-gas-emitting activities? In this chapter we look at the answer provided by decision theory. Some scholars think that the right answer is given by interactive decision theory, or game theory; and moreover think (...) that since private climate decisions are instances of the prisoner’s dilemma, one rationally should engage in these activities provided that one enjoys them. Others think that the right answer is given by expected utility theory, the best-known version of individual decision theory under risk and uncertainty. In this chapter we review these different answers, with a special focus on the latter answer and the debate it has generated. (shrink)

Many philosophers assume that, when making moral decisions under uncertainty, we should choose the option that has the greatest expected moral value, regardless of how risky it is. But their arguments for maximizing expected moral value do not support it over rival, risk-averse approaches. In this paper, I present a novel argument for maximizing expected value: when we think about larger series of decisions that each decision is a part of, all but the most risk-averse agents would prefer that we (...) consistently choose the option with the highest expected value. To the extent that what we choose on a given occasion should be guided by the entire series of choices we prefer, then on each occasion, we should choose the option with the highest expected moral value. (shrink)

Many healthcare practices expose people to risks of harmful outcomes. However, the major theories of moral philosophy struggle to assess whether, when and why it is ethically justifiable to expose individuals to risks, as opposed to actually harming them. Sven Ove Hansson has proposed an approach to the ethical assessment of risk imposition that encourages attention to factors including questions of justice in the distribution of advantage and risk, people’s acceptance or otherwise of risks, and the scope individuals have to (...) influence the practices that generate risk. This paper investigates the ethical justifiability of preventive healthcare practices that expose people to risks including overdiagnosis. We applied Hansson’s framework to three such practices: an ‘ideal’ breast screening service, a commercial personal genome testing service, and a guideline that lowers the diagnostic threshold for hypertension. The framework was challenging to apply, not least because healthcare has unclear boundaries and involves highly complex practices. Nonetheless, the framework encouraged attention to issues that would be widely recognised as morally pertinent. Our assessment supports the view that at least some preventive healthcare practices that impose risks including that of overdiagnosis are not ethically justifiable. Further work is however needed to develop and/or test refined assessment criteria and guidance for applying them. (shrink)

What moral reasons, if any, do we have to prevent the extinction of humanity? In “Unfinished Business,” Jonathan Knutzen argues that certain further developments in culture would make our history more ‘collectively meaningful,’ and that premature extinction would be bad because it would close off that possibility. Here, I critically examine this proposal. I argue that if collective meaningfulness is analogous to individual meaningfulness, then our meaning-based reasons to prevent the extinction of humanity are substantially different from the reasons discussed (...) by Knutzen. (shrink)

Recent claims that artificial intelligence (AI) systems demonstrate “emergent abilities” have fueled excitement but also fear grounded in the prospect that such systems may enable a wider range of parties to make unprecedented advances in areas that include the development of chemical or biological weapons. Ambiguity surrounding the term “emergent abilities” has added avoidable uncertainty to a topic that has the potential to destabilize the strategic landscape, including the perception of key parties about the viability of nonproliferation efforts. To avert (...) these problems in the future, scientists, developers, policymakers, and other parties should take credible steps to strengthen the health of the scientific ecosystem around AI. (shrink)

Since the release of ChatGPT, there has been a lot of debate about whether AI systems pose an existential risk to humanity. This paper develops a general framework for thinking about the existential risk of AI systems. We analyze a two-premise argument that AI systems pose a threat to humanity. Premise one: AI systems will become extremely powerful. Premise two: if AI systems become extremely powerful, they will destroy humanity. We use these two premises to construct a taxonomy of ‘survival (...) stories’, in which humanity survives into the far future. In each survival story, one of the two premises fails. Either scientific barriers prevent AI systems from becoming extremely powerful; or humanity bans research into AI systems, thereby preventing them from becoming extremely powerful; or extremely powerful AI systems do not destroy humanity, because their goals prevent them from doing so; or extremely powerful AI systems do not destroy humanity, because we can reliably detect and disable systems that have the goal of doing so. We argue that different survival stories face different challenges. We also argue that different survival stories motivate different responses to the threats from AI. Finally, we use our taxonomy to produce rough estimates of ‘P(doom)’, the probability that humanity will be destroyed by AI. (shrink)

Many of us think human extinction would be a very bad thing, and that we have moral reasons to prevent it. But there is disagreement over what would make extinction so bad, and thus over what grounds these moral reasons. Recently, several theorists have argued that our reasons to prevent extinction stem not just from the value of the welfare of future lives, but also from certain additional values relating to the existence of humanity itself (for example, humanity’s “final” value, (...) or the value of humanity in itself). In this paper, I argue against these “additional value” views. Despite their initial appeal, these views will inevitably face conflicts between the additional values to which they appeal, and the value of the welfare of future lives. And, I argue, the views cannot plausibly resolve these conflicts. In contrast, these conflicts do not arise for a rival view, on which our reasons to prevent extinction stem just from the value of the welfare of future lives. I conclude that this gives us reason to prefer the latter view, despite the greater initial plausibility of additional value views. (shrink)

The precautionary principle is invoked in a number of important personal and policy decision contexts. Peterson shows that certain ways of making the principle precise are inconsistent with other criteria of decision-making. Some object that the results do not apply to cases of deep uncertainty or value incommensurability which are alleged to be in the principle’s wheelhouse. First, I show that Peterson’s impossibility results can be generalized considerably to cover cases of both deep uncertainty and incommensurability. Second, I contrast an (...) alternative way of giving voice to the precautionary impulse. (shrink)

Longtermists have recently argued that it is overwhelmingly important to do what we can to mitigate existential risks to humanity. I consider three mistakes that are often made in calculating the value of existential risk mitigation. I show how correcting these mistakes pushes the value of existential risk mitigation substantially below leading estimates, potentially low enough to threaten the normative case for existential risk mitigation. I use this discussion to draw four positive lessons for the study of existential risk. -/- (...) . (shrink)

The ongoing evolution of advanced AI systems will have profound, enduring, and significant impacts on human existence that must not be overlooked. These impacts range from empowering humanity to achieve unprecedented transcendence to potentially causing catastrophic threats to our existence. To proactively and preventively mitigate these potential threats, it is crucial to establish clear redlines to prevent AI-induced existential risks by constraining and regulating advanced AI and their related AI actors. This paper explores different concepts of AI existential risk, connects (...) the enactment of AI red lines to broader efforts addressing AI's impacts, constructs a theoretical framework for analyzing the direct impacts of AI existential risk, and upon that proposes a set of exemplary AI red lines. By contemplating AI existential risks and formulating these red lines, we aim to foster a deeper and systematic understanding of the potential dangers associated with advanced AI and the importance of proactive risk management. We hope this work will contribute to the strengthening and refinement of a comprehensive AI redline system for preventing humanity from AI existential risks. (shrink)

Recent existential-risk thinkers have noted that the analysis of the fine-tuning argument for God’s existence, and the analysis of certain forms of existential risk, employ similar types of reasoning. This paper argues that insofar as the “many worlds objection” undermines the inference to God’s existence from universal fine-tuning, then a similar many worlds objection undermines the inference that the historic risk of global nuclear catastrophe has been low from the lack of such a catastrophe has occurred in our world. A (...) version of the fine-tuning argument applied to nuclear risk, The Nuclear Fine-Tuning Argument, utilizes the set of nuclear close calls to show that 1) conventional explanations fail to adequately explain how we have survived thus far and 2) the existence of many worlds provides an adequate explanation. This is because, if there are many worlds, observers are disproportionately more likely to reflect upon a world that hasn’t had a global nuclear catastrophe than upon one that has had a global nuclear catastrophe. This selection bias results from the catastrophic nature of such an event. This argument extends generally to all global catastrophic risks that both A) have been historic threats and B) would result in a significantly lower global population. (shrink)

Rawls famously argues that the parties in the original position would agree upon the two principles of justice. Among other things, these principles guarantee equal political liberty—that is, democracy—as a requirement of justice. We argue on the contrary that the parties have reason to reject this requirement. As we show, by Rawls’ own lights, the parties would be greatly concerned to mitigate existential risk. But it is doubtful whether democracy always minimizes such risk. Indeed, no one currently knows which political (...) systems would. Consequently, the parties—and we ourselves—have reason to reject democracy as a requirement of justice in favor of political experimentalism, a general approach to political justice which rules in at least some non-democratic political systems which might minimize existential risk. (shrink)

Longtermism is the view that the most urgent global priorities, and those to which we should devote the largest portion of our resources, are those that focus on (i) ensuring a long future for humanity, and perhaps sentient or intelligent life more generally, and (ii) improving the quality of the lives that inhabit that long future. While it is by no means the only one, the argument most commonly given for this conclusion is that these interventions have greater expected goodness (...) per unit of resource devoted to them than each of the other available interventions, including those that focus on the health and well-being of the current population. In this paper, I argue that, even if we grant the consequentialist ethics upon which this argument depends, and even if we grant one of the axiologies that are typically paired with that ethics to give the argument, we are not morally required to choose an option that maximises expected utility; indeed, we might not even be permitted to do so. Instead, I will argue, if the argument's consequentialism is correct, we should choose using a decision theory that is sensitive to risk, and allows us to give greater weight to worse-case outcomes than expected utility theory does. And, I will show, such decision theories do not always recommend longtermist interventions. Indeed, sometimes, they recommend exactly the opposite: sometimes, they recommend hastening human extinction. Many, though not all, will take this as a reductio of the consequentialism or the axiology of the argument. I remain agnostic on the conclusion we should draw. (shrink)

In this paper, we argue that our moral concern for future well-being should reduce over time due to important practical considerations about how humans interact with spacetime. After surveying several of these considerations (around equality, special duties, existential contingency, and overlapping moral concern) we develop a set of core principles that can both explain their moral significance and highlight why this is inherently bound up with our relationship with spacetime. These relate to the equitable distribution of (1) moral concern in (...) a universe that we can only causally affect in one temporal direction; (2) access to the benefits from using spacetime as a resource; and (3) the burdens of care given (1) and (2). We conclude by considering the practical implications of our argument and find that, while it is often assumed that a preference for present over future well-being weakens the case for existential risk mitigation, this likely is not the case. (shrink)

This paper examines a recent argument in favour of strong precautionary action—possibly including working to hasten human extinction—on the basis of a decision-theoretic view that accommodates the risk-attitudes of all affected while giving more weight to the more risk-averse attitudes. First, we dispute the need to take into account other people’s attitudes towards risk at all. Second we argue that a version of the non-identity problem undermines the case for doing so in the context of future people. Lastly, we suggest (...) that we should not work to hasten human extinction, even if significant risk aversion is warranted. (shrink)

In an effort to inform the discussion surrounding existential risks from AI, we formulate Extinction-level Goodhart’s Law as “Virtually any goal specification, pursued to the extreme, will result in the extinction of humanity”, and we aim to understand which formal models are suitable for investigating this hypothesis. Note that we remain agnostic as to whether Extinction-level Goodhart’s Law holds or not. As our key contribution, we identify a set of conditions that are necessary for a model that aims to be (...) informative for evaluating specific arguments for Extinction-level Goodhart’s Law. Since each of the conditions seems to significantly contribute to the complexity of the resulting model, formally evaluating the hypothesis might be exceedingly difficult. This raises the possibility that whether the risk of extinction from artificial intelligence is real or not, the underlying dynamics might be invisible to current scientific methods. (shrink)

In this commentary on Darrel Moellendorf’s Mobilizing Hope: Climate Change & Global Poverty (Oxford: Oxford University Press, 2022), I discuss his use of the precautionary principle, whether his hope for climate-friendly ‘green growth’ is realistic given the tendency for inequality to accelerate as it gets higher, and what I call his assumption of a liberal baseline. That is, I worry that the audience to whom the book is addressed are those who already accept the environmental and economic values to which (...) Moellendorf appeals, while those who are blocking effective action on climate change in fact reject those values and are therefore unlikely to be moved by the arguments the book presents. (shrink)

Philosophy &Public Affairs, Volume 51, Issue 4, Page 373-412, Fall 2023.

When our choice affects some other person and the outcome is unknown, it has been argued that we should defer to their risk attitude, if known, or else default to use of a risk avoidant risk function. This, in turn, has been claimed to require the use of a risk avoidant risk function when making decisions that primarily affect future people, and to decrease the desirability of efforts to prevent human extinction, owing to the significant risks associated with continued human (...) survival. I raise objections to the claim that respect for others’ risk attitudes requires risk avoidance when choosing for future generations. In particular, I argue that there is no known principle of interpersonal aggregation that yields acceptable results in variable population contexts and is consistent with a plausible ideal of respect for others’ risk attitudes in fixed population cases. GPI Working Paper No . 20-2022. (shrink)

Our Earth’s climate is changing. Any species living in the Earth’s ecosystem need to thrive to adapt to the new living conditions. Otherwise, extinction will be their outcome. In the race for adaptation, waterbirds (Aequorlitornithes), such as penguins, cormorants, and alcids, seem disadvantageous.

Recent advances in natural language processing have given rise to a new kind of AI architecture: the language agent. By repeatedly calling an LLM to perform a variety of cognitive tasks, language agents are able to function autonomously to pursue goals specified in natural language and stored in a human-readable format. Because of their architecture, language agents exhibit behavior that is predictable according to the laws of folk psychology: they function as though they have desires and beliefs, and then make (...) and update plans to pursue their desires given their beliefs. We argue that the rise of language agents significantly reduces the probability of an existential catastrophe due to loss of control over an AGI. This is because the probability of such an existential catastrophe is proportional to the difficulty of aligning AGI systems, and language agents significantly reduce that difficulty. In particular, language agents help to resolve three important issues related to aligning AIs: reward misspecification, goal misgeneralization, and uninterpretability. (shrink)

It is plausible that current generations owe something to future generations. One possibility is that we have a duty to not harm them. Another possibility is that we have a duty to protect them. In either case, however, to satisfy the duties to future generations from environmental or political degradation, we need to engage in widespread collective action. But, as we are, we have a limited ability to do so, in part because we lack the self-discipline necessary for successful collective (...) action. Given that having an obligation hinges on being able to satisfy it, the apparent duty to future generations is at odds with our apparent inability to satisfy it. Thus, we either need to rethink our duties to future generations or rethink the nature of our abilities. I argue in this chapter that we should rethink the nature of our abilities. Specifically, we should enhance our self-discipline, which is distinct from enhancements of motivation, judgment, or emotion. Even the most intelligent, motivated, and empathetic actor is likely to slip up, which encourages others to withdraw their cooperation. Without the necessary self-discipline to sustain our public cooperative behavior, collective action is likely to fail, leaving us with one alternative: to abandon our duties to future generations. (shrink)

Longtermists claim that what we ought to do is mainly determined by how our actions might affect the very long-run future. A natural objection to longtermism is that these effects may be nearly impossible to predict — perhaps so close to impossible that, despite the astronomical importance of the far future, the expected value of our present actions is mainly determined by near-term considerations. This paper aims to precisify and evaluate one version of this epistemic objection to longtermism. To that (...) end, I develop two simple models for comparing ‘longtermist’ and ‘neartermist’ interventions, incorporating the idea that it is harder to make a predictable difference to the further future. These models yield mixed conclusions: if we simply aim to maximize expected value, and don’t mind premising our choices on minuscule probabilities of astronomical payoffs, the case for longtermism looks robust. But on some prima facie plausible empirical worldviews, the expectational superiority of longtermist interventions depends heavily on these ‘Pascalian’ probabilities. So the case for longtermism may depend either on plausible but non-obvious empirical claims or on a tolerance for Pascalian fanaticism. (shrink)

Why, if at all, should we object to economic inequality? Some central arguments – the argument from decreasing marginal utility for example – invoke instrumental reasons and object to inequality because of its effects. Such instrumental arguments, however, often concern only the static effects of inequality and neglect its intertemporal conse- quences. In this article, we address this striking gap and investigate income inequality’s intertemporal consequences, including its potential effects on humanity’s (very) long-term future. Following recent arguments around future generations (...) and so-called longtermism, those effects might arguably matter more than inequality’s short-term con- sequences. We assess whether we have instrumental reason to reduce economic inequality based on its intertemporal effects in the short, medium, and the very long term. We find a good short and medium-term instrumental case for lower economic inequality. We then argue, somewhat speculatively, that we have instrumental reasons for inequality reduction from a longtermist perspective too, primarily because greater inequality could increase existential risk. We thus have instrumental reasons to reduce inequality, regardless of which time-horizon we take. We then argue that from most consequentialist perspectives, this pro tanto reason also gives us all-things-considered reason. And even across most non-consequentialist views in philosophy, this argument gives us either an all-things-considered or at least weighty pro tanto reason against inequality. (shrink)

A social planner who evaluates risky public policies in light of the other risks with which their society will be faced should judge favourably some such policies even though they would deem them too risky when considered in isolation. I suggest that a longtermist would—or at least should—evaluate risky polices in light of their prediction about future risks; hence, longtermism supports social risk-taking. I consider two formal versions of this argument, discuss the conditions needed for the argument to be valid, (...) and briefly compare these conditions to some risky policy options with which actual public decision-makers are faced. (shrink)

Longtermists have argued that humanity should significantly increase its efforts to prevent catastrophes like nuclear wars, pandemics, and AI disasters. But one prominent longtermist argument overshoots this conclusion: the argument also implies that humanity should reduce the risk of existential catastrophe even at extreme cost to the present generation. This overshoot means that democratic governments cannot use the longtermist argument to guide their catastrophe policy. In this paper, we show that the case for preventing catastrophe does not depend on longtermism. (...) Standard cost-benefit analysis implies that governments should spend much more on reducing catastrophic risk. We argue that a government catastrophe policy guided by cost-benefit analysis should be the goal of longtermists in the political sphere. This policy would be democratically acceptable, and it would reduce existential risk by almost as much as a strong longtermist policy. (shrink)

Like asteroids, hundred-year floods and pandemic disease, thermonuclear war is a low-frequency, high-impact threat. In the long run, catastrophe is inevitable if nothing is done − yet each successive government and generation may fail to address it. Drawing on risk perception research, this paper argues that psychological biases cause the threat of nuclear war to receive less attention than it deserves. Nuclear deterrence is, moreover, a ‘front-loaded good’: its benefits accrue disproportionately to proximate generations, whereas much of the expected cost (...) will be borne in the distant future. Recent surveys indicate that the US and Russian publics assign a surprisingly high likelihood to nuclear war. Nevertheless, earlier research suggests that it is probably not believed to be just around the corner. This, along with the absence of easy solutions, encourages governments and publics to give priority to more pressing concerns. The danger is that the pattern will continue clear up to the point that nuclear war arrives. (shrink)

According to an intriguing though somewhat enigmatic line of thought first proposed by Jonathan Bennett, if humanity went extinct any time soon this would be unfortunate because important business would be left unfinished. This line of thought remains largely unexplored. I offer an interpretation of the idea that captures its intuitive appeal, is consistent with plausible constraints, and makes it non-redundant to other views in the literature. The resulting view contrasts with a welfare-promotion perspective, according to which extinction would be (...) bad because it would come at a massive opportunity cost in terms of feasibly achievable welfare over the lifetime of our species. (shrink)

Current planetary defense policy prioritizes a probability assessment of risk of Earth impact by an asteroid or a comet in the planning of detection and mitigation strategies and in setting the levels of urgency and budgeting to operationalize them. The result has been a focus on asteroids of Tunguska size, which could destroy a city or a region, since this is the most likely sort of object we would need to defend against. However a complete risk assessment would consider not (...) only the probability of an impact but also the magnitude of its consequences, which in the case of an object of Chicxulub size could be the end of civilization or even human extinction. This paper argues that a planetary defense policy based on a complete (or one could say genuine) risk assessment would justify expenditures much higher than at present. (shrink)

“Existential risks” are risks that threaten the destruction of humanity’s long-term potential: risks of nuclear wars, pandemics, supervolcano eruptions, and so on. On standard utilitarianism, it seems, the reduction of such risks should be a key global priority today. Many effective altruists agree with this verdict. But how should the importance of these risks be assessed on a Christian moral theory? In this paper, I begin to answer this question – taking Thomas Aquinas as a reference, and the risks of (...) anthropogenic extinction as an example. I’ll suggest that on a Thomist framework, driving ourselves extinct would constitute a complete failure to fulfil our God-given nature, a radical form of hubris, and a loss of cosmological significance. So the reduction of such risks seems vital for Thomists, or Christians more broadly. Indeed, I end with the thought that these considerations are not parochially Christian, or indeed specifically religious: everyone with a similar view of mankind – as immensely valuable yet fallen and fragile – should agree that existential risks are a pressing moral concern. (shrink)

The standard argument to the conclusion that artificial intelligence (AI) constitutes an existential risk for the human species uses two premises: (1) AI may reach superintelligent levels, at which point we humans lose control (the ‘singularity claim’); (2) Any level of intelligence can go along with any goal (the ‘orthogonality thesis’). We find that the singularity claim requires a notion of ‘general intelligence’, while the orthogonality thesis requires a notion of ‘instrumental intelligence’. If this interpretation is correct, they cannot be (...) joined as premises and the argument for the existential risk of AI turns out invalid. If the interpretation is incorrect and both premises use the same notion of intelligence, then at least one of the premises is false and the orthogonality thesis remains itself orthogonal to the argument to existential risk from AI. In either case, the standard argument for existential risk from AI is not sound.—Having said that, there remains a risk of instrumental AI to cause very significant damage if designed or used badly, though this is not due to superintelligence or a singularity. (shrink)

Alan Turing, one of the fathers of computing, warned that Artificial Intelligence (AI) could one day pose an existential risk to humanity. Today, recent advancements in the field AI have been accompanied by a renewed set of existential warnings. But what exactly constitutes an existential risk? And how exactly does AI pose such a threat? In this chapter we aim to answer these questions. In particular, we will critically explore three commonly cited reasons for thinking that AI poses an existential (...) threat to humanity: the control problem, the possibility of global disruption from an AI race dynamic, and the weaponization of AI. (shrink)

The coronavirus pandemic, like its predecessors - AIDS, Ebola, etc., is evidence of the evolutionary instability of the socio-cultural and ecological niche created by mankind, as the main factor in the evolutionary success of our biological species and the civilization created by it. At least, this applies to the modern global civilization, which is called technogenic or technological, although it exists in several varieties. As we hope to show, the current crisis has less ontological as well as epistemological roots; its (...) reason lies in the main evolutionary trends in the development of science as a social institution. It was only later that epistemological factors were transformed into existential-ontological factors associated with the asymmetry of the existence of civilization and our biosocial nature. The perception or ignorance of a risk factor as a real fact is determined by the presence or absence of knowledge about it. In other words, risk is the result of the integration of the corresponding ontological concept into the general categorical structure. The plurality of such structures is a hallmark of multidisciplinary ontologies, each of which is associated with its own factual continuum. The aim of this study was to conceptually model for support relativistic parameter of evolutionary efficiency of stable evolutionary human strategy (SESH) in its techno-rationalistic module. The meaning of this term is equivalent to the category of scientific and technological development. (shrink)

This article assesses how autonomy and machine learning impact the existential risk of nuclear war. It situates the problem of cyber security, which proceeds by stealth, within the larger context of nuclear deterrence, which is effective when it functions with transparency and credibility. Cyber vulnerabilities poses new weaknesses to the strategic stability provided by nuclear deterrence. This article offers best practices for the use of computer and information technologies integrated into nuclear weapons systems. Focusing on nuclear command and control, avoiding (...) autonomy and machine learning is recommended as one means to reduce the existential risk of unintended nuclear conflict. (shrink)

Many researchers have argued that humanity will create artificial general intelligence (AGI) within the next twenty to one hundred years. It has been suggested that AGI may inflict serious damage to human well-being on a global scale ('catastrophic risk'). After summarizing the arguments for why AGI may pose such a risk, we review the fieldʼs proposed responses to AGI risk. We consider societal proposals, proposals for external constraints on AGI behaviors and proposals for creating AGIs that are safe due to (...) their internal design. (shrink)

Ian Stoner has recently argued that we ought not to colonize Mars because doing so would flout our pro tanto obligation not to violate the principle of scientific conservation, and there is no countervailing considerations that render our violation of the principle permissible. While I remain agnostic on, my primary goal in this article is to challenge : there are countervailing considerations that render our violation of the principle permissible. As such, Stoner has failed to establish that we ought not (...) to colonize Mars. I close with some thoughts on what it would take to show that we do have an obligation to colonize Mars and related issues concerning the relationship between the way we discount our preferences over time and projects with long time horizons, like space colonization. (shrink)

The possibility of social and technological collapse has been the focus of science fiction tropes for decades, but more recent focus has been on specific sources of existential and global catastrophic risk. Because these scenarios are simple to understand and envision, they receive more attention than risks due to complex interplay of failures, or risks that cannot be clearly specified. In this paper, we discuss the possibility that complexity of a certain type leads to fragility which can function as a (...) source of catastrophic or even existential risk. The paper first reviews a hypothesis by Bostrom about inevitable technological risks, named the vulnerable world hypothesis. This paper next hypothesizes that fragility may not only be a possible risk, but could be inevitable,and would therefore be a subclass or example of Bostrom’s vulnerable worlds. After introducing the titular fragile world hypothesis, the paper details the conditions under which it would be correct, and presents arguments for why the conditions may in fact may apply. Finally, the assumptions and potential mitigations of the new hypothesis are contrasted with those Bostrom suggests. (shrink)

Human civilisation faces a range of existential risks, including nuclear war, runaway climate change and superintelligent artificial intelligence run amok. As we show here with calculations for the New Zealand setting, large numbers of currently living and, especially, future people are potentially threatened by existential risks. A just process for resource allocation demands that we consider future generations but also account for solidarity with the present. Here we consider the various ethical and policy issues involved and make a case for (...) further engagement with the New Zealand public to determine societal values towards future lives and their protection. (shrink)

Many global catastrophic and existential risks (X-risks) threaten the existence of humankind. There are also many ideas for their prevention, but the meta-problem is that these ideas are not structured. This lack of structure means it is not easy to choose the right plan(s) or to implement them in the correct order. I suggest using a “Plan A, Plan B” model, which has shown its effectiveness in planning actions in unpredictable environments. In this approach, Plan B is a backup option, (...) implemented if Plan A fails. In the case of global risks, Plan A is intended to prevent a catastrophe and Plan B to survive it, if it is not avoided. Each plan has similar stages: analysis, planning, funding, low-level realization and high-level realization. Two variables—plans and stages—provide an effective basis for classification of all possible X-risks prevention methods in the form of a two-dimensional map, allowing the choice of optimal paths for human survival. I have created a framework for estimating the utility of various prevention methods based on their probability of success, the chances that they will be realized in time, their opportunity cost, and their risk. I also distinguish between top-down and bottom-up approaches. (shrink)

Existential risks threaten the future of humanity, but they are difficult to measure. However, to communicate, prioritize and mitigate such risks it is important to estimate their relative significance. Risk probabilities are typically used, but for existential risks they are problematic due to ambiguity, and because quantitative probabilities do not represent some aspects of these risks. Thus, a standardized and easily comprehensible instrument is called for, to communicate dangers from various global catastrophic and existential risks. In this article, inspired by (...) the Torino scale of asteroid danger, we suggest a color coded scale to communicate the magnitude of global catastrophic and existential risks. The scale is based on the probability intervals of risks in the next century if they are available. The risks’ estimations could be adjusted based on their severities and other factors. The scale covers not only existential risks, but smaller size global catastrophic risks. It consists of six color levels, which correspond to previously suggested levels of prevention activity. We estimate artificial intelligence risks as “red”, while “orange” risks include nanotechnology, synthetic biology, full scale nuclear war and a large global agricultural shortfall (caused by regional nuclear war, coincident extreme weather, etc.) The risks of natural pandemic, supervolcanic eruption and global warming are marked as “yellow” and the danger from asteroids is “green”. -/- Keywords: global catastrophic risks; existential risks; Torino scale; policy; risk probability . (shrink)

Some people think that the inevitability of human extinction renders life meaningless. Joshua Seachris has argued that naturalism can be conceptualized as a meta-narrative and that it narrates across important questions of human life, including what is the meaning of life and how life will end. How a narrative ends is important, Seachris argues. In the absence of God, and with knowledge that human extinction is a certainty, is there any way that humanity could be meaningful and have a good (...) ending? I will distinguish between two conceptions of how humanity could be meaningful: the traditional view and an alternative view, which I will outline. I will argue that this alternative view provides a plausible explanation for how humanity could become meaningful. I will also argue that coming to terms with our mortality and other limitations would add meaning to human life and provide humanity with a good ending. (shrink)

Papers from the conference on AI Risk (published in JETAI), supplemented by additional work. --- If the intelligence of artificial systems were to surpass that of humans, humanity would face significant risks. The time has come to consider these issues, and this consideration must include progress in artificial intelligence (AI) as much as insights from AI theory. -- Featuring contributions from leading experts and thinkers in artificial intelligence, Risks of Artificial Intelligence is the first volume of collected chapters dedicated to (...) examining the risks of AI. The book evaluates predictions of the future of AI, proposes ways to ensure that AI systems will be beneficial to humans, and then critically evaluates such proposals. 1 Vincent C. Müller, Editorial: Risks of Artificial Intelligence - 2 Steve Omohundro, Autonomous Technology and the Greater Human Good - 3 Stuart Armstrong, Kaj Sotala and Sean O’Heigeartaigh, The Errors, Insights and Lessons of Famous AI Predictions - and What they Mean for the Future - 4 Ted Goertzel, The Path to More General Artificial Intelligence - 5 Miles Brundage, Limitations and Risks of Machine Ethics - 6 Roman Yampolskiy, Utility Function Security in Artificially Intelligent Agents - 7 Ben Goertzel, GOLEM: Toward an AGI Meta-Architecture Enabling Both Goal Preservation and Radical Self-Improvement - 8 Alexey Potapov and Sergey Rodionov, Universal Empathy and Ethical Bias for Artificial General Intelligence - 9 András Kornai, Bounding the Impact of AGI - 10 Anders Sandberg, Ethics and Impact of Brain Emulations 11 Daniel Dewey, Long-Term Strategies for Ending Existential Risk from Fast Takeoff - 12 Mark Bishop, The Singularity, or How I Learned to Stop Worrying and Love AI -. (shrink)