Technological revolutions are among the most important things that happen to humanity. Ethical assessment in the incipient stages of a potential technological revolution faces several difficulties, including the unpredictability of their long‐term impacts, the problematic role of human agency in bringing them about, and the fact that technological revolutions rewrite not only the material conditions of our existence but also reshape culture and even – perhaps – human nature. This essay explores some of these difficulties and the challenges they pose (...) for a rational assessment of the ethical and policy issues associated with anticipated technological revolutions. (shrink)

Peter Singer has argued that there are good utilitarian reasons for rejecting the prospect of superlongevity: developing technology to double (or more) the average human lifespan. I argue against Singer's view on two fronts. First, empirical research on happiness indicates that the later years of life are (on average) the happiest, and there is no reason to suppose that this trend would not continue if superlongevity were realized. Second, it is argued that there are good reasons to suppose that there (...) will be a certain amount of self-selection: the happiest are more likely to adopt superlongevity technology. This means that the adoption of superlongevity technology will have the effect of raising the level of aggregate utility. (shrink)

Maximize Expected Choiceworthiness (MEC) is a theory of decision-making under moral uncertainty. It says that we ought to handle moral uncertainty in the way that Expected Value Theory (EVT) handles descriptive uncertainty. MEC inherits from EVT the problem of fanaticism. Roughly, a decision theory is fanatical when it requires our decision-making to be dominated by low-probability, high-payoff options. Proponents of MEC have offered two main lines of response. The first is that MEC should simply import whatever are the best solutions (...) to fanaticism on offer in decision theory. The second is to propose statistical normalization as a novel solution on behalf of MEC. This paper argues that the first response is open to serious doubt and that the second response fails. As a result, MEC appears significantly less plausible when compared to competing accounts of decision-making under moral uncertainty, which are not fanatical. (shrink)

This paper examines a recent argument in favour of strong precautionary action—possibly including working to hasten human extinction—on the basis of a decision-theoretic view that accommodates the risk-attitudes of all affected while giving more weight to the more risk-averse attitudes. First, we dispute the need to take into account other people’s attitudes towards risk at all. Second we argue that a version of the non-identity problem undermines the case for doing so in the context of future people. Lastly, we suggest (...) that we should not work to hasten human extinction, even if significant risk aversion is warranted. (shrink)

Longtermism is the view that positively influencing the long-term future is one of the key moral priorities of our time. Longtermists generally focus on humans, and neglect animals. This is a mistake. In this paper I will show that the basic argument for longtermism applies to animals at least as well as it does to humans, and that the reasons longtermists have given for ignoring animals do not withstand scrutiny. Because of their numbers, their capacity for suffering, and our ability (...) to influence their futures, animals ought to be a central concern of longtermists. Furthermore, I will suggest that longtermism is a fruitful framework for thinking about the wellbeing of animals, as it helps us to identify actions we can take now that have a reasonable chance of improving the wellbeing of animals over the very long term. (shrink)

The notion of existential catastrophe is increasingly appealed to in discussion of risk management around emerging technologies, but it is not completely clear what this notion amounts to. Here, I provide an opinionated survey of the space of plausibly useful definitions of existential catastrophe. Inter alia, I discuss: whether to define existential catastrophe in ex post or ex ante terms, whether an ex ante definition should be in terms of loss of expected value or loss of potential, and what kind (...) of probabilities should be involved in any appeal to expected value. (shrink)

Effective altruism sounds so innocuous—who could possibly be opposed to doing good more effectively? Yet it has inspired significant backlash in recent years. This paper addresses some common misconceptions and argues that the core “beneficentric” ideas of effective altruism are both excellent and widely neglected. Reasonable people may disagree on details of implementation, but all should share the basic goals or values underlying effective altruism.

This thesis is about the moral reasons we have to do less than best. It consists of six chapters. Part I of the thesis proposes, extends, and defends reasons to do less than best. In Chapter One (“The Conditional Obligation”) I outline and reject two recent arguments from Joe Horton and Theron Pummer for the claim that we have a conditional obligation to bring about the most good. In Chapter Two (“Agglomeration and Agent-Relative Costs”) I argue that agent-relative costs can (...) justify permissions to less than best in a far greater range of decisions contexts than we currently believe. In Chapter Three (“Risk and Aggregation”) I discuss how anti-aggregative moral theories deal with risk, whilst proposing an argument in favour of ex-post moral theories. -/- In Part II of the thesis, I discuss longtermism, presenting a series of reasons to believe we have both significant permissions and obligations to prioritise those in the short-term. In Chapter Four (“Listening to Future People”) I criticise a recent argument presented by Andreas Mogensen for the claim we cannot have obligations grounded in the wellbeing of future people. I then show that, even permitting the existence of obligations to future people, we still have significant reasons to resist longtermism on the grounds of partiality. In Chapter Five (“The Complaints of Future People”) I argue that scepticism about aggregation is inconsistent with an obligation, or indeed even a permission, to help far future people. In Chapter Six (“Leaving the Present Behind”) I explore the moral significance of “dooming”, whilst arguing that we have a pro tanto reason of fairness to prioritise aiding those currently in existence. (shrink)

Longtermists have recently argued that it is overwhelmingly important to do what we can to mitigate existential risks to humanity. I consider three mistakes that are often made in calculating the value of existential risk mitigation. I show how correcting these mistakes pushes the value of existential risk mitigation substantially below leading estimates, potentially low enough to threaten the normative case for existential risk mitigation. I use this discussion to draw four positive lessons for the study of existential risk. -/- (...) . (shrink)

Longtermists claim that what we ought to do is mainly determined by how our actions might affect the very long-run future. A natural objection to longtermism is that these effects may be nearly impossible to predict — perhaps so close to impossible that, despite the astronomical importance of the far future, the expected value of our present actions is mainly determined by near-term considerations. This paper aims to precisify and evaluate one version of this epistemic objection to longtermism. To that (...) end, I develop two simple models for comparing ‘longtermist’ and ‘neartermist’ interventions, incorporating the idea that it is harder to make a predictable difference to the further future. These models yield mixed conclusions: if we simply aim to maximize expected value, and don’t mind premising our choices on minuscule probabilities of astronomical payoffs, the case for longtermism looks robust. But on some prima facie plausible empirical worldviews, the expectational superiority of longtermist interventions depends heavily on these ‘Pascalian’ probabilities. So the case for longtermism may depend either on plausible but non-obvious empirical claims or on a tolerance for Pascalian fanaticism. (shrink)

Should we make significant sacrifices to ever-so-slightly lower the chance of extremely bad outcomes, or to ever-so-slightly raise the chance of extremely good outcomes? *Fanaticism* says yes: for every bad outcome, there is a tiny chance of extreme disaster that is even worse, and for every good outcome, there is a tiny chance of an enormous good that is even better. I consider two related recent arguments for Fanaticism: Beckstead and Thomas's argument from *strange dependence on space and time*, and (...) Wilkinson's *Indology* argument. While both arguments are instructive, neither is persuasive. In fact, the general principles that underwrite the arguments (a *separability* principle in the first case, and a *reflection* principle in the second) are *inconsistent* with Fanaticism. In both cases, though, it is possible to rehabilitate arguments for Fanaticism based on restricted versions of those principles. The situation is unstable: plausible general principles tell *against* Fanaticism, but restrictions of those same principles (with strengthened auxiliary assumptions) *support* Fanaticism. All of the consistent views that emerge are very strange. (shrink)

Philosophy &Public Affairs, Volume 51, Issue 4, Page 373-412, Fall 2023.

This paper considers questions that small animals and AI systems raise for utilitarianism. Specifically, if these beings have more welfare than humans and other large animals, then utilitarianism implies that we should prioritize them, all else equal. This could lead to a ‘rebugnant conclusion’, according to which we should, say, create large populations of small animals rather than small populations of large animals. It could also lead to a ‘Pascal’s bugging’, according to which we should, say, prioritize large populations of (...) small animals even if they have a low chance of being sentient. I suggest that utilitarians should accept these implications in theory but might be able to avoid some of them in practice. (shrink)

Our aim is to defend the Precautionary Principle (PP) against the main theoretical and practical criticisms that it has raised by proposing a novel conception and a specific formulation of the principle. We first address the theoretical concerns against the idea of there being a principle of precaution by arguing for a distinctively political conception of the PP as opposed to a moral one. Our claim is that the rationale of the PP is grounded in the fact that contemporary societies (...) wish to avoid morally unacceptable risks while at the same time pursuing other goals, aspirations and values. Following this political conception of the PP, we propose a specific formulation in terms of an argumentation scheme—the Argumentation Scheme for Appealing to Precaution—which is meant to solve, in turn, the main practical concerns that the PP has raised. We explain the advantages of our proposal by analysing appeals to precaution in the cases of (1) the European restrictions on the free movement of persons during the COVID-19 pandemic and (2) the discharge of nuclear wastes into the Irish Sea. (shrink)

We begin by showing that every theory of the value of uncertain prospects must have one of three unpalatable properties. _Reckless_ theories recommend giving up a sure thing, no matter how good, for an arbitrarily tiny chance of enormous gain; _timid_ theories permit passing up an arbitrarily large potential gain to prevent a tiny increase in risk; _non-transitive_ theories deny the principle that, if A is better than B and B is better than C, then A must be better than (...) C. Having set up this trilemma, we study its horns. Non-transitivity has been much discussed; we focus on drawing out the costs and benefits of recklessness and timidity when it comes to axiology, decision theory, and normative uncertainty. (shrink)

Advocates of longtermism point out that interventions which focus on improving the prospects of people in the very far future will, in expectation, bring about a significant amount of good. Indeed, in expectation, such long-term interventions bring about far more good than their short-term counterparts. As such, longtermists claim we have compelling moral reason to prefer long-term interventions. In this paper, I show that longtermism is in conflict with plausible deontic scepticism about aggregation. I do so by demonstrating that, from (...) both an ex-ante and ex-post perspective, longtermist interventions – and, in particular, those which aim to mitigate catastrophic risk – typically generate extremely weak claims of assistance from future people. (shrink)

According to an intriguing though somewhat enigmatic line of thought first proposed by Jonathan Bennett, if humanity went extinct any time soon this would be unfortunate because important business would be left unfinished. This line of thought remains largely unexplored. I offer an interpretation of the idea that captures its intuitive appeal, is consistent with plausible constraints, and makes it non-redundant to other views in the literature. The resulting view contrasts with a welfare-promotion perspective, according to which extinction would be (...) bad because it would come at a massive opportunity cost in terms of feasibly achievable welfare over the lifetime of our species. (shrink)

The possibility of seeding other planets with life poses a tricky dilemma. On the one hand, directed panspermia might be extremely good, while, on the other, it might be extremely bad depending on what factors are taken into consideration. Therefore, we need to understand better what is ethically at stake with planetary seeding. I map out possible conditions under which humanity should spread life to other solar systems. I identify two key variables that affect the desirability of propagating life throughout (...) the galaxy. The first is axiological and depends on which value theory in environmental ethics is correct. The second is empirical and depends on whether life is common or not in our region of the universe. I also consider two ethical objections to an interplanetary life-seeding mission: the risk of interfering with indigenous life and the risk of increasing suffering in our galaxy. (shrink)

Philosophy and Phenomenological Research, EarlyView.

Ethicists, policy-makers, and the general public have questioned whether artificial entities such as robots warrant rights or other forms of moral consideration. There is little synthesis of the research on this topic so far. We identify 294 relevant research or discussion items in our literature review of this topic. There is widespread agreement among scholars that some artificial entities could warrant moral consideration in the future, if not also the present. The reasoning varies, such as concern for the effects on (...) artificial entities and concern for the effects on human society. Beyond the conventional consequentialist, deontological, and virtue ethicist ethical frameworks, some scholars encourage “information ethics” and “social-relational” approaches, though there are opportunities for more in-depth ethical research on the nuances of moral consideration of artificial entities. There is limited relevant empirical data collection, primarily in a few psychological studies on current moral and social attitudes of humans towards robots and other artificial entities. This suggests an important gap for psychological, sociological, economic, and organizational research on how artificial entities will be integrated into society and the factors that will determine how the interests of artificial entities are considered. (shrink)

Artificial Intelligence (AI) must be directed at humane ends. The development of AI has produced great uncertainties of ensuring AI alignment with human values (AI value alignment) through AI operations from design to use. For the purposes of addressing this problem, we adopt the phenomenological theories of material values and technological mediation to be that beginning step. In this paper, we first discuss the AI value alignment from the relevant AI studies. Second, we briefly present what are material values and (...) technological mediation and reflect on the AI value alignment through the lenses of these theories. We conclude that a set of finite human values can be defined and adapted to the stable life tasks that AI systems will be called upon to accomplish. The AI value alignment can also be fostered between designers and users through technological mediation. Upon that foundation, we propose a set of common principles to understand the AI value alignment through phenomenological theories. This paper contributes the unique knowledge of phenomenological theories to the discourse on AI alignment with human values. (shrink)

In this entry, we discuss both the definition of effective altruism and objections to effective altruism, so defined.

A number of philosophers have claimed that we should take not just empirical uncertainty but also fundamental moral uncertainty into account in our decision-making, and that, despite widespread moral disagreement, doing so would allow us to draw robust lessons for some issues in practical ethics. In this article, I argue that, so far, the implications for practical ethics have been drawn too simplistically. First, the implications of moral uncertainty for normative ethics are far more wide-ranging than has been noted so (...) far. Second, one can't straightforwardly argue from moral uncertainty to particular conclusions in practical ethics, both because of ‘interaction’ effects between moral issues, and because of the variety of different possible intertheoretic comparisons that one can reasonably endorse. (shrink)

Ian Stoner has recently argued that we ought not to colonize Mars because doing so would flout our pro tanto obligation not to violate the principle of scientific conservation, and there is no countervailing considerations that render our violation of the principle permissible. While I remain agnostic on, my primary goal in this article is to challenge : there are countervailing considerations that render our violation of the principle permissible. As such, Stoner has failed to establish that we ought not (...) to colonize Mars. I close with some thoughts on what it would take to show that we do have an obligation to colonize Mars and related issues concerning the relationship between the way we discount our preferences over time and projects with long time horizons, like space colonization. (shrink)

Recent discussion of the alleged adverse consequences of space colonization by Phil Torres in this journal is critically assessed. While the concern for suffering risks should be part of any strategic discussion of the cosmic future of humanity, the Hobbesian picture painted by Torres is largely flawed and unpersuasive. Instead, there is a very real risk that the skeptical arguments will be taken too seriously and future human flourishing in space delayed or prevented.

A classification of the global catastrophic risks of AI is presented, along with a comprehensive list of previously identified risks. This classification allows the identification of several new risks. We show that at each level of AI’s intelligence power, separate types of possible catastrophes dominate. Our classification demonstrates that the field of AI risks is diverse, and includes many scenarios beyond the commonly discussed cases of a paperclip maximizer or robot-caused unemployment. Global catastrophic failure could happen at various levels of (...) AI development, namely, before it starts self-improvement, during its takeoff, when it uses various instruments to escape its initial confinement, or after it successfully takes over the world and starts to implement its goal system, which could be plainly unaligned, or feature-flawed friendliness. AI could also halt at later stages of its development either due to technical glitches or ontological problems. Overall, we identified around several dozen scenarios of AI-driven global catastrophe. The extent of this list illustrates that there is no one simple solution to the problem of AI safety, and that AI safety theory is complex and must be customized for each AI development level. (shrink)

Discussions about the possible consequences of creating superintelligence have included the possibility of existential risk, often understood mainly as the risk of human extinction. We argue that suffering risks (s-risks) , where an adverse outcome would bring about severe suffering on an astronomical scale, are risks of a comparable severity and probability as risks of extinction. Preventing them is the common interest of many different value systems. Furthermore, we argue that in the same way as superintelligent AI both contributes to (...) existential risk but can also help prevent it, superintelligent AI can both be a suffering risk or help avoid it. Some types of work aimed at making superintelligent AI safe will also help prevent suffering risks, and there may also be a class of safeguards for AI that helps specifically against s-risks. (shrink)

:There are manifold intriguing issues located within largely unexplored borderlands of bioethics, future studies, and astrobiology. Human enhancement has for quite some time been among the foci of bioethical debates, but the same cannot be said about its global, transgenerational, and even cosmological consequences. In recent years, discussions of posthuman and, in general terms, postbiological civilization have slowly gained a measure of academic respect, in parallel with the renewed interest in the entire field of future studies and the great strides (...) made in understanding of the origin and evolution of life and intelligence in their widest, cosmic context. These developments promise much deeper synergic answers to questions regarding the long-term future of enhancement: how far can it go? Is human enhancement a further step toward building a true postbiological civilization? Should we actively participate and help shape this process? Is the future of humanity “typical” in the same Copernican sense as our location in space and time is typical in the galaxy, and if so, can we derive important insights about the evolutionary pathways of postbiological evolution from astrobiological and Search for ExtraTerrestrial Intelligence studies? These and similar questions could be understood as parts of a possible unifying research program attempting to connect cultural and moral evolution with what we know and understand about their cosmological and biological counterparts. (shrink)

Because of accelerating technological progress, humankind may be rapidly approaching a critical phase in its career. In addition to well-known threats such as nuclear holocaust, the propects of radically transforming technologies like nanotech systems and machine intelligence present us with unprecedented opportunities and risks. Our future, and whether we will have a future at all, may well be determined by how we deal with these challenges. In the case of radically transforming technologies, a better understanding of the transition dynamics from (...) a human to a "posthuman" society is needed. Of particular importance is to know where the pitfalls are: the ways in which things could go terminally wrong. While we have had long exposure to various personal, local, and endurable global hazards, this paper analyzes a recently emerging category: that of existential risks. These are threats that could case our extinction or destroy the potential of Earth - originating intelligent life. Some of these threats are relatively well known while others, including some of the gravest, have gone almost unrecognized. Existential risks have a cluster of features that make ordinary risk management ineffective. A final section of this paper discusses several ethical and policy implications. A clearer understanding of the threat picture will enable us to formulate better strategies. (shrink)

Big historians are attempting to construct a general holistic narrative of human origins enabling an approach to studying the emergence of complexity, the relation between evolutionary processes, and the modern context of human experience and actions. In this paper I attempt to explore the past and future of cosmic evolution within a big historical foundation characterized by physical, biological, and cultural eras of change. From this analysis I offer a model of the human future that includes an addition and/or reinterpretation (...) of technological singularity theory with a new theory of biocultural evolution focused on the potential birth of technological life: the theory of atechnogenesis. Furthermore, I explore the potential deep futures of technological life and extrapolate towards two hypothetical versions of an ‘Omega Civilization’: expansion and compression. (shrink)

A small but growing number of studies have aimed to understand, assess and reduce existential risks, or risks that threaten the continued existence of mankind. However, most attention has been focused on known and tangible risks. This paper proposes a heuristic for reducing the risk of black swan extinction events. These events are, as the name suggests, stochastic and unforeseen when they happen. Decision theory based on a fixed model of possible outcomes cannot properly deal with this kind of event. (...) Neither can probabilistic risk analysis. This paper will argue that the approach that is referred to as engineering safety could be applied to reducing the risk from black swan extinction events. It will also propose a conceptual sketch of how such a strategy may be implemented: isolated, self-sufficient, and continuously manned underground refuges. Some characteristics of such refuges are also described, in particular the psychosocial aspects. Furthermore, it is argued that this implementation of the engineering safety strategy safety barriers would be effective and plausible and could reduce the risk of an extinction event in a wide range of possible scenarios. Considering the staggering opportunity cost of an existential catastrophe, such strategies ought to be explored more vigorously. (shrink)

As we learn more about the human brain, novel biotechnological means to modulate human behaviour and emotional dispositions become possible. These technologies could be used to enhance our morality. Moral bioenhancement, an instance of human enhancement, alters a person’s dispositions, emotions or behaviour in order to make that person more moral. I will argue that moral bioenhancement could be carried out in three different ways. The first strategy, well known from science fiction, is behavioural enhancement. The second strategy, favoured by (...) prominent defenders of moral bioenhancement, is emotional enhancement. The third strategy is the enhancement of moral dispositions, such as empathy and inequity aversion. I will argue that we ought to implement a combination of the second and third strategies. Furthermore, I will argue that the usual arguments against other instances of human enhancement do not apply to moral bioenhancement, or apply only to the first strategy, behavioural enhancement. (shrink)

Currently, some philosophers and technicians propose to change the fundamental constitution of Homo sapiens, as by significantly altering the genome, implanting microchips in the brain, and pursuing related techniques. Among these proposals are aspirations to guide humanity’s evolution into new species. Some philosophers have countered that such species alteration is unethical and have proposed international policies to protect species integrity; yet, it remains unclear on what basis such right to species integrity would rest. An answer may come from an unexpected (...) angle of rights issues: Some cultures have indicated that they want no part of our technological culture, preferring to retain their practices. Yet, rights documents do not explicitly establish that any individual has a right to species integrity. Careful interpretation of rights documents nonetheless reveals that such a right to species integrity is implicit. Interpreting these so as to reveal this needed right is also necessary to retain the foundations of rights documents and institutions. Further, acknowledging a right of species integrity could mean, because of practicalities, a limit to the freedom of proponents to implement proposals to manipulate the species. (shrink)

Recent discussions of theorigins of the thermodynamical temporal asymmetry (thearrow of time) by Huw Price and others arecritically assessed. This serves as amotivation for consideration of relationshipbetween thermodynamical and cosmologicalcauses. Although the project of clarificationof the thermodynamical explanandum is certainlywelcome, Price excludes another interestingoption, at least as viable as the sort ofAcausal-Particular approach he favors, andarguably more in the spirit of Boltzmannhimself. Thus, the competition of explanatoryprojects includes three horses, not two. Inaddition, it is the Acausal-Particular approachthat could benefit enormously (...) from dissociationfrom fanciful ideas of low-entropy futureboundary conditions entertained by Price. Novelrevolutionary developments in observationalcosmology, as well as in the nascentastrophysical discipline of physicaleschatology, have obliterated such hypotheses.Also, the Acausal-Anthropic approach wepropose, offers another clear instance ofdisteleological nature of the anthropicprinciple. (shrink)

In this paper, we argue that our moral concern for future well-being should reduce over time due to important practical considerations about how humans interact with spacetime. After surveying several of these considerations (around equality, special duties, existential contingency, and overlapping moral concern) we develop a set of core principles that can both explain their moral significance and highlight why this is inherently bound up with our relationship with spacetime. These relate to the equitable distribution of (1) moral concern in (...) a universe that we can only causally affect in one temporal direction; (2) access to the benefits from using spacetime as a resource; and (3) the burdens of care given (1) and (2). We conclude by considering the practical implications of our argument and find that, while it is often assumed that a preference for present over future well-being weakens the case for existential risk mitigation, this likely is not the case. (shrink)

Given the deep disagreement surrounding population axiology, one should remain uncertain about which theory is best. However, this uncertainty need not leave one neutral about which acts are better or worse. We show that, as the number of lives at stake grows, the Expected Moral Value approach to axiological uncertainty systematically pushes one toward choosing the option preferred by the Total View and critical-level views, even if one’s credence in those theories is low.

The advent of artificial intelligence (AI) systems has provoked a lot of discussions in both epistemological, bioethical and risk-analytic terms, much of it rather paranoid in nature. Unless one takes an extreme anthropocentric and chronocentric stance, this process can be safely regarded as part and parcel of the sciences of the origin. In this contribution, I would like to suggest that at least four different classes of arguments could be brought forth against the proposition that AI - either human-level or (...) superintelligent - represents in itself a credible existential threat to humanity in either nearby or distant future. Part of the same argumentation is applicable to the general notion of postbiological evolution, which has caused even more unnecessary confusion and uproar in both laymen and philosophical circles. While the due attention should be given to the risks and dangers of the transitional period, there are many reasons why we should openly support and cheer for humanity's transition into the postbiological epoch. (shrink)

The biosphere represents the global sum of all ecosystems. According to a prominent view in environmental ethics, ecocentrism, these ecosystems matter for their own sake, and not only because they contribute to human ends. As such, some ecocentrists are critical of the modern industrial civilization, and a few even argue that an irreversible collapse of the modern industrial civilization would be a good thing. However, taking a longer view and considering the eventual destruction of the biosphere by astronomical processes, we (...) argue that humans, a species with considerable technological know-how and industrial capacity could intervene to extend the lifespan of Earth’s biosphere, perhaps by several billion years. We argue that human civilization, despite its flaws and harmful impacts on many ecosystems, is the biosphere’s best hope of avoiding premature destruction. We argue that proponents of ecocentrism, even those who wholly disregard anthropocentric values, have a strong moral reason preserve the modern industrial civilization, for as long as needed to ensure biosphere survival. (shrink)

Artificial intelligence experts are currently divided into “presentist” and “futurist” factions that call for attention to near-term and long-term AI, respectively. This paper argues that the presentist–futurist dispute is not the best focus of attention. Instead, the paper proposes a reconciliation between the two factions based on a mutual interest in AI. The paper further proposes realignment to two new factions: an “intellectualist” faction that seeks to develop AI for intellectual reasons and a “societalist faction” that seeks to develop AI (...) for the benefit of society. The paper argues in favor of societalism and offers three means of concurrently addressing societal impacts from near-term and long-term AI: advancing societalist social norms, thereby increasing the portion of AI researchers who seek to benefit society; technical research on how to make any AI more beneficial to society; and policy to improve the societal benefits of all AI. In practice, it will often be advantageous to emphasize near-term AI due to the greater interest in near-term AI among AI and policy communities alike. However, presentist and futurist societalists alike can benefit from each others’ advocacy for attention to the societal impacts of AI. The reconciliation between the presentist and futurist factions can improve both near-term and long-term societal impacts of AI. (shrink)

I argue that many of the priority rankings that have been proposed by effective altruists seem to be in tension with apparently reasonable assumptions about the rational pursuit of our aims in the face of uncertainty. The particular issue on which I focus arises from recognition of the overwhelming importance and inscrutability of the indirect effects of our actions, conjoined with the plausibility of a permissive decision principle governing cases of deep uncertainty, known as the maximality rule. I conclude that (...) we lack a compelling decision theory that is consistent with a longtermist perspective and does not downplay the depth of our uncertainty, while also supporting orthodox effective altruist conclusions about cause prioritization. (shrink)

Is the overall value of a world just the sum of values contributed by each value-bearing entity in that world? Additively separable axiologies (like total utilitarianism, prioritarianism, and critical level views) say 'yes', but non-additive axiologies (like average utilitarianism, rank-discounted utilitarianism, and variable value views) say 'no'. This distinction appears to be practically important: among other things, additive axiologies generally assign great importance to large changes in population size, and therefore tend to strongly prioritize the long-term survival of humanity over (...) the interests of the present generation. Non-additive axiologies, on the other hand, need not assign great importance to large changes in population size. We show, however, that when there is a large enough `background population' unaffected by our choices, a wide range of non-additive axiologies converge in their implications with additive axiologies—for instance, average utilitarianism converges with critical-level utilitarianism and various egalitarian theories converge with prioritarianism. We further argue that real-world background populations may be large enough to make these limit results practically significant. This means that arguments from the scale of potential future populations for the astronomical importance of avoiding existential catastrophe, and other arguments in practical ethics that seem to presuppose additive separability, may succeed in practice whether or not we accept additive separability as a basic axiological principle. (shrink)

When our choice affects some other person and the outcome is unknown, it has been argued that we should defer to their risk attitude, if known, or else default to use of a risk‐avoidant risk function. This, in turn, has been claimed to require the use of a risk‐avoidant risk function when making decisions that primarily affect future people, and to decrease the desirability of efforts to prevent human extinction, owing to the significant risks associated with continued human survival. I (...) raise objections to the claim that respect for others' risk attitudes requires risk‐avoidance when choosing for future generations. In particular, I argue that there is no known principle of interpersonal aggregation that yields acceptable results in variable population contexts and is consistent with a plausible ideal of respect for others' risk attitudes in fixed population cases. (shrink)

The possibility of extraterrestrial intelligence (ETI) exists despite no observed evidence, and the risks and benefits of actively searching for ETI (Active SETI) have been debated. Active SETI has been criticized for potentially exposing humanity to existential risk, and a recent game-theoretical model highlights the Hobbesian trap that could occur following contact if mutual distrust leads to mutual destruction. We argue that observing a nearby ETI would suggest the existence of many unobserved ETI. This would expand the game and implies (...) that there may be a mechanism that prevents ETI from attacking us. As a result, the Hobbesian trap would be avoided if both parties assumed the other is not likely to attack. (shrink)

In the Anthropocene, human beings are capable of bringing about globally catastrophic outcomes that could damage conditions for present and future human life on Earth in unprecedented ways. This paper argues that the scale and severity of these dangers justifies a new international criminal offence of ‘postericide’ that would protect present and future people against wrongfully created dangers of near extinction. Postericide is committed by intentional or reckless systematic conduct that is fit to bring about near human extinction. The paper (...) argues that a proper understanding of the moral imperatives embodied in international criminal law shows that it ought to be expanded to incorporate a new law of postericide. (shrink)

The pre-modern world was one of gross inequalities and abject poverty. Yet, over the last two hundred years, social investments have unlocked the productive capacity and imagination of billions (Li...

Brad Hooker argues that the cost of inculcating in everyone the prevent disaster rule places a limit on its demandingness. My aim in this article is show that this is not true of existential risk reduction. However, this does not spell trouble for the reason that removing persistent global harms significantly improves our long-run chances of survival. We can expect things to get better, not worse, for our population.

Artificial general intelligence is a greatly anticipated technology with non-trivial existential risks, defined as machine intelligence with competence as great/greater than humans. To date, social scientists have dedicated little effort to the ethics of AGI or AGI researchers. This paper employs inductive discourse analysis of the academic literature of two intellectual groups writing on the ethics of AGI—applied and/or ‘basic’ scientific disciplines henceforth referred to as technicians (e.g., computer science, electrical engineering, physics), and philosophy-adjacent disciplines henceforth referred to as PADs (...) (e.g., philosophy, theology, anthropology). These groups agree that AGI ethics is fundamentally about mitigating existential risk. They highlight our moral obligation to future generations, demonstrate the ethical importance of better understanding consciousness, and endorse a hybrid of deontological/utilitarian normative ethics. Technicians favor technocratic AGI governance, embrace the project of ‘solving’ moral realism, and are more deontologically inclined than PADs. PADs support a democratic approach to AGI governance, are more skeptical of deontology, consider current AGI predictions as fundamentally imprecise, and are wary of using AGI for moral fact-finding. (shrink)

We outline an argument favoring voluntary moral bioenhancement as a response to existential risks humanity exposes itself to. We consider this type of enhancement a solution to the antithesis between the extinction of humanity and the imperative of humanity to survive at any cost. By opting for voluntary moral bioenhancement; we refrain from advocating illiberal or even totalitarian strategies that would allegedly help humanity preserve itself. We argue that such strategies; by encroaching upon the freedom of individuals; already inflict a (...) degree of existential harm on human beings. We also give some pointers as to the desirable direction for morally enhanced post-personhood. (shrink)