A morally acceptable course of AI development should avoid two dangers: creating unaligned AI systems that pose a threat to humanity and mistreating AI systems that merit moral consideration in their own right. This paper argues these two dangers interact and that if we create AI systems that merit moral consideration, simultaneously avoiding both of these dangers would be extremely challenging. While our argument is straightforward and supported by a wide range of pretheoretical moral judgments, it has far-reaching moral implications (...) for AI development. Although the most obvious way to avoid the tension between alignment and ethical treatment would be to avoid creating AI systems that merit moral consideration, this option may be unrealistic and is perhaps fleeting. So, we conclude by offering some suggestions for other ways of mitigating mistreatment risks associated with alignment. (shrink)

As ever more advanced digital systems are created, it becomes increasingly likely that some of these systems will be digital minds, i.e. digital subjects of experience. With digital minds comes the risk of digital suffering. The problem of digital suffering is that of mitigating this risk. We argue that the problem of digital suffering is a high stakes moral problem and that formidable epistemic obstacles stand in the way of solving it. We then propose a strategy for solving it: Access (...) Monitor Prevent (AMP). AMP uses a ‘dancing qualia’ argument to link the functional states of certain digital systems to their experiences—this yields epistemic access to digital minds. With that access, we can prevent digital suffering by only creating advanced digital systems that we have such access to, monitoring their functional profiles, and preventing them from entering states with functional markers of suffering. After introducing and motivating AMP, we confront limitations it faces and identify some options for overcoming them. We argue that AMP fits especially well with—and so provides a moral reason to prioritize—one approach to creating such systems: whole brain emulation. We also contend that taking other paths to digital minds would be morally risky. (shrink)

How all philosophical explanations of human consciousness and the fundamental structure of the cosmos are bizarre—and why that’s a good thing Do we live inside a simulated reality or a pocket universe embedded in a larger structure about which we know virtually nothing? Is consciousness a purely physical matter, or might it require something extra, something nonphysical? According to the philosopher Eric Schwitzgebel, it’s hard to say. In The Weirdness of the World, Schwitzgebel argues that the answers to these fundamental (...) questions lie beyond our powers of comprehension. We can be certain only that the truth—whatever it is—is weird. Philosophy, he proposes, can aim to open—to reveal possibilities we had not previously appreciated—or to close, to narrow down to the one correct theory of the phenomenon in question. Schwitzgebel argues for a philosophy that opens. According to Schwitzgebel’s “Universal Bizarreness” thesis, every possible theory of the relation of mind and cosmos defies common sense. According to his complementary “Universal Dubiety” thesis, no general theory of the relationship between mind and cosmos compels rational belief. Might the United States be a conscious organism—a conscious group mind with approximately the intelligence of a rabbit? Might virtually every action we perform cause virtually every possible type of future event, echoing down through the infinite future of an infinite universe? What, if anything, is it like to be a garden snail? Schwitzgebel makes a persuasive case for the thrill of considering the most bizarre philosophical possibilities. (shrink)

Purpose This paper aims to formalize long-term trajectories of human civilization as a scientific and ethical field of study. The long-term trajectory of human civilization can be defined as the path that human civilization takes during the entire future time period in which human civilization could continue to exist. -/- Design/methodology/approach This paper focuses on four types of trajectories: status quo trajectories, in which human civilization persists in a state broadly similar to its current state into the distant future; catastrophe (...) trajectories, in which one or more events cause significant harm to human civilization; technological transformation trajectories, in which radical technological breakthroughs put human civilization on a fundamentally different course; and astronomical trajectories, in which human civilization expands beyond its home planet and into the accessible portions of the cosmos. -/- Findings Status quo trajectories appear unlikely to persist into the distant future, especially in light of long-term astronomical processes. Several catastrophe, technological transformation and astronomical trajectories appear possible. -/- Originality/value Some current actions may be able to affect the long-term trajectory. Whether these actions should be pursued depends on a mix of empirical and ethical factors. For some ethical frameworks, these actions may be especially important to pursue. (shrink)

According to Longtermism, our acts’ expected influence on the expected value of the world is mainly determined by their effects in the far future. There is, given total utilitarianism, a straightforward argument for Longtermism due to the enormous number of people that might exist in the future, but this argument does not work on person-affecting views. In this paper, we will argue that these views might also lead to Longtermism if Prudential Longtermism is true. Prudential Longtermism holds for a person (...) if and only if our acts’ overall influence on that person’s expected well-being is mainly determined by the acts’ effects in the far future. We argue that (due to a small chance of anti-ageing and uploading) there could be an enormous amount of prudential value for some contemporary person in the far future and that value may be so large that it dominates their overall expectation of lifetime well-being. (shrink)

This chapter makes the case for strong longtermism: the claim that, in many situations, impact on the long-run future is the most important feature of our actions. Our case begins with the observation that an astronomical number of people could exist in the aeons to come. Even on conservative estimates, the expected future population is enormous. We then add a moral claim: all the consequences of our actions matter. In particular, the moral importance of what happens does not depend on (...) when it happens. That pushes us toward strong longtermism. We then address a few potential concerns, the first of which is that it is impossible to have any sufficiently predictable influence on the course of the long-run future. We argue that this is not true. Some actions can reasonably be expected to improve humanity’s long-term prospects. These include reducing the risk of human extinction, preventing climate change, guiding the development of artificial intelligence, and investing funds for later use. We end by arguing that these actions are more than just extremely effective ways to do good. Since the benefits of longtermist efforts are large and the personal costs are comparatively small, we are morally required to take up these efforts. (shrink)

Artificial intelligence experts are currently divided into “presentist” and “futurist” factions that call for attention to near-term and long-term AI, respectively. This paper argues that the presentist–futurist dispute is not the best focus of attention. Instead, the paper proposes a reconciliation between the two factions based on a mutual interest in AI. The paper further proposes realignment to two new factions: an “intellectualist” faction that seeks to develop AI for intellectual reasons and a “societalist faction” that seeks to develop AI (...) for the benefit of society. The paper argues in favor of societalism and offers three means of concurrently addressing societal impacts from near-term and long-term AI: advancing societalist social norms, thereby increasing the portion of AI researchers who seek to benefit society; technical research on how to make any AI more beneficial to society; and policy to improve the societal benefits of all AI. In practice, it will often be advantageous to emphasize near-term AI due to the greater interest in near-term AI among AI and policy communities alike. However, presentist and futurist societalists alike can benefit from each others’ advocacy for attention to the societal impacts of AI. The reconciliation between the presentist and futurist factions can improve both near-term and long-term societal impacts of AI. (shrink)

This article explores the proposition that the brain, normally seen as an organ of the human body, should be understood as a biologically based form of artificial intelligence, in the course of which the case is made for a new kind of ‘brain exceptionalism’. After noting that such a view was generally assumed by the founders of AI in the 1950s, the argument proceeds by drawing on the distinction between science—in this case neuroscience—adopting a ‘telescopic’ or a ‘microscopic’ orientation to (...) reality, depending on how it regards its characteristic investigative technologies. The paper concludes by recommending a ‘microscopic’ yet non-reductionist research agenda for neuroscience, in which the brain is seen as an underutilised organ whose energy efficiency is likely to outstrip that of the most powerful supercomputers for the foreseeable future. (shrink)

Discussions about the possible consequences of creating superintelligence have included the possibility of existential risk, often understood mainly as the risk of human extinction. We argue that suffering risks (s-risks) , where an adverse outcome would bring about severe suffering on an astronomical scale, are risks of a comparable severity and probability as risks of extinction. Preventing them is the common interest of many different value systems. Furthermore, we argue that in the same way as superintelligent AI both contributes to (...) existential risk but can also help prevent it, superintelligent AI can both be a suffering risk or help avoid it. Some types of work aimed at making superintelligent AI safe will also help prevent suffering risks, and there may also be a class of safeguards for AI that helps specifically against s-risks. (shrink)

How much value can our decisions create? We argue that unless our current understanding of physics is wrong in fairly fundamental ways, there exists an upper limit of value relevant to our decisions. First, due to the speed of light and the definition and conception of economic growth, the limit to economic growth is a restrictive one. Additionally, a related far larger but still finite limit exists for value in a much broader sense due to the physics of information and (...) the ability of physical beings to place value on outcomes. We discuss how this argument can handle lexicographic preferences, probabilities, and the implications for infinite ethics and ethical uncertainty. (shrink)

Over the coming century, the accelerating advance of bioenhancement technologies, robotics, and artificial intelligence (AI) may significantly broaden the qualitative range of sentient and intelligent beings. This article proposes a taxonomy of such beings, ranging from modified animals to bioenhanced humans to advanced forms of robots and AI. It divides these diverse beings into three moral and legal categories—animals, persons, and presumed persons—describing the moral attributes and legal rights of each category. In so doing, the article sets forth a framework (...) for extending the concept of personhood well beyond its current boundaries, assigning moral standing to a variety of biological and nonbiological beings. The author concludes that six of the eight subgroups of such beings deserve to be treated as persons or as if they were persons, with full consideration for their presumed interests, rights, obligations, and capabilities for ethically significant agency and patiency. (shrink)

Sideloading is the creation of a digital model of a person during their life via iterative improvements of this model based on the person's feedback. The progress of LLMs with large prompts allows the creation of very large, book-size prompts which describe a personality. We will call mind-models created via sideloading "sideloads"; they often look like chatbots, but they are more than that as they have other output channels, like internal thought streams and descriptions of actions. -/- By arranging the (...) predictive power of the facts about a person, we can write down the most important facts first. This allows us to get a high level of fidelity in a person's model by writing down a finite number of facts and running it as an LLM prompt with instructions to create a person's chatbot. This instruction also includes an extensive general sideload prompt (we will call it a prompt-loader), which explains how the person's sideload should work, but it is universal and works as an operating system for different persons. -/- I created my personal sideload to test all this and it started working surprisingly quickly. We found that there are three main levels of facts about a person: core facts, long-term memory, and historical facts. Core facts should go to the main prompt of the LLM, long-term memory belongs to RAG, and historical facts – which a person often does not remember – should be only used to extract new relevant data. The most important thing in preparing data for sideloading is to get a full list of the core facts, and not to mix it with long-term memory and historical facts. The LLM-sideload project is inherently future-oriented. New state-of-the-art LLMs with large prompt capabilities emerge quarterly. This allows for the preparation of prompts that exceed current LLM capabilities, anticipating smooth execution in the following year. Below we present the key findings of this work in the form of an executive summary: Quality of sideloads is evaluated using three metrics: • Facts: Accuracy of the answers about the person's life • Vibe: How well the sideload captures the person's style and personality, as judged by the person themselves and their acquaintances • Brilliant insights: Ability to contribute unique, valuable ideas beyond the training data, while maintaining the author's style • Coarseness: Levels of details as percent of the total of recordable textual information from memory estimated to be 10000 pages. Similar to the jpeg level of compression. Estimated performance of Alexey Turchin's sideload: • Facts: 70% correct • Vibe: 20% accurate • Brilliant insights: near zero • Coarseness: 10% The project achieved these results in about one month of work. Vibe is harder to evaluate and regulate than facts. This approach focuses on informational personal identity, setting aside issues of observer identity and internal qualia. (shrink)

The possibility of social and technological collapse has been the focus of science fiction tropes for decades, but more recent focus has been on specific sources of existential and global catastrophic risk. Because these scenarios are simple to understand and envision, they receive more attention than risks due to complex interplay of failures, or risks that cannot be clearly specified. In this paper, we discuss the possibility that complexity of a certain type leads to fragility which can function as a (...) source of catastrophic or even existential risk. The paper first reviews a hypothesis by Bostrom about inevitable technological risks, named the vulnerable world hypothesis. This paper next hypothesizes that fragility may not only be a possible risk, but could be inevitable,and would therefore be a subclass or example of Bostrom’s vulnerable worlds. After introducing the titular fragile world hypothesis, the paper details the conditions under which it would be correct, and presents arguments for why the conditions may in fact may apply. Finally, the assumptions and potential mitigations of the new hypothesis are contrasted with those Bostrom suggests. (shrink)

There are two types of artificial general intelligence (AGI) safety solutions: global and local. Most previously suggested solutions are local: they explain how to align or “box” a specific AI (Artificial Intelligence), but do not explain how to prevent the creation of dangerous AI in other places. Global solutions are those that ensure any AI on Earth is not dangerous. The number of suggested global solutions is much smaller than the number of proposed local solutions. Global solutions can be divided (...) into four groups: 1. No AI: AGI technology is banned or its use is otherwise prevented; 2. One AI: the first superintelligent AI is used to prevent the creation of any others; 3. Net of AIs as AI police: a balance is created between many AIs, so they evolve as a net and can prevent any rogue AI from taking over the world; 4. Humans inside AI: humans are augmented or part of AI. We explore many ideas, both old and new, regarding global solutions for AI safety. They include changing the number of AI teams, different forms of “AI Nanny” (non-self-improving global control AI system able to prevent creation of dangerous AIs), selling AI safety solutions, and sending messages to future AI. Not every local solution scales to a global solution or does it ethically and safely. The choice of the best local solution should include understanding of the ways in which it will be scaled up. Human-AI teams or a superintelligent AI Service as suggested by Drexler may be examples of such ethically scalable local solutions, but the final choice depends on some unknown variables such as the speed of AI progress. (shrink)

Abstract. Death seems to be a permanent event, but there is no actual proof of its irreversibility. Here we list all known ways to resurrect the dead that do not contradict our current scientific understanding of the world. While no method is currently possible, many of those listed here may become feasible with future technological development, and it may even be possible to act now to increase their probability. The most well-known such approach to technological resurrection is cryonics. Another method (...) is indirect mind uploading, or digital immortality, namely the preservation of data about a person to allow for future reconstruction by powerful AI. More speculative ways to immortality include combinations of future superintelligence on a galactic scale, which could use simulation to resurrect all possible people, and new physical laws, which may include time-travel or obtaining information from the past. Acausal trade with parallel worlds could help combine random resurrection and reconstruction based on known data without loss of share of worlds where I exist (known as existence measure). Quantum immortality could help to increase the probability of success for cryonics and digital immortality. There many possible approaches to technological resurrection and thus if large-scale future technological development occurs, some form of resurrection is inevitable. (shrink)

The popular belief that our mind either depends on or (in stronger terms) is identical with brain functions and processes, along with the belief that advances in technology in virtual reality and computability will continue, has contributed to the contention that one-day (perhaps this century) it may be possible to transfer one’s mind (or a simulated copy) into another body (physical or virtual). This is called mind-uploading or whole brain emulation. This paper serves as an introduction to the area and (...) some of the major issues that arise from the notion of mind-uploading. The topic of mind/brain uploading is, at present, both a philosophical and scientific minefield. This paper ties together several thoughts on the topic in the popular, philosophical, and scientific literature, and brings up some related issues that might be worth exploring. A challenge is made that uploading a mind from a brain may have intrinsic limitations on what can be uploaded: whatever is uploaded may not be recognizably human, or will be a very limited version. (shrink)