Purpose This paper aims to formalize long-term trajectories of human civilization as a scientific and ethical field of study. The long-term trajectory of human civilization can be defined as the path that human civilization takes during the entire future time period in which human civilization could continue to exist. -/- Design/methodology/approach This paper focuses on four types of trajectories: status quo trajectories, in which human civilization persists in a state broadly similar to its current state into the distant future; catastrophe (...) trajectories, in which one or more events cause significant harm to human civilization; technological transformation trajectories, in which radical technological breakthroughs put human civilization on a fundamentally different course; and astronomical trajectories, in which human civilization expands beyond its home planet and into the accessible portions of the cosmos. -/- Findings Status quo trajectories appear unlikely to persist into the distant future, especially in light of long-term astronomical processes. Several catastrophe, technological transformation and astronomical trajectories appear possible. -/- Originality/value Some current actions may be able to affect the long-term trajectory. Whether these actions should be pursued depends on a mix of empirical and ethical factors. For some ethical frameworks, these actions may be especially important to pursue. (shrink)

Abstract: In the last decade, an urban legend about “glitches in the matrix” has become popular. As it is typical for urban legends, there is no evidence for most such stories, and the phenomenon could be explained as resulting from hoaxes, creepypasta, coincidence, and different forms of cognitive bias. In addition, the folk understanding of probability does not bear much resemblance to actual probability distributions, resulting in the illusion of improbable events, like the “birthday paradox”. Moreover, many such stories, even (...) if they were true, could not be considered evidence of glitches in a linear-time computer simulation, as the reported “glitches” often assume non-linearity of time and space—like premonitions or changes to the past. Different types of simulations assume different types of glitches; for example, dreams are often very glitchy. Here, we explore the theoretical conditions necessary for such glitches to occur and then create a typology of so-called “GITM” reports. One interesting hypothetical subtype is “viruses in the matrix”, that is, self-replicating units which consume computational resources in a manner similar to transposons in the genome, biological and computer viruses, and memes. (shrink)

Explainability and comprehensibility of AI are important requirements for intelligent systems deployed in real-world domains. Users want and frequently need to understand how decisions impacting them are made. Similarly it is important to understand how an intelligent system functions for safety and security reasons. In this paper, we describe two complementary impossibility results (Unexplainability and Incomprehensibility), essentially showing that advanced AIs would not be able to accurately explain some of their decisions and for the decisions they could explain people would (...) not understand some of those explanations. (shrink)

If an AI makes a significant contribution to a research paper, should it be listed as a co-author? The current guidelines in the field have been created to reduce duplication of credit between two different authors in scientific articles. A new computer program could be identified and credited for its impact in an AI research paper that discusses an early artificial intelligence system which is currently under development at Lawrence Berkeley National. One way to imagine the future of artificial intelligence (...) is that it will be much less expensive to develop new technologies than to create new ways of thinking. Now we have done this technology, and now we go and ask why in the end it is the artificial intelligence that takes over? Well, it is not that artificial intelligence is bad, but it is not as effective as human minds or as intelligent as machine minds. Even in the past, when computers were more intelligent than humans, not all the AI programs have been so intelligent as to be intelligent enough to be called intelligent. (shrink)

Abstract. Boltzmann brains (BBs) are minds which randomly appear as a result of thermodynamic or quantum fluctuations. In this article, the question of if we are BBs, and the observational consequences if so, is explored. To address this problem, a typology of BBs is created, and the evidence is compared with the Simulation Argument. Based on this comparison, we conclude that while the existence of a “normal” BB is either unlikely or irrelevant, BBs with some ordering may have observable consequences. (...) There are two types of such ordered BBs: Boltzmannian typewriters (including Boltzmannian simulations), and chains of observer moments. Notably, the existence or non-existence of BBs may have practical applications for measuring the size of the universe, achieving immortality, or even manipulating the observed probability of events. -/- Disclaimer and trigger warning: some people have emotional breakdowns when thinking about topics described in the article, especially the “flux universe”. However, everything eventually adds up to normality. (shrink)

One of the possible solutions of the Fermi paradox is that all civilizations go extinct because they hit some Late Great Filter. Such a universal Late Great Filter must be an unpredictable event that all civilizations unexpectedly encounter, even if they try to escape extinction. This is similar to the “Death in Damascus” paradox from decision theory. However, this unpredictable Late Great Filter could be escaped by choosing a random strategy for humanity’s future development. However, if all civilizations act randomly, (...) this could be the actual cause of universal extinction, as no one will choose the most optimal strategy. Using a meta-random strategy, where the decision to start random actions is also made randomly, solves this. (shrink)

Many researchers have argued that humanity will create artificial general intelligence (AGI) within the next twenty to one hundred years. It has been suggested that AGI may inflict serious damage to human well-being on a global scale ('catastrophic risk'). After summarizing the arguments for why AGI may pose such a risk, we review the fieldʼs proposed responses to AGI risk. We consider societal proposals, proposals for external constraints on AGI behaviors and proposals for creating AGIs that are safe due to (...) their internal design. (shrink)

The goal of the article is to explore what is the most probable type of simulation in which humanity lives (if any) and how this affects simulation termination risks. We firstly explore the question of what kind of simulation in which humanity is most likely located based on pure theoretical reasoning. We suggest a new patch to the classical simulation argument, showing that we are likely simulated not by our own descendants, but by alien civilizations. Based on this, we provide (...) classification of different possible simulations and we find that simpler, less expensive and one-person-centered simulations, resurrectional simulations, or simulations of the first artificial general intelligence’s (AGI’s) origin (singularity simulations) should dominate. Also, simulations which simulate the 21st century and global catastrophic risks are probable. We then explore whether the simulation could collapse or be terminated. Most simulations must be terminated after they model the singularity or after they model a global catastrophe before the singularity. Undeniably observed glitches, but not philosophical speculations could result in simulation termination. The simulation could collapse if it is overwhelmed by glitches. The Doomsday Argument in simulations implies termination soon. We conclude that all types of the most probable simulations except resurrectional simulations are prone to termination risks in a relatively short time frame of hundreds of years or less from now. (shrink)

The young field of AI Safety is still in the process of identifying its challenges and limitations. In this paper, we formally describe one such impossibility result, namely Unpredictability of AI. We prove that it is impossible to precisely and consistently predict what specific actions a smarter-than-human intelligent system will take to achieve its objectives, even if we know terminal goals of the system. In conclusion, impact of Unpredictability on AI Safety is discussed.