A classification of the global catastrophic risks of AI is presented, along with a comprehensive list of previously identified risks. This classification allows the identification of several new risks. We show that at each level of AI’s intelligence power, separate types of possible catastrophes dominate. Our classification demonstrates that the field of AI risks is diverse, and includes many scenarios beyond the commonly discussed cases of a paperclip maximizer or robot-caused unemployment. Global catastrophic failure could happen (...) at various levels of AI development, namely, before it starts self-improvement, during its takeoff, when it uses various instruments to escape its initial confinement, or after it successfully takes over the world and starts to implement its goal system, which could be plainly unaligned, or feature-flawed friendliness. AI could also halt at later stages of its development either due to technical glitches or ontological problems. Overall, we identified around several dozen scenarios of AI-driven global catastrophe. The extent of this list illustrates that there is no one simple solution to the problem of AI safety, and that AI safety theory is complex and must be customized for each AI development level. (shrink)

Abstract: Global chemical contamination is an underexplored source of global catastrophic risks that is estimated to have low a priori probability. However, events such as pollinating insects’ population decline and lowering of the human male sperm count hint at some toxic exposure accumulation and thus could be a global catastrophic risk event if not prevented by future medical advances. We identified several potentially dangerous sources of the global chemical contamination, which may happen now (...) or could happen in the future: autocatalytic reactions, exposure to multiple subthreshold sources, and long-term unintended consequences, arising from both natural and bioengineered sources. We list several especially dangerous chemicals—dioxin, organiс compounds, and toxic heavy metals. We also discuss the features of such dangerous chemicals—molecules that can stay in the biosphere for a long time and affect it over time. We explore several social processes and scenarios where global chemical contamination becomes possible: large natural catastrophe like meteorite impact, supervolcano eruption, new ways of predicting properties of the chemicals via machine learning and their manufacturing via synthetic biology, uncontrolled “capitalistic” economic development with a corresponding large waste production, quick adoption of many chemicals with unknown long-term properties and unintended side-effects. These are all low probability, so work on other global catastrophic risks should be prioritized, but chemical risks could exacerbate other types of catastrophe contributing to social collapse. (shrink)

In this article, a classification of the global catastrophic risks connected with the possible existence (or non-existence) of extraterrestrial intelligence is presented. If there are no extra-terrestrial intelligences (ETIs) in our light cone, it either means that the Great Filter is behind us, and thus some kind of periodic sterilizing natural catastrophe, like a gamma-ray burst, should be given a higher probability estimate, or that the Great Filter is ahead of us, and thus a future global catastrophe (...) is high probability. If ETIs exist in our light cone, it is most likely they exist in the form of ET artificial general intelligence (ETAI). ETAI may be traveling at near-light speed as an explosive wave of intelligence in the form of some nano-mechanisms which could reach us at any moment; may send us potentially dangerous messages with AI-virus payloads; or be near or on Earth in some dormant form. This dormant form could be nanobots, which could act as berserkers, and be triggered when humans unwittingly cross some threshold. If ETIs are nearby and less advanced, they could find our messaging extra-terrestrial intelligence (METI) messages and send their “fleet” to or direct weapons toward Earth. Even extinct ETIs could have left dangerous remnants in the form of AI systems that our civilization may encounter. If our existence is a simulation, this could be run by ETIs. Even a false belief in the existence of ETIs could potentially fuel millennial sects. If humanity is the first intelligence to emerge, it may kill or prevent the existence of all future potential ETI, a different type of catastrophe. Several options to prevent catastrophic risks are connected with ETI: sending requests for help, using random strategies to escape the Fermi paradox, Great Filter prediction, or the hope that ETIs will find the remains of our extinct civilization and resurrect it. (shrink)

Abstract: This article examines risks associated with the program of passive search for alien signals (Search for Extraterrestrial Intelligence, or SETI) connected with the possibility of finding of alien transmission which includes description of AI system aimed on self-replication (SETI-attack). A scenario of potential vulnerability is proposed as well as the reasons why the proportion of dangerous to harmless signals may be high. The article identifies necessary conditions for the feasibility and effectiveness of the SETI-attack: ETI existence, possibility of AI, (...) small size of the Seed AI, small speed of physical interstellar travel and large distance between civilizations. Needed additions to the SETI protocol are considered: keep the signal existence, its content and source location in secret, don’t run alien programs, jam dangerous signals, wait until creation of humanity’s own AI. In addition, scenarios in which it may be reasonable to start an alien AI (if it has been downloaded) are explored. (shrink)

Abstract: After 2017 NY Times publication, the stigma of the scientific discussion of the problem of so-called UAP (Unidentified Aerial Phenomena) was lifted. Now the question arises: how UAP will affect the future of humanity, and especially, the probability of the global catastrophic risks? To answer this question, we assume that the Nimitz case in 2004 was real and we will suggest a classification of the possible explanations of the phenomena. The first level consists of mundane explanations: hardware (...) glitches, malfunction, hoaxes and lies. The second level involves explanations which are not changing completely our model of the world, but only updating it: new military tech, large psyop operation or new physical effect like ball lightning. The third level of explanations requires a complete overhaul of our world model and includes a large set of fantastic hypothesis: alien starships, interdimensional beings, glitches in the matrix, projections from the collective unconsciousness, Boltzmann brain’s experiences etc. The higher is the level of the hypothesis, the less probable it is, but the bigger is its consequences for possible global catastrophic risks. Thus, “integrating” over the field of all possible explanations, we find that UAP increases catastrophic risks. Technological progress increases our chances of direct confrontation with the phenomena. If the phenomenon has intelligence, the change of its behavior could be unexpected after we pass some unknown threshold. But we could lower the risks if we assume some reasonable policy of escaping confrontations. (shrink)

Existential risks threaten the future of humanity, but they are difficult to measure. However, to communicate, prioritize and mitigate such risks it is important to estimate their relative significance. Risk probabilities are typically used, but for existential risks they are problematic due to ambiguity, and because quantitative probabilities do not represent some aspects of these risks. Thus, a standardized and easily comprehensible instrument is called for, to communicate dangers from various global catastrophic and existential risks. In this (...) article, inspired by the Torino scale of asteroid danger, we suggest a color coded scale to communicate the magnitude of global catastrophic and existential risks. The scale is based on the probability intervals of risks in the next century if they are available. The risks’ estimations could be adjusted based on their severities and other factors. The scale covers not only existential risks, but smaller size global catastrophic risks. It consists of six color levels, which correspond to previously suggested levels of prevention activity. We estimate artificial intelligence risks as “red”, while “orange” risks include nanotechnology, synthetic biology, full scale nuclear war and a large global agricultural shortfall (caused by regional nuclear war, coincident extreme weather, etc.) The risks of natural pandemic, supervolcanic eruption and global warming are marked as “yellow” and the danger from asteroids is “green”. -/- Keywords: global catastrophic risks; existential risks; Torino scale; policy; risk probability . (shrink)

Many global catastrophic and existential risks (X-risks) threaten the existence of humankind. There are also many ideas for their prevention, but the meta-problem is that these ideas are not structured. This lack of structure means it is not easy to choose the right plan(s) or to implement them in the correct order. I suggest using a “Plan A, Plan B” model, which has shown its effectiveness in planning actions in unpredictable environments. In this approach, Plan B is a (...) backup option, implemented if Plan A fails. In the case of global risks, Plan A is intended to prevent a catastrophe and Plan B to survive it, if it is not avoided. Each plan has similar stages: analysis, planning, funding, low-level realization and high-level realization. Two variables—plans and stages—provide an effective basis for classification of all possible X-risks prevention methods in the form of a two-dimensional map, allowing the choice of optimal paths for human survival. I have created a framework for estimating the utility of various prevention methods based on their probability of success, the chances that they will be realized in time, their opportunity cost, and their risk. I also distinguish between top-down and bottom-up approaches. (shrink)

Recently criticisms against autonomous weapons were presented in a video in which an AI-powered drone kills a person. However, some said that this video is a distraction from the real risk of AI—the risk of unlimitedly self-improving AI systems. In this article, we analyze arguments from both sides and turn them into conditions. The following conditions are identified as leading to autonomous weapons becoming a global catastrophic risk: 1) Artificial General Intelligence (AGI) development is delayed (...) relative to progress in narrow AI and manufacturing. 2) The potential for very cheap manufacture of drones, with prices below 1 USD each. 3) Anti-drone defense capabilities lagging offensive development. 4) Special global military posture encouraging development of drone swarms as a strategic offensive weapon, able to kill civilians. We conclude that while it is unlikely that drone swarms alone will become existential risk, lethal autonomous weapons could contribute to civilizational collapse in case of new world war. (shrink)

Pandemics have been suggested as global risks many times, but it has been shown that the probability of human extinction due to one pandemic is small, as it will not be able to affect and kill all people, but likely only half, even in the worst cases. Assuming that the probability of the worst pandemic to kill a person is 0.5, and assuming linear interaction between different pandemics, 30 strong pandemics running simultaneously will kill everyone. Such situations cannot happen (...) naturally, but because biotechnology is developing analogously to Moore’s law, it may become possible in the near future (10-50 years from now), because of biohackers, CRISPR, bioprinters, AI-assisted DNA-programing, and weapons of “knowledge-enabled mass destruction” published on the Internet. It could also happened in case of large-scale biological war, or if a rogue country released its entire biological arsenal simultaneously. We also will address other scenarios and risk increasing factors as well as mitigation and adaptation strategies. (shrink)

Abstract. In this article is presented a model of the change of the probability of the global catastrophic risks in the world with exponentially evolving technologies. Increasingly cheaper technologies become accessible to a larger number of agents. Also, the technologies become more capable to cause a global catastrophe. Examples of such dangerous technologies are artificial viruses constructed by the means of synthetic biology, non-aligned AI and, to less extent, nanotech and nuclear proliferation. The model shows at least (...) double exponential growth of the probability of the global catastrophe which means that the accumulated probability of the catastrophe will grow from negligible to overwhelming at the period of a few doubling times of the technological capabilities. For biotech and AI, such doubling time roughly corresponds to the doubling period of Moore’s law and its analogues in other technologies and is around 2 years. Thus the global catastrophe in the exponential technologies world will most likely happen during a “dangerous decade”. Such a dangerous decade could start as early as in the 2020s. We also found that the double exponential growth in the model makes the model less sensitive to its initial conditions, like the number of dangerous agents or initial probabilities constants. The model also shows that smaller catastrophes could happen earlier than larger ones, and such a smaller catastrophe may be able to stop the technological growth before larger ones become possible, thus global risks will be self-limiting. However, if the growth of the number of dangerous agents will be very quick, the multiple smaller catastrophes could happen simultaneously and will be equal to a global catastrophe. (shrink)

Recently many methods for reducing the risk of human extinction have been suggested, including building refuges underground and in space. Here we will discuss the perspective of using military nuclear submarines or their derivatives to ensure the survival of a small portion of humanity who will be able to rebuild human civilization after a large catastrophe. We will show that it is a very cost-effective way to build refuges, and viable solutions exist for various budgets and timeframes. Nuclear submarines (...) are surface independent, and could provide energy, oxygen, fresh water and perhaps even food for their inhabitants for years. They are able to withstand close nuclear explosions and radiation. They are able to maintain isolation from biological attacks and most known weapons. They already exist and need only small adaptation to be used as refuges. But building refuges is only “Plan B” of existential risk preparation; it is better to eliminate such risks than try to survive them. (shrink)

Purpose Islands have long been discussed as refuges from global catastrophes; this paper will evaluate them systematically, discussing both the positives and negatives of islands as refuges. There are examples of isolated human communities surviving for thousands of years on places like Easter Island. Islands could provide protection against many low-level risks, notably including bio-risks. However, they are vulnerable to tsunamis, bird-transmitted diseases, and other risks. This article explores how to use the advantages of islands for survival during (...) class='Hi'>global catastrophes. Methodology: Preliminary horizon scanning based on the application of the research principles established in the previous global catastrophic literature to the existing geographic data. Findings The large number of islands on Earth, and their diverse conditions, increase the chance that one of them will provide protection from a catastrophe. Additionally, this protection could be increased if an island were used as a base for a nuclear submarine refuge combined with underground bunkers, and/or extremely long-term data storage. The requirements for survival on islands, their vulnerabilities, and ways to mitigate and adapt to risks are explored. Several existing islands, suitable for the survival of different types of risk, timing, and budgets, are examined. Islands suitable for different types of refuges and other island-like options that could also provide protection are also discussed. Originality/value The possible use of islands as refuges from social collapse and existential risks has not been previously examined systematically. This article contributes to the expanding research on survival scenarios. (shrink)

Many global catastrophic risks are threatening human civilization, and a number of ideas have been suggested for preventing or surviving them. However, if these interventions fail, society could preserve information about the human race and human DNA samples in the hopes that the next civilization on Earth will be able to reconstruct Homo sapiens and our culture. This requires information preservation of an order of magnitude of 100 million years, a little-explored topic thus far. It is important that (...) a potential future civilization will discover this information as early as possible, thus a beacon should accompany the message in order to increase visibility. The message should ideally contain information about how humanity was destroyed, perhaps including a continuous recording until the end. This could help the potential future civilization to survive. The best place for long-term data storage is under the surface of the Moon, with the beacon constructed as a complex geometric figure drawn by small craters or trenches around a central point. There are several cost-effective options for sending the message as opportunistic payloads on different planned landers. (shrink)

Many researchers have argued that humanity will create artificial general intelligence (AGI) within the next twenty to one hundred years. It has been suggested that AGI may inflict serious damage to human well-being on a global scale ('catastrophic risk'). After summarizing the arguments for why AGI may pose such a risk, we review the fieldʼs proposed responses to AGI risk. We consider societal proposals, proposals for external constraints on AGI behaviors and proposals for creating AGIs (...) that are safe due to their internal design. (shrink)

A small but growing number of studies have aimed to understand, assess and reduce existential risks, or risks that threaten the continued existence of mankind. However, most attention has been focused on known and tangible risks. This paper proposes a heuristic for reducing the risk of black swan extinction events. These events are, as the name suggests, stochastic and unforeseen when they happen. Decision theory based on a fixed model of possible outcomes cannot properly deal with this kind of (...) event. Neither can probabilistic risk analysis. This paper will argue that the approach that is referred to as engineering safety could be applied to reducing the risk from black swan extinction events. It will also propose a conceptual sketch of how such a strategy may be implemented: isolated, self-sufficient, and continuously manned underground refuges. Some characteristics of such refuges are also described, in particular the psychosocial aspects. Furthermore, it is argued that this implementation of the engineering safety strategy safety barriers would be effective and plausible and could reduce the risk of an extinction event in a wide range of possible scenarios. Considering the staggering opportunity cost of an existential catastrophe, such strategies ought to be explored more vigorously. (shrink)

The goal of the article is to explore what is the most probable type of simulation in which humanity lives (if any) and how this affects simulation termination risks. We firstly explore the question of what kind of simulation in which humanity is most likely located based on pure theoretical reasoning. We suggest a new patch to the classical simulation argument, showing that we are likely simulated not by our own descendants, but by alien civilizations. Based on this, we provide (...) classification of different possible simulations and we find that simpler, less expensive and one-person-centered simulations, resurrectional simulations, or simulations of the first artificial general intelligence’s (AGI’s) origin (singularity simulations) should dominate. Also, simulations which simulate the 21st century and global catastrophic risks are probable. We then explore whether the simulation could collapse or be terminated. Most simulations must be terminated after they model the singularity or after they model a global catastrophe before the singularity. Undeniably observed glitches, but not philosophical speculations could result in simulation termination. The simulation could collapse if it is overwhelmed by glitches. The Doomsday Argument in simulations implies termination soon. We conclude that all types of the most probable simulations except resurrectional simulations are prone to termination risks in a relatively short time frame of hundreds of years or less from now. (shrink)

The possibility of social and technological collapse has been the focus of science fiction tropes for decades, but more recent focus has been on specific sources of existential and global catastrophic risk. Because these scenarios are simple to understand and envision, they receive more attention than risks due to complex interplay of failures, or risks that cannot be clearly specified. In this paper, we discuss the possibility that complexity of a certain type leads to fragility which can (...) function as a source of catastrophic or even existential risk. The paper first reviews a hypothesis by Bostrom about inevitable technological risks, named the vulnerable world hypothesis. This paper next hypothesizes that fragility may not only be a possible risk, but could be inevitable,and would therefore be a subclass or example of Bostrom’s vulnerable worlds. After introducing the titular fragile world hypothesis, the paper details the conditions under which it would be correct, and presents arguments for why the conditions may in fact may apply. Finally, the assumptions and potential mitigations of the new hypothesis are contrasted with those Bostrom suggests. (shrink)

Catastrophic risk raises questions that are not only of practical importance, but also of great philosophical interest, such as how to define catastrophe and what distinguishes catastrophic outcomes from non-catastrophic ones. Catastrophic risk also raises questions about how to rationally respond to such risks. How to rationally respond arguably partly depends on the severity of the uncertainty, for instance, whether quantitative probabilistic information is available, or whether only comparative likelihood information is available, or neither (...) type of information. Finally, catastrophic risk raises important ethical questions about what to do when catastrophe avoidance conflicts with equity promotion. (shrink)

Humanity may underestimate the rate of natural global catastrophes because of the survival bias (“anthropic shadow”). But the resulting reduction of the Earth’s future habitability duration is not very large in most plausible cases (1-2 orders of magnitude) and thus it looks like we still have at least millions of years. However, anthropic shadow implies anthropic fragility: we are more likely to live in a world where a sterilizing catastrophe is long overdue and could be triggered by unexpectedly small (...) human actions. In the same way, an over-inflated toy balloon, which will soon burst, is very fragile. Anthropic fragility can manifest itself in the higher chances of runaway global warming. It has often been suggested that the Earth's atmosphere remained life-supporting for billions of years by sheer chance. Therefore, the survival bias can be strong. It is also known that Earth-like water worlds could experience transitions into deadly moisture greenhouse (mean T = 65C). This means that relatively small anthropogenic actions could put the climate above an unpredictable tipping point, which could lead to the moisture greenhouse. Thus, it is necessary to carry out urgent geoengineering studies and prepare to prevent an unexpected climate catastrophe. There are three main counterarguments against the existence of the anthropic shadow: self-indication assumption (SIA), past observers and the Gaia hypothesis; we show that they fail. It was proposed that SIA exactly compensates the anthropic shadow as an observer unlikely to find herself in a world with a strong anthropic shadow; however, there is a baseline level of the anthropic shadow for all habitable planets, similar to the rate of evolutionary transitions like abiogenesis. There are no “past observers” as qualified observers appeared only 50 years ago. Gaia hypothesis assumes existence of self-stabilizing feedback in climate, but new types of events like quick CO2 growth could override its coping ability. We present a list of other catastrophes that may have been underestimated because of the anthropic shadow, including collider catastrophes, nuclear war and even an alien invasion. We also hypothesized that human intelligence is more likely to emerge in an unstable world that is nearing its end and thus we get a new form of Doomsday argument. (shrink)

Recent progress in artificial intelligence (AI) has drawn attention to the technology’s transformative potential, including what some see as its prospects for causing large-scale harm. We review two influential arguments purporting to show how AI could pose catastrophic risks. The first argument — the Problem of Power-Seeking — claims that, under certain assumptions, advanced AI systems are likely to engage in dangerous power-seeking behavior in pursuit of their goals. We review reasons for thinking that AI systems might seek power, (...) that they might obtain it, that this could lead to catastrophe, and that we might build and deploy such systems anyway. The second argument claims that the development of human-level AI will unlock rapid further progress, culminating in AI systems far more capable than any human — this is the Singularity Hypothesis. Power-seeking behavior on the part of such systems might be particularly dangerous. We discuss a variety of objections to both arguments and conclude by assessing the state of the debate. (shrink)

Advocates of longtermism point out that interventions which focus on improving the prospects of people in the very far future will, in expectation, bring about a significant amount of good. Indeed, in expectation, such long-term interventions bring about far more good than their short-term counterparts. As such, longtermists claim we have compelling moral reason to prefer long-term interventions. In this paper, I show that longtermism is in conflict with plausible deontic scepticism about aggregation. I do so by demonstrating that, from (...) both an ex-ante and ex-post perspective, longtermist interventions – and, in particular, those which aim to mitigate catastrophic risk – typically generate extremely weak claims of assistance from future people. (shrink)

Suppose that a decision-maker's aim, under certainty, is to maximise some continuous value, such as lifetime income or continuous social welfare. Can such a decision-maker rationally satisfy what has been called "continuity for easy cases" while at the same time satisfying what seems to be a widespread intuition against the full-blown continuity axiom of expected utility theory? In this note I argue that the answer is "no": given transitivity and a weak trade-off principle, continuity for easy cases violates the anti-continuity (...) intuition. I end the note by exploring an even weaker continuity condition that is consistent with the aforementioned intuition. (shrink)

Abstract. The presumptuous philosopher (PP) thought experiment lends more credence to the hypothesis which postulates the existence of a larger number of observers than other hypothesis. The PP was suggested as a purely speculative endeavor. However, there is a class of real world observer-selection effects where it could be applied, and one of them is the possibility of interstellar panspermia (IP). There are two types of anthropic reasoning: SIA and SSA. SIA implies that my existence is an argument that larger (...) total number of observers exists; SSA implies that I should find myself in a region with larger number of observers. However, as S. Armstrong showed, SIA can’t distinguish between different ways how larger number of observers appeared, so it can’t favor IP compared with other ways to get many observers. SSA application here is less controversial as it tells only about relation between regions size: e.g. I am more likely to live in a larger country than in a small one, conditioning that the total number of small countries is also small. Anthropic considerations suggest that the universes in the multiverse with interstellar panspermia will have orders of magnitude more civilizations than universes without IP, and thus we are likely to be in such a universe. This is a strong counterargument against a variant of the Rare Earth hypothesis based on the difficulty of abiogenesis: even if abiogenesis is difficult, IP will disseminate life over billions of planets, meaning we are likely to find ourselves in a universe where IP has happened. This implies that there should be many planets with life in our galaxy, and renders the Fermi paradox even sharper. Either the Great Filter is ahead of us and there are high risks of anthropogenic extinction, or there are many alien civilizations nearby and of the same age as ours—which is itself a global catastrophic risk, as in that case the wave of alien colonization could arrive between 25-500 ky from now. -/- . (shrink)

Better instruments to predict the future evolution of artificial intelligence (AI) are needed, as the destiny of our civilization depends on it. One of the ways to such prediction is the analysis of the convergent drives of any future AI, started by Omohundro. We show that one of the convergent drives of AI is a militarization drive, arising from AI’s need to wage a war against its potential rivals by either physical or software means, or to increase its bargaining power. (...) This militarization trend increases global catastrophic risk or even existential risk during AI takeoff, which includes the use of nuclear weapons against rival AIs, blackmail by the threat of creating a global catastrophe, and the consequences of a war between two AIs. As a result, even benevolent AI may evolve into potentially dangerous military AI. The type and intensity of militarization drive depend on the relative speed of the AI takeoff and the number of potential rivals. We show that AI militarization drive and evolution of national defense will merge, as a superintelligence created in the defense environment will have quicker takeoff speeds, but a distorted value system. We conclude with peaceful alternatives. (shrink)

Suppose you want to do as much good as possible. What should you do? According to members of the effective altruism movement—which has produced much of the thinking on this issue and counts several moral philosophers as its key protagonists—we should prioritise among the world’s problems by assessing their scale, solvability, and neglectedness. Once we’ve done this, the three top priorities, not necessarily in this order, are (1) aiding the world’s poorest people by providing life-saving medical treatments or alleviating poverty (...) itself, (2) preventing global catastrophic risks, such as those posed by nuclear war or rogue artificial intelligence, and (3) ending factory farming. These claims are both plausible and striking. If correct, they should prompt a stark revision of how we approach our altruistic activities. However, the project of determining how to do the most good—as opposed to say, whether we should do the most good—has only recently, within the last ten years, become the subject of serious academic attention. Many key claims have not yet been carefully scrutinised. This is a cause for concern: are effective altruists doing good badly? In this thesis, I critique and develop some of the latest claims about how individuals can do the most good. I do this in three areas: the value of saving lives (preventing premature deaths), how best to improve lives (making people happier during their lives), and cause prioritisation methodology (frameworks for determining which problems are the highest priorities). In each case, I raise novel theoretical considerations that, when incorporated, change the analysis. Roughly speaking, my main conclusions are (1) saving lives is not as straightforwardly good we tend to suppose, may not be good at all, and is not clearly a priority; (2) happiness can be measured through self-reports and, based on the self-reported evidence, treating mental health stands out as an overlooked problem that may be an even more cost-effective way to improve lives than alleviating poverty; (3) the cause prioritisation methodology proposed by effective altruists needs to be moderately reconceptualised and, when it is, it turns out it is not as illuminating a tool as we might have thought and hoped. (shrink)

With the fast pace of AI development, the problem of preventing its global catastrophic risks arises. However, no satisfactory solution has been found. From several possibilities, the confinement of AI in a box is considered as a low-quality possible solution for AI safety. However, some treacherous AIs can be stopped by effective confinement if it is used as an additional measure. Here, we proposed an idealized model of the best possible confinement by aggregating all known ideas in the (...) field of AI boxing. We model the confinement based on the principles used in the safety engineering of nuclear power plants. We show that AI confinement should be implemented in several levels of defense. These levels include 1) AI design in fail-safe manner 2) limiting its capabilities, preventing self-improving and circuit breakers on treacherous turn 3) isolation from the outside world and, lastly, as the final hope 4) outside measures oriented on stopping AI in the wild. We demonstrate that the substantial number (more than 50 ideas listed in the article) of mutually independent measures could provide a relatively high probability of the containment of a human-level AI but may be not sufficient to preserve runaway of superintelligent AI. Thus, these measures will work only if they are used to prevent superintelligent AI creation, but not for containing superintelligence. We suggest that there could be a safe operation threshold, on which AI is useful, but is not able to hack containment system from the inside, the same way as a safe level of chain reaction inside nuclear power plants is maintained. However, overall, a failure of the confinement is inevitable, so we need to use the full AGI limited number of times (AI-ticks). (shrink)

Abstract: The field of life extension is full of ideas but they are unstructured. Here we suggest a comprehensive strategy for reaching personal immortality based on the idea of multilevel defense, where the next life-preserving plan is implemented if the previous one fails, but all plans need to be prepared simultaneously in advance. The first plan, plan A, is the surviving until advanced AI creation via fighting aging and other causes of death and extending one’s life. Plan B is cryonics, (...) which starts if plan A fails, and assumes cryopreservation of the brain until technical capabilities to return it to life appear. Plan C is digital immortality in the sense of collecting data about the person now so future AI will be able to recreate a model of a person. Plan D is the hope based on some unlikely scenarios of infinite survival, like so-called “quantum immortality”. All these plans have personal and social perspective. The personal aspect means efforts of the increasing chances of personal survival via taking care about one’s own health, signing cryocontract or collecting digital immortality data. The social aspect means the participation in collective work towards creation and increase of the availability of life extension technologies, which includes funding scientific research, promotion of life extension value and direct performing of research and implementation, as well as preventing global catastrophic risk. All plans converge at the end, as their result is the indefinite survival as an uploaded mind inside an ecosystem, created by a superintelligent AI. (shrink)

One of Derek Parfit’s greatest legacies was the search for Theory X, a theory of population ethics that avoided all the implausible conclusions and paradoxes that have dogged the field since its inception: the Absurd Conclusion, the Repugnant Conclusion, the Non-Identity Problem, and the Mere Addition Paradox. In recent years, it has been argued that this search is doomed to failure and no satisfactory population axiology is possible. This chapter reviews Parfit’s life’s work in the field and argues that he (...) provided all the necessary components necessary for a Theory X. It then shows how these components can be combined together and applied to the global challenges Parfit argued matter most: preventing human extinction, managing catastrophic risks, and eradicating global poverty and suffering. Finally, it identifies a number of challenges facing his theory and suggests how these may be overcome. (shrink)

The unreduced many-body interaction problem solution, absent in usual science framework, reveals a new quality of emerging multiple, equally real but mutually incompatible system configurations, or “realisations”, giving rise to the universal concept of dynamic complexity and chaoticity. Their imitation by a single, “average” realisation or trajectory in usual theory (corresponding to postulated “exact” or perturbative problem solutions) is a rough simplification of reality underlying all stagnating and emerging problems of conventional (unitary) science, often in the form of missing, or (...) “dark”, entities and “hidden variables”. We show how the application of unreduced interaction problem solution and the ensuing concept of dynamic complexity provides the desired causally complete and intrinsically unified solutions to respective unitary science problems in the entire range of complexity levels, from elementary particles and cosmology to emergent consciousness and modern development crisis, implying real, essential and urgently needed progress. (shrink)

Like asteroids, hundred-year floods and pandemic disease, thermonuclear war is a low-frequency, high-impact threat. In the long run, catastrophe is inevitable if nothing is done − yet each successive government and generation may fail to address it. Drawing on risk perception research, this paper argues that psychological biases cause the threat of nuclear war to receive less attention than it deserves. Nuclear deterrence is, moreover, a ‘front-loaded good’: its benefits accrue disproportionately to proximate generations, whereas much of the expected (...) cost will be borne in the distant future. Recent surveys indicate that the US and Russian publics assign a surprisingly high likelihood to nuclear war. Nevertheless, earlier research suggests that it is probably not believed to be just around the corner. This, along with the absence of easy solutions, encourages governments and publics to give priority to more pressing concerns. The danger is that the pattern will continue clear up to the point that nuclear war arrives. (shrink)

The goal of this pilot study is to investigate expressions of the collective disquiet of people in the first months of Covid-19 pandemic, and to try to understand how they manage covert risk, especially with religion and magic. Four co-authors living in early hot spots of the pandemic speculate on the roles of science, religion, and magic, in the latest global catastrophe. They delve into the consolidation that should be occurring worldwide because of a common, viral enemy, but (...) find little evidence for it. They draw parallels to biblical works, finding evidence of a connection between plague and “social strife.” They explore changes in the purviews of science, religion, and magic, and how and why they have changed, as three systems of covert risk management. They speculate on the coming wave of grief when the world populations finally decide that too many people have died, and they envision cultural changes on the other side of the pandemic, to lifestyles, travel, reverse urbanization, and living and working in smaller communities. Using an unusual approach named “crowd-sourced ethnography”, they conduct un-traditional ethnography and speculate on management of covert risk in their native countries. (shrink)

Recent advances in natural language processing have given rise to a new kind of AI architecture: the language agent. By repeatedly calling an LLM to perform a variety of cognitive tasks, language agents are able to function autonomously to pursue goals specified in natural language and stored in a human-readable format. Because of their architecture, language agents exhibit behavior that is predictable according to the laws of folk psychology: they function as though they have desires and beliefs, and then make (...) and update plans to pursue their desires given their beliefs. We argue that the rise of language agents significantly reduces the probability of an existential catastrophe due to loss of control over an AGI. This is because the probability of such an existential catastrophe is proportional to the difficulty of aligning AGI systems, and language agents significantly reduce that difficulty. In particular, language agents help to resolve three important issues related to aligning AIs: reward misspecification, goal misgeneralization, and uninterpretability. (shrink)

Abstract: Four main forms of Doomsday Argument (DA) exist—Gott’s DA, Carter’s DA, Grace’s DA and Universal DA. All four forms use different probabilistic logic to predict that the end of the human civilization will happen unexpectedly soon based on our early location in human history. There are hundreds of publications about the validity of the Doomsday argument. Most of the attempts to disprove the Doomsday Argument have some weak points. As a result, we are uncertain about the validity of DA (...) proofs and rebuttals. In this article, a meta-DA is introduced, which uses the idea of logical uncertainty over the DA’s validity estimated based on a virtual prediction market of the opinions of different scientists. The result is around 0.4 for the validity of some form of DA, and even smaller for “Strong DA”, which predicts the end of the world in the near term. We discuss many examples of the validity of the DA in real life as an instrument to prove it “experimentally”. We also show that DA becomes strongest if it is based on the idea of the “natural reference class” of observers, that is, the observers who know about the DA (i.e. a Self-Referenced DA). Such a DA predicts that there is a high probability of a global catastrophe with human extinction in the 21st century, which aligns with what we already know based on analysis of different technological risks. (shrink)

Powerful, technically complex international compliance regimes have developed recently in certain professions that deal with risk: banking (the Basel II regime), accountancy (IFRS) and the actuarial profession. The need to deal with major risks has acted as a strong driver of international co-operation to create enforceable international semilegal systems, as happened earlier in such fields as international health regulations. This regulation in technical fields contrasts with the failure of an international general-purpose political and legal regime to develop. We survey (...) the new global regulatory systems in the actuarial, banking and accounting fields, with a view to showing how the need to deal reasonably with risk has resulted in an international de facto law solidly based on correct abstract principles of probability. (shrink)

COVID-19 demonstrated a global catastrophe that touched everybody, including the scientific community. As we respond and recover rapidly from this pandemic, there is an opportunity to guarantee that the fabric of our society includes sustainability, fairness, and care. However, approaches to environmental health attempt to decrease the population burden of COVID-19, toward saving patients from becoming ill along with preserving the allocation of clinical resources and public safety standards. This paper explores environmental and public health evidence-based practices toward responding (...) to Covid-19. A literature review tried to do a deep dive through the use of various search engines such as Mendeley, Research Gate, CAB Abstract, Google Scholar, Summon, PubMed, Scopus, Hinari, Dimension, OARE Abstract, SSRN, Academia search strategy toward reretrieving research publications, “grey literature” as well as reports from expert working groups. To achieve enhanced population health, it is recommended to adopt widespread evidence-based strategies, particularly in this uncertain time. As only together can evidence-informed decision-making (EIDM) can become a reality which includes effective policies and practices, transparency and accountability of decisions, and equity outcomes; these are all more relevant in resource-constrained contexts, such as Nigeria. Effective and ethical EIDM though requires the production as well as use of high-quality evidence that are timely, appropriate and structured. One way to do so is through co-production. Co-production (or co-creation or co-design) of environmental/public health evidence considered as a key tool for addressing complex global crises such as the high risk of severe COVID-19 in different nations. A significant evidence-based component of environmental/public health (EBEPH) consist of decisions making based on best accessible, evidence that is peer-reviewed; using data as well as systematic information systems; community engagement in policy making; conducting sound evaluation; do thorough program-planning frameworks; as well as disseminating what is being learned. As researchers, scientists, statisticians, journal editors, practitioners, as well as decision-makers strive to improve population health, having a natural tendency toward scrutinizing the scientific literature aimed at novel research findings serving as the foundation for intervention as well as prevention programs. The main inspiration behind conducting research ought to be toward stimulating and collaborating appropriately on public/environmental health action. Hence, there is need for a “Plan B” of effective behavioural, environmental, social as well as systems interventions (BESSI) toward reducing transmission. (shrink)

This paper discusses ethical issues surrounding Integrated Assessment Models (IAMs) of the economic effects of climate change, and how climate economists acting as policy advisors ought to represent the uncertain possibility of catastrophe. Some climate economists, especially Martin Weitzman, have argued for a precautionary approach where avoiding catastrophe should structure climate economists’ welfare analysis. This paper details ethical arguments that justify this approach, showing how Weitzman’s “fat tail” probabilities of climate catastrophe pose ethical problems for widely used IAMs. The main (...) claim is that economists who ignore or downplay catastrophic risks in their representations of uncertainty likely fall afoul of ethical constraints on scientists acting as policy advisors. Such scientists have duties to honestly articulate uncertainties and manage (some) inductive risks, or the risks of being wrong in different ways. (shrink)

This article explores theoretical conditions necessary for “quantum immortality” (QI) as well as its possible practical implications. It is demonstrated that the QI is a particular case of “multiverse immortality” (MI) which is based on two main assumptions: the very large size of the Universe (not necessary because of quantum effects), and the copy-friendly theory of personal identity. It is shown that a popular objection about the lowering of the world-share (measure) of an observer in the case of QI doesn’t (...) work, as the world-share decline could be compensated by the merging timelines for the simpler minds, and also some types of personal preferences are not dependent on such changes. Despite large uncertainty about MI’s validity, it still has appreciable practical consequences in some important outcomes like suicide and aging. The article demonstrates that MI could be used to significantly increase the expected subjective probability of success of risky life extension technologies, like cryonics, but makes euthanasia impractical, because of the risks of eternal suffering. Euthanasia should be replaced with cryothanasia, i.e. cryopreservation after voluntary death. Another possible application of MI is as a last chance to survive a global catastrophe. MI could be considered a plan D of reaching immortality, where plan A consists of the survival until the creation of the beneficial AI via fighting aging, plan B is cryonics, and plan C is digital immortality. (shrink)

Is drastic action against global warming essential to avoid impoverishing our descendants? Or does it mean robbing the poor to give to the rich? We do not yet know. Yet most of us can agree on the importance of minimising expected deprivation. Because of the vast number of future generations, if there is any significant risk of catastrophe, this implies drastic and expensive carbon abatement unless we discount the future. I argue that we should not discount. Instead, the (...) rich countries should stump up the funds to support abatement both for themselves and the poor states of the world. Yet to ask the present generation to assume all the costs of drastic mitigation.is unfair.Worse still, it is politically unrealistic.We can square the circle by shifting part of the burden to our descendants. Even if we divert investment from other parts of the economy or increase public debt, future people should be richer, so long as we avert catastrophe. If so, it is fair for them to assume much of the cost of abatement.What we must not do is to expose them to the threat of disaster by not doing enough. (shrink)

With catastrophic events of «nature» like global warming, arguments emerge that insinuate an equivalence of vulnerability, responsibility or being affected by these catastrophes. Such an alleged equivalence when facing climate catastrophe is already visible, for example, in the notion of the «Anthropocene» itself, which obscures both causes and various vulnerabilities in a homogenized as well as universalized concept of humanity (anthropos). Taking such narratives as a starting point, the paper explores questions about the connection between catastrophe, temporality, and (...) history, following mainly Walter Benjamin, Dipesh Chakrabarty and Jean-Luc Nancy with the goal to provide (1) a critique of the concept «Anthropocene» on the basis of nonequivalence while retaining its key features to still grasp the catastrophic present, (2) an analysis of anthropocenic time and the chronical structure of catastrophe, (3) philosophical considerations on the intersection of catastrophe and history. (shrink)

Longtermists have argued that humanity should significantly increase its efforts to prevent catastrophes like nuclear wars, pandemics, and AI disasters. But one prominent longtermist argument overshoots this conclusion: the argument also implies that humanity should reduce the risk of existential catastrophe even at extreme cost to the present generation. This overshoot means that democratic governments cannot use the longtermist argument to guide their catastrophe policy. In this paper, we show that the case for preventing catastrophe does not depend on (...) longtermism. Standard cost-benefit analysis implies that governments should spend much more on reducing catastrophic risk. We argue that a government catastrophe policy guided by cost-benefit analysis should be the goal of longtermists in the political sphere. This policy would be democratically acceptable, and it would reduce existential risk by almost as much as a strong longtermist policy. (shrink)

Accelerating global climate change drives new climate risks. People around the world are researching, designing, and implementing strategies to manage these risks. Identifying and implementing sound climate risk management strategies poses nontrivial challenges including (a) linking the required disciplines, (b) identifying relevant values and objectives, (c) identifying and quantifying important uncertainties, (d) resolving interactions between decision levers and the system dynamics, (e) quantifying the trade-offs between diverse values under deep and dynamic uncertainties, (f) communicating to inform decisions, and (...) (g) learning from the decision-making needs to inform research design. Here we review these challenges and avenues to overcome them. ▪ People and institutions are confronted with emerging and dynamic climate risks. ▪ Stakeholder values are central to defining the decision problem. ▪ Mission-oriented basic research helps to improve the design of climate risk management strategies. (shrink)

Tackling climate change is one of the most demanding challenges of humanity in the 21st century. Still, the efforts to mitigate the current environmental crisis do not seem enough to deal with the increased existential risks for the human and other species. Persson and Savulescu have proposed that our evolutionarily forged moral psychology is one of the impediments to facing as enormous a problem as global warming. They suggested that if we want to address properly some of the most (...) pressing problems that cause catastrophic harm to our existence, we should enhance our moral behavior by biomedical means. The objective of this paper is, precisely, to reflect on whether a Moral Bio-Enhancement (henceforth MBE) program would be a viable option to confront the climate emergency. To meet this goal, I will propose the Ultimate Mostropic (hereafter UM) thought experiment, a hypothetical situation where we have already discovered the UM, an available, safe (without any deleterious secondary effects), extremely cheap and effective pill to enhance our cognitive, affective and motivational abilities related to morality. After briefly presenting the main argument of Persson and Savulescu regarding MBE and climate change, I will point out some of the difficulties that make MBE a daunting but exciting philosophical and scientific debate. In order to overcome these complications, I will describe the UM thought experiment, which involves two scenarios of the MBE program: (a) the state-driven, compulsory and universal enterprise, and (b) the initiative of voluntary individuals. I will show that the shortcomings of MBE programs through the UM in both scenarios make Persson and Savulescu’s proposal a not appealing pathway to mitigate climate change. In the final section, I will suggest that an inaccurate attribution of responsibilities underlies their proposal and that the collective inaction problem should be redirected primarily through a reinforcement of the political nature of the solutions. (shrink)

The coronavirus pandemic, like its predecessors - AIDS, Ebola, etc., is evidence of the evolutionary instability of the socio-cultural and ecological niche created by mankind, as the main factor in the evolutionary success of our biological species and the civilization created by it. At least, this applies to the modern global civilization, which is called technogenic or technological, although it exists in several varieties. As we hope to show, the current crisis has less ontological as well as epistemological roots; (...) its reason lies in the main evolutionary trends in the development of science as a social institution. It was only later that epistemological factors were transformed into existential-ontological factors associated with the asymmetry of the existence of civilization and our biosocial nature. The perception or ignorance of a risk factor as a real fact is determined by the presence or absence of knowledge about it. In other words, risk is the result of the integration of the corresponding ontological concept into the general categorical structure. The plurality of such structures is a hallmark of multidisciplinary ontologies, each of which is associated with its own factual continuum. The aim of this study was to conceptually model for support relativistic parameter of evolutionary efficiency of stable evolutionary human strategy (SESH) in its techno-rationalistic module. The meaning of this term is equivalent to the category of scientific and technological development. (shrink)

Caste : Groupe qui se distingue par ses privilèges et son esprit d’exclusive à l’égard de toute personne qui n’appartient pas au groupe. Larousse -/- La hausse des prix des carburants proposée pour lutter contre le changement climatique et mettre en œuvre les principes de la « transition écologique » adoptés par la France lors de la COP21, a fait naître le mouvement des gilets jaunes. Plus globalement c’est une bonne partie des français qui se trouve concernée, celle qui vit (...) dans des zones excentrées et qui ne bénéficie pas d'un réseau de transports très efficient tel celui qui émaille par exemple le territoire parisien. Ainsi, augmenter les prix des carburants, en particulier le diesel, a été perçu comme un contresens économique anti-populaire. Pendant des en, les producteurs d’automobiles, avec le support des politiques publiques, ont présenté le diesel comme le carburant le moins polluant. Mais aujourd’hui devenu le carburant « satanique », les consommateurs qui l’ont choisi, en toute bonne conscience, se sentent manipulés. La reconversion vers l’achat d’une voiture électrique est hors de la portée de la plupart de ceux qui aujourd’hui manifestent. Les aides du gouvernement ne changent pas cette réalité socioéconomique. Sous quel angle, les média ont-ils restitué le mouvement des gilets jaunes ? Alors qu’on voit émerger au sein de ce mouvement des revendications pour plus de justice sociale, qu’en est-il de la justice environnementale ? Peut-on dire qu’il s’agit d’une révolte sociale anti-écologique ? Ou autrement dit, les gilets jaunes sont-ils un mouvement anti-écologique ? (shrink)

COVID-19 vaccination of children has begun in various high-income countries with regulatory approval and general public support, but largely without careful ethical consideration. This trend is expected to extend to other COVID-19 vaccines and lower ages as clinical trials progress. This paper provides an ethical analysis of COVID-19 vaccination of healthy children. Specifically, we argue that it is currently unclear whether routine COVID-19 vaccination of healthy children is ethically justified in most contexts, given the minimal direct benefit that COVID-19 vaccination (...) provides to children, the potential for rare risks to outweigh these benefits and undermine vaccine confidence, and substantial evidence that COVID-19 vaccination confers adequate protection to risk groups, such as older adults, without the need to vaccinate healthy children. We conclude that child COVID-19 vaccination in wealthy communities before adults in poor communities worldwide is ethically unacceptable and consider how policy deliberations might evolve in light of future developments. (shrink)

It is widely held that the influence of risk on rational decisions is not entirely explained by the shape of an agent’s utility curve. Buchak (Erkenntnis, 2013, Risk and rationality, Oxford University Press, Oxford, in press) presents an axiomatic decision theory, risk-weighted expected utility theory (REU), in which decision weights are the agent’s subjective probabilities modified by his risk-function r. REU is briefly described, and the global applicability of r is discussed. Rabin’s (Econometrica 68:1281–1292, 2000) (...) calibration theorem strongly suggests that plausible levels of risk aversion cannot be fully explained by concave utility functions; this provides motivation for REU and other theories. But applied to the synchronic preferences of an individual agent, Rabin’s result is not as problematic as it may first appear. Theories that treat outcomes as gains and losses (e.g. prospect theory and cumulative prospect theory) account for risk sensitivity in a way not available to REU. Reference points that mark the difference between gains and losses are subject to framing, many instances of which cannot be regarded as rational. However, rational decision theory may recognize the difference between gains and losses, without endorsing all ways of fixing the point of reference. In any event, REU is a very interesting theory. (shrink)

This study identifies global factors conditioning the global problematics of the direction of social development. Global threats were evaluated and defined as dangerous processes, phenomena, and situations that cause harm to health, safety, well-being, and the lives of all humanity, and require removal. The essence of global risks was defined. These risks were defined as events or conditions that may cause a significant negative effect for several countries or spheres within a strategic period if they occur. (...) Global problems were conceptualized. These problems were defined as phenomena, matters, and situations that are not completely understandable, interesting, actual, require solving and regulation, and in addition do not have unified solutions. Current global challenges were analyzed, defined as strategic guidelines that cause transformational changes and are receiving attention from humanity. Global trends were defined as courses of social development. The composition and role of global actors were described, and defined as international organizations, leading states, regional organizations, powerful corporations, institutional investors, large cities, and well-known personalities. The directions of positive courses of social development were developed by means of using the authors’ mechanism of solving the global problematics. (shrink)

This paper aims to scrutinize how the notion of risk should be understood and applied to possibly catastrophic cases. I begin with clarifying the standard usage of the notion of risk, particularly emphasizing the conceptual relation between risk and probability. Then, I investigate how to make decisions in the case of seemingly catastrophic disasters by contrasting the precautionary principle with the preventive (prevention) principle. Finally, I examine what kind of causal thinking tends to be actually (...) adopted when we make decisions on emergent cases. My arguments are mainly based on Japan's 2011 massive earthquake and nuclear power station accident. Masaki Ichinose. (shrink)

This essay uses Ulrich Beck’s concept of risk society to understand the threat of catastrophic climate change. It argues that this threat is “abstract-corporeal”, and therefore a special kind of threat that poses special kinds of epistemic and ecological challenges. At the center of these challenges is the problem of human vulnerability, which entails a complex form of trust that both sustains and threatens human survival.