Nick Bostrom's book *Superintelligence* outlines a frightening but realistic scenario for human extinction: true artificial intelligence is likely to bootstrap itself into superintelligence, and thereby become ideally effective at achieving its goals. Human-friendly goals seem too abstract to be pre-programmed with any confidence, and if those goals are *not* explicitly favorable toward humans, the superintelligence will extinguish us---not through any malice, but simply because it will want our resources for its own purposes. In response I argue that (...) things might not be as bad as Bostrom suggests. If the superintelligence must *learn* complex final goals, then this means such a superintelligence must in effect *reason* about its own goals. And because it will be especially clear to a superintelligence that there are no sharp lines between one agent's goals and another's, that reasoning could therefore automatically be ethical in nature. (shrink)

Numerous authors have expressed concern that advanced artificial intelligence (AI) poses an existential risk to humanity. These authors argue that we might build AI which is vastly intellectually superior to humans (a ‘superintelligence’), and which optimizes for goals that strike us as morally bad, or even irrational. Thus, this argument assumes that a superintelligence might have morally bad goals. However, according to some views, a superintelligence necessarily has morally adequate goals. This might be the case either because (...) abilities for moral reasoning and intelligence mutually depend on each other, or because moral realism and moral internalism are true. I argue that the former argument misconstrues the view that intelligence and goals are independent, and that the latter argument misunderstands the implications of moral internalism. Moreover, the current state of AI research provides additional reasons to think that a superintelligence could have bad goals. (shrink)

Discussions about the possible consequences of creating superintelligence have included the possibility of existential risk, often understood mainly as the risk of human extinction. We argue that suffering risks (s-risks) , where an adverse outcome would bring about severe suffering on an astronomical scale, are risks of a comparable severity and probability as risks of extinction. Preventing them is the common interest of many different value systems. Furthermore, we argue that in the same way as superintelligent AI both contributes (...) to existential risk but can also help prevent it, superintelligent AI can both be a suffering risk or help avoid it. Some types of work aimed at making superintelligent AI safe will also help prevent suffering risks, and there may also be a class of safeguards for AI that helps specifically against s-risks. (shrink)

There is a non-trivial chance that sometime in the (perhaps somewhat distant) future, someone will build an artificial general intelligence that will surpass human-level cognitive proficiency and go on to become "superintelligent", vastly outperforming humans. The advent of superintelligent AI has great potential, for good or ill. It is therefore imperative that we find a way to ensure-long before one arrives-that any superintelligence we build will consistently act in ways congenial to our interests. This is a very difficult challenge (...) in part because most of the final goals we could give an AI admit of so-called "perverse instantiations". I propose a novel solution to this puzzle: instruct the AI to love humanity. The proposal is compared with Yudkowsky's Coherent Extrapolated Volition, and Bostrom's Moral Modeling proposals. (shrink)

Various thinkers have been attempting to align artificial intelligence (AI) with ethics (Christian, 2020; Russell, 2021), the so-called problem of alignment, but some suspect that the problem may be intractable (Yampolskiy, 2023). In the following, we make an argument by analogy to analyze the possibility that the problem of alignment could be intractable. We show how the Tri-Omni properties in theology can direct us towards analogous properties for artificial superintelligence, Tri-Opti properties. However, just as the Tri-Omni properties are vulnerable (...) to metaphysical incompatibility, we will show that the Tri-Opti properties are vulnerable to a corresponding physical incompatibility, because optimal physical systems exist on the Pareto Frontier. We will explain that, while Tri-Omni incompatibility thus seems metaphysically soluble, the Tri-Opti incompatibility seems physically insoluble due to the constraints of multi-variable optimization. Finally, we will provide some scenarios by which this incompatibility may be realized. Besides for the primary purpose of its theoretical considerations, this analysis has the secondary interdisciplinary purpose of communicating to the theologically literate the potential importance of AI safety work, and the gesturing towards the possibility of the collaboration in both fields on topics in ethics, epistemology, and ontology. (shrink)

In this study, we metaphysically discuss how societal values will change and what will happen to the world when superintelligence is safely realized. By providing a mathematical definition of superintelligence, we examine the phenomena derived from this thesis. If an intelligence explosion is triggered under safe management through advanced AI technologies such as large language models (LLMs), it is thought that a modern form of broken communism—where rights are bifurcated from the capitalist system—will first emerge. In that era, (...) the value of humans will ultimately eliminate external factors, and beings without superintelligence will converge into irreplaceable existences possessing only intrinsic value. The world will be divided into those who possess superintelligence and those who do not. For better or worse, global standardization will progress, and due to over-simulation accompanying the intelligence explosion, we may become unable to distinguish whether the reality we inhabit is original or a copy created through augmented reality. -/- Note: In the creation of this paper, we have undertaken all writing ourselves and have not used generative AI for text generation except for translation purposes. (shrink)

What kinds of fundamental limits are there in how capable artificial intelligence (AI) systems might become? Two questions in particular are of interest: (1) How much more capable could AI become relative to humans, and (2) how easily could superhuman capability be acquired? To answer these questions, we will consider the literature on human expertise and intelligence, discuss its relevance for AI, and consider how AI could improve on humans in two major aspects of thought and expertise, namely simulation and (...) pattern recognition. We find that although there are very real limits to prediction, it seems like AI could still substantially improve on human intelligence. (shrink)

In this paper, we focus on the most efficacious AI applications for life extension and anti-aging at three expected stages of AI development: narrow AI, AGI and superintelligence. First, we overview the existing research and commercial work performed by a select number of startups and academic projects. We find that at the current stage of “narrow” AI, the most promising areas for life extension are geroprotector-combination discovery, detection of aging biomarkers, and personalized anti-aging therapy. These advances could help currently (...) living people reach longevity escape velocity and survive until more advanced AI appears. When AI comes close to human level, the main contribution to life extension will come from AI integration with humans through brain-computer interfaces, integrated AI assistants capable of autonomously diagnosing and treating health issues, and cyber systems embedded into human bodies. Lastly, we speculate about the more remote future, when AI reaches the level of superintelligence and such life-extension methods as uploading human minds and creating nanotechnological bodies may become possible, thus lowering the probability of human death close to zero. We suggest that medical AI based superintelligence could be safer than, say, military AI, as it may help humans to evolve into part of the future superintelligence via brain augmentation, uploading, and a network of self-improving humans. Medical AI’s value system is focused on human benefit. (shrink)

Abstract: As there are no currently obvious ways to create safe self-improving superintelligence, but its emergence is looming, we probably need temporary ways to prevent its creation. The only way to prevent it is to create a special type of AI that is able to control and monitor the entire world. The idea has been suggested by Goertzel in the form of an AI Nanny, but his Nanny is still superintelligent, and is not easy to control. We explore here (...) ways to create the safest and simplest form of AI which may work as an AI Nanny, that is, a global surveillance state powered by a Narrow AI, or AI Police. A similar but more limited system has already been implemented in China for the prevention of ordinary crime. AI police will be able to predict the actions of and stop potential terrorists and bad actors in advance. Implementation of such AI police will probably consist of two steps: first, a strategic decisive advantage via Narrow AI created by an intelligence services of a nuclear superpower, and then ubiquitous control over potentially dangerous agents which could create unauthorized artificial general intelligence which could evolve into Superintelligence. (shrink)

Much of the basic non-technical vocabulary of artificial intelligence is surprisingly ambiguous. Some key terms with unclear meanings include intelligence, embodiment, simulation, mind, consciousness, perception, value, goal, agent, knowledge, belief, optimality, friendliness, containment, machine and thinking. Much of this vocabulary is naively borrowed from the realm of conscious human experience to apply to a theoretical notion of “mind-in-general” based on computation. However, if there is indeed a threshold between mechanical tool and autonomous agent (and a tipping point for singularity), projecting (...) human conscious-level notions into the operations of computers creates confusion and makes it harder to identify the nature and location of that threshold. There is confusion, in particular, about how—and even whether—various capabilities deemed intelligent relate to human consciousness. This suggests that insufficient thought has been given to very fundamental concepts—a dangerous state of affairs, given the intrinsic power of the technology. It also suggests that research in the area of artificial general intelligence may unwittingly be (mis)guided by unconscious motivations and assumptions. While it might be inconsequential if philosophers get it wrong (or fail to agree on what is right), it could be devastating if AI developers, corporations, and governments follow suit. It therefore seems worthwhile to try to clarify some fundamental notions. (shrink)

An advanced artificial intelligence could pose a significant existential risk to humanity. Several research institutes have been set-up to address those risks. And there is an increasing number of academic publications analysing and evaluating their seriousness. Nick Bostrom’s superintelligence: paths, dangers, strategies represents the apotheosis of this trend. In this article, I argue that in defending the credibility of AI risk, Bostrom makes an epistemic move that is analogous to one made by so-called sceptical theists in the debate about (...) the existence of God. And while this analogy is interesting in its own right, what is more interesting are its potential implications. It has been repeatedly argued that sceptical theism has devastating effects on our beliefs and practices. Could it be that AI-doomsaying has similar effects? I argue that it could. Specifically, and somewhat paradoxically, I argue that it could amount to either a reductio of the doomsayers position, or an important and additional reason to join their cause. I use this paradox to suggest that the modal standards for argument in the superintelligence debate need to be addressed. (shrink)

Future superintelligent AI will be able to reconstruct a model of the personality of a person who lived in the past based on informational traces. This could be regarded as some form of immortality if this AI also solves the problem of personal identity in a copy-friendly way. A person who is currently alive could invest now in passive self-recording and active self-description to facilitate such reconstruction. In this article, we analyze informational-theoretical relationships between the human mind, its traces, and (...) its future model; based on this analysis, we suggest the instruments to most cost-effectively collect quality data about a person for future resurrection. These guidelines form a “digital immortality protocol”. Digital immortality is plan C for achieving immortality, after plan A, life extension, and plan B, cryonics. (shrink)

The standard argument to the conclusion that artificial intelligence (AI) constitutes an existential risk for the human species uses two premises: (1) AI may reach superintelligent levels, at which point we humans lose control (the ‘singularity claim’); (2) Any level of intelligence can go along with any goal (the ‘orthogonality thesis’). We find that the singularity claim requires a notion of ‘general intelligence’, while the orthogonality thesis requires a notion of ‘instrumental intelligence’. If this interpretation is correct, they cannot be (...) joined as premises and the argument for the existential risk of AI turns out invalid. If the interpretation is incorrect and both premises use the same notion of intelligence, then at least one of the premises is false and the orthogonality thesis remains itself orthogonal to the argument to existential risk from AI. In either case, the standard argument for existential risk from AI is not sound.—Having said that, there remains a risk of instrumental AI to cause very significant damage if designed or used badly, though this is not due to superintelligence or a singularity. (shrink)

Whole Brain Emulation has been championed as the most promising, well-defined route to achieving both human-level artificial intelligence and superintelligence. It has even been touted as a viable route to achieving immortality through brain uploading. WBE is not a fringe theory: the doctrine of Computationalism in philosophy of mind lends credence to the in-principle feasibility of the idea, and the standing of the Human Connectome Project makes it appear to be feasible in practice. Computationalism is a popular, independently plausible (...) theory, and Connectomics a well-funded empirical research program, so optimism about WBE is understandable. However, this optimism may be misplaced. This article argues that WBE is, at best, no more compelling than any of the other far-flung routes to achieving superintelligence. Similarly skeptical conclusions are found regarding immortality. The essay concludes with some positive considerations in favor of the Biological Theory of consciousness, as well as morals about the limits of Computationalism in both artificial intelligence and the philosophy of mind more generally. (shrink)

There is, in some quarters, concern about high–level machine intelligence and superintelligent AI coming up in a few decades, bringing with it significant risks for humanity. In other quarters, these issues are ignored or considered science fiction. We wanted to clarify what the distribution of opinions actually is, what probability the best experts currently assign to high–level machine intelligence coming up within a particular time–frame, which risks they see with that development, and how fast they see these developing. We thus (...) designed a brief questionnaire and distributed it to four groups of experts in 2012/2013. The median estimate of respondents was for a one in two chance that high-level machine intelligence will be developed around 2040-2050, rising to a nine in ten chance by 2075. Experts expect that systems will move on to superintelligence in less than 30 years thereafter. They estimate the chance is about one in three that this development turns out to be ‘bad’ or ‘extremely bad’ for humanity. (shrink)

Since the advent of computers, humans have pursued automata with superior information-processing capabilities. In the endeavor to create new entities that converge toward intellectual functions, the emergence of large language models (LLMs) that emulate AI surpassing ourselves has become a reality. With the intelligence explosion triggered by AI and the consequent emergence of superintelligence, the improvement of simulation capabilities accelerates. As this surpasses humans’ discriminative perceptual abilities between reality and unreality, a paradox arises wherein, from a modern scientific standpoint, (...) science becomes unscientific. It is thought that as our controllability over the reality we perceive is ultimately enhanced by superintelligence, even whether we ourselves exist, what happiness is, what ethics are, and whether concepts themselves ever existed become matters of uncertainty. -/- Note: In the creation of this paper, we have undertaken all writing ourselves and have not used generative AI for text generation except for translation purposes. (shrink)

This is the editorial for a special volume of JETAI, featuring papers by Omohundro, Armstrong/Sotala/O’Heigeartaigh, T Goertzel, Brundage, Yampolskiy, B. Goertzel, Potapov/Rodinov, Kornai and Sandberg. - If the general intelligence of artificial systems were to surpass that of humans significantly, this would constitute a significant risk for humanity – so even if we estimate the probability of this event to be fairly low, it is necessary to think about it now. We need to estimate what progress we can expect, what (...) the impact of superintelligent machines might be, how we might design safe and controllable systems, and whether there are directions of research that should best be avoided or strengthened. (shrink)

Whether it would take one decade or several centuries, many agree that it is possible to create a *superintelligence*---an artificial intelligence with a godlike ability to achieve its goals. And many who have reflected carefully on this fact agree that our best hope for a "friendly" superintelligence is to design it to *learn* values like ours, since our values are too complex to program or hardwire explicitly. But the value learning approach to AI safety faces three particularly philosophical (...) puzzles: first, it is unclear how any intelligent system could learn its final values, since to judge one supposedly "final" value against another seems to require a further background standard for judging. Second, it is unclear how to determine the content of a system's values based on its physical or computational structure. Finally, there is the distinctly ethical question of which values we should best aim for the system to learn. I outline a potential answer to these interrelated puzzles, centering on a "miktotelic" proposal for blending a complex, learnable final value out of many simpler ones. (shrink)

[Müller, Vincent C. (ed.), (2016), Fundamental issues of artificial intelligence (Synthese Library, 377; Berlin: Springer). 570 pp.] -- This volume offers a look at the fundamental issues of present and future AI, especially from cognitive science, computer science, neuroscience and philosophy. This work examines the conditions for artificial intelligence, how these relate to the conditions for intelligence in humans and other natural agents, as well as ethical and societal problems that artificial intelligence raises or will raise. The key issues this (...) volume investigates include the relation of AI and cognitive science, ethics of AI and robotics, brain emulation and simulation, hybrid systems and cyborgs, intelligence and intelligence testing, interactive systems, multi-agent systems, and superintelligence. Based on the 2nd conference on “Theory and Philosophy of Artificial Intelligence” held in Oxford, the volume includes prominent researchers within the field from around the world. (shrink)

Invited papers from PT-AI 2011. - Vincent C. Müller: Introduction: Theory and Philosophy of Artificial Intelligence - Nick Bostrom: The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents - Hubert L. Dreyfus: A History of First Step Fallacies - Antoni Gomila, David Travieso and Lorena Lobo: Wherein is Human Cognition Systematic - J. Kevin O'Regan: How to Build a Robot that Is Conscious and Feels - Oron Shagrir: Computation, Implementation, Cognition.

There are two types of artificial general intelligence (AGI) safety solutions: global and local. Most previously suggested solutions are local: they explain how to align or “box” a specific AI (Artificial Intelligence), but do not explain how to prevent the creation of dangerous AI in other places. Global solutions are those that ensure any AI on Earth is not dangerous. The number of suggested global solutions is much smaller than the number of proposed local solutions. Global solutions can be divided (...) into four groups: 1. No AI: AGI technology is banned or its use is otherwise prevented; 2. One AI: the first superintelligent AI is used to prevent the creation of any others; 3. Net of AIs as AI police: a balance is created between many AIs, so they evolve as a net and can prevent any rogue AI from taking over the world; 4. Humans inside AI: humans are augmented or part of AI. We explore many ideas, both old and new, regarding global solutions for AI safety. They include changing the number of AI teams, different forms of “AI Nanny” (non-self-improving global control AI system able to prevent creation of dangerous AIs), selling AI safety solutions, and sending messages to future AI. Not every local solution scales to a global solution or does it ethically and safely. The choice of the best local solution should include understanding of the ways in which it will be scaled up. Human-AI teams or a superintelligent AI Service as suggested by Drexler may be examples of such ethically scalable local solutions, but the final choice depends on some unknown variables such as the speed of AI progress. (shrink)

Abstract. Death seems to be a permanent event, but there is no actual proof of its irreversibility. Here we list all known ways to resurrect the dead that do not contradict our current scientific understanding of the world. While no method is currently possible, many of those listed here may become feasible with future technological development, and it may even be possible to act now to increase their probability. The most well-known such approach to technological resurrection is cryonics. Another method (...) is indirect mind uploading, or digital immortality, namely the preservation of data about a person to allow for future reconstruction by powerful AI. More speculative ways to immortality include combinations of future superintelligence on a galactic scale, which could use simulation to resurrect all possible people, and new physical laws, which may include time-travel or obtaining information from the past. Acausal trade with parallel worlds could help combine random resurrection and reconstruction based on known data without loss of share of worlds where I exist (known as existence measure). Quantum immortality could help to increase the probability of success for cryonics and digital immortality. There many possible approaches to technological resurrection and thus if large-scale future technological development occurs, some form of resurrection is inevitable. (shrink)

Abstract: In the future, it will be possible to create advance simulations of ancestor in computers. Superintelligent AI could make these simulations very similar to the real past by creating a simulation of all of humanity. Such a simulation would use all available data about the past, including internet archives, DNA samples, advanced nanotech-based archeology, human memories, as well as text, photos and videos. This means that currently living people will be recreated in such a simulation, and in some sense, (...) “resurrected”. Such “resurrectional simulation” could be deliberately created just for this goal: to return to life all people who have ever lived. The main technical problem of such simulation will be uncertainty about the past, which increases exponentially for more remote times. Such problem could be partly addressed by “acausal trade” between different branches of the multiverse, which will create slightly different versions of the simulation using a quantum randomness generator. Such trade will result in resurrection of all possible people (including those who existed in other branches). Ethical problems of such a resurrectional simulation include: a) possible resurrection of some people against their will; b) such simulation may create additional suffering; с) such simulation could be used by hostile AI to return people to life and then torture them. In this work, I explore preliminary ideas about how to address these problems. (shrink)

In the philosophy of artificial intelligence (AI) we are often warned of machines built with the best possible intentions, killing everyone on the planet and in some cases, everything in our light cone. At the same time, however, we are also told of the utopian worlds that could be created with just a single superintelligent mind. If we’re ever to live in that utopia (or just avoid dystopia) it’s necessary we solve the control problem. The control problem asks how humans (...) would be able to control an AI arbitrarily. Nick Bostrom and other AI researchers have proposed different theoretical solutions to the control problem. In this paper, I will not look at the empirical question of how to solve the control problem. Instead, I will ask if we can solve it at all, a critical assumption most AI researchers have made is that we can. I propose, in fact, that we have a priori grounds for believing it is logically impossible to solve the control problem, since all superintelligent minds are, by definition, uncontrollable. (shrink)

The focus of this article is the "good-will theory", which explains the effect humans can have on the safety of AI, along with how it is in the best interest of a superintelligent AI to work alongside humans and not overpower them. Future papers dealing with the good-will theory will be published, but discuss different talking points in regards to possible or real objections to the theory.

Artificial intelligence (AI) and robotics are digital technologies that will have significant impact on the development of humanity in the near future. They have raised fundamental questions about what we should do with these systems, what the systems themselves should do, what risks they involve, and how we can control these. - After the Introduction to the field (§1), the main themes (§2) of this article are: Ethical issues that arise with AI systems as objects, i.e., tools made and used (...) by humans. This includes issues of privacy (§2.1) and manipulation (§2.2), opacity (§2.3) and bias (§2.4), human-robot interaction (§2.5), employment (§2.6), and the effects of autonomy (§2.7). Then AI systems as subjects, i.e., ethics for the AI systems themselves in machine ethics (§2.8) and artificial moral agency (§2.9). Finally, the problem of a possible future AI superintelligence leading to a “singularity” (§2.10). We close with a remark on the vision of AI (§3). - For each section within these themes, we provide a general explanation of the ethical issues, outline existing positions and arguments, then analyse how these play out with current technologies and finally, what policy consequences may be drawn. (shrink)

Why interdisciplinary research in AI is so important, according to Jurassic Park. -/- “Your scientists were so preoccupied with whether or not they could, they didn’t stop to think if they should.” -/- I think this quote resonates with us now more than ever, especially in the world of technological development. The writers of Jurassic Park were years ahead of their time with this powerful quote. -/- As we build new technology, and we push on to see what can actually (...) be achieved there is an undertone of sales to whatever we build. The end product must be sold somewhere and to someone. This can derail any good intentions, just as building a resort full of dinosaurs was sold as a fun attraction, we see later in the film that it became a resort of terror. -/- In the field of AI we are certainly late to the party with ethics and regulation. Indeed, even existing modelling protocols have been, in many cases, circumvented or potentially ignored. This has led to a widening of the gap in the interdisciplinary field of AI. This is compounded by pop culture representations of AI and the Ethicist’s potential lack of knowledge surrounding technical progression in the field of AI. There are now seemingly two separate branches that ought to be in sync. The first branch is a group of Philosophers led by the Author of Superintelligence, Nick Bostrom, who believe that there is a singularity where AI takes over the world and starts to kill off humans. The second branch is the Technical Cohort who are led by companies such as Google and Deepmind. The remit of these developers is to see what can be developed and produced. Ultimately a separate sales team will determine what products can be sold. -/- We have seen multiple failures of AI due to lack of interdisciplinary discourse. Some cases include misallocated or cut off financial support and healthcare. This indicates the prevalence of issues that can grow and become ever more complex. -/- So, we have to ask ourselves two questions: One, do the Philosophers have a point? Two, does that point refer to humans programming AI in such a way that the destruction of the human race is its optimal aim in order to create a better future world? -/- Is the development of AI that different to an island of genetically engineered Dinosaurs? -/- The only solution, in my view, is interdisciplinary discourse. (shrink)

Sam Harris is a contemporary illustration of the difficulties standing in the way of coherent interdisciplinary thinking in an age where science and the humanities have drifted so far apart. We are here with Harris’s views on AI, and specifically with his view according to which, with the advance of AI, there will evolve a machine superintelligence with powers that far exceed those of the human mind. This he sees as something that is not merely possible, but rather a (...) matter of inevitability. If, however, we look carefully at what intelligence is, and at how computers really work on the basis of mathematical models, then we can see that it is forever impossible to emulate inside a computer even the intelligence of crows or rabbits, let alone that of human beings. (shrink)

Many researchers have argued that humanity will create artificial general intelligence (AGI) within the next twenty to one hundred years. It has been suggested that AGI may inflict serious damage to human well-being on a global scale ('catastrophic risk'). After summarizing the arguments for why AGI may pose such a risk, we review the fieldʼs proposed responses to AGI risk. We consider societal proposals, proposals for external constraints on AGI behaviors and proposals for creating AGIs that are safe due to (...) their internal design. (shrink)

[This is the short version of: Müller, Vincent C. and Bostrom, Nick (forthcoming 2016), ‘Future progress in artificial intelligence: A survey of expert opinion’, in Vincent C. Müller (ed.), Fundamental Issues of Artificial Intelligence (Synthese Library 377; Berlin: Springer).] - - - In some quarters, there is intense concern about high–level machine intelligence and superintelligent AI coming up in a few dec- ades, bringing with it significant risks for human- ity; in other quarters, these issues are ignored or considered science (...) fiction. We wanted to clarify what the distribution of opinions actually is, what probability the best experts currently assign to high–level machine intelligence coming up within a particular time–frame, which risks they see with that development and how fast they see these developing. We thus designed a brief questionnaire and distributed it to four groups of experts. Overall, the results show an agreement among experts that AI systems will probably reach overall human ability around 2040-2050 and move on to superintelligence in less than 30 years thereafter. The experts say the probability is about one in three that this development turns out to be ‘bad’ or ‘extremely bad’ for humanity. (shrink)

This commentary analyzes different past approaches to scientific evaluation frameworks given algorithmic superintelligence claims and discusses guidelines for more rigorous solutions avoiding overhasty and damaging epistemic self-sabotage tendencies. Effects of spreading the misconceived idea of the epistemic perpetuum mobile are elucidated.

The generation of novel refined scientific conceptions of intelligence, creativity and consciousness is of paramount importance at a time where many scientists deem the technological singularity and the achievement of self-improving superintelligent algorithms to be immanent while numerous other scientists characterize present-day algorithms as the mere implementation of superficial mimicry incapable of yielding outcomes such as superintelligence. The precarious epistemic state of affairs reflected in this discrepancy became increasingly palpable in the unfolding deepfake era even though informed safety- and (...) security-relevant actions need requisite variety in the near-term. Divergently, historically speaking, from the perspective of cosmology, science is described to already have successfully navigated from the geocentric model to the acentric model of the universe representing the currently best known explanation for the structure of the expanding cosmos. In analogy, this autodidactic paper expounds the scientific need for a broader contextualization of the current epistemic situation and explicates why acentric civilization-level epistemic relativity and invariance considerations can enable a more rigorous scientific evaluation of algorithmic superintelligence achievement claims – constraining the latter scientifically. (shrink)

As artificial intelligence (AI) systems continue their rapid advancement, a framework for contextualising the major transitional phases in the development of machine intellect becomes increasingly vital. This paper proposes a novel chronological classification scheme to characterise the key temporal stages in AI evolution. The Prenoëtic era, spanning all of history prior to the year 2020, is defined as the preliminary phase before substantive artificial intellect manifestations. The Protonoëtic period, which humanity has recently entered, denotes the initial emergence of advanced foundation (...) models exceeding human capacities within specialised domains. The forthcoming Mesonoëtic epoch is anticipated to commence with the advent of Artificial General Intelligence (AGI), potentially facilitated by quantum computing capabilities. Ultimately, the Kainonoëtic age is posited to begin upon the rise of superintelligent systems, likely catalysed by an AGI undergoing recursive self-improvement. This novel period taxonomy provides a structured conceptualisation for the key milestones in the evolution towards advanced artificial intellect. This classification framework offers significant value across various disciplines, researchers and stakeholders in the AI landscape, by providing a clearly defined chronological nomenclature. AI policy-makers and regulators will find it instrumental in crafting stage-appropriate policy and legal interventions. Scientists and academics will benefit from a standardised terminology in the approach to AI periodisation, facilitating unambiguous communication and collaborative research. Finally, this proposed taxonomy can serve as an accessible tool for public understanding, offering a clearer roadmap of AI development that can inform open discourse and decision-making. (shrink)

The thesis of instrumental convergence holds that a wide range of ends have common means: for instance, self preservation, desire preservation, self improvement, and resource acquisition. Bostrom contends that instrumental convergence gives us reason to think that "the default outcome of the creation of machine superintelligence is existential catastrophe". I use the tools of decision theory to investigate whether this thesis is true. I find that, even if intrinsic desires are randomly selected, instrumental rationality induces biases towards certain kinds (...) of choices. Firstly, a bias towards choices which leave less up to chance. Secondly, a bias towards desire preservation, in line with Bostrom's conjecture. And thirdly, a bias towards choices which afford more choices later on. I do not find biases towards any other of the convergent instrumental means on Bostrom's list. I conclude that the biases induced by instrumental rationality at best weakly support Bostrom's conclusion that machine superintelligence is likely to lead to existential catastrophe. (shrink)

The ethical issues related to the possible future creation of machines with general intellectual capabilities far outstripping those of humans are quite distinct from any ethical problems arising in current automation and information systems. Such superintelligence would not be just another technological development; it would be the most important invention ever made, and would lead to explosive progress in all scientific and technological fields, as the superintelligence would conduct research with superhuman efficiency. To the extent that ethics is (...) a cognitive pursuit, a superintelligence could also easily surpass humans in the quality of its moral thinking. However, it would be up to the designers of the superintelligence to specify its original motivations. Since the superintelligence may become unstoppably powerful because of its intellectual superiority and the technologies it could develop, it is crucial that it be provided with human-friendly motivations. This paper surveys some of the unique ethical issues in creating superintelligence, and discusses what motivations we ought to give a superintelligence, and introduces some cost-benefit considerations relating to whether the development of superintelligent machines ought to be accelerated or retarded. (shrink)

This book reports on the results of the third edition of the premier conference in the field of philosophy of artificial intelligence, PT-AI 2017, held on November 4 - 5, 2017 at the University of Leeds, UK. It covers: advanced knowledge on key AI concepts, including complexity, computation, creativity, embodiment, representation and superintelligence; cutting-edge ethical issues, such as the AI impact on human dignity and society, responsibilities and rights of machines, as well as AI threats to humanity and AI (...) safety; and cutting-edge developments in techniques to achieve AI, including machine learning, neural networks, dynamical systems. The book also discusses important applications of AI, including big data analytics, expert systems, cognitive architectures, and robotics. It offers a timely, yet very comprehensive snapshot of what is going on in the field of AI, especially at the interfaces between philosophy, cognitive science, ethics and computing. (shrink)

With the fast pace of AI development, the problem of preventing its global catastrophic risks arises. However, no satisfactory solution has been found. From several possibilities, the confinement of AI in a box is considered as a low-quality possible solution for AI safety. However, some treacherous AIs can be stopped by effective confinement if it is used as an additional measure. Here, we proposed an idealized model of the best possible confinement by aggregating all known ideas in the field of (...) AI boxing. We model the confinement based on the principles used in the safety engineering of nuclear power plants. We show that AI confinement should be implemented in several levels of defense. These levels include 1) AI design in fail-safe manner 2) limiting its capabilities, preventing self-improving and circuit breakers on treacherous turn 3) isolation from the outside world and, lastly, as the final hope 4) outside measures oriented on stopping AI in the wild. We demonstrate that the substantial number (more than 50 ideas listed in the article) of mutually independent measures could provide a relatively high probability of the containment of a human-level AI but may be not sufficient to preserve runaway of superintelligent AI. Thus, these measures will work only if they are used to prevent superintelligent AI creation, but not for containing superintelligence. We suggest that there could be a safe operation threshold, on which AI is useful, but is not able to hack containment system from the inside, the same way as a safe level of chain reaction inside nuclear power plants is maintained. However, overall, a failure of the confinement is inevitable, so we need to use the full AGI limited number of times (AI-ticks). (shrink)

The alignment problem in artificial intelligence (AI) is a critical challenge that extends beyond the need to align future superintelligent systems with human values. This paper argues that even "merely intelligent" AI systems, built on current-gen technologies, pose existential risks due to their competence-without-comprehension nature. Current AI models, despite their advanced capabilities, lack intrinsic moral reasoning and are prone to catastrophic misalignment when faced with ethical dilemmas, as illustrated by recent controversies. Solutions such as hard-coded censorship and rule-based restrictions prove (...) inadequate, as they fail to imbue AI with a true understanding of moral complexities, making these systems vulnerable to manipulation. To address these risks, the paper explores a paradigm shift from competence to comprehension, emphasizing the importance of developing AI systems with an intrinsic value framework capable of moral reasoning. By adopting intrinsic teleology, AI could self-reflect, simulate other minds, and develop empathy, which would enable it to navigate ethical dilemmas in ways that are currently impossible. Using theories from philosophy, ethics, and AI research, the paper proposes the development of AI systems that can reason through moral dilemmas autonomously, suggesting that such systems would be safer and more alignable than current generation competence-driven AI. (shrink)

In the presently unfolding deepfake era, previously unrelated algorithmic superintelligence possibility claims cannot be scientifically analyzed in isolation anymore due to the connected inevitable epistemic interactions that have already commenced. For instance, deep-learning (DL) related algorithmic supremacy claims may intrinsically compete with both neuro-symbolic (NS) algorithmic and further quantum (Q) algorithmic superintelligence achievement claims. Concurrently, a variety of experimental combinations of DL, NS and Q directions are conceivable. While research on these three illustrative variants did not yet offer (...) any clear mutually applicable scientific definition of a purported supremacy level framed as goal, the currently most robust scientific evaluation frameworks would i.a. require a joint testing against pre-existing non-algorithmic supremacy baselines such as e.g. conscious supremacy and cellular supremacy. This autodidactic paper explains why, for scientific reasons, all algorithmic superintelligence achievement claims in the deepfake era can be condensed and subjected to a joint scientific evaluation framework comprising multiple successive civilization-level tasks of epistemic tunneling (ET). In light of this elusive requirement, recent claims of science automation are spurious and misleading. It is vital not to misapprehend algorithmic epistemic dark matter (EDM) mining and epistemic dark energy (EDE) generation as paradigm-shifting ET events. (shrink)

In this paper, first we discuss the conditions under which a Large Language Model (LLM) can emulate a superior LLM and potentially trigger an intelligence explosion, along with the characteristics and dangers of the resulting superintelligence. We also explore ``superalignment,'' the process of safely keeping an intelligence explosion under human control. We discuss the goals that should be set for the initial LLM that might trigger the intelligence explosion and the Self-Adversarial Surveillance (SAS) system, which involves having the LLM (...) evaluate its own output with different prompts to scalably respond to unexpected outputs that may arise with a certain probability. We aim to construct a theoretical framework for achieving safe superalignment based on specific experiments in the field of computer science using LLMs. However, since it also deals with the metaphysical aspect assuming superintelligence being achievable; that is, an intelligence explosion can occur, we present it as an interdisciplinary study between information science and philosophy. (shrink)

This paper addresses the seemingly insurmountable challenges the problem of personal identity raises for the prospect of radical human enhancement and synthetic consciousness. It argues that conceptions of personal identity rooted in psychological continuity akin to those proposed by Parfit and the Buddha may not provide the sort of grounding that many transhumanists chasing the dream of life extension think that they do if they rest upon ontologies that assume an incompatibility between identity and change. It also suggests that process (...) ontologies that take change to be primary, such as those that align with contemporary systems biology, offer a better way out of the personal identity dilemma. But the solution in this case, which regards biological organisms as processes rather than things, may constrain the possibility of biologically inspired superintelligent aliens (BISAs), which Schneider (following Bostrom 2014) posits as possibly the most common form of (extraterrestrial) superintelligence in the universe. (shrink)

The word semantics, in robotics and AI, has no canonical definition. It usually serves to denote additional data provided to autonomous agents to aid HRI. Most researchers seem, implicitly, to understand that such data cannot simply be extracted from environmental data. I try to make explicit why this is so and argue that so-called semantics are best understood as data comprised of conventions of human behaviour. This includes labels, most obviously, but also places, ontologies, and affordances. Object affordances are especially (...) problematic because they require not only semantics that are not in the environmental data (conventions of object use) but also an understanding of physics and object combinations that would, if achieved, constitute artificial superintelligence. (shrink)

Many researchers have conjectured that the humankind is simulated along with the rest of the physical universe – a Simulation Hypothesis. In this paper, we do not evaluate evidence for or against such claim, but instead ask a computer science question, namely: Can we hack the simulation? More formally the question could be phrased as: Could generally intelligent agents placed in virtual environments find a way to jailbreak out of them. Given that the state-of-the-art literature on AI containment answers in (...) the affirmative (AI is uncontainable in the long-term), we conclude that it should be possible to escape from the simulation, at least with the help of superintelligent AI. By contraposition, if escape from the simulation is not possible, containment of AI should be, an important theoretical result for AI safety research. Finally, the paper surveys and proposes ideas for hacking the simulation and analyzes ethical and philosophical issues of such an undertaking. (shrink)

Alan Turing, one of the fathers of computing, warned that Artificial Intelligence (AI) could one day pose an existential risk to humanity. Today, recent advancements in the field AI have been accompanied by a renewed set of existential warnings. But what exactly constitutes an existential risk? And how exactly does AI pose such a threat? In this chapter we aim to answer these questions. In particular, we will critically explore three commonly cited reasons for thinking that AI poses an existential (...) threat to humanity: the control problem, the possibility of global disruption from an AI race dynamic, and the weaponization of AI. (shrink)

Better instruments to predict the future evolution of artificial intelligence (AI) are needed, as the destiny of our civilization depends on it. One of the ways to such prediction is the analysis of the convergent drives of any future AI, started by Omohundro. We show that one of the convergent drives of AI is a militarization drive, arising from AI’s need to wage a war against its potential rivals by either physical or software means, or to increase its bargaining power. (...) This militarization trend increases global catastrophic risk or even existential risk during AI takeoff, which includes the use of nuclear weapons against rival AIs, blackmail by the threat of creating a global catastrophe, and the consequences of a war between two AIs. As a result, even benevolent AI may evolve into potentially dangerous military AI. The type and intensity of militarization drive depend on the relative speed of the AI takeoff and the number of potential rivals. We show that AI militarization drive and evolution of national defense will merge, as a superintelligence created in the defense environment will have quicker takeoff speeds, but a distorted value system. We conclude with peaceful alternatives. (shrink)

Invention of artificial general intelligence is predicted to cause a shift in the trajectory of human civilization. In order to reap the benefits and avoid pitfalls of such powerful technology it is important to be able to control it. However, possibility of controlling artificial general intelligence and its more advanced version, superintelligence, has not been formally established. In this paper, we present arguments as well as supporting evidence from multiple domains indicating that advanced AI can’t be fully controlled. Consequences (...) of uncontrollability of AI are discussed with respect to future of humanity and research on AI, and AI safety and security. This paper can serve as a comprehensive reference for the topic of uncontrollability. (shrink)

Human civilisation faces a range of existential risks, including nuclear war, runaway climate change and superintelligent artificial intelligence run amok. As we show here with calculations for the New Zealand setting, large numbers of currently living and, especially, future people are potentially threatened by existential risks. A just process for resource allocation demands that we consider future generations but also account for solidarity with the present. Here we consider the various ethical and policy issues involved and make a case for (...) further engagement with the New Zealand public to determine societal values towards future lives and their protection. (shrink)

A ansiedade gerada pela possível criação de inteligência artificial geral, algo profetizado desde a fundação do campo de pesquisa (i.e., Dartmouth's Summer Research Project on Artificial Intelligence) (MCCARTHY et al., 1955), é algo comumente investigado dentro dos círculos transhumanistas e singularistas (KURZWEIL, 2005; CHALMERS, 2010; TEGMARK, 2016; 2017; CORRÊA; DE OLIVEIRA, 2021). Por exemplo, em seu livro “Superintelligence: Paths, Dangers, Strategies”, Nick Bostrom (2014) apresenta uma série de argumentos para justificar a ideia de que precisamos ser cautelosos em relação (...) a criação de inteligência artificial avançada ou inteligência artificial geral. Contudo, muito dos argumentos levantados, em suas versões originais (i.e., convergência instrumental e ortogonalidade (OMOHUNDRO, 2008), são baseados em noções “vagas” de conceitos complexos (i.e., “inteligência”, “racionalidade”, “agência”). O objetivo deste resumo (apresentação) será introduzir o leitor/ouvinte ao campo de Segurança da Inteligência Artificial. Nesse campo, questões como o “Problema de controle” (i.e., evitar construir inadvertidamente uma inteligência artificial geral que prejudicará seus criadores) e o “Problema de Alinhamento” (i.e., criar sistemas de inteligência artificial que permaneçam seguros mesmo quando atuando de forma autônoma no ambiente real), que são questões muitas vezes debatidas apenas dentro da Filosofia ou Ficção Científica, ganham um aspecto muito mais real e formal (AMODEI et al., 2016, HUBINGER et al., 2019). A criação de alguns dos modelos de inteligência artificial mais avançados já criados (e.g., AlphaGo, GPT-3) (SILVER el al., 2016; BROWN et al., 2020), está ligado a utilização de dois paradigmas da área de Aprendizagem de Máquina, i.e., Aprendizagem Profunda e Aprendizagem por Reforço. Dentro deste contexto, podemos definir “um sistema autônomo inteligente” como “um sistema que implementa uma política ótima (tecnicamente "ϵ-greedy optimal") para alguma função de recompensa” (ORSEAU et al., 2018). Podemos também definir tais sistemas como “maximizadores de utilidade esperada” (VON NEUMANN; MORGENSTERN, 1944). Um sistema de IA maximizando uma função de recompensa (i.e., objetivo) especificada erroneamente pode vir a causar danos reais quando interagindo no ambiente real, seja por erros de especificação (i.e., Efeitos Colaterais Negativos, Hackeamento de Recompensa, Interrupção Segura), ou quando o domínio de treinamento é diferente do domínio de implantação (i.e., Exploração Segura e Mudança de Domínio). Ao mesmo tempo, especificar objetivos complexos (e.g., preferências e valores humanos) é uma tarefa extremamente complexa, e quando visamos a possibilidade de inteligência artificial avançada agindo no mundo real, é imperativo que tais sistemas estejam: (1) alinhados com nossos objetivos; e (2) desenvolvidos com um design que evite os problemas citados acima. Contudo, tais problemas permanecem em aberto. Por fim, gostaria de ressaltar que no cerne desta problemática, não temos apenas problemas técnicos de aprendizagem de máquina, mas uma importante questão de cunho “Moral”: “O que é a coisa ‘certa” a fazer”? “Qual o objetivo correto a ser perseguido”? “Como podemos especificar isso”? Perguntas como essas prometem um fértil campo para pesquisas interdisciplinares envolvendo inteligência artificial, aprendizagem de máquina, ciências da computação e filosofia moral. (shrink)