In 2000 Bill Joy proposed that the best way to prevent technological apocalypse was to "relinquish" emerging bio-, info- and nanotechnologies. His essay introduced many watchdog groups to the dangers that futurists had been warning of for decades. One such group, ETC, has called for a moratorium on all nanotechnological research until all safety issues can be investigated and social impacts ameliorated. In this essay I discuss the differences and similarities of regulating bio- and nanotechnological innovation to the efforts to (...) regulate nuclear and biological weapons of mass destruction. I then suggest the creation of a global technology regulatory regime to ensure the safe and equitable diffusion of genetic, molecular and information technologies, and point out the principal political obstacles to implementing such a regime. (shrink)

The use of advanced technologies, even of so-called transhuman technology, does not make militaries transhuman. Transhumanism includes dimensions of ethics that are themselves in direct conflict with many transhuman capabilities of soldiers in warfare. The use of advanced weapons of mass destruction represents an anti-humanism that undermines the modern, open, and high-tech nation state.

I survey four categories of factors that might give a digital mind, such as an upload or an artificial general intelligence, an advantage over humans. Hardware advantages include greater serial speeds and greater parallel speeds. Self-improvement advantages include improvement of algorithms, design of new mental modules, and modification of motivational system. Co-operative advantages include copyability, perfect co-operation, improved communication, and transfer of skills. Human handicaps include computational limitations and faulty heuristics, human-centric biases, and socially motivated cognition. The shape of hardware (...) growth curves, as well as the ease of modifying minds, are found to have a major impact on how quickly a digital mind may take advantage of these factors. (shrink)

There is no strong reason to believe that human-level intelligence represents an upper limit of the capacity of artificial intelligence, should it be realized. This poses serious safety issues, since a superintelligent system would have great power to direct the future according to its possibly flawed motivation system. Solving this issue in general has proven to be considerably harder than expected. This paper looks at one particular approach, Oracle AI. An Oracle AI is an AI that does not act in (...) the world except by answering questions. Even this narrow approach presents considerable challenges. In this paper, we analyse and critique various methods of controlling the AI. In general an Oracle AI might be safer than unrestricted AI, but still remains potentially dangerous. (shrink)

The human desire to acquire new capacities is as ancient as our species itself. We have always sought to expand the boundaries of our existence, be it socially, geographically, or mentally. There is a tendency in at least some individuals always to search for a way around every obstacle and limitation to human life and happiness.

Scientific research is subject to a number of regulations which impose incidental (time, place), rather than substantive (type of research), restrictions on scientific research and the knowledge created through such research. In recent years, however, the premise that scientific research and knowledge should be free from substantive regulation has increasingly been called into question. Some have suggested that the law should be used as a tool to substantively restrict research which is dual-use in nature or which raises moral objections. There (...) are, however, some problems with using law to restrict or prohibit certain types of scientific research, including (i) the inherent imprecision of law for regulating complex and rapidly evolving scientific research; (ii) the difficulties of enforcing legal restrictions on an activity that is international in scope; (iii) the limited predictability of the consequences of restricting specific branches of scientific research; (iv) inertia in the legislative process; and (v) the susceptibility of legislators and regulators to inappropriate factors and influence. Rather than using law to restrict scientific research, it may be more appropriate and effective to use a combination of non-traditional legal tools including norms, codes of conduct, restrictions on publication, and scientist-developed voluntary standards to regulate problematic scientific research. (shrink)

Effective altruism has a strategy problem. Overreliance on a strategy of donating to the most effective charities keeps us on the firefighter's treadmill, continually pursuing the next-highest quantifiable marginal gain. But on its own, this is politically shortsighted. Without any long-term framework within which these individual rescues fit together to bring about the greatest overall impact, we are almost certainly leaving a lot of value on the table. Thus, effective altruists' preferred means undercut their professed aims. Alongside the charity framework, (...) the more effective altruist ought to consider a mutual aid framework, which better acknowledges and honors the unavoidably political commitments of effective altruism to reimagine and remake the world. (shrink)

We argue that two long-term goals of AI research stand in tension with one another. The first involves creating AI that is safe, where this is understood as solving the problem of value alignment. The second involves creating artificial general intelligence, meaning AI that operates at or beyond human capacity across all or many intellectual domains. Our argument focuses on the human capacity to make what we call “existential choices”, choices that transform who we are as persons, including transforming what (...) we most deeply value or desire. It is a capacity for a kind of value misalignment, in that the values held prior to making such choices can be significantly different from (misaligned with) the values held after making them. Because of the connection to existentialist philosophers who highlight these choices, we call the resulting form of risk “existentialist risk.” It is, roughly, the risk that results from AI taking an active role in authoring its own values rather than passively going along with the values given to it. On our view, human-like intelligence requires a human-like capacity for value misalignment, which is in tension with the possibility of guaranteeing value alignment between AI and humans. (shrink)

Abstract. In this article is presented a model of the change of the probability of the global catastrophic risks in the world with exponentially evolving technologies. Increasingly cheaper technologies become accessible to a larger number of agents. Also, the technologies become more capable to cause a global catastrophe. Examples of such dangerous technologies are artificial viruses constructed by the means of synthetic biology, non-aligned AI and, to less extent, nanotech and nuclear proliferation. The model shows at least double exponential growth (...) of the probability of the global catastrophe which means that the accumulated probability of the catastrophe will grow from negligible to overwhelming at the period of a few doubling times of the technological capabilities. For biotech and AI, such doubling time roughly corresponds to the doubling period of Moore’s law and its analogues in other technologies and is around 2 years. Thus the global catastrophe in the exponential technologies world will most likely happen during a “dangerous decade”. Such a dangerous decade could start as early as in the 2020s. We also found that the double exponential growth in the model makes the model less sensitive to its initial conditions, like the number of dangerous agents or initial probabilities constants. The model also shows that smaller catastrophes could happen earlier than larger ones, and such a smaller catastrophe may be able to stop the technological growth before larger ones become possible, thus global risks will be self-limiting. However, if the growth of the number of dangerous agents will be very quick, the multiple smaller catastrophes could happen simultaneously and will be equal to a global catastrophe. (shrink)

Climate change is often described as an existential risk to the human species, but this terminology has generally been avoided in the climate-justice literature in analytic philosophy. I investigate the source of this disconnect and explore the prospects for incorporating the idea of climate change as an existential risk into debates about climate justice. The concept of existential risk does not feature prominently in these discussions, I suggest, because assumptions that structure ‘ideal’ accounts of climate justice ensure that the prospect (...) of climate change as an extinction-level threat does not arise. Given persistent noncompliance with mitigation duties, however, we have reason to revisit these assumptions. I argue that the most promising way for theories of climate justice to account for the significance of existential risk is to look to the practices of protest and resistance in which a concern about extinction or global catastrophe is frequently given expression. (shrink)

When discussing the extinction of humanity, there does not seem to be any clear agreement about what ‘humanity’ really means. One aim of this paper is to show that it is a more slippery concept than it might at first seem. A second aim is to show the relationship between what constitutes or defines humanity and what gives it value. Often, whether and how we ought to prevent human extinction depends on what we take humanity to mean, which in turn (...) depends on what we value about humans. Finally, I will offer a definition of humanity that I argue offers a plausible account of when we might say humanity has gone extinct, and reflects what we value about humans, and relatedly, what many actually care about in terms of why we should be preserved. (shrink)

In this chapter we take stock of some of the complexities of the sprawling field of AI ethics. We consider questions like "what is the proper scope of AI ethics?" And "who counts as an AI ethicist?" At the same time, we flag several potential uses and abuses of AI ethics. These include challenges for the AI ethicist, including what qualifications they should have; the proper place and extent of futuring and speculation in the field; and the dilemmas concerning how (...) we use our public and academic resources. (shrink)

Ikigai is a Japanese concept, which, in brief, refers to the “reason or purpose to live”. I-risks have been identified as a category of risks complementing x- risks, i.e., existential risks, and s-risks, i.e., suffering risks, which describes undesirable future scenarios in which humans are deprived of the pursuit of their individual ikigai. While some developments in AI increase i-risks, there are also AI-driven virtual opportunities, which reduce i-risks by increasing the space of potential ikigais, largely due to developments in (...) generative AI, virtual worlds as well as AI-driven hyper-personalization. The purpose of this paper is to present a first attempt to map the potential AI-driven virtual hyper-personalised ikigai universe. Moreover, challenges and further ideas are presented. (shrink)

Extinction is a concept of rapidly growing importance, with the world currently in the sixth mass extinction event and a biodiversity crisis. However, the concept of extinction has itself received surprisingly little attention from philosophers. I will first argue that in practice there is no single unified concept of extinction, but instead that its usage divides between descriptive, epistemic, and declarative concepts. I will then consider the epistemic challenges that arise in ascertaining whether a species has gone extinct, and how (...) these lead to serious limitations on responsibility and accountability for preventing extinctions. I will propose two conceptual engineering changes to our understanding of extinction. Firstly, to use twin epistemic concepts of extinction, corresponding to high and low epistemic demands. Secondly, to explicitly recognize that in many contexts extinction is a thick concept that contains an evaluative component, which therefore motivates explicit ethical considerations such as the use of precautionary principles. (shrink)

This paper analyzes the political, philosophical, societal, legal, educational, biological, psychological and technological reasons why there is an urgent need for basic intercultural and interfaith ethics in the world and whether it is possible to formulate a valid code of such ethics. It is shown that universal ethics could be founded on natural law, which can be understood in both religious and secular ways. Alternatively, universal ethics could be based on a single supreme principle that is independent of worldview and (...) culture: human dignity. In accordance with these concepts, a minimalist and normative code of essential, self-evident universal ethical principles and norms is proposed. The implementation of universal ethics in society is a long-term political task that could be achieved by including universal ethics in the compulsory school curriculum of all countries and in the UNESCO agenda of Global Citizen Education. (shrink)

In general, existential threats are those that may potentially result in the extinction of the entire human species, if not significantly endanger its living population. Among the said threats include, but not limited to, pandemics and the impacts of a technological singularity. As regards pandemics, significant work has already been done on how to mitigate, if not prevent, the aftereffects of this type of disaster. For one, certain problem areas on how to properly manage pandemic responses have already been identified, (...) like the following: (a) not being able to learn from previous experiences, (b) the inability to act on warning signals, and (c) the failure to reach a global consensus on a problem (i.e., in a timely manner). In terms of a singularity, however, it may be said that further research is still needed, specifically on how to aptly respond to its projected negative outcomes. In this paper, by treating the three problem areas noted above as preliminary assessment measures of a country’s capacity to coordinate a national response to large-scale disasters, we examine the readiness of the Philippines in preparing for an intelligence explosion. By citing certain instances of how the said country, specifically its national government, faced the coronavirus disease 2019 pandemic, it puts forward the idea that the likely Philippine disaster response towards a singularity needs to be worked on, appealing for a more comprehensive assessment of such for a more informed response plan. (shrink)

Enlightenment values are built around the presumption of an independent rational self, citizen, consumer and pursuer of self-interest. Even the authoritarian and communitarian variants of the Enlightenment presumed the existence of autonomous individuals, simply arguing for greater weight to be given to their collective interests. Since Hume, however, radical Enlightenment empiricists have called into question the existence of a discrete, persistent self. Today neuroscientific reductionism has contributed to the rejection of an essentialist model of personal identity. Contemporary transhumanism has yet (...) to grapple with the radical consequences of the erosion of liberal individualism on their projects of individually chosen enhancement and longevity. Most transhumanists still reflect an essentialist idea of personal identity, even as they embrace projects of radical cognitive enhancement that would change every constituent element of consciousness. Transhumanists need to grapple with how their projects and ethics would change if personal identity is an arbitrary, malleable fiction. (shrink)

The standard argument to the conclusion that artificial intelligence (AI) constitutes an existential risk for the human species uses two premises: (1) AI may reach superintelligent levels, at which point we humans lose control (the ‘singularity claim’); (2) Any level of intelligence can go along with any goal (the ‘orthogonality thesis’). We find that the singularity claim requires a notion of ‘general intelligence’, while the orthogonality thesis requires a notion of ‘instrumental intelligence’. If this interpretation is correct, they cannot be (...) joined as premises and the argument for the existential risk of AI turns out invalid. If the interpretation is incorrect and both premises use the same notion of intelligence, then at least one of the premises is false and the orthogonality thesis remains itself orthogonal to the argument to existential risk from AI. In either case, the standard argument for existential risk from AI is not sound.—Having said that, there remains a risk of instrumental AI to cause very significant damage if designed or used badly, though this is not due to superintelligence or a singularity. (shrink)

This essay interrogates the common assumption that good care is necessarily human care. It looks to disruptive fictional representations of robot care to assist its development of a theory of posthuman care that jettisons the implied anthropocentrism of ethics of care philosophy but retains care’s foregrounding of entanglement, embodiment and obligation. The essay reads speculative representations of robot care, particularly the Swedish television programme Äkta människor (Real Humans), alongside ethics of care philosophy and critical posthumanism to highlight their synergetic critiques (...) of neoliberal affective economies and humanist hierarchies that treat some bodies and affects as more real than others. These texts and discourses assist me in proposing a theory of care that regards vulnerability as the normative effect of posthuman vital embodiment, as opposed to an anomalous state that can be overcome or corrected via neoliberal practice. (shrink)

In the philosophy of artificial intelligence (AI) we are often warned of machines built with the best possible intentions, killing everyone on the planet and in some cases, everything in our light cone. At the same time, however, we are also told of the utopian worlds that could be created with just a single superintelligent mind. If we’re ever to live in that utopia (or just avoid dystopia) it’s necessary we solve the control problem. The control problem asks how humans (...) would be able to control an AI arbitrarily. Nick Bostrom and other AI researchers have proposed different theoretical solutions to the control problem. In this paper, I will not look at the empirical question of how to solve the control problem. Instead, I will ask if we can solve it at all, a critical assumption most AI researchers have made is that we can. I propose, in fact, that we have a priori grounds for believing it is logically impossible to solve the control problem, since all superintelligent minds are, by definition, uncontrollable. (shrink)

What does Cyberpunk and AI Ethics have to do with each other? Cyberpunk is a sub-genre of science fiction that explores the post-human relationships between human experience and technology. One similarity between AI Ethics and Cyberpunk literature is that both seek a dialogue in which the reader may inquire about the future and the ethical and social problems that our technological advance may bring upon society. In recent years, an increasing number of ethical matters involving AI have been pointed and (...) debated, and several ethical principles and guides have been suggested as governance policies for the tech industry. However, would this be the role of AI Ethics? To serve as a soft and ambiguous version of the law? I would like to promote in this article a more Cyberpunk way of doing AI Ethics, whit a more anarchic way of governance. In this study, I will seek to expose some of the deficits of the underlying power structures of our society, and suggest that AI governance be subject to public opinion, so that ‘good AI’ can become ‘good AI for all’. (shrink)

In the fields of artificial intelligence and robotics, the term “autonomy” is generally used to mean the capacity of an artificial agent to operate independently of human guidance. It is thereby assumed that the agent has a fixed goal or “utility function” with respect to which the appropriateness of its actions will be evaluated. From a philosophical perspective, this notion of autonomy seems oddly weak. For, in philosophy, the term is generally used to refer to a stronger capacity, namely the (...) capacity to “give oneself the law,” to decide by oneself what one’s goal or principle of action will be. The predominant view in the literature on the long-term prospects and risks of artificial intelligence is that an artificial agent cannot exhibit such autonomy because it cannot rationally change its own final goal, since changing the final goal is counterproductive with respect to that goal and hence undesirable. The aim of this paper is to challenge this view by showing that it is based on questionable assumptions about the nature of goals and values. I argue that a general AI may very well come to modify its final goal in the course of developing its understanding of the world. This has important implications for how we are to assess the long-term prospects and risks of artificial intelligence. (shrink)

Many researchers have argued that humanity will create artificial general intelligence (AGI) within the next twenty to one hundred years. It has been suggested that AGI may inflict serious damage to human well-being on a global scale ('catastrophic risk'). After summarizing the arguments for why AGI may pose such a risk, we review the fieldʼs proposed responses to AGI risk. We consider societal proposals, proposals for external constraints on AGI behaviors and proposals for creating AGIs that are safe due to (...) their internal design. (shrink)

The likely near future creation of artificial superintelligence carries significant risks to humanity. These risks are difficult to conceptualise and quantify, but malicious use of existing artificial intelligence by criminals and state actors is already occurring and poses risks to digital security, physical security and integrity of political systems. These risks will increase as artificial intelligence moves closer to superintelligence. While there is little research on risk management tools used in artificial intelligence development, the current global standard for risk management, (...) ISO 31000:2018, is likely used extensively by developers of artificial intelligence technologies. This paper argues that risk management has a common set of vulnerabilities when applied to artificial superintelligence which cannot be resolved within the existing framework and alternative approaches must be developed. Some vulnerabilities are similar to issues posed by malicious threat actors such as professional criminals and terrorists. Like these malicious actors, artificial superintelligence will be capable of rendering mitigation ineffective by working against countermeasures or attacking in ways not anticipated by the risk management process. Criminal threat management recognises this vulnerability and seeks to guide and block the intent of malicious threat actors as an alternative to risk management. An artificial intelligence treachery threat model that acknowledges the failings of risk management and leverages the concepts of criminal threat management and artificial stupidity is proposed. This model identifies emergent malicious behaviour and allows intervention against negative outcomes at the moment of artificial intelligence’s greatest vulnerability. (shrink)

In this essay I will explore an understanding of the potential moral agency of robots; arguing that the key characteristics of physical embodiment; adaptive learning; empathy in action; and a teleology toward the good are the primary necessary components for a machine to become a moral agent. In this context; other possible options will be rejected as necessary for moral agency; including simplistic notions of intelligence; computational power; and rule-following; complete freedom; a sense of God; and an immaterial soul. I (...) argue that it is likely that such moral machines may be able to be built; and that this does not diminish humanity or human personhood. (shrink)

A probable solution of the Fermi paradox; and a necessary step in humanity’s cosmic development; is the construction of a Bracewell-Von Neumann Artificial General Intelligence. The use of BN probes is the most plausible method of initial galactic exploration and communication for advanced ET civilizations; and our own cosmic evolution lies firmly in the utilization of; and cooperation with; AGI agents. To establish these claims; I explore the most credible developmental path from carbon-based life forms to planetary civilizations and AI (...) creation. I consider the likely physical characteristics of extraterrestrial AI probes and propose ways to predict their behavior. Lastly; I ponder the possible trajectories for humanity’s cosmic future. (shrink)

The greatest existential threats to humanity stem from increasingly powerful advanced technologies. Yet the “risk potential” of such tools can only be realized when coupled with a suitable agent who; through error or terror; could use the tool to bring about an existential catastrophe. While the existential risk literature has provided many accounts of how advanced technologies might be misused and abused to cause unprecedented harm; no scholar has yet explored the other half of the agent-tool coupling; namely the agent. (...) This paper aims to correct this failure by offering a comprehensive overview of what we could call “agential riskology.” Only by studying the unique properties of different agential risk types can one acquire an accurate picture of the existential danger before us. (shrink)

Scientific and technological progress might change people’s capabilities or incentives in ways that would destabilize civilization. For example, advances in DIY biohacking tools might make it easy for anybody with basic training in biology to kill millions; novel military technologies could trigger arms races in which whoever strikes first has a decisive advantage; or some economically advantageous process may be invented that produces disastrous negative global externalities that are hard to regulate. This paper introduces the concept of a vulnerable world: (...) roughly, one in which there is some level of technological development at which civilization almost certainly gets devastated by default, i.e. unless it has exited the “semi-anarchic default condition”. Several counterfactual historical and speculative future vulnerabilities are analyzed and arranged into a typology. A general ability to stabilize a vulnerable world would require greatly amplified capacities for preventive policing and global governance. The vulnerable world hypothesis thus offers a new perspective from which to evaluate the risk-benefit balance of developments towards ubiquitous surveillance or a unipolar world order. (shrink)

Historically, the hypothesis that our world is a computer simulation has struck many as just another improbable-but-possible “skeptical hypothesis” about the nature of reality. Recently, however, the simulation hypothesis has received significant attention from philosophers, physicists, and the popular press. This is due to the discovery of an epistemic dependency: If we believe that our civilization will one day run many simulations concerning its ancestry, then we should believe that we are probably in an ancestor simulation right now. This essay (...) examines a troubling but underexplored feature of the ancestor-simulation hypothesis: the termination risk posed by both ancestor-simulation technology and experimental probes into whether our world is an ancestor simulation. This essay evaluates the termination risk by using extrapolations from current computing practices and simulation technology. The conclusions, while provisional, have great implications for debates concerning the fundamental nature of reality and the safety of contemporary physics. (shrink)

A classification of the global catastrophic risks of AI is presented, along with a comprehensive list of previously identified risks. This classification allows the identification of several new risks. We show that at each level of AI’s intelligence power, separate types of possible catastrophes dominate. Our classification demonstrates that the field of AI risks is diverse, and includes many scenarios beyond the commonly discussed cases of a paperclip maximizer or robot-caused unemployment. Global catastrophic failure could happen at various levels of (...) AI development, namely, before it starts self-improvement, during its takeoff, when it uses various instruments to escape its initial confinement, or after it successfully takes over the world and starts to implement its goal system, which could be plainly unaligned, or feature-flawed friendliness. AI could also halt at later stages of its development either due to technical glitches or ontological problems. Overall, we identified around several dozen scenarios of AI-driven global catastrophe. The extent of this list illustrates that there is no one simple solution to the problem of AI safety, and that AI safety theory is complex and must be customized for each AI development level. (shrink)

Recently many methods for reducing the risk of human extinction have been suggested, including building refuges underground and in space. Here we will discuss the perspective of using military nuclear submarines or their derivatives to ensure the survival of a small portion of humanity who will be able to rebuild human civilization after a large catastrophe. We will show that it is a very cost-effective way to build refuges, and viable solutions exist for various budgets and timeframes. Nuclear submarines are (...) surface independent, and could provide energy, oxygen, fresh water and perhaps even food for their inhabitants for years. They are able to withstand close nuclear explosions and radiation. They are able to maintain isolation from biological attacks and most known weapons. They already exist and need only small adaptation to be used as refuges. But building refuges is only “Plan B” of existential risk preparation; it is better to eliminate such risks than try to survive them. (shrink)

Discussions about the possible consequences of creating superintelligence have included the possibility of existential risk, often understood mainly as the risk of human extinction. We argue that suffering risks (s-risks) , where an adverse outcome would bring about severe suffering on an astronomical scale, are risks of a comparable severity and probability as risks of extinction. Preventing them is the common interest of many different value systems. Furthermore, we argue that in the same way as superintelligent AI both contributes to (...) existential risk but can also help prevent it, superintelligent AI can both be a suffering risk or help avoid it. Some types of work aimed at making superintelligent AI safe will also help prevent suffering risks, and there may also be a class of safeguards for AI that helps specifically against s-risks. (shrink)

As the aim of the responsible robotics initiative is to ensure that responsible practices are inculcated within each stage of design, development and use, this impetus is undergirded by the alignment of ethical and legal considerations towards socially beneficial ends. While every effort should be expended to ensure that issues of responsibility are addressed at each stage of technological progression, irresponsibility is inherent within the nature of robotics technologies from a theoretical perspective that threatens to thwart the endeavour. This is (...) because the concept of responsibility, despite being treated as such, is not monolithic: rather this seemingly unified concept consists of converging and confluent concepts that shape the idea of what we colloquially call responsibility. From a different perspective, robotics will be simultaneously responsible and irresponsible depending on the particular concept of responsibility that is foregrounded: an observation that cuts against the grain of the drive towards responsible robotics. This problem is further compounded by responsible design and development as contrasted to responsible use. From a different perspective, the difficulty in defining the concept of responsibility in robotics is because human responsibility is the main frame of reference. Robotic systems are increasingly expected to achieve the human-level performance, including the capacities associated with responsibility and other criteria which are necessary to act responsibly. This subsists within a larger phenomenon where the difference between humans and non-humans, be it animals or artificial systems, appears to be increasingly blurred, thereby disrupting orthodox understandings of responsibility. This paper seeks to supplement the responsible robotics impulse by proposing a complementary set of human rights directed specifically against the harms arising from robotic and artificial intelligence technologies. The relationship between responsibilities of the agent and the rights of the patient suggest that a rights regime is the other side of responsibility coin. The major distinction of this approach is to invert the power relationship: while human agents are perceived to control robotic patients, the prospect for this to become reversed is beginning. As robotic technologies become ever more sophisticated, and even genuinely complex, asserting human rights directly against robotic harms become increasingly important. Such an approach includes not only developing human rights that ‘protect’ humans but also ‘strengthen’ people against the challenges introduced by robotics and AI [This distinction parallels Berlin’s negative and positive concepts of liberty ], by emphasising the social and reflective character of the notion of humanness as well as the difference between the human and nonhuman. This will allow using the human frame of reference as constitutive of, rather than only subject to, the robotic and AI technologies, where it is human and not technology characteristics that shape the human rights framework in the first place. (shrink)

Transhumanism is a modern expression of ancient and transcultural aspirations to radically transform human existence, socially and bodily. Before the Enlightenment these aspirations were only expressed in religious millennialism, magical medicine, and spiritual practices. The Enlightenment channeled these desires into projects to use science and technology to improve health, longevity, and human abilities, and to use reason to revolutionize society. Since the Enlightenment, techno‐utopian movements have dynamically interacted with supernaturalist millennialism, sometimes syncretically, and often in violent opposition. Today the transhumanist (...) movement, a modern form of Enlightenment techno‐utopianism, has evolved a number of subsects, from the libertarian utopians funded by billionaire Peter Thiel, to religious syncretists like the Mormon Transhumanist Association, to the left‐wing technoprogressives and their bioliberal intellectual allies. In reaction to accelerating technological innovation and transhumanist ideas, apocalyptic Christians, and even secular catastrophists, have begun to incorporate human enhancement into their End Times scenarios. With all sides believing that the fate of humanity hangs in the balance, there is a growing likelihood of violent confrontation. (shrink)

With very advanced technology, a very large population of people living happy lives could be sustained in the accessible region of the universe. For every year that development of such technologies and colonization of the universe is delayed, there is therefore a corresponding opportunity cost: a potential good, lives worth living, is not being realized. Given some plausible assumptions, this cost is extremely large. However, the lesson for standard utilitarians is not that we ought to maximize the pace of technological (...) development, but rather that we ought to maximize its safety, i.e. the probability that colonization will eventually occur. This goal has such high utility that standard utilitarians ought to focus all their efforts on it. Utilitarians of a ‘person-affecting’ stripe should accept a modified version of this conclusion. Some mixed ethical views, which combine utilitarian considerations with other criteria, will also be committed to a similar bottom line. (shrink)

We describe a significant practical consequence of taking anthropic biases into account in deriving predictions for rare stochastic catastrophic events. The risks associated with catastrophes such as asteroidal/cometary impacts, supervolcanic episodes, and explosions of supernovae/gamma-ray bursts are based on their observed frequencies. As a result, the frequencies of catastrophes that destroy or are otherwise incompatible with the existence of observers are systematically underestimated. We describe the consequences of the anthropic bias for estimation of catastrophic risks, and suggest some directions for (...) future work. (shrink)

Die Übernahme des KI-Unternehmens DeepMind durch Google für rund eine halbe Milliarde US-Dollar signalisierte vor einem Jahr, dass von der KI-Forschung vielversprechende Ergebnisse erwartet werden. Spätestens seit bekannte Wissenschaftler wie Stephen Hawking und Unternehmer wie Elon Musk oder Bill Gates davor warnen, dass künstliche Intelligenz eine Bedrohung für die Menschheit darstellt, schlägt das KI-Thema hohe Wellen. Die Stiftung für Effektiven Altruismus (EAS, vormals GBS Schweiz) hat mit der Unterstützung von Experten/innen aus Informatik und KI ein umfassendes Diskussionspapier zu den Chancen (...) und Risiken der künstlichen Intelligenz verfasst und geht darin auf aktuelle, mittel- und langfristige Herausforderungen im Bereich der KI-Entwicklung ein. (shrink)

This article casts light on the moral implications of the possibility of human extinction, with a specific focus on extinction caused by an interruption in human reproduction. In the first two paragraphs, I show that moral philosophy has not yet given promising explanations for the wrongness of this kind of extinction. Specifically, the second paragraph contains a detailed rejection of John Leslie’s main claims on the morality of extinction. In the third paragraph, I offer a demonstration of the fact that (...) a moral view which stresses the permissibility of some types of human extinction can be effectively constructed, while in the following paragraph I propose a prima facie defense of a new approach to the topic which is likely to provide us with reasons in favor of the wrongness of a premature departure of humankind. (shrink)

The purpose of this paper is to explore the impact that developments in transhumanist technologies may have upon human cultures, and to do so by exploring a potential debate between Habermas and the transhumanists. Transhumanists, such as Nick Bostrom, typically see the potential in genetic and other technologies for positively expanding and transcending human nature. In contrast, Habermas is a representative of those who are fearful of this technology, suggesting that it will compound the deleterious effects of the colonisation of (...) the lifeworld, further constraining human autonomy and undermining the meaningfulness of the lifeworld by expanding the technological control and manipulation of humanity. It will be argued that these opposed positions are grounded in fundamentally different understandings of the consequences of scientific and technological advance. On one level, the transhumanists remain confident that the lifeworld has within it the resources necessary to find meaning and purpose in a society deeply infused by genetic technology. Habermas disagrees. On another level, the difference is articulated by Horkheimer and Adorno in Dialectic of Enlightenment, primarily by challenging what may be understood as a Baconian faith in science as a project for the domination of nature. While the transhumanists broadly embrace this faith, Habermas returns to something akin to Horkheimer and Adorno’s pessimistic scepticism. (shrink)

Human extinction is something generally deemed as undesirable, although some scholars view it as a potential solution to the problems of the Earth since it would reduce the moral evil and the suffering that are brought about by humans. We contend that humans collectively have absolute intrinsic value as sentient, conscious and rational entities, and we should preserve them from extinction. However, severe threats, such as climate change and incurable viruses, might push humanity to the brink of extinction. Should that (...) occur, it might be useful to envision a successor to humans able to preserve and hand down its value. One option would be to resort to humanoid robots that reproduce our salient characteristics by imitation, thanks to AI powered by machine learning. However, the question would arise of how to select the characteristics needed for our successors to thrive. This could prove to be particularly challenging. A way out might come from an algorithm entrusted with this choice. In fact, an algorithmic selection both at the social and at the individual level could be a preferred choice than other traditional ways of making decisions. In this sense, reflecting on human extinction helps us to identify solutions that are also suitable for the problems we face today. (shrink)

There are a number of recent attempts to introduce Confucian values to the ethical analysis of technology. These works, however, have not attended sufficiently to one central aspect of Confucianism, namely Ritual (‘Li’). Li is central to Confucian ethics, and it has been suggested that the emphasis on Li in Confucian ethics is what distinguishes it from other ethical traditions. Any discussion of Confucian ethics for technology, therefore, remains incomplete without accounting for Li. This chapter aims to elaborate on the (...) concept of Confucian Li and discuss its relevance to ethics of technology. Particularly, by referring to Li’s communicative, formative, and aesthetic function, I formulate an approach to ethics of technology with an emphasis on community, performance, and the aesthetic and demonstrate how this approach proceeds with the ethical analysis of technology. In doing so, I attempt to answer the question: why Confucianism matters in ethics of technology. (shrink)

Effective altruism’s identity as both a philosophy and a social movement requires effective altruists to consider which philosophical commitments are essential, such that one must embrace them in order to count as an effective altruist, at least in part in the light of the goal of building a robust social movement capable of advancing its aims. The goal of building a social movement provides a strong reason for effective altruists to embrace an ecumenical set of core commitments. At the same (...) time, there are risks involved in adopting an ecumenical approach to social movement building. In this paper, I develop a view about how effective altruists should characterize the movement’s core philosophical commitments, in light of the challenges of social movement building. I suggest that John Rawls’s idea of an “overlapping consensus” provides a useful model for thinking about the philosophical core and broader structure of the movement. (shrink)

The goal of this paper is to describe the mechanism of the public perception of risk of artificial intelligence. For that we apply the social amplification of risk framework to the public perception of artificial intelligence using data collected from Twitter from 2007 to 2018. We analyzed when and how there appeared a significant representation of the association between risk and artificial intelligence in the public awareness of artificial intelligence. A significant finding is that the image of the risk of (...) AI is mostly associated with existential risks that became popular after the fourth quarter of 2014. The source of that was the public positioning of experts who happen to be the real movers of the risk perception of AI so far instead of actual disasters. We analyze here how this kind of risk was amplified, its secondary effects, what are the varieties of risk unrelated to existential risk, and what is the dynamics of the experts in addressing their concerns to the audience of lay people. (shrink)

This paper provides a critique of Bostrom’s concern with existential risks, a critique which relies on Adorno and Horkheimer’s interpretation of the Enlightenment. Their interpretation is used to elicit the inner contradictions of transhumanist thought and to show the invalid premises on which it is based. By first outlining Bostrom’s position this paper argues that transhumanism reverts to myth in its attempt to surpass the human condition. Bostrom’s argument is based on three pillars, Maxipok, Parfitian population ethics and a universal (...) notion of general human values. By attempting to transcend the human condition, to achieve post-humanity, transhumanism reverts to myth. Thus, the aim of this paper is to provide a critical examination of transhumanism which elicits its tacit contradictions. It will also be argued that transhumanism’s focus on a universal, all-encompassing, notion of humanity neglects any concern with actual lived lives. This absence is problematic because it clearly shows that there is a discrepancy, between transhumanism’s claimed concern for all of humanity and the practical implications of proposing a universal notion of humanity. This paper will conclude, that transhumanism’s lack of concern with actual lives is due to its universal and totalising gestures. Gestures which allow for universal claims such as general values or Earth-originating intelligent life. (shrink)

The thinkers who have reflected on the problem of a coming superintelligence have generally seen the issue as a technological problem, a problem of how to control what the superintelligence will do. I argue that this approach is probably mistaken because it is based on questionable assumptions about the behavior of intelligent agents and, moreover, potentially counterproductive because it might, in the end, bring about the existential catastrophe that it is meant to prevent. I contend that the problem posed by (...) a future superintelligence will likely be a political problem, that is, one of establishing a peaceful form of coexistence with other intelligent agents in a situation of mutual vulnerability, and not a technological problem of control. (shrink)

The purpose of this paper is to explore the impact that developments in transhumanist technologies may have upon human cultures (and thus upon the lifeworld), and to do so by exploring a potential debate between Habermas and the transhumanists. Transhumanists, such as Nick Bostrom, typically see the potential in genetic and other technologies for positively expanding and transcending human nature. In contrast, Habermas is a representative of those who are fearful of this technology, suggesting that it will compound the deleterious (...) effects of the colonisation of the lifeworld, further constraining human autonomy and undermining the meaningfulness of the lifeworld by expanding the technological control and manipulation of humanity. It will be argued that these opposed positions are grounded in fundamentally different understandings of the consequences of scientific and technological advance. On one level, the transhumanists remain confident that the lifeworld has within it the resources necessary to find meaning and purpose in a society deeply infused by genetic technology. Habermas disagrees. On another level, the difference is articulated by Horkheimer and Adorno in Dialectic of Enlightenment, primarily by challenging what may be understood as a Baconian faith in science as a project for the domination of nature (where nature is an infinitely malleable material, to be dominated and shaped, without adverse consequences, purely for the purposes of human survival). While the transhumanists broadly embrace this faith, Habermas returns to something akin to Horkheimer and Adorno’s pessimistic scepticism. (shrink)

Systems evolve in order to adjust and survive. The paper’s contribution is that this evolvement is inadequate without an evolutionary telos. It is argued that without the presence of self-destruction in multiple levels of our existence and surroundings, our survival would have been impossible. This paper recognises an appreciation of auto-catastrophe at the cell level, in human attitudes (both as an individual and in societies), and extended to Earth and out to galaxies. Auto-Catastrophic Theory combines evolution with auto-catastrophic behaviours and (...) innovative machinery by offering new themes of understanding their merging such as (a) protogenic and deuterogenic auto-catastrophic processes, (b) protogenic and deuterogenic survival processes, and (c) ‘alive’, partially ‘alive’, and non-‘alive’ systems. The value of self-destruction relies on a numerical stability within a system and correlational relationships between systems. Auto-Catastrophic Theory provides novel justifications why artificial intelligence: (a) is crucial to overcome the extinction of humanity (via H+), (b) is crucial to offer indirect survival of humanity (by saving its history), and (c) is at the same time a threat for humanity. These novel justifications and themes, developed by combining evolution with catastrophe as well as innovative machinery, expand our knowledge of how best to handle fresh challenges ahead. (shrink)