This chapter argues that dual-use emerging technologies are distributing unprecedented offensive capabilities to nonstate actors. To counteract this trend, some scholars have proposed that states become a little “less liberal” by implementing large-scale surveillance policies to monitor the actions of citizens. This is problematic, though, because the distribution of offensive capabilities is also undermining states’ capacity to enforce the rule of law. I will suggest that the only plausible escape from this conundrum, at least from our present vantage point, is (...) the creation of a “supersingleton” run by a friendly superintelligence, founded upon a “post-singularity social contract.” In making this argument, the present chapter offers a novel reason for prioritizing the “control problem,” i.e., the problem of ensuring that a greaterthan-human-level AI will positively enhance human well-being. (shrink)

Abstract. In this article is presented a model of the change of the probability of the global catastrophic risks in the world with exponentially evolving technologies. Increasingly cheaper technologies become accessible to a larger number of agents. Also, the technologies become more capable to cause a global catastrophe. Examples of such dangerous technologies are artificial viruses constructed by the means of synthetic biology, non-aligned AI and, to less extent, nanotech and nuclear proliferation. The model shows at least double exponential growth (...) of the probability of the global catastrophe which means that the accumulated probability of the catastrophe will grow from negligible to overwhelming at the period of a few doubling times of the technological capabilities. For biotech and AI, such doubling time roughly corresponds to the doubling period of Moore’s law and its analogues in other technologies and is around 2 years. Thus the global catastrophe in the exponential technologies world will most likely happen during a “dangerous decade”. Such a dangerous decade could start as early as in the 2020s. We also found that the double exponential growth in the model makes the model less sensitive to its initial conditions, like the number of dangerous agents or initial probabilities constants. The model also shows that smaller catastrophes could happen earlier than larger ones, and such a smaller catastrophe may be able to stop the technological growth before larger ones become possible, thus global risks will be self-limiting. However, if the growth of the number of dangerous agents will be very quick, the multiple smaller catastrophes could happen simultaneously and will be equal to a global catastrophe. (shrink)

In slogan form, the thesis of this dissertation is that shaping the far future is overwhelmingly important. More precisely, I argue that: Main Thesis: From a global perspective, what matters most is that we do what is best for the general trajectory along which our descendants develop over the coming millions, billions, and trillions of years. The first chapter introduces some key concepts, clarifies the main thesis, and outlines what follows in later chapters. Some of the key concepts include: existential (...) risk, the world's development trajectory, proximate benefits and ripple effects, speeding up development, trajectory changes, and the distinction between broad and targeted attempts to shape the far future. The second chapter is a defense of some methodological assumptions for developing normative theories which makes my thesis more plausible. In the third chapter, I introduce and begin to defend some key empirical and normative assumptions which, if true, strongly support my main thesis. In the fourth and fifth chapters, I argue against two of the strongest objections to my arguments. These objections come from population ethics, and are based on Person-Affecting Views and views according to which additional lives have diminishing marginal value. I argue that these views face extreme difficulties and cannot plausibly be used to rebut my arguments. In the sixth and seventh chapters, I discuss a decision-theoretic paradox which is relevant to my arguments. The simplest plausible theoretical assumptions which support my main thesis imply a view I call fanaticism, according to which any non-zero probability of an infinitely good outcome, no matter how small, is better than any probability of a finitely good outcome. I argue that denying fanaticism is inconsistent with other normative principles that seem very obvious, so that we are faced with a paradox. I have no solution to the paradox; I instead argue that we should continue to use our inconsistent principles, but we should use them tastefully. We should do this because, currently, we know of no consistent set of principles which does better. (shrink)

Climate change is often described as an existential risk to the human species, but this terminology has generally been avoided in the climate-justice literature in analytic philosophy. I investigate the source of this disconnect and explore the prospects for incorporating the idea of climate change as an existential risk into debates about climate justice. The concept of existential risk does not feature prominently in these discussions, I suggest, because assumptions that structure ‘ideal’ accounts of climate justice ensure that the prospect (...) of climate change as an extinction-level threat does not arise. Given persistent noncompliance with mitigation duties, however, we have reason to revisit these assumptions. I argue that the most promising way for theories of climate justice to account for the significance of existential risk is to look to the practices of protest and resistance in which a concern about extinction or global catastrophe is frequently given expression. (shrink)

The possibility of extraterrestrial intelligence (ETI) exists despite no observed evidence, and the risks and benefits of actively searching for ETI (Active SETI) have been debated. Active SETI has been criticized for potentially exposing humanity to existential risk, and a recent game-theoretical model highlights the Hobbesian trap that could occur following contact if mutual distrust leads to mutual destruction. We argue that observing a nearby ETI would suggest the existence of many unobserved ETI. This would expand the game and implies (...) that there may be a mechanism that prevents ETI from attacking us. As a result, the Hobbesian trap would be avoided if both parties assumed the other is not likely to attack. (shrink)

When discussing the extinction of humanity, there does not seem to be any clear agreement about what ‘humanity’ really means. One aim of this paper is to show that it is a more slippery concept than it might at first seem. A second aim is to show the relationship between what constitutes or defines humanity and what gives it value. Often, whether and how we ought to prevent human extinction depends on what we take humanity to mean, which in turn (...) depends on what we value about humans. Finally, I will offer a definition of humanity that I argue offers a plausible account of when we might say humanity has gone extinct, and reflects what we value about humans, and relatedly, what many actually care about in terms of why we should be preserved. (shrink)

Human extinction is something generally deemed as undesirable, although some scholars view it as a potential solution to the problems of the Earth since it would reduce the moral evil and the suffering that are brought about by humans. We contend that humans collectively have absolute intrinsic value as sentient, conscious and rational entities, and we should preserve them from extinction. However, severe threats, such as climate change and incurable viruses, might push humanity to the brink of extinction. Should that (...) occur, it might be useful to envision a successor to humans able to preserve and hand down its value. One option would be to resort to humanoid robots that reproduce our salient characteristics by imitation, thanks to AI powered by machine learning. However, the question would arise of how to select the characteristics needed for our successors to thrive. This could prove to be particularly challenging. A way out might come from an algorithm entrusted with this choice. In fact, an algorithmic selection both at the social and at the individual level could be a preferred choice than other traditional ways of making decisions. In this sense, reflecting on human extinction helps us to identify solutions that are also suitable for the problems we face today. (shrink)

This article is about a specific, but so far neglected peril of AI, which is that AI systems may become existential as well as causing suffering risks for nonhuman animals. The AI value alignment problem has now been acknowledged as critical for AI safety as well as very hard. However, currently it has only been attempted to align the values of AI systems with human values. It is argued here that this ought to be extended to the values of nonhuman (...) animals since it would be speciesism not to do so. The article focuses on the two subproblems—value extraction and value aggregation—discusses challenges for the integration of values of nonhuman animals and explores approaches to how AI systems could address them. (shrink)

Machine ethics and robot rights are quickly becoming hot topics in artificial intelligence and robotics communities. We will argue that attempts to attribute moral agency and assign rights to all intelligent machines are misguided, whether applied to infrahuman or superhuman AIs, as are proposals to limit the negative effects of AIs by constraining their behavior. As an alternative, we propose a new science of safety engineering for intelligent artificial agents based on maximizing for what humans value. In particular, we challenge (...) the scientific community to develop intelligent systems that have human-friendly values that they provably retain, even under recursive self-improvement. (shrink)

A classification of the global catastrophic risks of AI is presented, along with a comprehensive list of previously identified risks. This classification allows the identification of several new risks. We show that at each level of AI’s intelligence power, separate types of possible catastrophes dominate. Our classification demonstrates that the field of AI risks is diverse, and includes many scenarios beyond the commonly discussed cases of a paperclip maximizer or robot-caused unemployment. Global catastrophic failure could happen at various levels of (...) AI development, namely, before it starts self-improvement, during its takeoff, when it uses various instruments to escape its initial confinement, or after it successfully takes over the world and starts to implement its goal system, which could be plainly unaligned, or feature-flawed friendliness. AI could also halt at later stages of its development either due to technical glitches or ontological problems. Overall, we identified around several dozen scenarios of AI-driven global catastrophe. The extent of this list illustrates that there is no one simple solution to the problem of AI safety, and that AI safety theory is complex and must be customized for each AI development level. (shrink)

The thinkers who have reflected on the problem of a coming superintelligence have generally seen the issue as a technological problem, a problem of how to control what the superintelligence will do. I argue that this approach is probably mistaken because it is based on questionable assumptions about the behavior of intelligent agents and, moreover, potentially counterproductive because it might, in the end, bring about the existential catastrophe that it is meant to prevent. I contend that the problem posed by (...) a future superintelligence will likely be a political problem, that is, one of establishing a peaceful form of coexistence with other intelligent agents in a situation of mutual vulnerability, and not a technological problem of control. (shrink)

In the fields of artificial intelligence and robotics, the term “autonomy” is generally used to mean the capacity of an artificial agent to operate independently of human guidance. It is thereby assumed that the agent has a fixed goal or “utility function” with respect to which the appropriateness of its actions will be evaluated. From a philosophical perspective, this notion of autonomy seems oddly weak. For, in philosophy, the term is generally used to refer to a stronger capacity, namely the (...) capacity to “give oneself the law,” to decide by oneself what one’s goal or principle of action will be. The predominant view in the literature on the long-term prospects and risks of artificial intelligence is that an artificial agent cannot exhibit such autonomy because it cannot rationally change its own final goal, since changing the final goal is counterproductive with respect to that goal and hence undesirable. The aim of this paper is to challenge this view by showing that it is based on questionable assumptions about the nature of goals and values. I argue that a general AI may very well come to modify its final goal in the course of developing its understanding of the world. This has important implications for how we are to assess the long-term prospects and risks of artificial intelligence. (shrink)

In this paper, I draw on Hannah Arendt’s notion of ‘banality of evil’ to argue that as long as AI systems are designed to follow codes of ethics or particular normative ethical theories chosen by us and programmed in them, they are Eichmanns destined to commit evil. Since intelligence alone is not sufficient for ethical decision making, rather than strive to program AI to determine the right ethical decision based on some ethical theory or criteria, AI should be concerned with (...) avoiding making the wrong decisions, and this requires hardwiring the thinking activity as a prerequisite for decision making. (shrink)

I survey four categories of factors that might give a digital mind, such as an upload or an artificial general intelligence, an advantage over humans. Hardware advantages include greater serial speeds and greater parallel speeds. Self-improvement advantages include improvement of algorithms, design of new mental modules, and modification of motivational system. Co-operative advantages include copyability, perfect co-operation, improved communication, and transfer of skills. Human handicaps include computational limitations and faulty heuristics, human-centric biases, and socially motivated cognition. The shape of hardware (...) growth curves, as well as the ease of modifying minds, are found to have a major impact on how quickly a digital mind may take advantage of these factors. (shrink)

This article casts light on the moral implications of the possibility of human extinction, with a specific focus on extinction caused by an interruption in human reproduction. In the first two paragraphs, I show that moral philosophy has not yet given promising explanations for the wrongness of this kind of extinction. Specifically, the second paragraph contains a detailed rejection of John Leslie’s main claims on the morality of extinction. In the third paragraph, I offer a demonstration of the fact that (...) a moral view which stresses the permissibility of some types of human extinction can be effectively constructed, while in the following paragraph I propose a prima facie defense of a new approach to the topic which is likely to provide us with reasons in favor of the wrongness of a premature departure of humankind. (shrink)

The goal of this paper is to describe the mechanism of the public perception of risk of artificial intelligence. For that we apply the social amplification of risk framework to the public perception of artificial intelligence using data collected from Twitter from 2007 to 2018. We analyzed when and how there appeared a significant representation of the association between risk and artificial intelligence in the public awareness of artificial intelligence. A significant finding is that the image of the risk of (...) AI is mostly associated with existential risks that became popular after the fourth quarter of 2014. The source of that was the public positioning of experts who happen to be the real movers of the risk perception of AI so far instead of actual disasters. We analyze here how this kind of risk was amplified, its secondary effects, what are the varieties of risk unrelated to existential risk, and what is the dynamics of the experts in addressing their concerns to the audience of lay people. (shrink)

The standard argument to the conclusion that artificial intelligence (AI) constitutes an existential risk for the human species uses two premises: (1) AI may reach superintelligent levels, at which point we humans lose control (the ‘singularity claim’); (2) Any level of intelligence can go along with any goal (the ‘orthogonality thesis’). We find that the singularity claim requires a notion of ‘general intelligence’, while the orthogonality thesis requires a notion of ‘instrumental intelligence’. If this interpretation is correct, they cannot be (...) joined as premises and the argument for the existential risk of AI turns out invalid. If the interpretation is incorrect and both premises use the same notion of intelligence, then at least one of the premises is false and the orthogonality thesis remains itself orthogonal to the argument to existential risk from AI. In either case, the standard argument for existential risk from AI is not sound.—Having said that, there remains a risk of instrumental AI to cause very significant damage if designed or used badly, though this is not due to superintelligence or a singularity. (shrink)

Scientific research is subject to a number of regulations which impose incidental (time, place), rather than substantive (type of research), restrictions on scientific research and the knowledge created through such research. In recent years, however, the premise that scientific research and knowledge should be free from substantive regulation has increasingly been called into question. Some have suggested that the law should be used as a tool to substantively restrict research which is dual-use in nature or which raises moral objections. There (...) are, however, some problems with using law to restrict or prohibit certain types of scientific research, including (i) the inherent imprecision of law for regulating complex and rapidly evolving scientific research; (ii) the difficulties of enforcing legal restrictions on an activity that is international in scope; (iii) the limited predictability of the consequences of restricting specific branches of scientific research; (iv) inertia in the legislative process; and (v) the susceptibility of legislators and regulators to inappropriate factors and influence. Rather than using law to restrict scientific research, it may be more appropriate and effective to use a combination of non-traditional legal tools including norms, codes of conduct, restrictions on publication, and scientist-developed voluntary standards to regulate problematic scientific research. (shrink)

As the aim of the responsible robotics initiative is to ensure that responsible practices are inculcated within each stage of design, development and use, this impetus is undergirded by the alignment of ethical and legal considerations towards socially beneficial ends. While every effort should be expended to ensure that issues of responsibility are addressed at each stage of technological progression, irresponsibility is inherent within the nature of robotics technologies from a theoretical perspective that threatens to thwart the endeavour. This is (...) because the concept of responsibility, despite being treated as such, is not monolithic: rather this seemingly unified concept consists of converging and confluent concepts that shape the idea of what we colloquially call responsibility. From a different perspective, robotics will be simultaneously responsible and irresponsible depending on the particular concept of responsibility that is foregrounded: an observation that cuts against the grain of the drive towards responsible robotics. This problem is further compounded by responsible design and development as contrasted to responsible use. From a different perspective, the difficulty in defining the concept of responsibility in robotics is because human responsibility is the main frame of reference. Robotic systems are increasingly expected to achieve the human-level performance, including the capacities associated with responsibility and other criteria which are necessary to act responsibly. This subsists within a larger phenomenon where the difference between humans and non-humans, be it animals or artificial systems, appears to be increasingly blurred, thereby disrupting orthodox understandings of responsibility. This paper seeks to supplement the responsible robotics impulse by proposing a complementary set of human rights directed specifically against the harms arising from robotic and artificial intelligence technologies. The relationship between responsibilities of the agent and the rights of the patient suggest that a rights regime is the other side of responsibility coin. The major distinction of this approach is to invert the power relationship: while human agents are perceived to control robotic patients, the prospect for this to become reversed is beginning. As robotic technologies become ever more sophisticated, and even genuinely complex, asserting human rights directly against robotic harms become increasingly important. Such an approach includes not only developing human rights that ‘protect’ humans but also ‘strengthen’ people against the challenges introduced by robotics and AI [This distinction parallels Berlin’s negative and positive concepts of liberty ], by emphasising the social and reflective character of the notion of humanness as well as the difference between the human and nonhuman. This will allow using the human frame of reference as constitutive of, rather than only subject to, the robotic and AI technologies, where it is human and not technology characteristics that shape the human rights framework in the first place. (shrink)

Systems evolve in order to adjust and survive. The paper’s contribution is that this evolvement is inadequate without an evolutionary telos. It is argued that without the presence of self-destruction in multiple levels of our existence and surroundings, our survival would have been impossible. This paper recognises an appreciation of auto-catastrophe at the cell level, in human attitudes (both as an individual and in societies), and extended to Earth and out to galaxies. Auto-Catastrophic Theory combines evolution with auto-catastrophic behaviours and (...) innovative machinery by offering new themes of understanding their merging such as (a) protogenic and deuterogenic auto-catastrophic processes, (b) protogenic and deuterogenic survival processes, and (c) ‘alive’, partially ‘alive’, and non-‘alive’ systems. The value of self-destruction relies on a numerical stability within a system and correlational relationships between systems. Auto-Catastrophic Theory provides novel justifications why artificial intelligence: (a) is crucial to overcome the extinction of humanity (via H+), (b) is crucial to offer indirect survival of humanity (by saving its history), and (c) is at the same time a threat for humanity. These novel justifications and themes, developed by combining evolution with catastrophe as well as innovative machinery, expand our knowledge of how best to handle fresh challenges ahead. (shrink)

Transhumanism is a modern expression of ancient and transcultural aspirations to radically transform human existence, socially and bodily. Before the Enlightenment these aspirations were only expressed in religious millennialism, magical medicine, and spiritual practices. The Enlightenment channeled these desires into projects to use science and technology to improve health, longevity, and human abilities, and to use reason to revolutionize society. Since the Enlightenment, techno‐utopian movements have dynamically interacted with supernaturalist millennialism, sometimes syncretically, and often in violent opposition. Today the transhumanist (...) movement, a modern form of Enlightenment techno‐utopianism, has evolved a number of subsects, from the libertarian utopians funded by billionaire Peter Thiel, to religious syncretists like the Mormon Transhumanist Association, to the left‐wing technoprogressives and their bioliberal intellectual allies. In reaction to accelerating technological innovation and transhumanist ideas, apocalyptic Christians, and even secular catastrophists, have begun to incorporate human enhancement into their End Times scenarios. With all sides believing that the fate of humanity hangs in the balance, there is a growing likelihood of violent confrontation. (shrink)

Enlightenment values are built around the presumption of an independent rational self, citizen, consumer and pursuer of self-interest. Even the authoritarian and communitarian variants of the Enlightenment presumed the existence of autonomous individuals, simply arguing for greater weight to be given to their collective interests. Since Hume, however, radical Enlightenment empiricists have called into question the existence of a discrete, persistent self. Today neuroscientific reductionism has contributed to the rejection of an essentialist model of personal identity. Contemporary transhumanism has yet (...) to grapple with the radical consequences of the erosion of liberal individualism on their projects of individually chosen enhancement and longevity. Most transhumanists still reflect an essentialist idea of personal identity, even as they embrace projects of radical cognitive enhancement that would change every constituent element of consciousness. Transhumanists need to grapple with how their projects and ethics would change if personal identity is an arbitrary, malleable fiction. (shrink)

This paper provides a critique of Bostrom’s concern with existential risks, a critique which relies on Adorno and Horkheimer’s interpretation of the Enlightenment. Their interpretation is used to elicit the inner contradictions of transhumanist thought and to show the invalid premises on which it is based. By first outlining Bostrom’s position this paper argues that transhumanism reverts to myth in its attempt to surpass the human condition. Bostrom’s argument is based on three pillars, Maxipok, Parfitian population ethics and a universal (...) notion of general human values. By attempting to transcend the human condition, to achieve post-humanity, transhumanism reverts to myth. Thus, the aim of this paper is to provide a critical examination of transhumanism which elicits its tacit contradictions. It will also be argued that transhumanism’s focus on a universal, all-encompassing, notion of humanity neglects any concern with actual lived lives. This absence is problematic because it clearly shows that there is a discrepancy, between transhumanism’s claimed concern for all of humanity and the practical implications of proposing a universal notion of humanity. This paper will conclude, that transhumanism’s lack of concern with actual lives is due to its universal and totalising gestures. Gestures which allow for universal claims such as general values or Earth-originating intelligent life. (shrink)

Many of the existing ethical analyses of genetic engineering technologies (GET) focus on how they can be used to enhance individuals—to improve individual well-being, health and cognition. There is a gap in the current literature about the specific ways enhancement technologies could be used to improve our populations and species, viewed as a whole. In this paper, I explore how GET may be used to enhance the species through improvements in the gene pool. I argue one aspect of the species (...) that may be desirable to enhance is ‘persistence’ or long-term viability. I then look at some of the ways in which GET could be used to improve human persistence and argue that the use of GET to secure benefits for individuals may compromise persistence. This suggests conflicts between uses of GET to enhance individuals and uses to promote the persistence of the species may occur. As GET are further developed, the likelihood that these conflicts will actually arise, and how we should resolve them if they do, will need to be considered. (shrink)

One argument that is sometimes made against pursuing radical forms of human life extension is that such interventions will make the species less evolvable, which would be morally undesirable. In this article, I discuss the empirical and evaluative claims of this argument. I argue that radical increases in life expectancy could, in principle, reduce the evolutionary potential of human populations through both biological and cultural mechanisms. I further argue that if life extension did reduce the evolvability of the species, this (...) will be undesirable for three reasons: it may increase the species’ susceptibility to extinction risks, it may adversely affect institutions and practices that promote well-being, and it may impede moral progress. (shrink)

Historically, the hypothesis that our world is a computer simulation has struck many as just another improbable-but-possible “skeptical hypothesis” about the nature of reality. Recently, however, the simulation hypothesis has received significant attention from philosophers, physicists, and the popular press. This is due to the discovery of an epistemic dependency: If we believe that our civilization will one day run many simulations concerning its ancestry, then we should believe that we are probably in an ancestor simulation right now. This essay (...) examines a troubling but underexplored feature of the ancestor-simulation hypothesis: the termination risk posed by both ancestor-simulation technology and experimental probes into whether our world is an ancestor simulation. This essay evaluates the termination risk by using extrapolations from current computing practices and simulation technology. The conclusions, while provisional, have great implications for debates concerning the fundamental nature of reality and the safety of contemporary physics. (shrink)

The covid-19 pandemic has revealed a partially forgotten vulnerability. This very hostile period has made us more aware of the finitude of our life as well as the dangers to which we are exposed as the biological organisms that we are. Faced with such a panorama, transhumanism has taken advantage of this situation to claim, more forcefully than ever, the benefits that the application of technologies on human beings could have. Yet, at the same time, opponents of this movement have (...) pointed out, with equal intensity, how the pandemic has wakened many of its promises. We show how both the former and the latter continue to hold radically different positions regarding how to address vulnerability. However, both have seen the pandemic as an opportunity to highlight what should be the right direction for the future of our species. This event, which is not rooted in the field of science-fiction, could be decisive in rethinking the great challenges that await us. (shrink)

The purpose of this paper is to explore the impact that developments in transhumanist technologies may have upon human cultures (and thus upon the lifeworld), and to do so by exploring a potential debate between Habermas and the transhumanists. Transhumanists, such as Nick Bostrom, typically see the potential in genetic and other technologies for positively expanding and transcending human nature. In contrast, Habermas is a representative of those who are fearful of this technology, suggesting that it will compound the deleterious (...) effects of the colonisation of the lifeworld, further constraining human autonomy and undermining the meaningfulness of the lifeworld by expanding the technological control and manipulation of humanity. It will be argued that these opposed positions are grounded in fundamentally different understandings of the consequences of scientific and technological advance. On one level, the transhumanists remain confident that the lifeworld has within it the resources necessary to find meaning and purpose in a society deeply infused by genetic technology. Habermas disagrees. On another level, the difference is articulated by Horkheimer and Adorno in Dialectic of Enlightenment, primarily by challenging what may be understood as a Baconian faith in science as a project for the domination of nature (where nature is an infinitely malleable material, to be dominated and shaped, without adverse consequences, purely for the purposes of human survival). While the transhumanists broadly embrace this faith, Habermas returns to something akin to Horkheimer and Adorno’s pessimistic scepticism. (shrink)

The purpose of this paper is to explore the impact that developments in transhumanist technologies may have upon human cultures, and to do so by exploring a potential debate between Habermas and the transhumanists. Transhumanists, such as Nick Bostrom, typically see the potential in genetic and other technologies for positively expanding and transcending human nature. In contrast, Habermas is a representative of those who are fearful of this technology, suggesting that it will compound the deleterious effects of the colonisation of (...) the lifeworld, further constraining human autonomy and undermining the meaningfulness of the lifeworld by expanding the technological control and manipulation of humanity. It will be argued that these opposed positions are grounded in fundamentally different understandings of the consequences of scientific and technological advance. On one level, the transhumanists remain confident that the lifeworld has within it the resources necessary to find meaning and purpose in a society deeply infused by genetic technology. Habermas disagrees. On another level, the difference is articulated by Horkheimer and Adorno in Dialectic of Enlightenment, primarily by challenging what may be understood as a Baconian faith in science as a project for the domination of nature. While the transhumanists broadly embrace this faith, Habermas returns to something akin to Horkheimer and Adorno’s pessimistic scepticism. (shrink)

This essay interrogates the common assumption that good care is necessarily human care. It looks to disruptive fictional representations of robot care to assist its development of a theory of posthuman care that jettisons the implied anthropocentrism of ethics of care philosophy but retains care’s foregrounding of entanglement, embodiment and obligation. The essay reads speculative representations of robot care, particularly the Swedish television programme Äkta människor (Real Humans), alongside ethics of care philosophy and critical posthumanism to highlight their synergetic critiques (...) of neoliberal affective economies and humanist hierarchies that treat some bodies and affects as more real than others. These texts and discourses assist me in proposing a theory of care that regards vulnerability as the normative effect of posthuman vital embodiment, as opposed to an anomalous state that can be overcome or corrected via neoliberal practice. (shrink)

This article argues that an artificial superintelligence (ASI) emerging in a world where war is still normalised constitutes a catastrophic existential risk, either because the ASI might be employed by a nation–state to war for global domination, i.e., ASI-enabled warfare, or because the ASI wars on behalf of itself to establish global domination, i.e., ASI-directed warfare. Presently, few states declare war or even war on each other, in part due to the 1945 UN Charter, which states Member States should “refrain (...) in their international relations from the threat or use of force”, while allowing for UN Security Council-endorsed military measures and self-defense. As UN Member States no longer declare war on each other, instead, only ‘international armed conflicts’ occur. However, costly interstate conflicts, both hot and cold and tantamount to wars, still take place. Further, a New Cold War between AI superpowers looms. An ASI-directed/enabled future conflict could trigger total war, including nuclear conflict, and is therefore high risk. Via conforming instrumentalism, an international relations theory, we advocate risk reduction by optimising peace through a Universal Global Peace Treaty (UGPT), contributing towards the ending of existing wars and prevention of future wars, as well as a Cyberweapons and Artificial Intelligence Convention. This strategy could influence state actors, including those developing ASIs, or an agential ASI, particularly if it values conforming instrumentalism and peace. (shrink)

The likely near future creation of artificial superintelligence carries significant risks to humanity. These risks are difficult to conceptualise and quantify, but malicious use of existing artificial intelligence by criminals and state actors is already occurring and poses risks to digital security, physical security and integrity of political systems. These risks will increase as artificial intelligence moves closer to superintelligence. While there is little research on risk management tools used in artificial intelligence development, the current global standard for risk management, (...) ISO 31000:2018, is likely used extensively by developers of artificial intelligence technologies. This paper argues that risk management has a common set of vulnerabilities when applied to artificial superintelligence which cannot be resolved within the existing framework and alternative approaches must be developed. Some vulnerabilities are similar to issues posed by malicious threat actors such as professional criminals and terrorists. Like these malicious actors, artificial superintelligence will be capable of rendering mitigation ineffective by working against countermeasures or attacking in ways not anticipated by the risk management process. Criminal threat management recognises this vulnerability and seeks to guide and block the intent of malicious threat actors as an alternative to risk management. An artificial intelligence treachery threat model that acknowledges the failings of risk management and leverages the concepts of criminal threat management and artificial stupidity is proposed. This model identifies emergent malicious behaviour and allows intervention against negative outcomes at the moment of artificial intelligence’s greatest vulnerability. (shrink)

Technological revolutions are among the most important things that happen to humanity. Ethical assessment in the incipient stages of a potential technological revolution faces several difficulties, including the unpredictability of their long‐term impacts, the problematic role of human agency in bringing them about, and the fact that technological revolutions rewrite not only the material conditions of our existence but also reshape culture and even – perhaps – human nature. This essay explores some of these difficulties and the challenges they pose (...) for a rational assessment of the ethical and policy issues associated with anticipated technological revolutions. (shrink)

With very advanced technology, a very large population of people living happy lives could be sustained in the accessible region of the universe. For every year that development of such technologies and colonization of the universe is delayed, there is therefore a corresponding opportunity cost: a potential good, lives worth living, is not being realized. Given some plausible assumptions, this cost is extremely large. However, the lesson for standard utilitarians is not that we ought to maximize the pace of technological (...) development, but rather that we ought to maximize its safety, i.e. the probability that colonization will eventually occur. This goal has such high utility that standard utilitarians ought to focus all their efforts on it. Utilitarians of a ‘person-affecting’ stripe should accept a modified version of this conclusion. Some mixed ethical views, which combine utilitarian considerations with other criteria, will also be committed to a similar bottom line. (shrink)

Effective altruism’s identity as both a philosophy and a social movement requires effective altruists to consider which philosophical commitments are essential, such that one must embrace them in order to count as an effective altruist, at least in part in the light of the goal of building a robust social movement capable of advancing its aims. The goal of building a social movement provides a strong reason for effective altruists to embrace an ecumenical set of core commitments. At the same (...) time, there are risks involved in adopting an ecumenical approach to social movement building. In this paper, I develop a view about how effective altruists should characterize the movement’s core philosophical commitments, in light of the challenges of social movement building. I suggest that John Rawls’s idea of an “overlapping consensus” provides a useful model for thinking about the philosophical core and broader structure of the movement. (shrink)

In this article, I address the issue of evil and roboethics in the context of management studies and suggest that management scholars should locate evil in the realm of the human rather than of the artificial. After discussing the possibility of addressing the reality of evil machines in ontological terms, I explore users’ reaction to robots in a social context. I conclude that the issue of evil machines in management is more precisely a case of technology anthropomorphization.

The idea that the phenomenon of morality and, consequently, our ability to distinguish between vice and virtue can be explained by sympathy has been challenged as a highly controversial hypothesis, since sympathy appears to be easily influenced by proximity and selective, and would therefore seem incompatible with the possibility of taking an impartial, objective point of view. We intend to show that even a sentimentalist moral perspective such as the ‘Humean’ one, which places empathy (or ‘sympathy’, as Hume calls it) (...) at the core of morality, is capable of accounting for and justifying moral responsibility to future generations. Moreover, if we consider things from a Humean perspective, it is immoral not only to harm future generations, but also not to care at all about the extinction of the human species. For in the former case, we perform actions that cause misery, pain and suffering, whilst in the latter, we are not at all concerned about the welfare (i.e., happiness) of possible people who may, in the near future, come into the world and have a life worth or very much worth living. That is, in both cases, we show no ability to empathize with the possible people who will or may be born tomorrow. (shrink)

In this paper, the author aims to show that transhumanists are confused about their own conception of the posthuman: transhumanists anticipate radical transformation of the human through technology and at the same time assume that the criteria to determine what is “normal” and what is “enhanced” are univocal, both in our present time and in the future. Inspired by Nietzsche’s notion of the Overhuman, the author argues that the slightest “historical and phenomenological sense” discloses copious variations of criteria, both diachronic (...) and synchronic, for what can be considered “normal” and “enhanced.” Radical transformation through technology does not simply enable us to become “stronger,” “smarter,” or “healthier,” but it can and often will also change the very standard or yardstick with which we measure what counts as “stronger,” “smarter,” or “healthier.” Put yet differently: new and emerging technologies are not neutral means but often bring about different and, from our current perspective, foreign standards for determining what are “normal” and “enhanced” capacities. Since the qualitative meanings of these terms are themselves not fixed, it is unintelligible and too reassuring to simply predict that new technologies will enhance human beings. (shrink)

There is no strong reason to believe that human-level intelligence represents an upper limit of the capacity of artificial intelligence, should it be realized. This poses serious safety issues, since a superintelligent system would have great power to direct the future according to its possibly flawed motivation system. Solving this issue in general has proven to be considerably harder than expected. This paper looks at one particular approach, Oracle AI. An Oracle AI is an AI that does not act in (...) the world except by answering questions. Even this narrow approach presents considerable challenges. In this paper, we analyse and critique various methods of controlling the AI. In general an Oracle AI might be safer than unrestricted AI, but still remains potentially dangerous. (shrink)

Many researchers have argued that humanity will create artificial general intelligence (AGI) within the next twenty to one hundred years. It has been suggested that AGI may inflict serious damage to human well-being on a global scale ('catastrophic risk'). After summarizing the arguments for why AGI may pose such a risk, we review the fieldʼs proposed responses to AGI risk. We consider societal proposals, proposals for external constraints on AGI behaviors and proposals for creating AGIs that are safe due to (...) their internal design. (shrink)

This paper analyzes the political, philosophical, societal, legal, educational, biological, psychological and technological reasons why there is an urgent need for basic intercultural and interfaith ethics in the world and whether it is possible to formulate a valid code of such ethics. It is shown that universal ethics could be founded on natural law, which can be understood in both religious and secular ways. Alternatively, universal ethics could be based on a single supreme principle that is independent of worldview and (...) culture: human dignity. In accordance with these concepts, a minimalist and normative code of essential, self-evident universal ethical principles and norms is proposed. The implementation of universal ethics in society is a long-term political task that could be achieved by including universal ethics in the compulsory school curriculum of all countries and in the UNESCO agenda of Global Citizen Education. (shrink)

In this chapter we take stock of some of the complexities of the sprawling field of AI ethics. We consider questions like "what is the proper scope of AI ethics?" And "who counts as an AI ethicist?" At the same time, we flag several potential uses and abuses of AI ethics. These include challenges for the AI ethicist, including what qualifications they should have; the proper place and extent of futuring and speculation in the field; and the dilemmas concerning how (...) we use our public and academic resources. (shrink)

Scientific and technological progress might change people’s capabilities or incentives in ways that would destabilize civilization. For example, advances in DIY biohacking tools might make it easy for anybody with basic training in biology to kill millions; novel military technologies could trigger arms races in which whoever strikes first has a decisive advantage; or some economically advantageous process may be invented that produces disastrous negative global externalities that are hard to regulate. This paper introduces the concept of a vulnerable world: (...) roughly, one in which there is some level of technological development at which civilization almost certainly gets devastated by default, i.e. unless it has exited the “semi-anarchic default condition”. Several counterfactual historical and speculative future vulnerabilities are analyzed and arranged into a typology. A general ability to stabilize a vulnerable world would require greatly amplified capacities for preventive policing and global governance. The vulnerable world hypothesis thus offers a new perspective from which to evaluate the risk-benefit balance of developments towards ubiquitous surveillance or a unipolar world order. (shrink)

We outline an argument favoring voluntary moral bioenhancement as a response to existential risks humanity exposes itself to. We consider this type of enhancement a solution to the antithesis between the extinction of humanity and the imperative of humanity to survive at any cost. By opting for voluntary moral bioenhancement; we refrain from advocating illiberal or even totalitarian strategies that would allegedly help humanity preserve itself. We argue that such strategies; by encroaching upon the freedom of individuals; already inflict a (...) degree of existential harm on human beings. We also give some pointers as to the desirable direction for morally enhanced post-personhood. (shrink)

Recently many methods for reducing the risk of human extinction have been suggested, including building refuges underground and in space. Here we will discuss the perspective of using military nuclear submarines or their derivatives to ensure the survival of a small portion of humanity who will be able to rebuild human civilization after a large catastrophe. We will show that it is a very cost-effective way to build refuges, and viable solutions exist for various budgets and timeframes. Nuclear submarines are (...) surface independent, and could provide energy, oxygen, fresh water and perhaps even food for their inhabitants for years. They are able to withstand close nuclear explosions and radiation. They are able to maintain isolation from biological attacks and most known weapons. They already exist and need only small adaptation to be used as refuges. But building refuges is only “Plan B” of existential risk preparation; it is better to eliminate such risks than try to survive them. (shrink)

In general, existential threats are those that may potentially result in the extinction of the entire human species, if not significantly endanger its living population. Among the said threats include, but not limited to, pandemics and the impacts of a technological singularity. As regards pandemics, significant work has already been done on how to mitigate, if not prevent, the aftereffects of this type of disaster. For one, certain problem areas on how to properly manage pandemic responses have already been identified, (...) like the following: (a) not being able to learn from previous experiences, (b) the inability to act on warning signals, and (c) the failure to reach a global consensus on a problem (i.e., in a timely manner). In terms of a singularity, however, it may be said that further research is still needed, specifically on how to aptly respond to its projected negative outcomes. In this paper, by treating the three problem areas noted above as preliminary assessment measures of a country’s capacity to coordinate a national response to large-scale disasters, we examine the readiness of the Philippines in preparing for an intelligence explosion. By citing certain instances of how the said country, specifically its national government, faced the coronavirus disease 2019 pandemic, it puts forward the idea that the likely Philippine disaster response towards a singularity needs to be worked on, appealing for a more comprehensive assessment of such for a more informed response plan. (shrink)