Many researchers have argued that humanity will create artificial general intelligence (AGI) within the next twenty to one hundred years. It has been suggested that AGI may inflict serious damage to human well-being on a global scale ('catastrophic risk'). After summarizing the arguments for why AGI may pose such a risk, we review the fieldʼs proposed responses to AGI risk. We consider societal proposals, proposals for external constraints on AGI behaviors and proposals for creating AGIs that are safe due to (...) their internal design. (shrink)

Because of accelerating technological progress, humankind may be rapidly approaching a critical phase in its career. In addition to well-known threats such as nuclear holocaust, the propects of radically transforming technologies like nanotech systems and machine intelligence present us with unprecedented opportunities and risks. Our future, and whether we will have a future at all, may well be determined by how we deal with these challenges. In the case of radically transforming technologies, a better understanding of the transition dynamics from (...) a human to a "posthuman" society is needed. Of particular importance is to know where the pitfalls are: the ways in which things could go terminally wrong. While we have had long exposure to various personal, local, and endurable global hazards, this paper analyzes a recently emerging category: that of existential risks. These are threats that could case our extinction or destroy the potential of Earth - originating intelligent life. Some of these threats are relatively well known while others, including some of the gravest, have gone almost unrecognized. Existential risks have a cluster of features that make ordinary risk management ineffective. A final section of this paper discusses several ethical and policy implications. A clearer understanding of the threat picture will enable us to formulate better strategies. (shrink)

In some situations a number of agents each have the ability to undertake an initiative that would have significant effects on the others. Suppose that each of these agents is purely motivated by an altruistic concern for the common good. We show that if each agent acts on her own personal judgment as to whether the initiative should be undertaken, then the initiative will be undertaken more often than is optimal. We suggest that this phenomenon, which we call the unilateralist’s (...) curse, arises in many contexts, including some that are important for public policy. To lift the curse, we propose a principle of conformity, which would discourage unilateralist action. We consider three different models for how this principle could be implemented, and respond to an objection that could be raised against it. (shrink)

There is, in some quarters, concern about high–level machine intelligence and superintelligent AI coming up in a few decades, bringing with it significant risks for humanity. In other quarters, these issues are ignored or considered science fiction. We wanted to clarify what the distribution of opinions actually is, what probability the best experts currently assign to high–level machine intelligence coming up within a particular time–frame, which risks they see with that development, and how fast they see these developing. We thus (...) designed a brief questionnaire and distributed it to four groups of experts in 2012/2013. The median estimate of respondents was for a one in two chance that high-level machine intelligence will be developed around 2040-2050, rising to a nine in ten chance by 2075. Experts expect that systems will move on to superintelligence in less than 30 years thereafter. They estimate the chance is about one in three that this development turns out to be ‘bad’ or ‘extremely bad’ for humanity. (shrink)

Unfit for the Future argues that the future of our species depends on radical enhancement of the moral aspects of our nature. Population growth and technological advances are threatening to undermine the conditions of worthwhile life on earth forever. We need to modify the biological bases of human motivation to deal with this challenge.

A cornerstone of modern western philosophy, addressing the role of man in government, society and religion In 1651, Hobbes published his work about the relationship between the government and the individual. More than four centuries old, this brilliant yet ruthless book analyzes not only the bases of government but also physical nature and the roles of man. Comparable to Plato's Republic in depth and insight, Leviathan includes two society-changing phenomena that Plato didn't dare to dream of -- the rise of (...) great nation-states with their claims to absolute sovereignty, and modern science, with its unprecedented analytic power. To Hobbes, the leviathan -- a mythical sea creature described in the Old Testament -- represented his central thesis: that the state must be strong in order to control and protect its citizens. Even today, Hobbes's thesis in Leviathan is debated among scholars and philosophy aficionados around the globe. One of the earliest attempts at a genuinely scientific understanding of politics and society in their modern form, this book also remains one of the most stimulating. In his timeless work, Hobbes outlines his ideas about the passions and the conduct of man, and how his theories are realized in every individual. Addressing free will and religion, Hobbes constructs an intelligent argument for the basis of religion within government and how to organize a peaceful and successful Christian commonwealth. Like Plato's Republic, this book contains ideas on psychology, ethics, law, language, and religion that continue to challenge modern thinkers and exercise a profound influence on Western thought. A classic treatise of philosophy, Leviathan is critical reading for anyone who wishes to examine the human mind through the prisms of government and society. (shrink)

Policy disputes arise at all scales of governance: in clubs, non-profits, firms, nations, and alliances of nations. Both the means and ends of policy are disputed. While many, perhaps most, policy disputes arise from conflicting ends, important disputes also arise from differing beliefs on how to achieve shared ends. In fact, according to many experts in economics and development, governments often choose policies that are “inefficient” in the sense that most everyone could expect to gain from other feasible policies. Many (...) other kinds of experts also see existing policies as often clearly inferior to known alternatives. (shrink)

The human brain has some capabilities that the brains of other animals lack. It is to these distinctive capabilities that our species owes its dominant position. Other animals have stronger muscles or sharper claws, but we have cleverer brains. If machine brains one day come to surpass human brains in general intelligence, then this new superintelligence could become very powerful. As the fate of the gorillas now depends more on us humans than on the gorillas themselves, so the fate of (...) our species then would come to depend on the actions of the machine superintelligence. But we have one advantage: we get to make the first move. Will it be possible to construct a seed AI or otherwise to engineer initial conditions so as to make an intelligence explosion survivable? How could one achieve a controlled detonation? To get closer to an answer to this question, we must make our way through a fascinating landscape of topics and considerations. Read the book and learn about oracles, genies, singletons; about boxing methods, tripwires, and mind crime; about humanity's cosmic endowment and differential technological development; indirect normativity, instrumental convergence, whole brain emulation and technology couplings; Malthusian economics and dystopian evolution; artificial intelligence, and biological cognitive enhancement, and collective intelligence. (shrink)

Bestselling author Sam Harris dismantles the most common justification for religious faith-that a moral system cannot be based on science.

While there has been much interest in this topic, no generally accepted definition of dual use has been forthcoming. As a contribution to this issue, it is maintained that three related kinds of things comprise the category of dual use: research, technologies and artefacts. In regard to all three kinds, difficulties are identified in making clear distinctions between those that are and are not dual use. It is suggested that our classification should take account of actual capacities and willingness to (...) make use of these objects for ‘bad ends’ and not the mere possibility that this could be done, and here three ‘contextual factors’ are identified. A (provisional) definition is proposed that takes account of threats and risks. (shrink)

Ever since its first publication in 1992, The End of History and the Last Man has provoked controversy and debate. Francis Fukuyama's prescient analysis of religious fundamentalism, politics, scientific progress, ethical codes, and war is as essential for a world fighting fundamentalist terrorists as it was for the end of the Cold War. Now updated with a new afterword, The End of History and the Last Man is a modern classic.

Consisting of two essays, this work by a Harvard professor offers his thoughts on the idea of a social contract regulating people's behavior toward one another.

The future of humanity is often viewed as a topic for idle speculation. Yet our beliefs and assumptions on this subject matter shape decisions in both our personal lives and public policy – decisions that have very real and sometimes unfortunate consequences. It is therefore practically important to try to develop a realistic mode of futuristic thought about big picture questions for humanity. This paper sketches an overview of some recent attempts in this direction, and it offers a brief discussion (...) of four families of scenarios for humanity’s future: extinction, recurrent collapse, plateau, and posthumanity. (shrink)

Transhumanism is a loosely defined movement that has developed gradually over the past two decades. [1] It promotes an interdisciplinary approach to understanding and evaluating the opportunities for enhancing the human condition and the human organism opened up by the advancement of technology. Attention is given to both present technologies, like genetic engineering and information technology, and anticipated future ones, such as molecular nanotechnology and artificial intelligence.

Where does the mind stop and the rest of the world begin? The question invites two standard replies. Some accept the demarcations of skin and skull, and say that what is outside the body is outside the mind. Others are impressed by arguments suggesting that the meaning of our words "just ain't in the head", and hold that this externalism about meaning carries over into an externalism about mind. We propose to pursue a third position. We advocate a very different (...) sort of externalism: an _active externalism_, based on the active role of the environment in driving cognitive processes. (shrink)

The truth of the matter is that our deficiency does not lie in the want of well-verified "facts." What we lack is our bearings. The contemporary experience of things technological has repeatedly confounded our vision, our expectations, and our capacity to make intelligent judgments. Categories, arguments, conclusions, and choices that would have been entirely obvious in earlier times are obvious no longer. Patterns of perceptive thinking that were entirely reliable in the past now lead us systematically astray. Many of our (...) standard conceptions of technology reveal a disorientation that borders on dissociation from reality. And as long as we lack the ability to make our situation intelligible, all of the "data" in the world will make no difference.;From the Introduction. (shrink)

Sam Harris dismantles the most common justification for religious faith--that a moral system cannot be based on science.

In Unfit for the Future, Ingmar Persson and Julian Savulescu argue that our collective existetial predicment is unprecedentedly dangerous due to climate change and terrorism. Given these global risks to human prosperity and survival, Persson and Savulescu argue that we should explore the radical possibility of moral bioenhancement in addition to cognitive enhancement. In this article, I argue that moral bioenhancements could nontrivially exacerbate the threat posed by certain kinds of malicious agents, while reducing the threat of other kinds. This (...) introduces a previously undiscussed complication to Persson and Savulescu's proposal. In the final section, I present a novel argument for why moral bioenhancement should either be compulsory or not be made available to the public at all. (shrink)

Neuroscientific studies have shown that brain activity correlated with a decision to move can be observed before a person reports being consciously aware of having made that decision . Given that a later event cannot cause an earlier one , such results have been interpreted as evidence that decisions are made unconsciously . We argue that this interpretation depends upon an all-or-none view of consciousness, and we offer an alternative interpretation of the early decision-related brain activity based on models in (...) which conscious awareness of the decision to move develops gradually up to the level of a reporting criterion. Under this interpretation, the early brain activity reflects sub-criterion levels of awareness rather than complete absence of awareness and thus does not suggest that decisions are made unconsciously. (shrink)

A major approach to the ethics of artificial intelligence is to use social choice, in which the AI is designed to act according to the aggregate views of society. This is found in the AI ethics of “coherent extrapolated volition” and “bottom–up ethics”. This paper shows that the normative basis of AI social choice ethics is weak due to the fact that there is no one single aggregate ethical view of society. Instead, the design of social choice AI faces three (...) sets of decisions: standing, concerning whose ethics views are included; measurement, concerning how their views are identified; and aggregation, concerning how individual views are combined to a single view that will guide AI behavior. These decisions must be made up front in the initial AI design—designers cannot “let the AI figure it out”. Each set of decisions poses difficult ethical dilemmas with major consequences for AI behavior, with some decision options yielding pathological or even catastrophic results. Furthermore, non-social choice ethics face similar issues, such as whether to count future generations or the AI itself. These issues can be more important than the question of whether or not to use social choice ethics. Attention should focus on these issues, not on social choice. (shrink)

The Law of Peoples, with “The Idea of Public Reason Revisited”. john rawls. William James and the Metaphysics of Experience. david c. lamberth.