Longtermists have argued that humanity should significantly increase its efforts to prevent catastrophes like nuclear wars, pandemics, and AI disasters. But one prominent longtermist argument overshoots this conclusion: the argument also implies that humanity should reduce the risk of existential catastrophe even at extreme cost to the present generation. This overshoot means that democratic governments cannot use the longtermist argument to guide their catastrophe policy. In this paper, we show that the case for preventing catastrophe does not depend on longtermism. (...) Standard cost-benefit analysis implies that governments should spend much more on reducing catastrophic risk. We argue that a government catastrophe policy guided by cost-benefit analysis should be the goal of longtermists in the political sphere. This policy would be democratically acceptable, and it would reduce existential risk by almost as much as a strong longtermist policy. (shrink)

This paper proposes that existing computational modeling research programs may be combined into platforms for the information of public policy. The main idea is that computational models at select levels of organization may be integrated in natural terms describing biological cognition, thereby normalizing a platform for predictive simulations able to account for both human and environmental costs associated with different action plans and institutional arrangements over short and long time spans while minimizing computational requirements. Building from established research programs, the (...) proposal aims to take advantage of current momentum in the direction of the integration of the cognitive with social and natural sciences, reduce start-up costs and increase speed of development. These are all important upshots given rising unease over the potential for AI and related technologies to shape the world going forward. (shrink)

This paper examines and analyzes five definitions of ‘existential risk.’ It tentatively adopts a pluralistic approach according to which the definition that scholars employ should depend upon the particular context of use. More specifically, the notion that existential risks are ‘risks of human extinction or civilizational collapse’ is best when communicating with the public, whereas equating existential risks with a ‘significant loss of expected value’ may be the most effective definition for establishing existential risk studies as a legitimate field of (...) scientific and philosophical inquiry. In making these arguments, the present paper hopes to provide a modicum of clarity to foundational issues relating to the central concept of arguably the most important discussion of our times. (shrink)

Many believe that a suitably programmed computer could act for its own goals and experience feelings. I challenge this view and argue that agency, mental causation and qualia are all founded in the unique, homeostatic nature of living matter. The theory was formulated for coherence with the concept of an agent, neuroscientific data and laws of physics. By this method, I infer that a successful action is homeostatic for its agent and can be caused by a feeling - which does (...) not motivate as a force, but as a control signal. From brain research and the locality principle of physics, I surmise that qualia are a fundamental, biological form of energy generated in specialized neurons. Subjectivity is explained as thermodynamically necessary on the supposition that, by converting action potentials to feelings, the neural cells avert damage from the electrochemical pulses. In exchange for this entropic benefit, phenomenal energy is spent as and where it is produced - which precludes the objective observation of qualia. (shrink)

There are two primary camps in which nanotechnology today can be categorized normal nanotechnology and speculative nanotechnology. The birth of nanotechnology proper was conceived through discourses of speculative nanotechnology. However, current nanotech-nology research has detracted from its speculative promises in favour of more attainable material products. Nonetheless, normal nanotechnology has leveraged the popular support and consequential funding it needs to conduct research and development (R&D) as a result of popular conceptions of speculative nanotechnology and its promises. Similarly, the scholarly literature (...) has shifted its focus away from speculative nanofutures towards normal nanotech-nology R&D. This paper shows that there is an incongruence between the representation of nanotechnology in the media, scholarly journals and industry. (shrink)

There are two types of artificial general intelligence (AGI) safety solutions: global and local. Most previously suggested solutions are local: they explain how to align or “box” a specific AI (Artificial Intelligence), but do not explain how to prevent the creation of dangerous AI in other places. Global solutions are those that ensure any AI on Earth is not dangerous. The number of suggested global solutions is much smaller than the number of proposed local solutions. Global solutions can be divided (...) into four groups: 1. No AI: AGI technology is banned or its use is otherwise prevented; 2. One AI: the first superintelligent AI is used to prevent the creation of any others; 3. Net of AIs as AI police: a balance is created between many AIs, so they evolve as a net and can prevent any rogue AI from taking over the world; 4. Humans inside AI: humans are augmented or part of AI. We explore many ideas, both old and new, regarding global solutions for AI safety. They include changing the number of AI teams, different forms of “AI Nanny” (non-self-improving global control AI system able to prevent creation of dangerous AIs), selling AI safety solutions, and sending messages to future AI. Not every local solution scales to a global solution or does it ethically and safely. The choice of the best local solution should include understanding of the ways in which it will be scaled up. Human-AI teams or a superintelligent AI Service as suggested by Drexler may be examples of such ethically scalable local solutions, but the final choice depends on some unknown variables such as the speed of AI progress. (shrink)

This chapter proposed a novel design methodology called Value-Sensitive Design and its potential application to the field of artificial intelligence research and design. It discusses the imperatives in adopting a design philosophy that embeds values into the design of artificial agents at the early stages of AI development. Because of the high risk stakes in the unmitigated design of artificial agents, this chapter proposes that even though VSD may turn out to be a less-than-optimal design methodology, it currently provides a (...) framework that has the potential to embed stakeholder values and incorporate current design methods. The reader should begin to take away the importance of a proactive design approach to intelligent agents. (shrink)

Discussions about the possible consequences of creating superintelligence have included the possibility of existential risk, often understood mainly as the risk of human extinction. We argue that suffering risks (s-risks) , where an adverse outcome would bring about severe suffering on an astronomical scale, are risks of a comparable severity and probability as risks of extinction. Preventing them is the common interest of many different value systems. Furthermore, we argue that in the same way as superintelligent AI both contributes to (...) existential risk but can also help prevent it, superintelligent AI can both be a suffering risk or help avoid it. Some types of work aimed at making superintelligent AI safe will also help prevent suffering risks, and there may also be a class of safeguards for AI that helps specifically against s-risks. (shrink)

New technologies with military applications may demand new modes of governance. In this article, we develop a taxonomy of technology governance forms, outline their strengths, and red-team their weaknesses. In particular, we consider the challenges and opportunities posed by advancing artificial intelligence, which is likely to have substantial dual-use properties. We conclude that subnational governance, though prevalent and mitigating some risks, is insufficient when the individual rewards from societally harmful actions outweigh normative sanctions, as is likely to be the case (...) with AI. Nationally enforced standards are promising ways to govern AI deployment, but they are less viable in the “race-to-the-bottom” environments that are becoming common. When it comes to powerful technologies with military implications, there is only one multilateral option with a strong historical precedent: a non-proliferation plus norms-of-use regime, which we call NPT+. We believe that a non-proliferation regime may, therefore, be the necessary foundation for AI governance. However, AI may exhibit characteristics that would make a non-proliferation regime less effective than it has proven for nuclear weapons. As an alternative, verification-backed restrictions on AI development and use would address more risks, but they face challenges in the case of advanced AI, and we show how these challenges may not have technical solutions. Perhaps more importantly, we show that there is no clear example of major powers restricting the development of a powerful military technology when that technology lacks a ready substitute. We, therefore, turn to a final alternative, International Monopoly, which was the preferred solution of many scholars and policymakers in the early nuclear era. It should be considered again for governing AI: a monopoly would require less-invasive monitoring, though at the possible cost of eroding national sovereignty. Ultimately, we conclude that it is too soon to tell whether a non-proliferation regime, a verification-based regime, or an International Monopoly is most feasible for governing AI. Nonetheless, a variety of policies would yield a high return across all three scenarios, and we conclude by identifying some of these steps that could be taken today. (shrink)

Tonkens has issued a seemingly impossible challenge, to articulate a comprehensive ethical framework within which artificial moral agents satisfy a Kantian inspired recipe—"rational" and "free"—while also satisfying perceived prerogatives of machine ethicists to facilitate the creation of AMAs that are perfectly and not merely reliably ethical. Challenges for machine ethicists have also been presented by Anthony Beavers and Wendell Wallach. Beavers pushes for the reinvention of traditional ethics to avoid "ethical nihilism" due to the reduction of morality to mechanical causation. (...) Wallach pushes for redoubled efforts toward a comprehensive account of ethics to guide machine ethicists on the issue of artificial moral agency. Options, thus, present themselves: reinterpret traditional ethics in a way that affords a comprehensive account of moral agency inclusive of both artificial and natural agents, or give up on the possibility and “muddle through” regardless. This series of papers pursues the first option, meets Tonkens' "challenge" and pursues Wallach's ends through Beavers’ proposed means, by "landscaping" traditional moral theory in resolution of a comprehensive account of moral agency. This first paper sets out the challenge and establishes the tradition that Kant had inherited from Aristotle, briefly entertains an Aristotelian AMA, fields objections, and ends with unanswered questions. The next paper in this series responds to the challenge in Kantian terms, and argues that a Kantian AMA is not only a possibility for Machine ethics research, but a necessary one. (shrink)

An arms race for an artificial general intelligence would be detrimental for and even pose an existential threat to humanity if it results in an unfriendly AGI. In this paper, an all-pay contest model is developed to derive implications for public policy to avoid such an outcome. It is established that, in a winner-takes-all race, where players must invest in R&D, only the most competitive teams will participate. Thus, given the difficulty of AGI, the number of competing teams is unlikely (...) ever to be very large. It is also established that the intention of teams competing in an AGI race, as well as the possibility of an intermediate outcome, is important. The possibility of an intermediate prize will raise the probability of finding the dominant AGI application and, hence, will make public control more urgent. It is recommended that the danger of an unfriendly AGI can be reduced by taxing AI and using public procurement. This would reduce the pay-off of contestants, raise the amount of R&D needed to compete, and coordinate and incentivize co-operation. This will help to alleviate the control and political problems in AI. Future research is needed to elaborate the design of systems of public procurement of AI innovation and for appropriately adjusting the legal frameworks underpinning high-tech innovation, in particular dealing with patenting by AI. (shrink)

This paper discusses means for promoting artificial intelligence that is designed to be safe and beneficial for society. The promotion of beneficial AI is a social challenge because it seeks to motivate AI developers to choose beneficial AI designs. Currently, the AI field is focused mainly on building AIs that are more capable, with little regard to social impacts. Two types of measures are available for encouraging the AI field to shift more toward building beneficial AI. Extrinsic measures impose constraints (...) or incentives on AI researchers to induce them to pursue beneficial AI even if they do not want to. Intrinsic measures encourage AI researchers to want to pursue beneficial AI. Prior research focuses on extrinsic measures, but intrinsic measures are at least as important. Indeed, intrinsic factors can determine the success of extrinsic measures. Efforts to promote beneficial AI must consider intrinsic factors by studying the social psychology of AI research communities. (shrink)

From his preliminary analysis in 1965, Hubert Dreyfus projected a future much different than those with which his contemporaries were practically concerned, tempering their optimism in realizing something like human intelligence through conventional methods. At that time, he advised that there was nothing “directly” to be done toward machines with human-like intelligence, and that practical research should aim at a symbiosis between human beings and computers with computers doing what they do best, processing discrete symbols in formally structured problem domains. (...) Fast-forward five decades, and his emphasis on the difference between two essential modes of processing, the unconscious yet purposeful mode fundamental to situated human cognition, and the “minded” sense of conscious processing characterizing symbolic reasoning that seems to lend itself to explicit programming, continues into the famous Dreyfus–McDowell debate. The present memorial reviews Dreyfus’ early projections, asking if the fears that punctuate current popular commentary on AI are warranted, and in light of these if he would deliver similar practical advice to researchers today. (shrink)