Many researchers and intellectuals warn about extreme risks from artificial intelligence. However, these warnings typically came without systematic arguments in support. This paper provides an argument that AI will lead to the permanent disempowerment of humanity, e.g. human extinction, by 2100. It rests on four substantive premises which it motivates and defends: first, the speed of advances in AI capability, as well as the capability level current systems have already reached, suggest that it is practically possible to build AI systems (...) capable of disempowering humanity by 2100. Second, due to incentives and coordination problems, if it is possible to build such AI, it will be built. Third, since it appears to be a hard technical problem to build AI which is aligned with the goals of its designers, and many actors might build powerful AI, misaligned powerful AI will be built. Fourth, because disempowering humanity is useful for a large range of misaligned goals, such AI will try to disempower humanity. If AI is capable of disempowering humanity and tries to disempower humanity by 2100, then humanity will be disempowered by 2100. This conclusion has immense moral and prudential significance. (shrink)

One response to the existential threat posed by a super-intelligent AI is to design it to be friendly to us. Some have argued that even if this were possible, the resulting AI would treat us as we do our pets. Sparrow (AI & Soc. https://doi.org/10.1007/s00146-023-01698-x, 2023) argues that this would be a bad outcome, for such an AI would dominate us—resulting in our freedom being diminished (Pettit in Just freedom: A moral compass for a complex world. WW Norton & Company, (...) 2014). In this paper, I consider whether this would be such a bad outcome. (shrink)