With very advanced technology, a very large population of people living happy lives could be sustained in the accessible region of the universe. For every year that development of such technologies and colonization of the universe is delayed, there is therefore a corresponding opportunity cost: a potential good, lives worth living, is not being realized. Given some plausible assumptions, this cost is extremely large. However, the lesson for standard utilitarians is not that we ought to maximize the pace of technological (...) development, but rather that we ought to maximize its safety, i.e. the probability that colonization will eventually occur. This goal has such high utility that standard utilitarians ought to focus all their efforts on it. Utilitarians of a ‘person-affecting’ stripe should accept a modified version of this conclusion. Some mixed ethical views, which combine utilitarian considerations with other criteria, will also be committed to a similar bottom line. (shrink)

In this century a number of events could extinguish humanity. The probability of these events may be very low, but the expected value of preventing them could be high, as it represents the value of all future human lives. We review the challenges to studying human extinction risks and, by way of example, estimate the cost effectiveness of preventing extinction-level asteroid impacts.

Humans have slowly built more productive societies by slowly acquiring various kinds of capital, and by carefully matching them to each other. Because disruptions can disturb this careful matching, and discourage social coordination, large disruptions can cause a “social collapse,” i.e., a reduction in productivity out of proportion to the disruption. For many types of disasters, severity seems to follow a power law distribution. For some of types, such as wars and earthquakes, most of the expected harm is predicted to (...) occur in extreme events, which kill most people on Earth. So if we are willing to worry about any war or earthquake, we should worry especially about extreme versions. If individuals varied little in their resistance to such disruptions, events a little stronger than extreme ones would eliminate humanity, and our only hope would be to prevent such events. If individuals vary a lot in their resistance, however, then it may pay to increase the variance in such resistance, such as by creating special sanctuaries from which the few remaining humans could rebuild society. (shrink)

Clear-cut cases of decision-making under risk (known probabilities) are unusual in real life. The gambler’s decisions at the roulette table are as close as we can get to this type of decision-making. In contrast, decision-making under uncertainty (unknown probabilities) can be exemplified by a decision whether to enter a jungle that may contain unknown dangers. Life is usually more like an expedition into an unknown jungle than a visit to the casino. Nevertheless, it is common in decision-supporting disciplines to proceed (...) as if reasonably reliable probability estimates were available for all possible outcomes, i.e. as if the prevailing epistemic conditions were analogous to those of gambling at the roulette table. This mistake can be called the tuxedo fallacy . It is argued that traditional engineering practices such as safety factors and multiple safety barriers avoid this fallacy and that they therefore manage uncertainty better than probabilistic risk analysis (PRA). PRA is a useful tool, but it must be supplemented with other methods in order not to limit the analysis to dangers that can be assigned meaningful probability estimates. (shrink)

A search is under way for a theory that can accommodate our intuitions in population axiology. The object of this search has proved elusive. This is not surprising since, as we shall see, any welfarist axiology that satisfies three reasonable conditions implies at least one of three counter-intuitive conclusions. I shall start by pointing out the failures in three recent attempts to construct an acceptable population axiology. I shall then present an impossibility theorem and conclude with a short discussion of (...) how it might be extended to pluralist axiologies, that is, axiologies that take more values than welfare into account. (shrink)

Clear-cut cases of decision-making under risk (known probabilities) are unusual in real life. The gambler’s decisions at the roulette table are as close as we can get to this type of decision-making. In contrast, decision-making under uncertainty (unknown probabilities) can be exemplified by a decision whether to enter a jungle that may contain unknown dangers. Life is usually more like an expedition into an unknown jungle than a visit to the casino. Nevertheless, it is common in decision-supporting disciplines to proceed (...) as if reasonably reliable probability estimates were available for all possible outcomes, i.e. as if the prevailing epistemic conditions were analogous to those of gambling at the roulette table. This mistake can be called the tuxedo fallacy. It is argued that traditional engineering practices such as safety factors and multiple safety barriers avoid this fallacy and that they therefore manage uncertainty better than probabilistic risk analysis (PRA). PRA is a useful tool, but it must be supplemented with other methods in order not to limit the analysis to dangers that can be assigned meaningful probability estimates. (shrink)

We describe a significant practical consequence of taking anthropic biases into account in deriving predictions for rare stochastic catastrophic events. The risks associated with catastrophes such as asteroidal/cometary impacts, supervolcanic episodes, and explosions of supernovae/gamma-ray bursts are based on their observed frequencies. As a result, the frequencies of catastrophes that destroy or are otherwise incompatible with the existence of observers are systematically underestimated. We describe the consequences of the anthropic bias for estimation of catastrophic risks, and suggest some directions for (...) future work. (shrink)