Reflections on The Precipice
Among the things that most people worry about on a day-to-day basis, the possibility of human extinction is generally pretty far down the list. On typical days we might worry somewhat about our own futures, or those of loved ones, or perhaps others more distant from us, but rarely the fate of humans thousands or millions of years in the future. Even during the ongoing Covid-19 pandemic, most of our focus has remained on the human face of disaster in the here and now. A recent book by Toby Ord, however, seeks to trouble this complacency, and asks us to focus more intently on the far distant future and how to ensure the survival of humanity over the very long term.
Released in 2020, The Precipice is both the name of Ord's book, as well as his name for the period in which we are now living — one which he believes represents an time of unusually high risk, but which, if we can survive it, might give way to a time of much greater safety, maturity, and flourishing. To Ord, the Precipice is thus a kind of bottleneck, one which he argues is not receiving enough attention. Crucially, Ord is focused here on what he calls “existential” risks, by which he means potential dangers that threaten not just many people, but which might lead to the complete extinction of all humans, or otherwise greatly diminish the future prospects for our species.
Although highly readable, the Precipice is simultaneously a thorough compendium of details about the risks to human civilization, an earnest philosophical meditation on what is possible for humanity, and a surprisingly bold vision for reshaping the future. Ord is an excellent writer, and this book would be worth reading for the scientific material alone. However, because Ord goes so far towards trying to push humanity in a very particular direction, his work deserves special scrutiny, and invites response.
Professionally, Ord is a philosopher and researcher at Oxford's Future of Humanity Institute, and at the outset it is worth noting that his ethical credentials are unimpeachable. He has personally founded or helped to found multiple organizations devoted to the greater good, including Effective Altruism and Giving What We Can. He has also previously written extensively on the cause of international poverty, and he has personally committed to donating at least 10% of everything he earns to effective charities, including 100% of his advance and royalties from sales of this book.
Existential risk might seem like an odd topic for someone who has devoted so much of his time and effort towards ameliorating suffering in the present, but in many ways it is a natural outgrowth of a philosophical position of non-selfish objectivity. Just as philosophers such as Peter Singer make a convincing case that there is no good reason to discount the lives of people simply because they happen to live far away, Ord takes this to its logical conclusion, which is that there is little reason to privilege the lives of people who are alive today over those who will (or might) exist in the future.
Indeed, because so many more people might yet be born, in some sense the total moral weight tilts in the direction of the future. To borrow a thought experiment which Ord gets from Derek Parfit, an event which kills 100% of people is clearly worse than one which kills 99% of people, but how much worse? According to Parfit, the difference between the death of all people and the death of 99% of people is actually much greater than the difference between the death of 99% of people and the death of none. Building on this idea, Ord joins his colleague Nick Bostrom in suggesting that the possibility of human extinction is the most pressing concern of our time.
Given that Ord is a philosopher, I had expected this to be a work of popular philosophy, presumably one which would make the case for why we should worry about existential risk, and how to think about it. It turns out, however, to be much closer to a popular science book, replete with copious detail (about half of which is in footnotes) about the Earth, the universe, the history of humanity, and the risks to all three. Ord began his training in computer science, and it shows here, in terms of the emphasis on logical thinking and numerical estimates. Indeed, for a philosophical work about existential risk, this book puts surprisingly little effort into trying to convince the reader that existential risk is something we should care about.
Rather, the core of the book is concerned with presenting the evidence for different types of risks that threaten humanity, which Ord divides into “natural” risks (such as asteroids and supervolcanoes), and “anthropogenic” risks (nuclear weapons, engineering pandemics, “unaligned” artificial intelligence, and the like). Like any good computer scientist, Ord tries valiantly to quantify the probabilities that we might suffer each of these existential catastrophes sometime in the next century, coming up with an overall assessment that our chances of surviving the coming 100 years are about 5 in 6 -- the same, in other words, as a game of Russian roulette.
The final chapter is a much more speculative exploration of what the future of humanity might hold over the very long term (i.e., expanding throughout the galaxy, and beyond), and contains Ord's most operatic vision for what is possible, and how he thinks we should proceed. While there is much to agree with here (it's hard not to be inspired by a level of hope and optimism which surges above and beyond even utopian episodes of Star Trek: The Next Generation), I believe it useful and necessary to think through several key issues that arise from Ord's book, specifically the implications of focusing specifically on existential risks, how he handles risk and probability, and his particular vision for the long term future of humanity.
I'll return to all of these ideas below, but first, it's too much fun not to begin with a classic “big history” summary of where we are at in space in time. For anyone who doesn't have the numbers at their fingertips (I definitely include myself in this), having Ord present all of this in one place is, in a very particular sense, awesome. The length of a long-lived human life is around 100 years. Almost all of our current technology and understanding of the universe has been developed in the past 1,000 years, with the origins of agriculture dating back roughly 10,000 years. Many of the earliest unambiguous human artistic creations are around 40,000 years old, with anatomically modern humans emerging about 200,000 years ago. Based on the fossil record, mammalian species typically exist for around 1,000,000 years before going extinct, with some of the most enduring species (e.g., the horseshoe crab) existing, essentially unchanged, for close to 500 million years. Bacteria have been present on earth for billions of years, with the earth itself having formed around 4.7 billion years ago, and the age of the universe being something like 13.8 billion years (to the best of our current knowledge and understanding).
Going forward in time, current models predict human population will reach a peak of around 11 billion people in about 100 years. The Voyager 1 space probe, launched in 1977, will soon cease functioning, but will nevertheless approach one of the nearest stars to our sun in about 40,000 years. The sun itself will last for around 8 billion years, though it’s gradually increasing brightness may make Earth uninhabitable for certain types of life in as little as 800 million years. The universe itself, however, might well continue for trillions of years, though galactic clusters may become permanently isolated — utterly unreachable from each other, thanks to the expansion of space — much sooner than that. Our own galaxy contains around 100 billion stars, with something like 20 billion galaxies being theoretically within reach given the time available, as an upper bound. If we could manage to travel just a few light years at a time, we could in principle reach nearly anywhere in our galaxy in a series of short leaps. Depending on your perspective, the time available for such expansion might seem to be either generously long, or comparatively short. To borrow a metaphor from this book, the universe (and perhaps humanity) is still very much in its infancy, but the Earth itself is decidedly middle-aged.
Although Ord's hopeful vision for the future does not entirely depend on humans colonizing the galaxy (and beyond), it is an important part of his imaginings, and he makes it clear that if we fail to do so, it will have been, from his perspective, a great loss. Far from rushing into things, however, Ord urgently wants us to take our time, think a lot, and address the ways in which we are presently vulnerable to the possibility of complete annihilation. If humanity is a child, Ord is very much the concerned parent, cautioning prudence and forbearance, worried that we might squander our potential in our recklessness and haste. Once we have passed through this phase, however, and established existential security, then we can turn our attention to finding ourselves, and deciding what we want to do with ourselves.
As mentioned, Ord doesn't try to cross every t and dot every i when it comes to making the case for why we should care about existential risk, perhaps because he will assume that most people will generally agree with him.1 Or rather, it is perhaps because you only need to at least partly agree with him for his case that we are not devoting enough attention to this problem to seem quite strong. Indeed, he leans fairly heavily on the idea that we really don't know (yet) what is best, but that the long term future plausibly has great importance, and so prudence would dictate that we try harder to ameliorate the risk of complete annihilation, at least so as to preserve the chance to find out.
Nevertheless, the second chapter does lay out a basic argument, based on three ideas. First, past generations have invested their hopes and efforts into making a better world, and all that would have been wasted if we fail to survive. Not only do we have a duty to maintain the continuity of progress, but also an opportunity to make things right, given enough time. To quote Ord on this, “if we drop the baton, succumbing to an existential catastrophe, we would fail our ancestors in a multitude of ways. We would fail to achieve the dreams they hoped for; we would betray the trust they placed in us, their heirs; and we would fail in any duty we had to pay forward the work they did for us.”
Second, because the future is potentially so long, the number of people yet to be born could be vastly larger than the number who have already lived (which is estimated to be on the order of 100 billion people in total). Moreover, life today is much better on average than it was for our ancestors, and the future might be better still. Thus on the “total view”, if we think in strictly numeric terms, the total value of future generations (the potential positive conscious experiences of future people) is immense, even after discounting this value to account for the possibility that we might not survive.2
Third, while not completely separate from the above two ideas, Ord places a special emphasis on the idea of what he calls “our longterm potential”. Just as a parent has great hopes for their child, and wants to give them every opportunity to have the greatest life they can have (the analogy is explicit and sustained throughout the book), Ord sees humanity as potentially having a unique role in the universe. We don't know what the future of humanity looks like, but given that all we have accomplished so far, there seems to be no reason to place any limits on what might be possible. Given enough time, we might finally create a fully just society, repair all damage to the Earth and restore biodiversity, and discover new and more intense states of consciousness beyond ordinary imagining. (If this seems rather utopian, that is no doubt the point). Moreover, humanity, in Ord's view, plays a unique role in terms of moral potential — the only chance, as far as we know, of ensuring that as much of the universe as possible is transformed from lifeless matter into conscious beings. Ord sums this up nicely, saying,
"Because, in expectation, almost all of humanity's life lies in the future, almost everything of value lies in the future as well: almost all of the flourishing; almost all of the beauty; our greatest achievements; our most just societies; our most profound discoveries. We can continue our progress on prosperity, health, justice, freedom and moral thought. We can create a world of wellbeing and flourishing that challenges our capacity to imagine. And if we protect the world from catastrophe, it could last millions of centuries. This is our potential -- what we could achieve if we pass through the Precipice and continue striving for a better world."
In the end, it is hard to dispute that there is good reason to devote more thought and effort to the things which threaten humanity, not least because such efforts would also help to mitigate the risks of a less-than-existential catastrophe. Nevertheless, it is worthwhile pulling at the strings of this a bit more.
Ord makes it clear that he is explicitly focused on existential risks, as opposed to dangers which merely threaten a large portion of humanity. While these do tend to overlap strongly (e.g., a large asteroid might kill everyone, or just 99% of people), it does make a difference in terms of relative importance and reasoning. Not only does this criterion determine which risks Ord does and does not include in the book, and the time horizon he focuses on, it also shapes how he analyzes some of these risks. In particular, while he acknowledges that climate change may involve mass disruption and death, in the end he finds it quite unlikely that it will cause the actual extinction of humanity, with the largest part of the risk arising from the possibility of climate change disrupting our resilience to other risks (e.g., asteroids).
Again, Ord's moral credentials are beyond reproach. After all, he has almost certainly devoted more effort, thought, and resources towards helping the world’s poorest people than anyone reading this essay. Nevertheless, it is worth thinking through how a focus on the possibility of annihilation might distort one's view. In particular, there is clearly the potential for a kind of philosophical mugging of one's priorities to arise when confronted with an extremely improbable but consequential event, and I worry this distortion may be strongest when it comes to Ord's discussion of artificial intelligence (which I'll return to below).
At the same time, this specific focus perhaps leads to too great an emphasis on humanity, which might, in the end, be somewhat parochial. Again, Ord is on strong ground in asserting that as far as we know, Earth is the only place in the universe where complex life exists, and so it would be prudent to proceed as if that is the case. (There is not much explicit discussion of the Fermi paradox here, though Ord has written about it elsewhere). However, this view depends quite strongly on the idea that not only is Earth the only place that intelligent life has developed, but that humans are the only case in which it will develop. Although it is possible that nothing like the evolutionary process which led to modern human life and culture has ever happened before (anywhere), and never will happen again, this seems to strain credulity, given the unimaginably vast extent of the universe.
If only humans went extinct tomorrow, it does not seem entirely implausible that a new hairless ape would eventually evolve on Earth, and perhaps come to equal or exceed us in capabilities (or “potential”). On the other hand, if all complex life on Earth were to suddenly go extinct, the timeline for the emergence of something as complex as modern humans on Earth would begin to feel a bit tight (less than a billion years!), though that's not to say that such life might not yet evolve elsewhere in the universe.
Ord doesn’t necessarily assume that humans are the only entities capable of acting morally — as he notes, “the nature of such significance would depend on the ways in which we are unique” — but he does essentially proceed as if that were the case. For example, Ord writes, “It is not that I think only humans count. Instead, it is that humans are the only beings we know of that are responsive to moral reasons and moral argument—the beings who can examine the world and decide to do what is best. If we fail [to survive], that upward force, that capacity to push toward what is best or what is just, will vanish from the world.” Again, it would potentially be reckless to ignore the possibility that humans truly are unique in full sweep of the universe, but I'm not convinced that Ord devotes enough thought to the possibility of a successor or sibling coming along (on Earth or elsewhere) in calculating the potential downside risk of extinction.3
If nothing else, I expect Ord would agree that if we learned of the existence of an advanced civilization elsewhere in the universe, we would have to massively revise all our estimates. The existence of only a single species of intelligent life has a kind of narrative plausibility to it, but the existence of precisely two (and not more) in so vast a universe would surely be too improbable to accept.
Finally, although he is extremely thoughtful with respect to many objections, he doesn't devote much space to considering truly dissenting opinions. Although he generally seems to think that most people will come to agree with him, if they simply follow his reasoning, one can certainly imagine people holding other positions. In particular, while it is perhaps true that most people want the human race to continue forever, I strongly suspect there are many who would also prefer for it to come to an end at some point in the future. Extending Ord's own analogy of youth and maturity of the species, one could easily imagine someone placing more value on old age and death, if for no other reason than to make way for what is yet to come.
In terms of the actual risks, Ord divides the existential threats facing humanity into “natural” and “anthropogenic”. He spends a great deal of space on both, going through the ways in which they pose a threat, reviewing what we know about them, and trying to quantify the danger that each represents (in terms of the likelihood of leading to human extinction).4 Although he treats them in comparable fashion, there is a fundamental difference between natural and anthropogenic risks, in that for most (though not all) of the former, we can get a rough estimate of the frequency at which they occur, which serves as a decent baseline for how likely they are to pose a serious danger in the future. Ord tries valiantly to provide analogous estimates for the anthropogenic risks, but it is much more difficult, for a variety of reasons.
Going into this book, I (like many readers, I expect) had assumed that asteroids were one of the most serious risks that people should probably worry more about. It turns out, however, that this is the one that is perhaps best understood, and of the least concern of all the risks that Ord presents. Fascinatingly, according to Ord, people didn't even begin to consider the possibility that all of humanity might be wiped out by a massive asteroid until a few decades ago. Since then, however, rapid progress has been made. By examining various geological records, we can very precisely quantify just how frequent asteroids of different sizes have hit the earth. Moreover, NASA has been working hard at detecting and tracking asteroids, and can place fairly high confidence on the likelihood that there are any it has missed, scaled by size. In particular, it is apparently now confident that it is tracking nearly all asteroids with diameter greater than 10km, and none pose any serious threat in the near future.
Other natural risks are somewhat less tractable. It turns out that supervolcanoes pose a surprisingly serious threat (by filling the atmosphere with soot and blocking out the sun), and are much harder to forecast with any degree of accuracy. However, we also know that the frequency of truly massive eruptions is very low, based on the historical record. There are many more possibilities one could choose to worry about (many of which people are actively working on), such as comets, gamma ray bursts, and all manner of unknown phenomena. However, after considering all natural risks, Ord places the odds at only 1 in 10,000 that any such event would cause the end of humanity over the next 100 years.
The anthropogenic risks, on the other hand, are much harder to quantify. These include risks that might occur even without deliberate human intervention, like climate change, as well as purely human driven dangers, such as nuclear weapons and “unaligned” artificial intelligence. It is much harder to assign probabilities to these threats, precisely because humans are involved in the causes, not just in the consequences. As such, we cannot rely on simple long run frequencies, both because many of these risks are unprecedented, and because the underlying situation is dependent on human action, and constantly in flux. Moreover, our potential to respond is also constantly evolving, and so the risks can change dramatically from one year to the next. Finally, because so many of these technologies have the potential to be used as weapons (or actually are weapons), there is a real concern that we probably don't have access to all the relevant information.
Nuclear war provides an excellent example. There is some reason to think that an all-out nuclear exchange between the US and Russia would lead to the end of humanity — not due to the direct casualties, but because the resulting fires would poison the atmosphere and make the earth uninhabitable. This, however, is only the prediction of a model. Moreover, as Ord points out, there has been surprisingly little research into this, with just two papers being published on the topic since the end of the Cold War. We don't actually know what would happen with certainty, because we have never experienced anything quite like it, and the systems involved are very complex. (Others have argued that even an all-out exchange between India and Pakistan could lead to such an outcome, though this is disputed).
Unfortunately, even if we could predict the climatic effects of nuclear weapons with higher accuracy, there is much greater uncertainty about how likely it is that humans would take actions that would lead to such an outcome. The two obvious routes are aggression and error. For the former, it's hard to imagine that a country like the US or Russia would choose to launch a large scale assault, knowing that mutual destruction would almost certainly result, and yet it's incredible how close the world (seemingly) came to such an all-out nuclear war during the Cuban Missile Crisis. The whole period of the Cold War was of course a time of horrifyingly high risk, but those few days of crisis seem to have been particularly momentous, with most of Kennedy's advisors urging him to attack, and Castro ready to respond, even though it would mean the destruction of Cuba, and then likely, the world. Certainly things seem much farther from the brink now than they did back then (better safeguards, better communication channels, less desperation), but how certain can we be that other insane regimes might not arise (or indeed, that current insanities might not prevail)?
Even if we ignore the possibility of the deliberate initiation of nuclear war, there is still the possibility of error. There have been a disturbing number of accidents involving nuclear weapons over the years.5 Most of these were relatively innocuous, but there have also been higher profile events that many are likely familiar with, such as the choices made by Stanislav Petrov, a Soviet officer who chose not to respond to an alarm of an incoming American strike, deeming it (correctly) to be a glitch. Again, there seems to be good reason to think that the technology is safer now, but it is hard to be sure, and that is not to say that things might yet get worse, especially as information security and cyberwarfare become more serious concerns. Regardless, this does seem to be an area where there is a growing amount of interest in trying to mitigate the risks, but also one in which there is great difficulty in making progress.
Climate change is actually strangely similar. We know it is already reshaping the world and causing additional death and destruction. We also can fully expect for things to get worse. Ultimately, however, we don't know how much worse things will get, how people will respond, or how bad the consequences will be. This is partly due to uncertainty about the climate models, and partly due to uncertainties about how people will behave. It is a completely understandable position that some feel we must act now and do whatever we can to prevent the worst outcomes, no matter what. Unfortunately climate change is not the only potential threat, practical solutions are difficult, and resources are always limited, and so it really does matter how much we are able to quantify the uncertainties involved.
Although Ord's book is a good source of information on both of these risks (with extensive footnotes and references to other sources), the Precipice can at times feel more like the work of an impassioned autodidact (which in some sense it is), who has brought together material from many of the smartest people, rather than someone who has truly specialized in one area. This is not necessarily a bad thing; it means you can get a broad sampling of detailed information about these various risks in one go. In the end, however, I was left feeling a bit uncertain about how much to trust Ord's research on any one topic, especially climate change, given the sheer breadth of material involved. For whatever it's worth, Ord ultimately settles on placing a risk of about 1 in 1,000 over the next century for the prospect of either nuclear war or climate change leading to human extinction.
Among the anthropogenic risks, two get the lion's share of attention — pandemics and artificial intelligence. As it happens, the Precipice was released on March 5, 2020 in the United Kingdom, and a few weeks later in the United States. In other words, it's hard to imagine a more relevant time for people to be reading about the possibility of annihilation. Interestingly, however, I feel like reading this book during a pandemic has led me to think differently about it that I otherwise would have. Clearly pandemics still remain one of the most important risks in terms of the potential for mass death and suffering around the globe (both directly, and indirectly via economic effects). However, using Ord's strict criteria, it is not obvious to me how concerned we should be about pandemics if we were thinking only about existential risks.
To be clear, Covid-19 is far from the worst virus we might ever face. However, at least for natural pandemics, it does seem like there exists a kind of a frontier of lethality. On the one end are viruses like measles, which spread extremely easily, but are not very likely to be fatal. On the other end is something like Ebola, which is extremely deadly, but (partly as a consequence of its lethality), much less likely to spread, especially given modern knowledge and technology.
The worst spot on this frontier could still of course be very bad. According to Ord, the Black Death killed between one-quarter and one-half of all Europeans within six years. Especially for a small or isolated population, it is easy to see how a pandemic could be a true existential threat. However, given the amount of knowledge and technology that now exists, and the extent to which humanity is widely distributed, it is hard to imagine how a naturally occurring pandemic could truly pose an existential threat to humanity as a whole.
Of course, one of the reasons that pandemics are discussed as part of anthropogenic risks is because of the possibility of engineered pandemics. Along with our knowledge of how to prevent and cure infections has come a vast increase in our ability to create viruses and other biological weapons. One could imagine someone tinkering with the frontier to try to produce maximum casualties, such as by engineering a virus that would have an extremely long dormant period before causing a fatal illness. However, even that would involve a longer period of time in which one could potentially intervene, and it is unclear how to truly transcend the frontier described above.6
Covid-19 has proved to be such a huge problem in part due to societal failure, in ways that now seem tragic and absurd. However, it has also shown how capable societies are of being inventive in combating such illnesses, especially when governments act with skill and purpose. It is hard to overstate the extent to which our knowledge of these things has advanced in recent years, such that we can now conceive of simply coming up with and distributing a vaccine for a completely novel illness within a year. As with so many risks, the threat from pandemics is likely to be highly unevenly distributed. But the idea that no community of people could manage to weather such a storm now seems far less probable to me.
To be clear, pandemics (either natural or engineered) still clearly have the potential to cause mass death and destruction, at a scale much larger than we are currently experiencing. Pandemic preparedness should clearly be a top priority for governments (though it seems hard to maintain the political will over a sufficiently long period of time). In addition, there is some possibility that Ord has access to sensitive information which leads him to have greater concern, and which he prefers not to share for reasons of not wanting to increase the risk. (Ord has emphasized in interviews that he likes to stick to well-known examples, such as smallpox). However, based on the information he presents, and our collective experience over the past year, I simply can’t understand how Ord arrives at his estimate that there is a 1 in 30 chance that humanity will be entirely killed off by a pandemic within the next 100 years.
The risk that Ord ultimately places the most weight on is that of artificial intelligence (AI). This section really deserves a more lengthy treatment, and so I won't discuss it at any length here. I will say, however, that while I don't completely discount the issues that Ord raises, I feel that he unfortunately falls into the same trap as many others who have written about this topic, such as Nick Bostrom, in which he focuses exclusively on one very particular narrative, and simply asserts that this is how things will be. In particular, if we imagine something like an all-powerful demon (which we'll call “AI”) suddenly coming into existence, it would be very difficult (indeed, impossible!) to stop.
The problem is that it is far from clear that this is how the future will play out, and so Ord's discussion ends up being just one minor part of a much larger conversation. Moreover, Ord unfortunately lets the skepticism and rigor he applied to other risks fall away in this section, generally taking things at face value and not doing a great job of questioning assumptions, in a way that ultimately undermines his credibility.
Part of the problem is revealed by statements that seem to betray a lack of understanding of the difficulties involved. For example, Ord suggests that “perhaps the most important sign of things to come [from AI] is their ability to learn to play games”, which ignores all the ways in which games are an exceptionally easy domain for AI (compared to social systems). Another part of the problem is the way Ord extrapolates from narrow or speculative scenarios to confidently making general claims about how thing will be (e.g., “An intelligent agent would also resist attempts to change its reward function to something more aligned with human values — for it can predict that this would lead it to get less of what it currently sees as rewarding.”)
In addition (and similar to others who have written carelessly about this topic), Ord ascribes near perfection to an imagined piece of software, such as an ability to perfectly understand and predict human motivations and behaviour (e.g., "Since humans would predictably interfere with all these instrumental goals, it [an AI system] would be motivated to hide them from us until it was too late for us to be able to put up meaningful resistance.”) There are clearly interesting thought experiments and difficult technical questions that arise from this domain, but I worry that someone reading this book might end up treating Ord’s prescriptions as dogma, equivalent in weight and credibility to his reading of the literature on asteroids or nuclear weapons, which I think would be a serious mistake.7
Rather than trying to unpack this entire complicated conversation here, I would prefer to use it as an excuse for a brief digression into Ord's treatment of probability more broadly. One of the great strengths of Ord's book is that he is willing to put numbers to things. He also, to a limited extent, explains how he arrives at these numbers. For natural risks such as supervolcanoes, the reasoning is quite clear. Based on geological evidence, we can approximately calculate the frequency with which the Earth has been subject to volcanic eruptions on the scale that would have the potential to destroy all of humanity. Based on that, and absent any other information, we can assume that the risk in the coming years is proportional to the rate at which such events have occurred in the past. This is a classic frequentist estimate, and is on quite solid ground, assuming that such processes continue to operate in the future as they have in the past.8
For AI, by contrast, Ord gives a probability of 1 in 10 that humanity will be destroyed by unaligned AI within the next 100 years, without fully explaining how he arrives at this number. Based on interviews he has given, however, his reasoning goes something like this: based on surveys of experts, intuition, and other information, Ord estimates the probability that “superhuman AI” will be developed within the next 100 years to be 1 in 5. Second, if superhuman AI is developed, Ord assume that there is a 50% chance that it will destroy humanity. Multiplying these together gives us odds of destruction equal to 1 in 10. That's it. That's the model.
To be clear, this is not necessarily a bad way of obtaining an estimate, as far as it goes. Indeed, it's greatest virtue is its simplicity. Making one's reasoning explicit in this way is extremely useful, as it allows others to question or improve upon the model, or better estimate the probabilities involved. (Indeed, I wish Ord had been much more explicit about his probabilities models for other risks such as engineered pandemics, even if they are simple and heuristic).
The important thing to emphasize here, however, is that these predictions are derived from the combination of a model and estimates of the relevant probabilities within that model, each of which can be potentially improved. In this case, the model says that either superhuman AI will be invented or it won't, and that if it is invented, it will either destroy humanity or it won't. That is certainly one way of dividing up the space of possibilities, though it does to some extent beg the question of what exactly will count as “superhuman AI” and when we would causally attribute human destruction to it. More importantly however, the vast majority of the work is being done by two numbers (chance of invention and chance of destruction conditional on invention), both of which, it seems to me, involve a massive amount of uncertainty.9
Given how this section of the book is written, there are many issues in it that are deserving of greater scrutiny. The main point I want to emphasize however, is that I think Ord does a disservice to the reader in failing to flag the extent to which his estimates of 1 in 10,000 for natural risks and 1 in 10 for AI are not really comparable.10 Unsurprisingly, coverage of the book (for example, a good interview with Ezra Klein) has picked up on these as headline numbers, and repeated them without sufficient scrutiny. I’m not particularly worried that this book will have a massively distorting effect on funding priorities or government policy. I do believe, however, that it is important to recognize the mutually reinforcing nature of certain narratives, especially about AI, and that asserting such confidence about these imagined scenarios risks unhelpfully narrowing the conversation, blinding us to the many other ways in which things may develop.
The last part of Ord's book I want to discuss is perhaps the most interesting, namely his vision for what the far distant future of humanity might look like. Although it is not presented in full until the end of the book, this vision is actually the main motivation for all that came before. The reason that Ord thinks human extinction would be such a tragedy is that he believes there could be a future which would represent an almost unimaginably greater universe — one in which humanity has spread throughout the galaxy, or even beyond, and in which people are enjoying the most pleasurable heights of conscious experience, beyond what we can currently imagine.
This sort of vision is of course a staple of science fiction — the gradual expansion of humanity beyond the Earth, the colonization of other worlds, the development of technology which allows colonies to further propagate themselves, the reaching of a higher order of existence. Like any good “hard” science fiction, Ord devotes a lot of space in the text and footnotes to working through to what extent we have reason to believe that such a thing could be possible.
In many ways, his numbers are truly compelling: if we could just develop the technology to travel, say six light years, and to remotely deploy the mining and manufacturing technologies to gather resources from local asteroids and build the infrastructure required for the next stage of the journey, we could gradually reach the entire galaxy, without requiring any longer hop. Such developments are themselves rather enormous technological leaps. They do not, however, seem insurmountable, especially if we assume that we have tens or hundreds of thousands of years in which to figure things out. Thus, Ord's main concern is that we avoid destroying ourselves in the meantime, and continue developing our technology to the point where this might be possible.
Despite its familiarity, it is well worth digging into this vision a bit more. In its most benign version, it feels like another variation on the stern parent metaphor — we as a species should think about where we want to go, and avoid making any choices that would lock us in to a suboptimal future. This is a dangerous time in our (collective) lives, but if we take it seriously, and avoid making any risky choices, we can open up the potential of a truly great future. Moreover, this future might contain planes of existence that we cannot yet even conceive, and could last for millions or even billions of years. (How many parents can say that!)
The more complete vision, however, is in fact one of the most extraordinary high modernist visions I have ever encountered — extraordinary for it's scope, but also for the extent to which it embraces an ideal of rational planning and governance that would make Hamilton blush. Ord's main vehicle for navigating the future is what he calls (borrowing a term coined by William McAskill), “the long reflection”. The idea is that once we have mitigated the existential risk, such that there is a negligible chance of being annihilated, then we should take some time to think about what is best, and make that vision a reality. Rather than an ongoing process, Ord thinks we should use the time to come to some sort of philosophical conclusion as to what is best: “The ultimate aim of the Long Reflection would be to achieve a final answer to the question of which is the best kind of future for humanity.”
Although Ord's stated vision is inclusive and egalitarian, this is the section of the book in which Ord's “we” feels most tenuous. Like many enlightenment thinkers, Ord, believes this should be a democratic process, but with limits. “While the conversation should be courteous and respectful to all perspectives, it is even more important that it be robust and rigorous. For its ultimate aim is not just to win the goodwill of those alive at the time, but to deliver a verdict that stands the test of eternity.” Naturally, philosophers would play a central role, accompanied by “science, engineering, economics, political theory and beyond.”
Although I expect he would dispute this characterization, the more Ord writes about this, the more he seems to be tending towards a top-down, technocratic ideal, where the lines have been drawn and will be strictly enforced: Ord imagines “new institutions, filled with people of keen intellect and sound judgment, endowed with a substantial budget and real influence over policy.” Recognizing that power of shaping the decision making process, Ord comments that “Achieving existential security would be like writing the safeguarding of our potential into our constitution. The Long Reflection would then flesh out this constitution, setting the directions and limits in which our future will unfold.”
Perhaps tellingly, Ord compares his vision to the Renaissance, revealing similarities in terms of both influence and elitism: “A useful comparison is to the Renaissance. Most people in Europe were not actively involved in this rebirth of culture and learning, yet it is this grand project for which Europe in the fourteenth to seventeenth centuries is best remembered. Notable differences include that the Long Reflection would be a global project and that it should be more open to everyone to participate.” Although Ord has clearly read the temperature of the room, and wants to position his long reflection as a process in which many will be able to participate, he doesn't actually confront the inherent difficulties and messiness of democracy or choosing in groups. At a minimum, any deliberative system needs ground rules, but those who write those ground rules command enormous power.
Ord admits that he is simplifying things by treating humanity as a single entity, while brushing the internal conflict and diversity of opinions under the rug. This is certainly effective for allowing him to imagine what he takes to be a great future — one in which a well-reasoned process gives rise to a Constitution which will govern the unfolding of the future, such that all beings who exist experience the best possible lives, and a great many such beings will exist. The main difficulty is that this overlooks the possibility that the most valuable existence entails deciding — of not simply following the prescribed path laid out by the founders, but by innovating and in some cases challenging the system under which one is governed. Indeed, although I admit to getting caught up in the grandeur of Ord's vision at times (which is in many ways yet another manifestation of the mythical Western frontier narrative), in the end it reminded me of nothing so strongly as James C. Scott's unforgettable portraits of modernist social engineering — from the Bolshevik revolution to Le Corbusier.
Thinking that one can foresee and prevent existential catastrophes is bold and ambitious. Thinking that one can foresee the entire future and shape it to be optimal seems like hubris, and is, in my opinion, bound to fail (or at least, not to succeed in a way that was expected). No matter how carefully a system is designed, it cannot anticipate all possible future developments, and will break if it cannot bend. This means building in the possibility of change and adaptation, and this in turn entails the possible loss of control. Indeed, the only way to ensure a perfectly orderly future would be to bring about the kind of annihilation or dystopia that Ord is so eager to prevent. I appreciate Ord's optimism, but expect that any attempt to corral the messy organic nature of a complex system in which many people participate necessarily eliminates the most important feature that makes such a system valuable and alive.
In the end, Ord is clearly on strong ground in arguing that humanity is being collectively quite careless when it comes to existential risk, but I would argue that we get to a similar place if we start from merely trying to prevent catastrophes which would imply mass death, destruction, and disruption, without actually having to go so far as to focus exclusively on those risks that threaten the wholesale destruction of the species. Not only does this involve far less speculation about how things will unfold thousands of years from now, it also helps to avoid messy philosophical conundrums in population ethics, and would be, I expect, more persuasive to most people in terms of motivating action.
In fact, in terms of short term policy implications, there would seem to be fairly little difference between focusing on existential as opposed to merely catastrophic risks (or at least, it's not entirely clear what the differences would be). Most of Ord's concrete short-term policy suggestions seem eminently reasonable and commonplace (e.g., spend more money on planning for and preventing scenarios such as global pandemics and nuclear war, don't regulate prematurely, etc.). But where it leads Ord over the long term seems vastly different, and I find it far less plausible that there should be a global democratic process of debate among philosophers and economists that will decide and determine for all future generations how it would be best for them to live.
Indeed, the one place where Ord's book arguably falls short is in his lack of consideration of the multifarious ways in which humanity itself might change over the distant and not-so-distant future. We are currently on the edge of such a vast array of possible technological developments, from precise gene editing to neural interfaces to planetary management, and I would not be completely surprised if our notion of what humanity is is vastly different in a few hundred years from what it is today. If anything, despite Ord's surprising confidence in the possibility of superintelligent AI being created in that time, he seems to strangely underestimate the potential for fundamental changes in the realm of life itself.
I am a fan of bold, imaginative visions of what might be possible, and I strongly believe we need more of such visions, not less, but I begin to worry when such visions place much greater emphasis on security and control than growth and change. I absolutely agree with Ord that the world is currently devoting too few resources to trying to prevent and prepare for potential catastrophes, including things that might threaten the survival of all of humanity, but I also believe that the potential loss of 99% of life is more than enough to justify action on those issues. Although the latter does not connect us so strongly to the majesty (or horror, depending on your perspective) of Ord's vision of galactic colonization and perpetual dominance, a more smoothly varying level of concern sidesteps the question of exactly what does or does not count as an existential risk, and avoids the hubris of thinking that we can foresee how things will unfold hundreds of years hence, let alone trying to shape the future for thousands of generations to come.
Such an assumption might be presumptuous. Indeed, a recent study, also from researchers at the University of Oxford, found that not only do people not see human extinction as uniquely bad (compared to 4 out of 5 dying), only 78% of respondents considered human extinction to be a bad thing! (as opposed to “not bad”). The latter statistic was from a preliminary study, and only included 183 participants, but is obviously still far from a clear consensus.
Perhaps the most serious objection to this sort of utilitarian calculus is that while it is easy enough to (conceptually) compare many good lives to many bad lives, or many good lives to a few good lives, it is less clear how to compare many good lives to non-existence. Ord delves into this in more detail in an appendix, though he ultimately falls back upon the idea that we don't really know what the answer is, therefore we should hedge our bets.
In a footnote, he does mention the possibility that if we were to encounter another civilization, it could pose either a threat or a moral test for humanity, depending on whether it was more or less technologically advanced.
Ord also briefly discusses the risk of society being somehow “locked in” to an unfortunate state, such as if a totalitarian regime were somehow able to maintain control literally forever, which he still considers an existential risk, even though it does not directly involve the wholesale extinction of humanity. This is less of an easy fit within his framework, however, and it does not receive the same degree of consideration.
Eric Schlosser’s Command and Control is another excellent book on this topic.
It is interesting to reflect here on HIV, which has exactly that kind of long-gestation period, with the property of being both highly transmissible and highly lethal. Somewhere around 35 million people have died from HIV since the 1980s, and millions of people are newly infected each year (including 35,000 new infections in the US annually). The lethality has declined dramatically over time, however, thanks to better treatments.
This is not to say that this section is not worth reading, just that it should not be the only source one reads on the topic. Both Stuart Russell and Mex Tegmark provide somewhat more balanced (though still tendentious) treatments in their books. There are also an exceptionally large number of brilliant people focused on more immediate concerns from AI, though the two communities tend to not directly engage much with each other.
Technically, Ord likes to frame his thinking as Bayesian, although the distinction has little significance here.
One thing I greatly appreciate is that Ord fully admits to the incoherence of some of the evidence he relies on, specifically the surveys of AI researchers. In the main text, Ord reports that “Asked when an AI system would be ‘able to accomplish every task better and more cheaply than human workers,’ on average they [AI researchers] estimated a 50 percent chance of this happening by 2061 and a 10 percent chance of it happening as soon as 2025.” But in a footnote, he expands on this, saying "Note also that this estimate [of people’s timelines for AI] may be quite unstable. A subset of the participants were asked a slightly different question instead (emphasizing the employment consequences by talking of all occupations instead of all tasks). Their time by which there would be a 50% chance of this standard being met was 2138, with a 10% chance of it happening as early as 2036. I don’t know how to interpret this discrepancy, but it suggests we take these estimates cautiously."
Ord does emphasize that his estimates are only ballpark figures, and could be off by multiple orders of magnitude, but I feel he still doesn’t do enough to communicate the varying degrees of uncertainty, and perhaps still understates just how wrong he could be about some of these. Relatedly, Ord also makes a somewhat incoherent statement that “for low-probability high-stakes risks there is more room for the true probability to be higher than the estimate, than to be lower”, which requires considerable more transparency about its unstated assumptions in order for it to stand.