Forecasting the End of the World

New estimates of p(doom)

Detail of Michelangelo’s The Last Judgment.

The Forecasting Research Institute recently brought superforecasters and experts together to estimate the risk of severe catastrophes and of human extinction. The superforecasters were a lot more optimistic than the experts.

Doom is in the air. Earlier this year the Bulletin of Atomic Scientists announced it had moved its Doomsday Clock forward to 90 seconds to midnight—the closest we’ve ever been to global catastrophe—saying that Russia’s war on Ukraine makes this “a time of unprecedented danger.” Climate change threatens to render parts of the Earth uninhabitable. A few years ago Oxford philosopher Toby Ord estimated there’s a 1 in 6 chance of an “existential catastrophe”—an event that would permanently destroy much of humanity’s long-term potential—before 2120.¹ The Center for AI Safety recently published an open letter signed by a large number of eminent AI experts comparing the risk of an AI catastrophe to the risk from global pandemics or nuclear war. Timothée Chalamet thinks “it smells like” societal collapse.

There are certainly reasons to worry. As a former existential risk researcher, I’m not going to ridicule or dismiss concerns about the long-term survival of humanity. As I wrote a couple weeks ago, humanity may actually be at greater risk of extinction now than at any other time in our history because our ability to govern ourselves has not kept pace with our ability to harm ourselves. But while I believe the risk of global catastrophe is much too high—given that more or less everything anyone cares about is at stake—that doesn’t mean a catastrophe is likely in absolute terms.

Last year, the Forecasting Research Institute sponsored a tournament in which participants were asked to forecast the long-range probability of severe catastrophe. The Existential-risk Persuasion Tournament (XPT) brought together superforecasters like me with a record of accurately forecasting shorter-range questions and domain experts in subjects relevant to catastrophic risk. They were asked to work together to forecast the probability of catastrophes (defined as the death of 10% of the global population over a five-year period) and the probability of extinction (defined for this purpose as the human population falling below 5,000 people) before 2100. Estimating the long-range probability of severe catastrophes is particularly hard because they don’t happen regularly enough for us to know how often they are likely to happen or to evaluate the accuracy of our forecasts. The XPT tried to address this problem by using a number of novel techniques to incentivize participants to produce good-faith forecasts backed by compelling rationales.

In the end—in spite of their incentives to persuade one another—superforecasters and domain experts were far apart on their estimates of the risk. The two groups were furthest apart on their estimates of the probability of a disaster caused by AI, although both agreed it was the primary driver of the total risk. The domain experts estimated the probability of a catastrophe that kills 10% of the global population before 2100 is 20% and the probability of an extinction event before 2100 is an alarming 6%. The superforecasters estimated the probability of a catastrophe that kills 10% of the global population before 2100 is 9% and the probability of an extinction event before 2100 is just 1%.² It perhaps shouldn’t be surprising that two groups with different approaches and different expertise didn’t reach consensus. The unprecedented nature of what they were trying to predict makes these questions extremely uncertain. Because the signal of the underlying probability is likely to be weak, we shouldn’t be confident in anyone’s end-of-the-world forecasts.

But I’m with the superforecasters on this. I didn’t participate in the tournament, but the experts’ forecasts seem clearly too high to me. There has never been a catastrophe that killed 10% of the human population in a five year period in all of recorded history, although two plague pandemics—the Plague of Justinian in the sixth century and the Black Death in fourteenth—are near misses. WWII doesn’t come close. While the chance of such a catastrophe may be substantially higher now than it was in the past, the historical base rate suggests the chance of a catastrophe of that scale in the next 80 years should be low. The chance something will cause the human population to fall below 5,000 is much lower. There are very few realistic catastrophe scenarios that could kill more than 99.9999% of humans in short order. Nuclear war probably wouldn’t wipe humans out so quickly and completely. Superintelligent AI might be able to exterminate us, but for all the ways AI development could go wrong I think scenarios in which AI methodically kills humans are unlikely. People have been confidently predicting the end of the world is near throughout history, but it should take extraordinary evidence to convince us that it really is near. As superforecaster tournament participant Kjirste Morrell said in The Economist, “It’s just not easy to kill everybody.”

In fact, I think the superforecaster forecasts are probably—and not just because I otherwise agree with them—the best estimate of these risks we have right now. It’s certainly possible that superforecasters are simply not good at forecasting rare, extraordinary events like global catastrophes. But the same ability to estimate probability that allows superforecasters to make accurate short-term forecasts should at least help them make accurate long-term forecasts. Because they score highly on measures of open-minded thinking, they should have been able to incorporate the experts’ insight into their forecasts. Those superforecasters who, like me, have some expertise of their own—there’s a fair amount of overlap between the forecasting and existential risk communities—thought the risk was much lower than the experts who weren’t also skilled forecasters. While I wouldn’t dismiss the experts’ forecasts, there are more reasons to be skeptical of their value. Experts haven’t been able to translate their knowledge into accurate forecasts in other contexts. As a group, they tend to be overconfident and to weight their own expertise too heavily. Even experts who recognize superforecasters have a skill they lack may find it difficult to adequately update long-held views in light of the judgment of skilled forecasters.

All of which is good news. We should absolutely do everything we can do to ensure humanity’s long-term survival and to reduce the risk of a catastrophe caused by AI. But we should also take heart in the knowledge that skilled forecasters think the chance of a severe catastrophe isn’t as high as it seems.

My Forecast

• 10% chance of a catastrophe that kills at least 10% of the global population by 2100

• <1% chance the global population falls below 5,000 by 2100

At this time a year ago I was in a rehab hospital recovering from a pair of strokes that had left me unable to walk or focus my eyes. My initial prognosis was good, but many stroke victims don’t survive a year. I didn’t know whether I’d make it this long, much less that I’d be able to walk without assistance again. Thank you for all your kindness and for your continuing support over the last year.

1. Toby Ord, The Precipice: Existential Risks and the Future of Humanity (Hachette, 2020).

2. Ezra Karger et al., “Forecasting Existential Risks: Evidence From a Long-Run Forecasting Tournament,” FRI Working Paper #1 (2023): 16.