Notes on Forecasting the Invasion of Ukraine
Should we have seen it coming more clearly?
Should we have seen Russia’s invasion of Ukraine coming? Some forecasters were confident that Russia would invade, but that doesn’t necessarily mean they were right to be confident. Wars on this scale are rare, so it’s appropriate to be skeptical they will happen. Because the question depended to a large extent on what Putin would decide—and because he deliberately obscured his intentions—it was hard to say with confidence whether Russia would invade. But Putin had signaled his intentions more clearly earlier and the massive mobilization of forces was a strong signal he planned to invade. In hindsight, I think we should have realized an invasion was very likely.
Clay Graubard deserves some credit. Clay is a grad student in international relations at Oxford who has been doing invaluable work tracking forecasts about the crisis in Ukraine. On February 12, when I wrote that there was a 65% chance Russia would invade Ukraine, Clay thought there was an 85% chance. When I lowered my forecast briefly later in the week after both sides seemed to soften their rhetoric, Clay raised his forecast to over 90%. Some superforecasters agreed with Clay, but others thought an invasion was unlikely. At the time, I told Clay I didn’t see how he could be that certain. But in fact Russia not only invaded Ukraine, but seems to have been planning to invade all along. Clay has said he doesn’t think the forecasting community did a good job forecasting this question. In hindsight, I think he might be right.
Were we wrong? If you say there’s a high probability of something, and it ultimately happens, it doesn’t necessarily mean you were right. If you say there’s a high probability of something that doesn’t ultimately happen, it doesn’t necessarily mean you were wrong. If you tell me, for example, there’s a 100% chance a coin flip will come up heads, and it does come up heads, were you right to say there was a 100% chance that would happen? Maybe, if you had supernatural foresight or some inside knowledge the coin was fixed. But typically we should assume—because of our prior knowledge of how coin flips work—that the true probability the coin would come up heads was 50%. You’re going to have to predict a bunch of coin flips in a row before you convince me it wasn’t just a lucky guess.
So we can’t draw strong conclusions from a single forecast. Clay’s high forecast is evidence that he knew something the rest of us didn’t, but on its own it’s fairly weak evidence. If we just look at cases when we were skeptical about something that ended up happening, of course our forecasting will look bad. But a fair analysis would also have to also include those cases when we were skeptical about something that ended up not happening. Ideally, of course, we’d like to do a better job of discriminating between the two cases—to say with 100% certainty that one will happen and the other won’t—but with the information available to us that may generally not be possible. The question we have to ask ourselves is, in this particular case, should we actually have known that Russia would invade?
There are strong arguments that 85% was too high. Wars on this scale are historically rare, so we should probably be skeptical they’ll happen. It is also hard to say with certainty what will happen when it largely depends on a single person’s decision—as it did in this case—unless you have a very clear window into that person’s thinking. The fact that Putin was trying to obscure his intentions made it even harder. Russia’s actions in Georgia and Crimea foreshadowed Russia’s recent invasion of Ukraine, but those invasions were also on a significantly smaller scale. They seemed calibrated to stay below the threshold of triggering a unified international response. Even Putin’s inner circle reportedly thought he was bluffing. The US intelligence assessments that Russia would invade were compelling—and were consistent with open-source intelligence—but were clearly part of a messaging campaign designed to box Russia in, so they couldn’t be taken entirely at face value. Even if the decision to invade had already been made, it could still have been stopped at any time. Was there really less than a 1-in-6 chance that it could be deterred or an agreement could be reached?
Nevertheless, I think Clay might be right. For one thing, the views he expressed in his speech recognizing the separatist republics are not new to people who follow Russian politics. Last July, Putin published a letter titled “On the Historical Unity of Russians and Ukrainians” that denied Ukraine’s legitimacy as a nation and compared modern Ukraine to a weapon of mass destruction aimed at Russia. The specific demands he ultimately made of NATO were clearly unreasonable and seemed designed to be rejected.
More significantly, I think the massive mobilization and detailed preparations for an invasion—setting up field hospitals and stocking up on blood supplies—were an extremely strong indicator of his intention to invade. As Jeff Edmonds told Talking Points Memo, it was “a very big force to deploy just to signal somebody”. If it was a bluff, it was a very compelling, costly bluff. When the invasion had effectively been set in motion—as it was by mid-February—it was probably reasonable to conclude, as Clay did, that the chance it would be stopped was fairly small.
Forecasting awful things is awful. One of the questions we looked at my second year forecasting was whether Egypt would execute a prominent political prisoner. Many of us didn’t forecast the question because we were uncomfortable speculating about a person’s death. I’m not sure whether my hope that Russia wouldn’t invade affected my forecast, but it is painful to conclude that a terrible thing is about to happen. I think it’s important to be realistic about the future, but sometimes it feels disgusting—and completely beside the point—to think clinically about awful things. So let me add here that the Russian invasion is a completely unjustifiable atrocity. It threatens the stability of our international system. Ukrainians are our brothers and sisters; we must all be Ukrainians today.
I think you may have inadvertently pointed to a potential blindspot when you conclude "It threatens the stability of our international system". Even if an international system is by definition for all and by all, it is unavoidable to recognize that Russia (beyond Putin himself) feels shortchanged by the European status quo of that system and in that respect would feel less constrained to challenge it as they did. The long history of their failed attempts to modify it peacefully has been obfuscated in the west, as is the disadvantageous position for Russia that came along with that, as well as other long term disadvantages that they have to deal with.
The issue was that the Base rates of saber rattling are much higher than the base rates of actual war. On feb 8th I gave russia a 45% chance of invasion, on feb 17th it was much higher (80%) (metaculus) because of looking not at rhetoric (which has a very low signal to noise ratio) but instead looking at troop movement (which had a high signal to noise ratio). But alas I think my brier score was too heavily impacted by rhetoric rather than by looking at the hard data. The hard data would show that while the base rates of sabre rattling are high, the large update to make was the physical movement of physical resources, which can't be faked.
I'm generally not very good at updating relative to other forecasters and tend to focus harder on establishing good base rates, but it's a good idea generally to update based on physical facts very hard and not update at all on Rhetoric (which is always nonsense).