How to Increase the Accuracy of Geopolitical Forecasts
The question of China’s position and influence on the Russo-Ukrainian war has remained on the geopolitical agenda since the beginning of the war. There are official Chinese government statements, that state that China encourages peace negotiations and promotes the application of international law. However, there are also geopolitical forecasts that predict that China will likely send military aid clandestinely to Russia within the next six months to prevent its defeat. For instance, Grey Dynamics is a British private intelligence firm that published a six-month forecast in March, which analyses the likelihood of Chinese lethal aid to Russia. This forecast predicts that it is unlikely that China will support Russia openly with military aid similar to Western states supporting Ukraine. However, if military assistance is being sent clandestinely, it will most likely be ammunition, drones, and infantry weapons. Additionally, the forecast predicts that non-lethal aid covertly sent to Russia will increase, which could lead to an extension of the war. There are numerous private intelligence companies that provide tailored geopolitical forecasts to clients such as individuals, businesses, or governments. In the past decade, extensive research has been conducted on how to increase the accuracy of geopolitical forecasts. This article focuses on falsifiability, quantifiability, and psychological factors such as training, teaming, tracking, and crowdsourcing.
Geopolitical forecasting is “the process of generating falsifiable predictions about future political events.” For instance, the US government uses them to facilitate policymaking, as their predictions about future geopolitical developments indicate which policies should be prioritized and how resources should be allocated. These forecasts include probability and risk assessments. One important characteristic of any improved forecast is its falsifiability, which means that it can be refuted or confirmed. Falsifiable statements facilitate the evaluation process of geopolitical forecasts by keeping a score of their accuracy. For instance, the statement that China is likely to send lethal aid to Russia within the next 6 months can be refuted or confirmed in 6 months. However, vague statements such as that China will be more aggressive in the next 6 months are more difficult to prove as true or false, as aggressive can be defined differently. This evaluation process can be facilitated by using quantified scorecards, as numerical performance averages of forecasters are more objective and easier to compare than qualitative feedback.
Quantifiability is also a key element for increasing the accuracy of geopolitical forecasts. However, a large number of foreign policy and intelligence analysts believe that the complexity of world politics and their subjective judgment make it impossible to create precise and quantified probability assessments. However, the informational value of geopolitical forecasts can be significantly improved by using “words of estimative probability” (WEPs) or numerical probabilities instead of estimative verbs and confidence levels, which are commonly used in intelligence reports. Estimative verbs include phrases such as “we estimate” or “we assess” that do not convey the level of certainty sufficiently. Confidence levels such as “low”, “moderate”, or “high” are used to convey how confident the analysts are in their judgments, but are often confused with likelihood. This ambiguity can lead to decision-makers interpreting geopolitical forecasts differently than it was intended. WEPs include spectrums of qualitative expressions that function as guidelines to convey certainty levels, Figure 1 shows three examples.
Figure 1 - “Words of estimative probability”
While the WEPs present a method to improve the informational value of geopolitical forecasts using qualitative expressions, there is evidence that suggests that quantification of probability assessments would increase the predictive accuracy even further. Probability assessments should be quantified by using numerical percentages or frequency representations (e.g. 1-in-10), as they become more precise. Additionally, this facilitates the re-evaluation of forecasts after receiving new information. For instance, imagine that an analyst assigns a 79% chance to China sending ammunition to Russia within the next six months and receives information that China has agreed to provide Russia with lethal aid but disguised as civilian items. The usage of numerical percentages enables the analyst to re-evaluate their probability assessment and increase it to 82%. These incremental adjustments would not be possible when using WEPs but over time, they could turn into significant changes from the original assessment.
Furthermore, the accuracy of geopolitical forecasting can be improved by three psychological factors: training, teaming, and tracking. Firstly, training includes methods to overcome cognitive biases to improve decision-making such as scenario training or probabilistic-reasoning training. Scenario training teaches geopolitical forecasters how to use decision trees and envision more possibilities. Probabilistic-reasoning training teaches forecasters when to extrapolate (extending a conclusion to an unknown situation justified by the assumption that existing trends will continue) and how to avoid overconfidence or confirmation bias. Confirmation bias refers to the tendency to seek evidence that verifies own existing beliefs. Secondly, teaming refers to forecasters working in a group which yields better results than working independently, as groups benefit from information-sharing and encouragement. Thirdly, tracking refers to forecasters being sorted into groups of similar ability levels, which accelerates the learning process. Results of a two-year geopolitical forecasting tournament have shown that the team of the best forecasters from the first year considerably outperformed all other participating groups in the second year. These individual differences in cognitive ability measured by cognitive reflection (tendency to re-evaluate an intuitive response and consider alternative solutions) have a significant effect on forecasting accuracy. This proves that to a certain degree, skill plays a role in geopolitical forecasting.
Moreover, the think tank Perry World House has published a white paper that states that the geopolitical forecasting capacity of the US government can be increased with a crowdsourced geopolitical forecasting platform. Crowdsourcing refers to including the general public in a project by asking them to contribute information and ideas. According to empirical evidence collected since 2010, the knowledge of the general public can be applied to national security problems. For instance, the Intelligence Advanced Research Projects Activity (IARPA) organizes tournaments, where the best forecasters performed 30% better while being limited to open-source intelligence than career intelligence analysts. Therefore, introducing an open forecasting platform as a complement to traditional forecasting methods would benefit the US government in increasing its geopolitical forecasting capacity.