Here are the results of our experiment on the evaluation of a sample of 2012 end of year predictions, following up on the post explaining the methodology used (spreadsheet and an interactive version of the charts can be found here).

Let us start with the bad news. As a whole, the percentage of success is relatively low, 27%, i.e. 44 predictions were correct out of the 165 made. However, this global figure hides very different results.

In terms of method, as shown below, classical analysis (that may cover the use of other methods or not) obtains the whole range of results, from complete inaccuracy to excellent. The validity of the judgement on the future depends upon the knowledge, understanding and genius of the analyst.

Risk analysis fares better than overall sample, but is still below 50%. This might be related to the absence of differentiation between likelihood and impact as explained in the previous post.

Our sole example of scenario is relatively unsuccessful. However, this is also linked to the very specific form and place scenarios have in terms of foresight: fictionalized narratives mainly aim at making one plausible version of the future real for the target audience. They intend to break cognitive biases and other lenses. They must be built upon a coherent model, which can be seen as the principle, the essence, but the unfolding discrete events themselves are only one example of what might happen. In Kant’s understanding, a scenario is a phenomenon, built upon noumena.

Unsurprisingly, analysis that includes, more or less, a part of recommendation and advocacy, what we could see as normative predictions, do not fare very well.

This brief evaluation, however, tells only one part of the story. As explained in the methodological post, we can draw much more interesting conclusions out of an assessment that is less drastic and marks each prediction first according to the plausibility of the content and second to the accuracy of the timing, despite the inherent subjectivity of the approach.

Issues and countries: a conventional view of national security

The first very interesting result this experiment gives us is about the topic of the predictions itself, what was deemed as relevant and interesting enough to be the object of anticipation.

The overwhelming majority of predictions were made according to countries, be they focused on economics, political economy, geopolitics or politics. The map below shows the intensity of the number of predictions made, the brightest the colour, the more numerous the prediction. Some countries were off the radar, when, for example, coups in Mali and Guinea-Bissau happened, as correctly predicted by Jay Ulfelder, whose forecasts were not included in the experiment. This underlines the danger to leave some countries out when making judgements on the future, because one will automatically tend to focus on those countries where events or problems occurred in the recent past, or on those that were of interest for one reason or another. The limited character of resources however most of the time forces such initial selection, which thus must be made with great care and kept in mind.

Very few assessments concerned other global problems, when they belong to what is called unconventional national security. Among those identified in our sample, we find: oil, water, gold, the virtual and digital world (although hardly with a cyber-security dimension), augmented reality, and the environment (but only in terms of regime and debates, not in terms of actual natural events and their impacts). Many issues such as most transformational technologies, from nano to biosecurity, health concerns, cyber threats, extreme weather events or resources competition beyond oil were thus left out. One possible explanations is that we are still operating within specializations inherited from the last three centuries, and that for each new issue appearing on the agenda of national security, a new sector of expertise is created, with serious potential adverse consequences on our identification of threats. We may very well become perfect in terms of predictions on old topics, this will always remain insufficient if interactions and feedbacks with new threats are ignored. For example, International Relations – or geopolitical – analysis must fully include the cyber dimension, and cyber-security in terms of national security cannot be fully understood without the international, geopolitical and political dimensions.

Systematically including horizon scanning for emergence of novel dangers and pluri-disciplinary/multi-expertise work would be needed. Another possible explanation is that those unconventional security issues were left out because they were estimated as beyond the 2012 time horizon. We may only wish this latter hypothesis to be correct.

Inaccurate timing and relatively plausible content

If we now look at the countries, object of predictions, and colour them first according to the plausibility of content of the predictions, and second according to the accuracy of the timing, we have the two following maps. The averaged accuracy of the results goes from deep red (inaccuracy) to deep green (accuracy).

The maps confirm the hunch I wished to test: our capacity to predict timing is less good than our ability to understand content and thus foresee coming evolutions. We know quite well what will most probably happen, but we do not know precisely when.

Interestingly, China, Russia and the U.S. fare relatively badly for both content and timing. This could be explained by strong cognitive and ideological biases existing for those three countries, including, for the U.S., which also ranks first for the number of predictions, those biases related to partisan politics… and analysis. Regarding our initial conclusions on methodology, and considering the lack of explanations given by authors, this shows that we should, ideally, and as underlined by forecaster, futurist and strategist Scott Smith in his Year-end lists are hazardous to your health, identify precisely who the author is, his/her target audience, and in which context the predictions were made. The category mixing classical and normative analysis would most probably swell as a result.

Timing for Brazil is completely wrong, and this would be even worse if the prediction made for the BRICS (0 on all counts) had been added, while the results would have been less good for all the other BRICS. Again, we are seeing an ideological bias at work, a “pro-BRICS” bias, which is also the reflect of a global power struggle we can see enacted in any international fora.

These results point towards the absolute necessity to struggle against all biases when making judgements on the future, if proper decisions are to result from this foresight (which is of most probably not the case with our sample, but we have to consider that many decision-makers also read open source predictions and may be influenced by them, knowingly or not).

Novelty and pace

Finally, let us observe the evaluation for all predictions, without aggregation and average (click here to open the chart in full in another window).

Besides the points we already made, what is most striking is the way various water issues were erroneously foreseen. If, true enough, only one author is concerned – and he had the merit to select this issue when the corresponding U.S. ICA was not yet published - we can always learn from all mistakes. This erroneous judgements on water security may underline the difficulty of properly estimating issues when those are relatively newly integrated in assessments. First, there is an insufficient accumulated knowledge and understanding. Second, the eagerness to promote a topic that may still be debated and belittled may lead to overstatement.

The wrong timing on various European countries stems most probably from our very imperfect knowledge of internal political dynamics, as those last decades mainstream political science has tended to focus on elite politics and public policy – one of the major cause of the warning failure regarding the “Arab Spring” – even more so in the case of the so-called rich countries. Furthermore, time is very rarely an object of research. Finally, we tend collectively to forget that the political time is long – even very long – and that much of our (recent) habits, approaches and institutions do not accommodate for it… but this will not change the reality of political dynamics.

————-

Nota: The surprising, at first glance, cases when timing gets a better mark than content correspond to predictions that were accurate (or almost accurate) in terms of timing, accompanied by explanation of dynamics that were partly or fully wrong, illogical, or inaccurate.