Category: Evaluation

Evaluating Scenarios and Indicators for the Syrian War

Every year, The Economist, in its “The World in…” series, assesses it successes and failures regarding its past yearly forecasts (e.g. for 2012). This is an exemplary behaviour that should be adopted by all practitioners: if we are to deliver good and actionable strategic foresight and warning, and to improve our process, methodology and thus our final products, then we should always evaluate our work. Having now completed our last series of updates on the state of play for the Syrian war, we can now start assessing how our own scenarios and indicators fared so far, if they need to be updated and the potential methodological improvements that we should endeavour.

Evaluating the scenarios

As the Geneva conference took place (see previous post), we have obviously been in the case of scenario 1 – Negotiating Peace for Syria in Geneva, in its version 1.2 – all but the Salafis.

Geneva

Thus the overall definition of this scenario and the possibility to see it happening were correctly anticipated, including the fact that the Salafi-Nationalists would refuse to participate in the negotiations. The “Jihadis” were, of course, excluded as foreseen. The fact that the regime of Bashar al-Assad was not represented by a person acceptable to both the regime and the “opposition” – as imagined in the scenario – implied that the negotiations could not be successful.

We moved thus to scenario 1.2.2 – Back to civil war – Salafis’ advantage?  The likelihood to see this scenario happening rather than its alternative, scenario 1.2.1 – An unlikely very fragile external peace was also correctly anticipated (the likelihood was portrayed through the thickness of the lines on the overall presentation for the scenarios – see also explanation in report in pdf – Potential Futures for Syria in the Fog of War).

The latest efforts by the Friends of Syria, notably the U.S. and Saudi Arabia to strengthen militarily the SMC and the “moderate opposition” (see previous post), may also be seen as an attempt, as far as the U.S. and Western countries belonging to the Friends of Syria, to counter the advantage we anticipated the Salafi-nationalists would have in this scenario and that appeared as likely before Geneva.

We are here at the heart of the difficulty of evaluation as evolving understanding by various actors and related decisions have the potential to alter the likelihood and the content of scenarios. In the case of a product delivered publicly by a think tank, thus without the capacity to trace with certainty any influence on decision-making, we cannot know if actions that could, in the future, infirm scenarios are a sign of failure (we failed to anticipate the main potential chains of decisions for actors and the related scenarios) or success (we anticipated properly and a change of understanding related to the scenarios led to new actions).  This would not (or should not) be the case if influence on decision-making can be traced. Indeed, in the case of a strategic foresight and warning process being established or created within an institutional framework (be it for a single product/issue or for the full organisation of units/offices) care will be taken to make sure this traceability can be ensured.

scenarios Syrian War, indicators, evaluating scenariosWhatever the case, methodologically, the new military strengthening of the “moderate opposition” emphasises – as underlined in the introduction of the full report on the future of Syria – that the scenarios we created and that were focused on the Syrian actors and the Syrian battlefield, should be complemented by a detailed state of play for all external actors, then leading to revised scenarios and potentially to more cases and narratives. The difficulty then will be to find the right balance in the number of scenarios, enough to cover most “lines of plausibility” but not too many that would be confusing to readers and decision-makers.

We also correctly considered that the Kurds would be involved in a peace conference, and that their participation would initially stem from the Supreme Kurdish Council (SKC) and estimated that their presence or absence would not influence the immediate prospects for the conference. In reality, in Geneva, we indeed had a Kurdish representation but if it evolved from the SKC as anticipated, this was only temporary, and it led the Kurdish National Council (KNC), which finally went to Geneva as only one part of the overall Syrian opposition delegation, to leave the SKC, while the SKC sponsored the implementation of the autonomous Rojava (see The Kurds and Rojava, State-Building in the Syrian War).

sipanhemo1 scThus, we underestimated the internecine fights within the Kurdish groups. We also potentially underplayed the Kurdish response to the lack of support and interest for the Kurdish issue both within the Syrian “opposition” and within the international “community”, considering their determination to see their fate taken into account and not to be again a powerless pawn in others’ game, as shown by the creation of Rojava. As a result of the evolution within the Kurdish nexus, the potential strategic importance of the Kurds has most probably changed, and the various existing scenarios should be supplemented to reflect this evolution. Existing scenarios should also be revisited in this light.

Scenarios, cicil war, SyriaConsidering furthermore our increasingly unstable world, as underlined by the crisis in Ukraine and Crimea, the potential scenarios regarding a partition (that was mentioned – see grey box – but not detailed as it was estimated to be outside the chosen time frame, i.e. 5 years starting April 2013), even if the Syrian Kurds make sure to emphasize this is not their aim, may similarly start to be addressed more in detail. We see here at work the difficulty to make scenarios under the fog of war as well as the complexity to address timeline and timeframe notably in a global, connected world in transition.

Checking the indicators

Let us now check the indicators we had identified to estimate the likelihood to see scenario 1 happening.

The question we had selected that presided over the choice of indicators that should be monitored in the future (i.e. once the scenarios finalized) is “What could enhance the likelihood to see such a scenario happening?”

When an indicator is monitored, the facts that are collected and correspond to the state of the indicator at a specific moment in time are called indications. In other words, an indicator is a variable that may take a range of values in reality. We then use the indications collected to make an assessment of the possible answer question related to the indicators.

To take an analogy, if we imagine many roads leading to scenario 1, each indicator could be seen as a traffic light opening or closing each road, and each indication would be the colour taken by the traffic light. Furthermore, at each traffic light, the longer the driver would stay the higher the odds it would choose another path or decide about doing something else than going to scenario 1. If all traffic lights show the colour green, then all drivers should arrive quickly to scenario 1 while the likelihood that each arrives are also enhanced. The arrival of as many drivers as possible enhances the likelihood to see scenario 1 happen. Of course, reality is more complex, most of the time there are more than one traffic light on the same road, while some roads and some drivers may be more important than others.

When we also asked the question “What are the supporting facts increasing the plausibility of scenario 1?” to be able to estimate what was the likelihood to see this scenario happening at the time of the creation of the scenarios, we were looking for the indications existing when the scenarios were written.

  1. The civil war in Syria drags on: Yes
    • meeting syrian weaponsWith specific evolution favouring diplomatic talks between international powers: Yes (notably as a consequence of the chemical disarmament following the August 2013 crisis – see corresponding part in Update 7 October 2013, Al-Assad regime groups). The type of evolution over the winter 2012/2013 that made scenario 1 plausible in the first place continued and thus reinforced the likelihood of the scenario, until making it a reality. For the sake of clarity for future monitoring, what was used to assess probability in May 2013 could have also been rephrased and made explicit for the future, rather than left in an implicit form.
  2. Fear by external actors to see further use and spread of chemical weapons: Yes, as exemplified during the August 2013 crisis.
  3. Heightened fear by external actors to see the Syrian conflict spilling over further: Yes, considering the evolution in both Lebanon and Iraq, which only worsened with time. Yes also, in the “diplomatic” field, when we consider the hardening of the Saudi position, for example.
  4. Difficulty to implement rapidly, efficiently and with a high likelihood of success any other solution: Yes, with the American and European fear and inefficiency (to moderate according to country) in reacting to the chemical weapons’ crisis, on the one hand, and with, on the other, potentially, the difficulty of Russia (and Iran) to always influence the regime of Bashar al-Assad (see for example Iran’s reaction to the chemical weapons crisis – e.g. Marcus George, 16 Sept 2013, Reuters, and the speed with which Russian minister of Foreign Affairs Lavrov picked upon and supported the chemical disarmament’s solution).
  5. Cost of intervention for intervening countries: Yes, especially for the U.S. and Europe, not only in economic terms, considering public expenses and budgets, which continue being reduced, but also in terms of popular support, as again shown during the chemical weapons’ crisis.
  6. Increased violence and multiplication of attacks during negotiations as actors will seek the strongest bargaining position possible at the negotiation table: Yes. Indeed, no ceasefire took place while new military positioning took place among the various actors’ supporters (see previous post).
  7. Discussions, declarations, bargains and twists regarding the participation of various actors: Yes, as the month leading to Geneva showed, including the “affair” of Iran’s invitation by the U.N. – e.g. BBC News 20 January 2014, or the impact of Geneva on the Kurdish groups (see above), or the reconfiguration of the Salafis groups and their relationship to the NC and the SMC. This was (and still is for the future, when another peace conference will take place) a particularly important indicator as from the composition of the delegations and their relative strength depends the result of the conference.

As all indicators turned “green and even greener” (see analogy above on traffic lights) as time went by, then it meant that the likelihood to see scenario 1 happen, as well as the proximity of the event on the timeline both increased. The content of the indications (those facts that are collected and correspond to the selected indicators) told us which sub-scenario would happen and enhanced the already identified  likelihood to see the peace talks succeed or fail and thus to see sub-scenario 1.2.2 take place, which is equivalent to a specific version of scenario 2.

This evaluation shows first that scenario 1 and its sub-scenarios, as well as the related indicators are valid and can thus be kept for the future, as one day another peace conference will take place.  It also shows how both strategic foresight and strategic warning should be utilized jointly. It then underlines that the scenarios should now be supplemented by more sub-scenarios, especially to consider the potential novel strategic strength of the Kurds, as well as the change in policy of some of the Friends of Syria. Meanwhile scenarios envisioning a partition of Syria may now be added in more detail.

By checking scenarios and indicators, updating and supplementing them accordingly, we make them a coherent organic whole, which takes stock of reality as time passes and thus remains useful to policy and decision-makers.

Featured image: U.S. Secretary of State John Kerry joins with attendees the Syrian Donors’ Conference for a group photo in Kuwait City, Kuwait, on January 15, 2014. [State Department photo/ Public Domain]

Assessing End of Year Predictions: How Did they Fare? (2)

Here are the results of our experiment on the evaluation of a sample of 2012 end of year predictions, following up on the post explaining the methodology used (spreadsheet and an interactive version of the charts can be found here).

Let us start with the bad news. As a whole, the percentage of success is relatively low, 27%, i.e. 44 predictions were correct out of the 165 made. However, this global figure hides very different results.

In terms of method, as shown below, classical analysis (that may cover the use of other methods or not) obtains the whole range of results, from complete inaccuracy to excellent. The validity of the judgement on the future depends upon the knowledge, understanding and genius of the analyst.

predictions, 2012, success rate, evaluationRisk analysis fares better than overall sample, but is still below 50%. This might be related to the absence of differentiation between likelihood and impact as explained in the previous post.

Our sole example of scenario is relatively unsuccessful. However, this is also linked to the very specific form and place scenarios have in terms of foresight: fictionalized narratives mainly aim at making one plausible version of the future real for the target audience. They intend to break cognitive biases and other lenses. They must be built upon a coherent model, which can be seen as the principle, the essence, but the unfolding discrete events themselves are only one example of what might happen. In Kant’s understanding, a scenario is a phenomenon, built upon noumena.

Unsurprisingly, analysis that includes, more or less, a part of recommendation and advocacy, what we could see as normative predictions, do not fare very well.

This brief evaluation, however, tells only one part of the story. As explained in the methodological post, we can draw much more interesting conclusions out of an assessment that is less drastic and marks each prediction first according to the plausibility of the content and second to the accuracy of the timing, despite the inherent subjectivity of the approach.

Issues and countries: a conventional view of national security

The first very interesting result this experiment gives us is about the topic of the predictions itself, what was deemed as relevant and interesting enough to be the object of anticipation.

The overwhelming majority of predictions were made according to countries, be they focused on economics, political economy, geopolitics or politics. The map below shows the intensity of the number of predictions made, the brightest the colour, the more numerous the prediction. Some countries were off the radar, when, for example, coups in Mali and Guinea-Bissau happened, as correctly predicted by Jay Ulfelder, whose forecasts were not included in the experiment. This underlines the danger to leave some countries out when making judgements on the future, because one will automatically tend to focus on those countries where events or problems occurred in the recent past, or on those that were of interest for one reason or another. The limited character of resources however most of the time forces such initial selection, which thus must be made with great care and kept in mind.

nbre per countries scaled1Very few assessments concerned other global problems, when they belong to what is called unconventional national security. Among those identified in our sample, we find: oil, water, gold, the virtual and digital world (although hardly with a cyber-security dimension), augmented reality, and the environment (but only in terms of regime and debates, not in terms of actual natural events and their impacts). Many issues such as most transformational technologies, from nano to biosecurity, health concerns, cyber threats, extreme weather events or resources competition beyond oil were thus left out. One possible explanations is that we are still operating within specializations inherited from the last three centuries, and that for each new issue appearing on the agenda of national security, a new sector of expertise is created, with serious potential adverse consequences on our identification of threats. We may very well become perfect in terms of predictions on old topics, this will always remain insufficient if interactions and feedbacks with new threats are ignored. For example, International Relations – or geopolitical – analysis must fully include the cyber dimension, and cyber-security in terms of national security cannot be fully understood without the international, geopolitical and political dimensions.

Systematically including horizon scanning for emergence of novel dangers and pluri-disciplinary/multi-expertise work would be needed. Another possible explanation is that those unconventional security issues were left out because they were estimated as beyond the 2012 time horizon. We may only wish this latter hypothesis to be correct.

Inaccurate timing and relatively plausible content

If we now look at the countries, object of predictions, and colour them first according to the plausibility of content of the predictions, and second according to the accuracy of the timing, we have the two following maps. The averaged accuracy of the results goes from deep red (inaccuracy) to deep green (accuracy).

2012, prediction, evaluation 2012, prediction, evaluation

The maps confirm the hunch I wished to test: our capacity to predict timing is less good than our ability to understand content and thus foresee coming evolutions. We know quite well what will most probably happen, but we do not know precisely when.

Interestingly, China, Russia and the U.S. fare relatively badly for both content and timing. This could be explained by strong cognitive and ideological biases existing for those three countries, including, for the U.S., which also ranks first for the number of predictions, those biases related to partisan politics… and analysis. Regarding our initial conclusions on methodology, and considering the lack of explanations given by authors, this shows that we should, ideally, and as underlined by forecaster, futurist and strategist Scott Smith in his Year-end lists are hazardous to your health, identify precisely who the author is, his/her target audience, and in which context the predictions were made. The category mixing classical and normative analysis would most probably swell as a result.

Timing for Brazil is completely wrong, and this would be even worse if the prediction made for the BRICS (0 on all counts) had been added, while the results would have been less good for all the other BRICS. Again, we are seeing an ideological bias at work, a “pro-BRICS” bias, which is also the reflect of a global power struggle we can see enacted in any international fora.

These results point towards the absolute necessity to struggle against all biases when making judgements on the future, if proper decisions are to result from this foresight (which is of most probably not the case with our sample, but we have to consider that many decision-makers also read open source predictions and may be influenced by them, knowingly or not).

Novelty and pace

Finally, let us observe the evaluation for all predictions, without aggregation and average (click here to open the chart in full in another window).

results 2012 predictions scaled 1

Besides the points we already made, what is most striking is the way various water issues were erroneously foreseen. If, true enough, only one author is concerned – and he had the merit to select this issue when the corresponding U.S. ICA was not yet published – we can always learn from all mistakes. This erroneous judgements on water security may underline the difficulty of properly estimating issues when those are relatively newly integrated in assessments. First, there is an insufficient accumulated knowledge and understanding. Second, the eagerness to promote a topic that may still be debated and belittled may lead to overstatement.

The wrong timing on various European countries stems most probably from our very imperfect knowledge of internal political dynamics, as those last decades mainstream political science has tended to focus on elite politics and public policy – one of the major cause of the warning failure regarding the “Arab Spring” – even more so in the case of the so-called rich countries. Furthermore, time is very rarely an object of research. Finally, we tend collectively to forget that the political time is long – even very long – and that much of our (recent) habits, approaches and institutions do not accommodate for it… but this will not change the reality of political dynamics.

————-

Nota: The surprising, at first glance, cases when timing gets a better mark than content correspond to predictions that were accurate (or almost accurate) in terms of timing, accompanied by explanation of dynamics that were partly or fully wrong, illogical, or inaccurate.

An Experiment in Assessing End of Year Predictions (1)

experiment, assessing, evaluating, foresight, forecast, prediction Evaluating predictions, or more broadly the end-products resulting from methodologies used to anticipate possible futures, should become the norm rather than the exception, as explained in a previous post. Such exercise should improve methods and processes and direct our efforts towards further research. We shall here make the experiment to assess a sample of open source predictions for the year 2012. This part will address the methodological problems encountered while creating the evaluation itself, and underline the related lessons learned. The second part (forthcoming) will discuss results.

Actually, there is nothing new here as estimating results for “predictions” is one of the fundamental principles of science (a scientific theory must have explanatory, descriptive and predictive power). If a theory does not fulfill the predictive criteria, then it must be disqualified. Things are relatively straightforward when dealing with hard science. They are much more complex when we are in he field of social science, and the very possibility to obtain predictive power is hotly discussed, debated and often discarded. If we consider the family of disciplines, sub-disciplines and methodologies – what we call here strategic foresight and warning (foresight for short) – that deal with future(s)-related analysis, then we are faced with even more challenges. Some methodologies will be considered as scientific, and among them, some are close to hard science, while others belong to the realm of social science. Other approaches will be seen as art and thus are considered as not having to be tested. Furthermore, everyone has her/his own vision of what constitutes good future(s) related analysis, what should be done and used, what is valuable and what is not.

Despite these difficulties, it is still worth our while evaluating those future(s) related efforts, which had the courage to make an evaluation for the future located on a timeline. This is, of course, a very small exercise and experiment, compared with what is done by The Good Judgement project, led by Philip Tetlock, Barbara Mellers, and Don Moore with funding by the Intelligence Advanced Research Projects Activity (IARPA) and explained in this article by Dan Gardner and Philip Tetlock, “Overcoming our aversion to acknowledging our ignorance.” Nevertheless, hopefully, it will also bring interesting results, and the reflection it imposes, the questions it brings are in themselves a very constructive practice.

For this experiment, the sample used is constituted of open source predictions for the year 2012 posted on the web from December 2011 to January 2012, as presented here.

The result of the evaluation, in a Google spreadsheet, can be downloaded here or viewed below. Explanations and discussion follow.

The sources used to evaluate the foresight are given in the seventh column (except when the answer is obvious or common knowledge, and thus does not necessitate reference to a specific source e.g. the European Union still exists).

The variety of format and methodologies, furthermore more or less explained, was a first challenge. How to evaluate consistently “predictions” delivered in ways as varied as classical analysis (e.g. The Financial Times – Beyond BRICS), scenarios (e.g. Tick by Tick Team), risks (e.g. CFR – Preventive Priorities Survey: 2012) or predictions mixed with policy recommendations and advocacy, what could be seen as a version of normative foresight (e.g. Foreign Policy with the International Crisis Group – 10 conficts to watch in 2012)?

The Council on Foreign Relations, risk and making likelihood explicit

The Council on Foreign Relations’ approach is a perfect example of this hurdle. Its risk list for 2012 was particularly difficult to evaluate considering the way it is formulated and the lack of information regarding the methodology (those challenges have been removed or to the least improved with the 2013 version, where we find more detailed explanations and where likelihood and impact are separated). To find out what the CFR exactly meant I had to turn to a companion article to the risk list published in the Atlantic, “Gauging Top Global Threats in 2012“. There we read:

“The contingencies that were introduced for the first time or elevated in terms of their relative importance and likelihood in 2012 included an intensification of the eurozone crisis, acute political instability in Saudi Arabia that threatens global oil supplies, and heightened unrest in Bahrain that spurs further military action.”

A contingency means “an event (as an emergency) that may but is not certain to occur”.

Thus we can deduce that the CFR saw all the events (their “risks”) listed as possible for the year 2012 – if not probable. This is on this basis that the evaluation was made. Hence, all the CFR “risk-statements” were mentally expanded as follows: “a mass casualty attack on the U.S. homeland or on a treaty ally” means, for evaluation, “a mass casualty attack on the U.S. homeland or on a treaty ally” in 2012 is possible and would have a major impact for US National Interest (according to the tier to which the risk belongs) or “a major military incident with China involving U.S. or allied forces” means “a major military incident with China involving U.S. or allied forces” is possible in 2012 and would have a major impact etc.

Making the “risk-statements” more explicit for evaluation (however not transforming the statement itself) immediately underlines how the fusion of likelihood and impact existing most commonly in the idea of risk (until the concept itself was revised by the new ISO31000: 2009 norm) creates supplementary difficulties in terms of evaluation, hence my personal reluctance to use the concept, despite its fashionable character. What are we to judge: a likelihood? an estimation of impact? a timing? As already mentioned, the CFR Preventive Priorities Survey tackled indeed this problem and now (2013) gives detailed results in terms of impact and likelihood.

This underlines how crucial it would be, ideally, to always include, for all results of future(s)-analysis an estimation of likelihood, as done, for example, in the Intelligence Assessments (see p.14 of the ICA on Global Water Security).

In our sample, each prediction, or series thereof, corresponds to one or another methodology. Yet, rather than trying to standardize thoughts, for example transforming what the authors wanted to write in a sentence easy to evaluate, I chose to keep the text as it was, breaking it down in various paragraphs most of the time, sometimes expanding it mentally as explained above for the CFR, and in agreement with their methodology, but not altering it, and to evaluate it as such. The exercise was constructive in itself and led to interesting points. We shall see with the next post if the results will also say something about the foresight methodology itself.

When the text was far too removed from something that looked like a judgement on the future, for example when it was only an opinion on what was happening, or when it was a 50/50 possibility, I excluded the sentence or paragraph from the sample (in red in the spreadsheet).

Scenarios, timing and content

As I started the concrete phase of evaluating statements with the fictionalized scenario made by Tick by Tick Team (Finance), it very quickly became clear that I had to make two types of assessment: one regarding the plausibility and logic of the content of the prediction itself, the other the accuracy of the timing. Indeed, some of the predictions made still sounded plausible, had not happened in 2012 but could not be ruled out for the short to medium term, e.g. “Greece leaves the Euro, returns to the Drachma.” (3002 – this number corresponds to the identification given in the database, to facilitate reference). To me there is a large difference with a prediction that is plainly wrong in terms of content and thus impossible in terms of timing: e.g. Syria deals with the “initial post-Assad stages” (2011) or “Obama decides not to run for elections” (3011).

Furthermore, this approach will allow me to test a hunch according to which we are in general much better to explain phenomena than to time them, would it be only because we hardly ever work on timing (outside the hard science realm).

Evaluating content and timing: a difficult, uncertain, never-ending task?

Thus, columns 4 and 5 display marks for content and timing, ranging from 0 (completely wrong) to 1 (completely accurate). There is, however, a major hurdle with this approach. First, by judging the content in terms of plausibility of dynamics, I evaluate one understanding (the author’s) against another (mine). There is little we can do about it as this is the core of research and debates in social science, besides giving evidence (column 7, the sources), developing a coherent argument and/or pointing out flaws in the argument subjected to the evaluation. A commissioned report would need to be more detailed and specified than I could be in the framework of a volunteered experiment.

foresight, prediction, evaluationSecond, it implies that by evaluating the plausibility of something happening in the future, then I am myself making a judgement on the future, thus a prediction. Ultimately, those challenges should be resolved through the happenstance of events and facts, which suggests that evaluations should themselves be reviewed and followed in time. This is certainly not ideal, but still better than to lose the information on timing and content, which would happen if one chose with black and white, true or false, 0 and 1 answers.

Objectivity (as much as biases allow) of the person assessing the predictions is crucial, and the use of teams that would discuss and confront their analyses would be best. Furthermore, the latter would also allow overcoming plain lack of knowledge on one issue or another.

This leads us to a last challenge that is not easily overcome for some predictions: the information that is available to the person doing the evaluation. Still using the CFR example, and more particularly the risk of “a mass casualty attack on the U.S. homeland or on a treaty ally”, some actions taken throughout the year by authorities may have prevented a risk to materialize, thus the prediction could be seen as false. Certainly, had no intelligence, defense and diplomatic actions existed, then such risk would have materialised. Such state’s actions are ongoing, and, as an outsider, we can only estimate (without complete certainty) that it is because of them that the threat did not materialize, not because the risk was incorrectly identified. An evaluation made by an insider with access to all classified documents would be made with more confidence. Here, I could only estimate the reality of the risk to the best of my understanding and knowledge, for example with the use of counterfactuals.

Should all those challenges, the existence of uncertainty even in evaluation, lead us to conclude that trying to evaluate foresight products is useless? My first answer, at this stage, is no because all the questions one asks or should ask oneself and that are forced by the evaluation are crucial and may only lead to better methodologies and thus to better judgements on the future. It is thus a gage of quality. We shall see next what the results of the assessment, keeping in mind all their imperfections, may tell us.

—–

“Gauging Top Global Threats in 2012” – Interviewee: Micah Zenko, Fellow for Conflict Prevention, 
Interviewer: Robert McMahon, Editor, 
December 8, 2011, The Atlantic.

2013 Predictions’ Yield (2)

predictions, forecast, foresight, future, windTo follow up on last week’s list of 2013 predictions, here is the yield for the first week of 2013, which should end the series for this year.

Nations prepare for cyber war, David Goldman @CNNMoneyTech, January 7, 2013.

2013: Lucky or unlucky for the world economy? Anastasiya Kostomarova, Business RT, 4 Jan 2013.

Analysis – Doom scenario far-fetched but euro gloom to deepen, Alan Wheatley, Reuters, Jan 7, 2013 (note: not exactly presented as a 2013 prediction, but not very far from it either, and interesting anyway).

2013 Outlook & Forecast, Kiron Sarkar, The Big Picture, January 5th, 2013.

Back To The Future, Zero Hedge, Jan 7 2013.

The 96 Charts That Have To Be Seen To Believed For 2013, Zero Hedge,  01/02/2013 and  corresponding original EYE ON THE MARKET, OUTLOOK 2013, Michael Cembalest, JPMorgan Asset Management’s Chairman of Market and Investment Strategy, J.P. Morgan Private Bank (pdf).

2013, Bruce Krasting, Zero Hedge, 12/29/2012.

10 National Security Threats in 2013, By Lamont Colucci, USNews World Report, January 3, 2013.

A Better Year: 2013 Forecast for Asian Economies, Anthony Fensom, The Diplomat, 3 January, 2013.

10 ASEAN Trends to Watch for in 2013, Prashanth Parameswaran, The Diplomat, January 4, 2013.

2013 Predictions: A Strengthening Economy, John Lounsbury and Steven Hansen, Global Economic Intersection, Dec 30, 2012.

Raising the curtain on 2013, FT writers, FT, December 30, 2012.

2013: Crunch time for the global economy, Joseph E. Stiglitz. World Economic Forum, Jan 2nd 2013.

Global Risks 2013 report, World Economic Forum, 8th January 2013 (forthcoming).

2013 the year ahead and 2013: the experts’ view series, The Guardian, notably:

Ten trends for 2013, ECFR & Nicholas Walton, The European Council on Foreign Relations, 1 Jan 2013.

Featured image: John Dee’s crystal ball. Now in the collection of the British Museum. By Vassil (Own work) [CC0], via Wikimedia Commons.

End of Year Predictions: the 2013 Yield

Predictions and Warning
End of year predictions

As last year, in this section, we shall gather open sources predictions for the New Year, published on the web and interesting for conventional and unconventional national security. As a foreword, we shall start with a few methodological posts, which have been recently published on predictions and forecasts, and that are worth reading. We shall then move to links to 2013 predictions.

On Forecasting and Predicting

Forecasting Round-Up No. 1, No. 2, No. 3, Jay Ulfelder, Dart-Throwing Chimp,  10 Sept, 30 Nov, 6 Dec 2012.

On “Prediction Accuracy”, Josh Calder, World Future Society Blogs, December 11, 2012.

Year-end lists are hazardous to your health, Scott Smith, Quartz, December 24, 2012.

2013 Predictions and Forecasts

The Economist, The World in 2013, with a twitter feed: @EconWorldin

New framework: 2013 and beyond – What will appear and disappear in our lives, Ross Dawson, December 17, 2012.

VMware’s Chief Technology Office series of predictions – Among them:

Global development podcast: hopes and fears for 2013, Presented by John Vidal and produced by Vivienne Perry, The Guardian – guardian.co.uk, Wednesday 19 December 2012.

Council on Foreign Relations

Coup Forecasts for 2013, Jay Ulfelder, Dart-Throwing Chimp, 21 December 2012.

High food prices could be here to stay: Crops lead the field for commodities, Javier Blas in London and Gregory Meyer in New York, Financial Times, 21 December 2012.

The 10 Most Important Economic Questions For 2013 – Business Insider, Dec. 26, 2012.

2013: The strategic agenda, from the Newspaper, , 23rd December, 2012, Dawn.com.

World Economic Situation and Prospects 2013, DESA, UN, 18 December 2012.

IMF’s Lagarde sees advanced economies up 1.6 percent in 2013, Reuters, Dec 16, 2012.

Energy in 2013: What’s next for oil, gas, renewables? David J. Unger, The Christian Science Monitor, December 23, 2012.

Foresight Africa: Top Priorities for the Continent in 2013, Julius Agbor et al. Brookings Institution, January 2013.

Predicting 2013 – Opportunities and Threats, written by wikistrat analysts, special to the Diplomatic Courier, 26 December 2012.

Predictions for 2013 – The Washington Post, 26 December 2012.

10 Conflicts to Watch in 2013, Louise Arbour, Foreign Policy, Dec 27, 2012.

Featured image: John Dee’s crystal ball. Now in the collection of the British Museum. By Vassil (Own work) [CC0], via Wikimedia Commons.

2012 Predictions Database

Last year we gathered as many open source, web-based predictions we could find, for further analysis. As 2012 ends, it is time to go back to them and see what we can learn from them.

This post is a first step as it presents the material gathered as a public usable database, that can be accessed and downloaded as a Google spreadsheet through Google drive. The text of each prediction has been subdivided in as many statements as necessary to allow for evaluation. However, the original text has, most of the time, not been changed to make sure not to introduce unintentional biases.

As a first result, the issues identified in 2011 as worthy of being mentioned in predictions for 2012 are displayed in the graph below.

prediction, 2012

Trial by Fire for Foresight: The 2012 Predictions of The Economist

The Economist, with “The prediction games: Our winners and losers from last year’s edition,” shows the lead in a courageous, crucial and yet hardly ever done exercise: going back to our own foresight and assessing, in the light of the present, what was right and what was wrong. This exercise is not about getting awards or apportioning blame, but about improving foresight. Thus, all results, be they right or wrong, should go with a detailed explanation for the success or failure. This approach is essential to obtain progress. Systematically implementing such a practice in all anticipatory processes should be a major goal for practitioners and a criteria of quality for users.

The Economist self-assessment provides us with a very interesting example of how such lessons learned could be endeavoured, underlines questions that should be asked and key challenges for anticipation, and exemplifies how biases can derail foresight.

Evaluating success

It is often said that it is impossible to validate success, because the very fact to have been accurate will change the world, thus the predicted events will not take place but be altered. This very logical argument depends actually upon the actor’s capacity to influence events. For actors that directly act on their foresight with sufficient might, this challenging problem can be overcome by a particular attention paid to the overall process, distinguishing analysis and anticipation from the response chain. In the case of media, as they are not meant to act but to inform, as their analysis should consider all actors involved and not depend upon the strategic aims of one specific player (see for an explanation Assessing the “Strategic” in Surprise), evaluating success is easier (or more difficult because the influence of mass media should be considered).

The fact is that The Economist underlines the accuracy of its predictions in many areas (and we shall take it at face value). Beyond polite modesty, it would actually be very interesting to understand why the analysts themselves are surprised by their success. Certainly, they were not betting, but relied on something to predict events, phenomena, and this something is what matters (even while betting we use some kind of model). This emphasises the importance of making models explicit, as only those give us the tools for precise assessment. If the model used (and, following Epstein, cognitive models belong to this category) led to successful predictions, then our reliance on this model should increase, and we should pay attention not to destroy it. Changes brought to the model should be documented and integrated cautiously.

Biases, the vexing problem of time and the case of Syria

“A more serious mistake was failing to foresee how bloody the conflict in Syria would become. We thought President Bashar Assad was unlikely to last the year in office.” The Economist, 21 November 2012.

The Economist’s sentence echoes a point made by anthropologist Andrew Turton in his work on Thailand and everyday politics, according to which we often tend to underestimate the power of coercion and violence. Considering the relatively peaceful and mild twenty or so last years, the general disinterest in war and politics (qua politics, not politician politics) because “only economy matters,” then it was – and still is – even more likely to see violence, war and politics grossly underestimated. As global tension is now rising, it would be crucial if we want to improve our foresight to revise the old models to integrate what should never have been forgotten, political dynamics, both international and domestic. This does not mean over-favouring them, but getting more balanced and thus adequate models.

In turn, this underestimation led to another mistake, an erroneous assessment of timing. Apprehending time properly is one of the most difficult problems of foresight and warning analysis (see for example Creating Evertime) and certainly one of the most neglected, as hardly anyone seems to be working directly on it. Here we might have one important element that should be integrated in research: time and timing depends on other variables and their interactions, which is congruent with one of the ways indications and warning deal with this challenge, through timeline indicators.*

Timing again

“A better call, for the progress of the Arab spring more broadly, was that Islamists would make ground but play a cautious and pragmatic game.” The Economist, 21 November 2012.

The Economist article was published on 21st November, just before Mohamed Morsi, Egypt’s President, and “a leader of the Muslim Brotherhood” decided to issue “a decree… granting himself broad powers above any court as the guardian of Egypt’s revolution” (New York Times, 2 Dec 2012; 22 Nov 2012), which led to concern and widespread domestic protest. It was also published before demonstrations flared again in Tunisia on 27th November, for reasons similar to those that triggered the Jasmine revolution (Sarah Mersch, Deutsche Welle, 2 Dec 2012; Amnesty International via Bikyamasr, 1 Dec 2012).

This questions the assessment of success for this specific prediction. On the day of the evaluation, The Economist was indeed successful. However, can a forecast be right one day and wrong the next day, when it is about political dynamics, and especially considering the types of decisions that could be taken according to the prediction?

It is the very framework that we use to evaluate the success or failure of prediction that is at stake here, and this framework is calendar time. We need this framework because all our activities, thus the responses we would design and implement, are planned according to it, but there is also an element of absurdity if we limit ourselves to specific dates. A more interesting way to put predictions when dealing with political dynamics would be to point those out, assessing the patterns at work, the rise or decrease of tension and, more difficult, their estimated timing, which brings us back to the challenge previously underlined.

Black swans or biases?

“As ever, we failed at big events that came out of the blue. We did not foresee the LIBOR scandal, for example, or the Bo Xilai affair in China or Hurricane Sandy.” The Economist, 21 November 2012.

In those cases, the explanation given for the failures shows cognitive biases, most probably the same ones that were at work during the analysis and led to the incapacity to foresee, thus we may expect the same mistake to be reproduced.

Starting with Sandy, the storm did not come out of the blue; it is neither a black swan event (a concept Nassim Nicholas Taleb borrowed from Karl Popper to describe an unpredictable event, which is, with hindsight, re-imagined as predictable) as suggested by The Economist sentence, nor even a wild card (a high impact, low probability event). Any attention paid to climate change, to the statistics and documents produced by Munich-re (e.g the video below on North American weather) or Allianz, for example, to say nothing about the host of related scientific studies, show that extreme weather events have become a reality and we are to expect more of them and more often, including in the so-called rich countries, whatever ideologists say.

It may be impossible to predict the exact event, the day and precise path of a storm, but the likelihood to see “Frankenstorms” in the Eastern part of the US at this time of the year is high and in no way can be seen as an unpredictable surprise. How many similar events and related signals need to occur before we start considering them as likely and thus integrating them systematically in our various forecasts, foresight analyses and warnings.

A similar logic may be applied to the LIBOR scandal and even to the Bo Xilai affair. In a world where the financial establishment believes (rightly because political authorities let it do it) it is all-powerful, where most shy away from the shadow banking liability, where regulation is seen as cumbersome at best, then financial institutions and those working for them can easily conceive of themselves as being above the laws, which means that manipulating the LIBOR becomes completely plausible and not surprising.

The methodological problem we are facing here is as follows: Are we trying to predict discrete events (hard but not impossible, however with some constraints and limitations according to cases) or are we trying to foresee dynamics, possibilities? The answer to this question will depend upon the type of actions that should follow from the anticipation, as predictions or foresight are not done in a vacuum but to allow for the best handling of change.

In an ideal world, it would thus be logical to start with the second goal, that would then allow for the creation of proper mitigating policies, as well as for the design of further directives in terms of surveillance of problems, specific intelligence requirements, etc.

Then one could move to the prediction of discrete events (within the dynamics previously identified), with resources and analytical methodologies correctly allocated and designed according to the nature and characteristics of the potential events. In many cases such as the LIBOR or Bo Xilai, this would imply systematic investigation and intelligence collection, and those have traditionally been part of the media role. In the case of Sandy, we are in the field of warning of natural events, which is handled by the scientific community, by state’s (e.g. meteorological offices) and international governmental organisations.

The Economist courageous and interesting self-assessment of last year’s predictions has thus pointed out the need to make explicit and revise our analytical models (including cognitive ones), notably to fully integrate political dynamics, violence and wars, the importance and difficulty of time’s evaluation, the necessity to think about the use various clients could make of foresight when endeavouring and then phrasing a forecast, while the struggle against all biases must remain constant.

———–

*See, for example, Grabo chapter 6, “‘Timing and Surprise”, who underlines the particular difficulty to foresee timing in the case of military attacks. Grabo, Cynthia M., Anticipating Surprise: Analysis for Strategic Warning, edited by Jan Goldman, (Lanham MD: University Press of America, May 2004).

Epstein, Joshua M. “Why Model?” Santa Fe Institute Working Papers, 2008.

Taleb, Nassim Nicholas, The Black Swan: the impact of the highly improbable (Random House (U.S.) Allen Lane (U.K.), 2007).

Turton, Andrew “Patrolling the middle ground: methodological perspectives on ‘everyday peasant resistance,’” in Everyday Forms of Peasant Resistance in South-East Asia, ed. James C. Scott and Benedict J. Tria Kerkvliet, (London: Frank Cass & Co.; 1986), pp. 36-48.

Featured image: Four Horsemen of Apocalypse, by Viktor Vasnetsov. Painted in 1887. Via wikimedia Commons.

2012 predictions (4)

Jennifer Mc Lean, What Will Happen In 2012? (video)

Various authors for beyondbrics: a series – 12 for 2012 – that beyondbrics is running on key emerging markets topics for the coming year, The Financial Times, starting Dec 27, 2011:

Louise Arbour, Next Year’s Wars: Ten conflicts to watch in 2012, Foreign Policy, Dec 27, 2011

Ross Dawson, Twelve tech trends to shape our future, Technology Spectator, 21 Dec 2011

Roert Reich, My Political Prediction for 2012: It’s Obama-Clinton, The Huffington Post, 12/28/11.

Things That Make You Go Hmmm – 28th December.

Gordon G. Chang, The Coming Collapse of China: 2012 Edition, Foreign Policy, Dec 29, 2011.

IFTF, A Multiverse of Exploration: The Future of Science 2021, Dec 5, 2011.

Bryan Lufkin, InnovationNewsDaily Contributor, Institute for the Future Unveils Its Future Predictions, IFTF, 29 December 2011.

Yadullah Hussain, Too many wild cards cloud 2012 oil price outlook, Financial Post, Dec 30, 2011.

2012 predictions (3)

2012 predictions (3)

Morgan Stanley, Global 2012 Outlook, Global economic Forum, December 15, 2011.

EconMatters, Debt Crisis 2012: Forget Europe, Check Out Japan, Zerohedge, 12/27/2011.

Council on Foreign Relations, The World Next Year: 2012 – A preview of world events in the coming year, (podcast), CFR multimedia, December 22, 2011.

Council on Foreign Relations, Preventive Priorities Survey: 2012, CFR.com, December 8, 2011.

Council on Foreign Relations, Five Economic Trends to Watch in 2012, CFR.com, December 28, 2011.

Sundeep Waslekar (), 12 Trends To Watch For 2012 – OpEd, Eurasia Review, December 27, 2011.

Lance Ulanoff, 5 Tech Trends to Watch in 2012, Mashable Social Media, December 28, 2011.

Zachary Karabell, 2012 Economic Outlook: Why Things Are Better Than We Think, the Daily Beast, Dec 27, 2011.

James Petras, A Doomsday View of 2012, The James Petras website, December 24,  2011.

 

2012 predictions (2)

2012 predictions (2)

ZeroHedge, Globalization, The Decade Ahead, And Asymmetric Returns, 12/26/2011

ZeroHedge, Jim Rogers 2012 Outlook: Pessimism With Scattered Crises, 12/26/2011

Derek Abma on 2012 predictions by Douglas Porter, deputy chief economist with BMO Capital Markets, “Canada to avoid recession next year despite Europe,” The Vancouver Sun, Financial Post, December 26, 2011

Ryan Mauro, Top 12 Threats to Watch in 2012, Family Security Matters, December 27, 2011

Tony Karon, “If 2011 Was a Turbulent Year for Obama’s Foreign Policy, 2012 Looks Set to Be Worse,” (Survey of the top ten global crisis issues facing the U.S. in the new year), Time.com Global Spin, December 27, 2011.

Moneycontrol bureau, “Keep your coats on! It’s going to be a stormy 2012,” (summary of financial and eco predictions by various financial firms/sources),  Moneycontrol.com, Dec 27, 2011.

Kurzweil AI, News, Cyber threats 2012: search poisoning, mobile Web attacks, selling social-media data, (through Futurist Foresight), October 12, 2011.

NASA, 2012: Beginning of the End or Why the World Won’t End?, 22 December, 2011.

Fiona Graham, Fantastic futures? Technology and business in 2012, BBC News Business, 27 December 2011.