Tags
Post-mortem analysis on Syria
Question #183 on the DAGGRE site resolved as “yes,” surprising many forecasters who gave a low probability for the question. Our Brier score, which refers to the distance between our forecast and the true answer, was high (0.81). The Brier score ranges from 0..2, with perfect predictions getting 0, while a coin toss gets 0.5. In short, we got surprised. That doesn’t by itself mean we had the wrong probability, but it should prompt us to look back and see what we might have missed.
In an article from the Guardian, Turkey was the focus of foreign tensions with Syria. Turkey broke off relations with Syria in August 2011. Some Middle Eastern nations hinted at taking a tougher stance against the Assad regime, and Turkey was part of that conversation. However, many of these nations did not appear to want to take a direct approach in dealing with the Syrian government. Much of the Syrian-Turkish tension arose from Syrian support of the Kurdistan Worker’s Party (PKK), which has been a growing source of violence within Turkey.
On September 19th, fighting between Syrian troops and rebels came close to spilling over the Turkish border. The rebels seized control of a border crossing and gunfire did cross the border. Officials in Turkey shut down local schools and urged residents to stay inside fearing for their safety. The influx of refugees from Syria to Turkey and the resultant instability was also a potential motivation for Turkey to take a more active stance against the Assad regime.
A Yahoo article from September 28th reported that a mortar bomb fired from Syria damaged an area in the Turkish Akcakale border area. Turkish areas bordering Syria had previously encountered some spilling over of the conflict. The mortar shell killed two women and three children, and prompted a response from the Turkish government that ended up resolving our question as “yes.”
The resolution of this question took many of our forecasters by surprise, indicating that it was a very difficult question to predict. Knowing why this question was so difficult is not easily definable. Current studies on question difficulty could potentially explain why so many forecasters had trouble with this question. This question encompassed many factors that could potentially influence the outcome of the question. All criteria are being observed on a preliminary basis and have a basis for consideration due to previous multidisciplinary research. One of the highest scoring criteria was decomposition, in which the question is considered more difficult when there is a higher degree of complexity and the question can be broken down into many smaller parts. This question involved many different forces, including a single or combined force from any number of nations, or even breaking the question down to an invasion, entrance, or firing on of Syria. Other factors that contributed heavily to the question’s difficulty were the question’s factors being partially observable to open source analysts, continuous answer possibilities, adversarial actions, and the lack of agreement among sources. Questions that are more continuous have a wider range of answers that could potentially be correct. Adversarial questions are those in which the information necessary to create intelligence is being actively blocked. When sources do not display a consensus opinion, this is referred to as source polarity. This criteria makes it more difficult for an analyst to display confidence in a forecasted course of action. The other criteria listed in the model are open class, stochastic, data synthesis, broad focus, and time constraint. Open class questions are those that cannot be answered without giving a full explanation and conditions. Stochastic questions are those that do not involve active decision making and are more subject to random change. Questions requiring high levels of data synthesis are more difficult because of the amount of information that must be reconciled. Broad focus questions are those that do not clearly define a single topic of focus, therefore introducing more elements to consider in the analysis of the question. Questions that have a strict time constraint are more difficult due to the stress that is put on the analyst to collect and analyze data in a constrained time frame. The consideration of all of these criteria and the resultant high score on the question difficulty model suggest that this question was more difficult to answer than others on the site, and therefore more difficult to accurately predict.
Was this the question of Turkey sending military troops into Syria? My understanding was that resolution to yes required an unwelcomed incursion of significant (I thought 1000+) Turkish troops on to Syrian soil and that cross boarder fire and bombings did not meet the requirements.
I have seen reports that troops are authorized to cross the boarder, that the number of troops at the boarder has been increased, and that cross boarder attacks have taken place but I have not seen any reports that Turkish troops have significantly crossed the boarder. Did I, and presumably other participants, misunderstand the question? Could that have been a simpler reason for the inaccurate predictions? I am probably thinking of the wrong question but please let me know.