Before we closed the DAGGRE project, one of our last experiments consisted of forecasting the number of civil unrest events in Mexico for a predetermined period of time (in this case, the month of May 2013). We used this question as an experiment for another type of Autotrader, that was “feeding” from daily social media reports.
When we closed this question on May 31st, we didn’t have the resolution at that time, since there is a lag in verifying the Twitter based data with the actual events. Now we have the verified data and there were 149 civil unrest protests in Mexico during the month of May. Our market closed with 65% of the trades on the 11-50 range, only 28% on the above 50 range and 6% on the 2-10 range.
The Brier score for the question was 0.26.
The Brier Score (Brier 1950) is a measurement of the accuracy of probabilistic predictions. As a distance metric, lower is the better. The Brier Score ranges from 0..2, and is the sum of the squared differences between the forecast and the outcome averaged over the number of forecasts. On a binary (Yes/No) question, simply guessing 50% all the time yields a “no courage” score of 0.5. We want to be firmly below 0.1.
The auto trader was responsible for 16% of the market activity on this question and was trading on the “over 50″ range upwards and on the other ranges downwards. Although it was moving the forecasts in the right directions, due to the limitations in changes it was allowed to make, the Brier score of the autotrader was still high, at 0.24. But it shows a score better than the overall score of the market, with an 8% improvement in accuracy.