Tags
ACE, Aggregation, Brier score, DAGGRE, IARPA, Intelligence Advanced Research Projects Activity, Research
DAGGREISTAS!
Y'all Rock! 24 official questions resolved in the last week – that's half the total – and according to the preliminary numbers from IARPA, we're in 1st place on those questions, and 65% better than baseline! Even counting all data from December (which includes some software-induced disasters), we're 17% better than the baseline, and ranked 2nd out of 6. Looking at just the Feb-Apr3 reports, we're 50% better than baseline, and still 2nd out of 6.
| Baseline | DAGGRE |
PDM |
Rank |
||||||||
|
All Data Average: |
0.375 |
0.343 |
0.315 |
0.225 |
0.364 |
0.318 |
17% |
2 |
|||
|
w/o MultiChoice: |
0.350 |
0.324 |
0.233 |
0.190 |
0.340 |
0.291 |
40% |
2 |
|||
|
Feb. Rpt. Average: |
0.222 |
0.208 |
0.093 |
0.079 |
0.274 |
0.137 |
82% |
2 |
|||
|
Mar. Rpt. Average: |
0.484 |
0.480 |
0.424 |
0.401 |
0.528 |
0.444 |
13% |
2 |
|||
|
Apr.3 Rpt. Average: |
0.351 |
0.307 |
0.179 |
0.188 |
0.304 |
0.275 |
65% |
1 |
|||
|
Feb-Mar-Apr3 Average: |
0.355 |
0.324 |
0.213 |
0.211 |
0.344 |
0.285 |
50% |
2 |
|||
All teams (including us) are checking the numbers, so this result could change, but wow! Keep it up! Resolutions are listed below. They include 10 unofficial (DAGGRE-only) questions. We expect another 7 questions to resolve later this week. We'll post all these results to the blog over the next week or so. New Engine: Around April 13 we plan to replace the core calculation engine with a faster, more robust one. We will replay the entire trade history from scratch and thereby fix several scoring issues that have accumulated. Scores will change some. Stay tuned.
Follow us: Blog – http://blog.daggre.org Twitter – https://twitter.com/#!/DAGGRE_
DAGGRE Questions Resolving by early April
Q.140 IFPID #1062 - Europe Brent Crude FOB - is resolved as "b" / "no" Q.182 IFPID #1097 – Tunisia State of Emergency – is resolved as “a” / “yes”
This was a very close thing only announced on the day of. IARPA delayed resolution until they could confirm in-country news sources, and verify that it met the US timezone deadline.
Q.186 IFPID #1101 - Laurent Lamothe - is resolved as "b" / "No, but he will remain the nominee"
The following IARPA Test & Evaluation (T&E) all resolved “no”:
Q.107 IFPID #1035 - UNSC Iran Resolution
Q.123 IFPID #1045 - Saif Gaddafi Trial
Q.133 IFPID #1055 - North Korea Nuclear device
Q.136 IFPID #1058 - Egypt Israel Peace Treaty
Q.137 IFPID #1059 - Kim Jong-un
Q.147 IFPID #1063 - Taliban Negotiations
Q.148 IFPID #1064 - Yousaf Raza Gillani
Q.151 IFPID #1067 - Traian Basescu
Q.152 IFPID #1068 - UNSC Syria Resolution
Q.153 IFPID #1069 - South Korea Iran Oil
Q.155 IFPID #1071 - Iran U.S. nuclear talks
Q.157 IFPID #1073 - Hungary IMF Loan
Q.160 IFPID #1076 - Tareq al-Hashemi
Q.170 IFPID #1085 - UNSC Haiti Force Reduction
Q.173 IFPID #1088 - Japan Parliamentary Elections
Q.176 IFPID #1091 - Japan S&P downgrade
Q.177 IFPID #1092 - Myanmar Political Prisoners
Q.181 IFPID #1096 - Syrian Civil War
Q.184 IFPID #1099 - Al-Saadi Gaddafi Extradition
Q.189 IFPID #1103 - Sudan Oil Transit Fees
Q.190 IFPID #1104 - Jaar Zinjibar
The following DAGGRE questions all resolved “yes”:
Q.46 Will Hugo Chavez still hold the title President of Venezuela as of 1 April 2012?
The following DAGGRE questions all resolved “no”:
Q.6 Will the World Health Organization (WHO) report more than 2 new actual or suspected cases of the Ebola virus anywhere in the world between 1 August 2011 and 1 April 2012?
Q.13 Before 1 April 2012, will either the Knesset or the Israeli president officially call for a new national election for Israel’s Knesset?
Q.41 Will the Nikkei 225 index finish trading on 1 April 2012 at or above 11,000?
Q.45 Will Iran successfully detonate a nuclear weapon, either atmospherically, underground, or underwater, before 1 April 2012?
Q.47 Will at least one country newly and officially declare that they possess nuclear weapons before 1 April 2012?
Q.48 Will China conduct an overt [previously announced] military or naval exercise in the South China Sea at any point between 1 September 2011 and 1 April 2012?
Q.50 Will there be an ‘authorized strength’ of more than 11000 military and civilian police personnel as part of the UN Mission in South Sudan (UNMISS) as of 1 April 2012?
Q.65 Will the North Korean (DPRK) military be reported as occupying, invading, or firing weapons on South Korean troops, citizens, or territory at any point before 1 April 2012?
Q.145 Will Iran successfully test-fire a ballistic missile between 13 January 2012 and 1 April 2012?
Nice work. Hope there are more new questions soon. Have a bigger balance than I can spend on subjects I’m comfortable evaluating
We launched two last week on Greece and the EU. The next batch from IARPA isn’t due for awhile, but we’re happy to take suggestions. We like new questions too.
Pingback: More forecasting feedback « Future Perfect