Future Perfect is the weblog of the DAGGRE Forecasting Project. (Please join, if you’re not already on another team.)
Prediction is hard, especially about the future.
Scientists have long known that the average of several estimates regularly beats the best single estimate. What they don’t yet know is how to beat that simple average.
As one of the research teams of IARPA’s ACE Program, we’re trying to break that barrier. Each team is tasked with beating the control group — a pool of analysts using an unweighted average. There are many good ideas, but without an objective test it would be hard to compare them. IARPA is providing the objective test: once the outcomes are known, our forecasts will be scored by how quickly and confidently they came to the right answer. We expect a lot of good research from Team DAGGRE and the other teams.
This blog will keep participants and researchers informed about the methodology, relevant literature, press items and current events, and published results for Team DAGGRE and others in the ACE program. It’s a team blog, and is not affiliated with IARPA. We hope to contribute to the larger discussion of forecasting, expertise, judgment, and decision-making.
The only relevant test of the validity of a hypothesis is comparison of prediction with experience.
The DAGGRE Approach
The DAGGRE approach is to Decompose, Analyze, Aggregate.
What is Decomposition?
Breaking the problems down into smaller pieces. We encourage participants to specialize, and we will provide tools to help you make conditional estimates. (See below.)
What is Aggregation?
Combining the estimates that all participants contribute. Doing this right is one of the core research problems. Given a group of participants estimating over time, can we devise a system that gives more weight to those who know, only when they know?
We are using several key ideas:
- Edit-based forecasting: you see a consensus estimate and risk some of your credibility (score) to correct it. The core mathematics are isomorphic to more familiar prediction markets.
- Combinatorial or conditional forecasts: Instead of estimating the chance of Chavez’ election directly, you can make it conditional upon his recovery from cancer, and rely on epidemiologists in the crowd to estimate that. Your score is only affected if your conditions are met. This lets you “case out” a forecast.
- Bayesian networks: Combinatorial systems quickly overload. But usually many variables are related weakly if at all. By assuming these independent, we can use Bayesian networks to update probabilities _much_ more efficiently. In addition, Bayesian networks stay coherent by design.
If you are not already on another team, we want you to help us learn how to better understand our world, prevent conflict, avoid pandemics, and predict disasters. [Please Join]
The DAGGRE project is supported by the Intelligence Advanced Research Projects Activity (IARPA) via Department of Interior National Business Center contract number D11PC20062. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon.
Disclaimer: The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of IARPA, DoI/NBC, or the U.S. Government.