The long-awaited combinatorial site is now live! Many thanks to our software team and testers.
Things you should know:
- Clear your browser cache! Otherwise you could end up with an odd mix of old and new javascript and CSS. (Things will look odd and/or break.)
- Visit the tutorials! (See the welcome screen after logging in, or go to http://help.daggre.org.) Look around.
- Note: The previous conditional questions were locked when switching to combo. The new way to edit conditionals is to edit the target after assuming the condition. For example, if the old question pair was, “If A, will B happen?” and “If not A, will B happen?”, you would now edit B assuming A. In steps:
- Select the target question (B).
- Click “Show Related Questions”
- Put (A) in the Current Assumptions Box.
- Assume A is True (or False).
- Edit B.
- We already did this to initialize the corresponding new-style conditional probabilities to the current values from the old ones.
- Your assets in the old-style conditionals are temporarily locked as well. We should be able to release them soon. In the worst case, they will release when the question resolves. If this causes hardship, let us know.
- The combo website is currently a little slower than the old flat one. It will probably take us a couple of weeks of tuning to get all response times under 3 seconds again. In the meantime, please be patient. Most are now 5-10s, sometimes longer. Remember, it’s doing a lot more updating than the old site.
- We will be posting several other conditional types of questions on the market in the near future, such as “Will A happen if B happens or if C happens or if D happens?” This is just an example. By posting 4 separate questions instead of 1, you will be able to make your own prediction and investments using the new features the combo market brings.
- We need your participation! The new features allow you to make your own predictions given your own assumptions! But don’t neglect just making “plain” unconditional predictions too.
- We are looking for your feedbackwith respect to any issues or bugs you encounter and also how you like the new combo market!
- Here’s a bug! The graph works fine with assumptions, but with no assumptions set, it doesn’t match the estimates. It’s showing the raw edit history, regardless of assumptions. That’s getting fixed soon!
Sincerely,
DAGGRE Team
Related articles
- Software Update This Week: Combo! (daggre.org)
- Analysis Training Workshop Update (daggre.org)
- Forecasting the US Elections (daggre.org)
- Conditional Reasoning (daggre.org)
1) The new site is very slow. But excellent in all other respects. Although I think it just crashed. At least, as of 8:05 AM pacific time, Nov. 1, it is not loading for me. Did I break it? I just bet that assuming the Euro is not below $1.20 at the end of the year, nor will gold be > $2500/oz.
2) I like that the assumptions allow me to “leverage” the points I will have when my expected outcome occurs. For example, I don’t think X will happen and have made a bet to that effect. When X settles, I will have more free points. So I can assume X and bet now on Y, whether or not I think X will affect Y. In effect, I am borrowing from my future self to bet on Y, but if my bet on X goes sour and my future self doesn’t have the points to lend, it’s as if my bet on Y never happened. If, like me, one is perpetually short of free points, this allows me to improve the accuracy of the market on Y _now_ which I would otherwise be unable to do for liquidity reasons. If Y settles at the same time as X (or very close to it), without this feature I would probably never edit Y, since I wouldn’t have the free points until too late.
3) As a corollary of (2), I think one should be allowed to impose Assumptions and then edit ARBITRARY questions, rather than only questions you dub “related”. I realize this violates the spirit of Assumptions as modelling possible states of the world and helping to estimate conditional probabilities. But the liquidity boost as in (2) is awesome, and in effect targeted only at active participants. It lets those who have already made edits make more edits, without implementing a generic point inflation.
4) Unrelatedly, where are the results of all IARPA contestants posted? I want to know how accurate Daggre is versus other markets (like GJP).
(1) Thank you. We’re speeding it up. It will likely never be as fast as the flat market, but core benchmarks suggest under 3s response time is achievable. No record of a reset near 0500 EST — must’ve just been slow.
(2) Good thinking!
(3) We’d like that too. But it’s hard. To do it exactly requires complicating the underlying Markov model, and the complexity grows exponentially. We have two approximations in mind, but there are tradeoffs.
(4) They’re not posted, because IARPA discourages framing it as a competition. However, I’ll see if we can post unnamed summaries. In short: we’ve been holding 2nd for awhile, but performance has dropped recently. We need participation, esp. on IARPA questions.
The big performance question this year: does the power of combo offsets its complexity? (Assuming the website were no slower, that is. Clearly with today’s lag time, the answer is no! But even running just as fast, all the extra options mean more chances for distraction. That’s one reason the CAB stays small — too discourage making too many assumptions at once.)
First speed boost in place. Browsing and making assumptions are now usually under 3 seconds in our testing.
A few suggestions.
1. Put an “i” or some symbol next to IARPA questions, this will hopefully reduce the number of people gaming those questions to improve their scores
2. On individual questions, under the history/”Direct Edits” consider making “When” the default that the edits are sorted by and in “Exchange” consider making the “claim number” the default questions are sorted by.
3. In blog posts discussing questions it would be grand if you could state the full question instead of just the question number so we know which question you are discussing.
4. Asking the site to do anything takes much more than 3 seconds, even clicking a link or logging in seems to take 20 seconds at least, additional speed boosts are necessary.
5. When I click on someones name, I am taken to their profile, which is great, but the Direct edits list I see under their profile shows my edits. It would make more sense if it showed their direct edits under their profile. On that same users profile, a section says “How would different outcomes affect my score?” but it actually show how their choices would affect their scores, if you change the Direct Edits section to show their edits this will not be confusing and it would be even less confusing if the wording was changed to “How would different outcomes affect the score?”
@halfpast:
1. Good idea. We’re not worried about gaming: if you understand the system and do something you expect to increase your score, it should also lead to better forecasts. But a more prominent sign might make people take them more seriously.
2. Yes! The patch is already waiting to be applied.
3. Agreed, but we’ve been asked not to publish many questions verbatim.
4. Alas yes — a tax on our most active participants. Bad bad bad. Fix is on the way. Design goal is <3s for most (95%) of actions.
5. (a) Yikes! Good catch. We missed that in testing. *Blush* (b) No, that section really should show how outcomes would affect their score. "Choice" is probably a bad column label.
Speed was still very bad for people invested in many questions. We’ve got a patch in testing that helps a lot by reducing what we show on the left-hand-side question list. We’re also finding faster ways to calculate that.
NB: Another user writes,
She did demonstrate a couple of Chrome glitches, notably clicking “Show All Questions” activates the Category menu instead. (?!). Chrome is an officially supported platform, so we’ll work on them.
New update launched in the last hour or so. It turns off the “position” indicators — up/down arrows showing if you would gain or lose on a particular question. They’ll return when we can speed them up.
Speed is now either about 3s (in cache) or 8s (engine) for an action. Some operations will get slower when there are many people editing at once.
History table sorts by date (reversed) by default.
Also, Equalize has some cases where it won’t work.
HAL9000 noticed the equalize bug:
Clear bug … I bought in by raising this from 1% to 2%, then Dopefer raised it from 2% to 3%. When I equalized out, I should have had a profit of 1 point or so, but in fact got NOTHING. My profit was just ERASED.
HAL9000 — yesterday
@HAL9000: Yes, we found that last night. If you look again, it’s not that your profit was erased — it’s that the Equalize DIDN’T DO ANYTHING. (It changed the estimate 0%.) It happens in questions with two “children” in the underlying Bayes net. For now we changed the interface so that when nothing happens, it *looks* like nothing is happening. A proper fix is in the works.
ctwardy — now
What exactly happens in questions with 2 (or more?) children in the Bayes net? I just noticed that my edits on 284 (assuming “False” for 62 and 144) are having no effect: the estimates are accepted, and (in the state where 62 and 144 resolve false) I have sufficient points available, but there is no effect on the estimate for 284.
Incidentally, has the DAGGRE group published anything regarding its algorithmic approach to updating the conditional probabilities in its bayes net? This seems to be an active area of research in computer science / machine learning, but I am curious how the application to DAGGRE is different from / similar to other prominent applications of probabilistic graphical models.
@sflicht: Not sure what’s going on with your edit. Q.284 only has one child in the Bayes net (but three parents), so it’s not likely to be the same Equalize bug. However, I see you had three edits apparently for 0% Change, which is odd. I might need you to work with someone.
Yes, our approach most certainly has to do with propagation in a Bayes net. We presented our algorithm at UAI 2012. The paper is available by itself from our site and as part of the proceedings. The market also has to efficiently propagate assets for each participant. The paper proves we can use a parallel junction tree structure (one per participant) to hold the assets, and do some clever things there.
The website is not being quite clever enough, and we’re trying to find places where it’s doing more computation than necessary.
@sflicht: it seems the core problem is that the system got behind in showing your actual points available. We’re looking into it.
NB: in general I’d like to encourage bug reports to use the Support Center (help.daggre.org), so that we can track them. Blog comments are easy to miss or lose track of.
Pingback: Site Updated! « Future Perfect
Pingback: And all that that entails…. « Future Perfect
Pingback: Bugs & Bonuses « Future Perfect