Scooped by The New Yorker

After scooping Time magazine earlier in the year, I now find myself scooped by The New Yorker. I have been planning to blog about knowledge markets, or predictive markets for a couple of weeks, and then, bam!, the New Yorker gets there first. The article they published is, of course, excellent (go ahead, read it).

The underlying ideas here are based on trading financial instruments called "Futures Contracts", which you can read all about on Wikipedia. If you are not familiar with these, the closest you have come is hearing about commodity futures, probably a news story about the price of oil futures or wheat futures. Using these you can, on a given day, buy (or sell) a future version of something (oil, wheat, credit risk, and many more). Farmers (and oil companies) use futures contracts to fix the future price of their products and eliminate the risk that the price will go down (although losing the upside if the price moves up). The other side of the deal is someone eliminating the risk the price will move up (but losing the opportunity of the price going down).

A knowledge market provides for buying and selling futures around knowledge or events. It is very similar to making a bet on an event happening in the future. However you are not dealing with a bookie or betting shop, you are dealing with a counter-party who is taking the opposite view.

I was introduced to knowledge markets, as have many people, when I first heard about the University of Iowa political futures exchange. The Iowa Electronic Markets allow people to buy and sell futures relating to the next Presidential election (and other political questions). It has been more accurate than most opinion polls in the last few elections. By providing a simple mechanism for people to speculate (sounds so much nicer than "bet") with each other, the price adjusts to level around which the participant pool "thinks" represents the expected outcome. I put "think" in quotes, because each participant is acting purely out of profit motive based on their belief about the likelihood of the event. As the event gets closer and closer, the price tends to push towards the actual outcome.

Similarly, the Hollywood Futures Exchange is accurate to within 16% for forecasting first weekend box-office revenue for new movies. 16% sounds like a high margin of error, but it is far better than any other forecast.

I have been playing recently with Inkling Markets which allows you to make these "bets" with play money. Inkling has a bunch of general interest contracts (sports, weather, politics) and gives you a feeling for how things work. I am also playing with play money on the Popular Science Futures Exchange which trades contracts on science and technology events (e.g., will the iPhone face a recall?). My friend Paul Bleicher sent me a note on another such play-money site called NewsFutures: today's highlighted contract is about whether and when the US will begin leaving Iraq.

There was one academic study from a few years ago that showed that sales executives in an internal corporate knowledge market generated a more accurate forecast of the company sales than did their own direct reporting to management. When sales executives report directly, they are concerned with how their reports make them look, and often inflate expectations. When acting anonymously in a knowledge market, driven only by profit-motive (and not the desire to look good to their manager), their true beliefs are expressed and produce a more accurate result. The profit-motive in these cases generally relate to very small incentives (gift certificates and the like). The idea is not for people to get rich by betting on (and, perhaps, "game fixing") corporate results, but just giving them enough incentive to participate in a fun game and expose the truth from the aggregate opinions of the group.

Inkling Markets and NewsFutures both allow organizations to generate their own knowledge markets. Google noted recently that they have been very successful with the internal use of knowledge markets.

At about this moment you may remember that there was a big public-relations disaster relating to knowledge markets in the US Defence Department after 9/11. Knowledge markets about particular threats were being used to aggregate knowledge from across the intelligence community. This is actually a very smart way to share knowledge without having to share intelligence (something which is tightly constrained by law and inter-agency rivalry). Unfortunately the scandal surrounding the idea that people were "betting on whether or not there will be another attack on America" killed the idea (unless it has re-emerged, as a secret program which the press does not know about).

If you are interested I recommend playing around with these sites (Hollywood Futures Exchange, Inkling or NewsFutures are great places to start, or PopSci for techies). I expect knowledge markets to become much more prevalent over time in our society. In fact, I am a buyer of a futures contract to that effect at 80% or better.