Predicting the future is an intensely difficult endeavor. Most of the time, when we want to predict the future, we consult an expert in the specific area that we want to predict.
However, there is another possible approach: Find people who are generically good at predicting, rather than experts on specific topics. Philip Tetlock’s research on Superforecasters shows that sort of approach can be very successful. Ability at prediction is a general skill. One way to find such people is by engaging in competitive prediction. In order to do so, one assembles a bunch of questions that would be interesting to predict the results of, and then allow people to make predictions, and track the results.
In order to encourage people to try their best to predict, these prediction platforms set up incentives for people to get the answer right. Prediction markets, such as PredictIt, have money on the line to encourage people to get the answer right. However, the overheads are large and there’s not much money at stake, so the predictions aren’t always self-consistent, much less particularly accurate.
Other prediction platforms use a reputation systems, so people can show that they’ve consistently gotten predictions right. An example of such of site is Metaculus.
One notable case in which this approach was very succesful was the Social Science Replication Project, where scientists were asked to predict (anonymously) whether social science experiments would replicate, via a prediction market. Then, the experiments were reperformed. The predictions matched the results perfectly, with every experiment that replicated being ranked more likely to replicate than every experiment that did not.
What aspects of a prediction platform make it successful? Is money a good incentive, reputation, or something else? What questions are amenable to generalist prediction, versus expert prediction, or a mixture of the two? How can we use competitive prediction to improve our forecasts and decision making? How can we make such prediction platforms robust, if the predictions they make have actual consequences?