Superforecasting: The Art and Science of Prediction

The Good Judgment Project was founded by Philip Tetlock and Barbara Mellers in 2011 and was essential to identifying and nurturing the superforecasters who are the subject of the book.

A multiyear forecasting study, the GJP began when IARPA was smarting from the shame of recent failures in American intelligence-gathering and thus launched a competition that would help it identify new ways to forecast relevant geopolitical events. The forecasting tournament comprised a competition among five scientific teams led by top world researchers, who would attempt to predict the types of issues intelligence analysts dealt with daily. As one of these teams, GJP had to answer IARPA’s 500 questions about world affairs in the span of four years. The GJP began to overshadow its competitors from the first year, during which it outperformed the control group by 60%. The GJP continued on this path of success, also beating the experts attached to academic institutions from other groups. Indeed, the GJP’s performance was so stellar that IARPA got rid of all other competitors after the first two years. Tetlock’s key insight from the competition was that “foresight is real” and that “some people […] have it in spades” (18). He also discovered that foresight is less an innate capacity than a skill that can be cultivated. At the end of the tournament, the GJP’s forecasts conclusively beat the control group’s intelligence analysts—even though the latter had access to classified data—thereby cementing the path for the GJP’s relevance and continuation.

The GJP’s key asset is its superforecasters. These individuals already possessed traits that were essential to good forecasting such as intelligence, numeracy, open-mindedness, a willingness to adapt to new information, and a commitment to self-improvement. By working on problems with the GJP, superforecasters were able to fine-tune their judgments and aggregate their predictions in teams to produce forecasts of unparalleled accuracy. Superforecasting tells the stories of these superforecasters, emphasizing aspects of their humanity and individuality, such as hailing from Nebraska or being a retiree who drives up and down the California coast in a red convertible. The book’s accounts also show how superforecasters learned from previous failures. For example, while Bill Flack failed to become a mathematics academic, he was able to use the problem-solving skills he garnered on the way to become a truly great forecaster. By offering these details, in addition to step-by-step accounts of superforecasters’ strategic approaches to ambiguous problems, the authors argue that it is not an individual’s knowledge that makes them such an asset, but how they think.

Beyond the scope of the book, Good Judgment Inc. is currently available to partner with commercial organizations, offering the talents of its superforecasters to help companies see into the future and aid their decision-making. While the original GJP forecasters were recruited via psychometric testing, the GJP continually searches for new talents and sets about finding them through the website Good Judgment Open. Good Judgment Open poses questions about current affairs and asks competitors to come up with a percentage of how likely certain events are to happen. Each question is bounded by a closing date and features the average of all predictions that have already been made. This approach emphasizes Tetlock’s value on giving probabilities a number so that forecasts might be more accurate or easily evaluated if they go wrong. Of all the forecasters who participate at Good Judgment Open, few are selected to be superforecasters. Superforecasters are chosen not only according to the accuracy of their answers but also for the quality of their problem-solving commentary. This latter aspect is important because it contributes to the continual feedback loop that enables superforecasters to continue on a path of excellence.