We have endured the wrong forecasts of pundits and experts without them experiencing any costs for their errors. Yet, Philip Tetlock ( University of Pennsylvania) highlighted this in 2005, when he released his 20-year study of 284 experts (professors, journalists, civil servants, etc.). According to “Intelligent Intelligence” (The Economist, July 19, 2014 edition) “their performance was abysmal.”
These results automatically created questions about intelligence agencies, causing David Mandel, of Defence Research and Development Canada, and Alan Barnes, a former intelligence analyst to publish, Accuracy of Forecasts in Strategic Intelligence. In this examination of intelligence analysts’ forecasts, they found significantly better forecasters than Tetlock did with pundits and experts.
When they dug deeper into analysts’ personalities, they found caution – even about their own abilities – pronounced especially among those classified as “superforecasters.” If they erred, it was on the side of uncertainty, meaning they were more likely to say, “I don’t know.”
More significantly, experienced analysts forecasted better than junior ones, meaning good forecasting is learnable as long as three protocols exist:
- Accountability for forecasts
- Skepticism by analysts’ managers
- Absence of self-serving biases
As James Surowiecki writes in “Punditonomics” (The New Yorker, April 7, 2014 edition), pundits and experts don’t forecast within these protocols. Their confidence is rewarded and errors unpunished. People can suffer from erroneous intelligence forecasts.
Businesses aren’t immune. Confidence and glowing forecasts easily seduce us. Style often trumps content and competence. Skepticism is difficult when hearing what we want to hear, and more so when seeing it as dissent or pessimism. That doesn’t even address interpreting prudence as underconfidence.
All of which I’m prudently confident will change . . . some day.