What’s in it for me? Learn what it takes to become a superforecaster.
Everyone’s a forecaster. You’re a forecaster, I’m a forecaster, we all are. Just think about it for a minute. In your everyday life, when you’re considering changing your job, buying a new house, moving to a new country, retiring, or any other major decision, you make predictions about what the future will hold for you.
Now, that doesn’t mean that we’re all good at forecasting, or that even experts are. Indeed, you’ve probably heard that the average “expert” is often no better than a dart-throwing chimpanzee at making predictions. But it turns out that some people are really good at it. These people can make predictions that are 60 percent more accurate than those of regular forecasters. Meet the superforecasters.
In this Blink to Superforecasting by Philip Tetlock and Dan Gardner, you’ll find out why superforecasters are so much more accurate.
What does it take to be a superforecaster?
Bill Flack, a 55-year-old retiree who used to work for the US Department of Agriculture in Arizona, has lots of free time. He uses some of that time to make forecasts.
He answers questions – important ones. They’re questions that companies, banks, embassies, and intelligence agencies have difficulty with. These are questions like, “Will Russia annex additional Ukraine territories in the next three months?” or “Will one of the EU countries withdraw from the Eurozone within the next year?”
The thing about Flack is that although he might not have any idea about the answers to such questions when he first sees them, he does his research and gets as much information about the topic as he can before he decides how to answer.
All of Flack’s answers have been documented and checked for accuracy by independent scientists. His success rate is remarkable. Unfortunately, nobody actually bases decisions on Flack’s predictions. You see, Flack is one of thousands of others who are all answering the same questions. They’re all volunteers, and only about two percent of them are as good as Flack at making predictions. They come from all walks of life, and they’re part of the Good Judgement Project, or GJP, a research project cocreated by Philip Tetlock and two other professors at the University of Pennsylvania.
The GJP was part of a larger research project carried out by the Intelligence Advanced Research Projects Activity, or IARPA, which set up a forecasting tournament. The GJP was one of five teams pitted against each other to answer nearly 500 questions on world affairs over a four-year period. In the first year, GJP outperformed the control group by 60 percent. In year two, that increased to 78 percent.
Out of the research came two key conclusions. First, some people have real foresight and can make accurate judgments about important events that might happen up to 18 months in the future. And second, it isn’t important who the person is, but rather what they do. Becoming good at forecasting is more about how you think, gather information, and update your beliefs – skills that any intelligent, thoughtful person who is determined enough can learn.
We’ll cover the techniques in the rest of this Blink.
The rigorous art of superforecasting
As we hinted at in the previous chapter, it’s not because superforecasters have genius-level IQs that they’re able to outperform experts. It’s the techniques that they use, including specific evidence-based reasoning methods.
Superforecasters break down difficult questions logically rather than rely solely on intuition. For example, in 2012, when superforecasters were asked to predict whether evidence of poisoning would be found in Yasser Arafat’s remains, they avoided intuition-based guesses about Israel’s role. Instead, they broke the question into parts. First, they researched whether polonium, the suspected poison, could be detected so many years after his death. It was only after confirming the technical feasibility that they considered political motivation – including other political actors such as Palestinian rivals, or factions wanting to cast suspicion on Israel. By decomposing the prediction into sub-questions, the superforecasters avoided bias and built an evidence-based framework for objective analysis.
Superforecasters also actively seek out different perspectives, discussing their reasoning to get feedback from others. They may even rephrase questions to check their own biases. For example, imagine a question asking whether the South African government will grant the Dalai Lama a visa within six months. Most people would only look for evidence suggesting he will get the visa, unconsciously confirming their bias.
Superforecasters are more cautious. They rephrase the question to also ask whether the government will deny the Dalai Lama a visa. This subtle rewording prompts them to consider contradictory evidence of reasons South Africa may deny the visa, like desires to preserve trade relations with China. Even though the predictions are mutually exclusive, by exploring both versions, superforecasters offset bias.
Superforecasters also understand that small shifts in perspective, like rephrasing a question, can reveal blind spots. Additionally, they update their views incrementally as new evidence comes in, rather than making one prediction and stubbornly sticking to it. And they synthesize all their evidence into forecasts in a balanced, objective manner.
Critically, superforecasters practice active open-mindedness. Their assumptions are hypotheses to be rigorously tested, not treasures to be protected from contradictory evidence. This avoids mistakes from confirmation bias.
These methods of careful objective reasoning and actively open-minded synthesis of evidence are teachable and learnable. With practice, you too can improve your forecasting abilities and achieve superforecaster accuracy levels. It's about modifying your behavior, not your innate talent.
The magic of predictive insight
In our age of Big Data, extracting meaningful insights can feel like a magical art. Lionel Levine, a mathematics professor at Cornell, stands as a “magician” at the forefront of this domain. Surprisingly, though, Levine and many other superforecasters don't just lean on intricate math. They blend this with keen judgment, proving that forecasting isn't merely a game of numbers.
This interplay between judgment and numbers was put to the test in the high-stakes hunt for Osama bin Laden. The film Zero Dark Thirty depicts officials grappling with varying probability estimates on bin Laden's location. While the movie portrays a desire for absolute certainty, real-life CIA Director Leon Panetta navigated the uncertainty, appreciating the spectrum of judgments rather than seeking absolute assurance. This real-world scenario contrasts with the movie, emphasizing the value of a holistic approach to prediction.
Journalist Mark Bowden's book, The Third Setting, portrays a meeting in the White House Situation Room where President Barack Obama and the CIA debate the likelihood of Osama bin Laden's presence in a Pakistani compound. Estimates of certainty vary from 30 to 95 percent. Obama, seemingly overwhelmed by the numbers, states that the situation is “fifty-fifty,” indicating his view of the matter as uncertain rather than a literal percentage. Bowden ties this to psychologist Amos Tversky's observation on human simplicity: we often reduce complexities to “gonna happen,” “not gonna happen,” or “maybe.”
This predilection for simplicity harks back to our ancestors who, when spotting a shadow, had mere seconds to decide if it was just grass or a lurking predator. Nuanced probabilities weren’t an option – it was a binary world of “threat or no threat.” Modern preferences echo this.
But the world isn't always black and white. Even experts can be tripped up by nuances in probabilities. Robert Rubin, once a US Treasury secretary, noted how people often misconstrue an 80 percent probability as a surefire bet. But in the evolving landscape of modern science, the quest for absolute certainty is being recognized as a mirage. Instead, there's a tendency to embrace the gray areas, understanding that all knowledge is essentially provisional.
Rubin himself epitomized this nuanced, probabilistic approach, pondering even the smallest distinctions in probability assessments. This contrasts sharply with the commonplace “fifty-fifty” mindset. Forecasts show that granularity, even in predictions, can yield more accurate results. Superforecasters, with their knack for sharp predictions, exemplify this precision.
While narratives of fate may comfort many, embracing probabilistic thinking empowers us with clearer insights and better decisions in an inherently uncertain world.
The balance of belief
Superforecasting is about making highly precise predictions regarding future events. It's not a step-by-step method, but does involve a systematic approach. First, forecasters break down questions into smaller parts. They then distinguish between what they already know and what remains uncertain. This means they don't just accept things at face value; they really dig into the details. They also take two main viewpoints – one that sees the issue as part of a broader category and another that considers its unique aspects. After gathering different perspectives, they merge them into one clear prediction. The key here is that once they make a prediction, they don't just forget about it. They revisit and revise it as new information comes in.
Remember our superforecaster from earlier, Bill Flack? He made predictions about whether polonium would be found in Yasser Arafat’s remains. As time went on and more news came out, he kept refining his prediction to be more accurate. Another forecaster, Devyn Duffy, is exceptionally good at this. He constantly monitors the news and tweaks his forecasts based on what he learns. What's interesting is that these superforecasters adjust their predictions more often than the average person, which seems to be a big part of their success.
But the process of superforecasting isn't infallible. There are instances when even the best forecasters can be thrown off by new data. For instance, despite his expertise, Flack made an error in predicting a visit by Japan’s then-prime minister, Shinzo Abe, to the Yasukuni Shrine. Initially, he believed the prime minister wouldn’t visit due to the shrine's controversial history and international reactions. But when an insider hinted that Abe would indeed visit, Flack, considering the broader political implications, doubted the veracity of this information and chose not to update his forecast. To his surprise, Abe did visit the shrine.
Another example is Doug Lorch, who was trying to predict Arctic sea ice levels. When he found a month-old report suggesting that the ice would be less than the previous year, he heavily adjusted his prediction. Yet, in the end, the ice levels turned out to be higher, indicating that he might have placed too much weight on the older data.
New information can either be given too much weight or not enough. Personal beliefs or existing knowledge can influence how fresh details are processed and responded to. There are two main tendencies: underreaction and overreaction. Some people might not adjust their beliefs sufficiently even when faced with new evidence. On the flip side, individuals can sometimes place excessive importance on new information, allowing it to sway their beliefs disproportionately.
Both these tendencies highlight the delicate balance of human cognition and decision-making, and how it's shaped by the interplay of new information and existing beliefs.
A growth mindset and continuous improvement
Mary Simpson, despite her extensive background in economics and a successful career, missed the signs leading up to the 2007 financial crisis. This personal failure, which significantly affected her retirement savings, motivated her to refine her forecasting abilities. Joining the Good Judgment Project, she became a superforecaster, exemplifying a “growth mindset.” This concept, first articulated by psychologist Carol Dweck, suggests that abilities are developed through effort. Contrarily, many possess a “fixed mindset,” believing that their capacities are unchangeable. Dweck's research shows that those with a growth mindset are more adaptive and learn from challenges, while fixed-mindset individuals often avoid difficulties.
John Maynard Keynes, known for his macroeconomic theories, is another example of this growth mindset in action. As an investor, Keynes experienced significant losses, but continuously adapted and learned from his failures. Rather than seeing failures as insurmountable, he viewed them as opportunities to enhance his strategies. This mindset, the essence of learning from your mistakes and persistently striving for improvement, is universal. From children learning basic skills to adults acquiring new ones, the cycle of “try, fail, analyze, adjust, try again” is crucial to human growth and advancement across various fields, from piloting jets to performing surgery.
There is, of course, a significant difference between theoretical understanding and hands-on experience. Take, for example, the physics behind riding a bicycle. Just because someone understands the science doesn't mean they can easily ride a bike. Similarly, while you might know all the theories about forecasting, genuine mastery comes from actual experience.
Just practicing isn't enough, though; it's essential that the practice is informed and paired with clear feedback – crucial for skill development. For instance, police officers often think they're adept at distinguishing between truths and lies, but they can be mistaken. This misjudgment happens because they don't always get immediate feedback on their assessments. On the other hand, professions like meteorologists and bridge players, who consistently receive feedback on their predictions and decisions, show improvement in their skills over time.
In essence, superforecasters always aim for improvement and are in a continuous state of growth and learning. They embody the spirit of perpetual self-improvement.
Final summary
Superforecasting isn’t about innate talent or having a genius-level IQ. Instead, it’s a systematic approach that requires breaking down questions into smaller components, actively seeking various perspectives, and continuously updating predictions as new information emerges.
Superforecasters are characterized by their rigorous, evidence-based reasoning methods, active open-mindedness, and precision in making predictions. They don't rely solely on intuition but employ a structured, evidence-based technique to achieve remarkable accuracy.
Key factors in their success include their ability to offset biases, practice incremental updating, and maintain a growth mindset – the belief that abilities can be developed through dedicated effort. Practical experience and consistent feedback are essential for honing these skills.
Ultimately, the art of superforecasting emphasizes the importance of how you think, gather information, and update beliefs, rather than who you are.