Superforecasting | Philip Tetlock & Dan Gardner


Published in: 2015

Amazon Goodreads

Philip Tetlock has tested a large population’s forecasting ability through tests performed over a long period of time. The results show that about 2% is what he calls “Superforecasters” (SF) – these people have a real, measurable ability to assess how high-stakes events are likely to develop three months, six months, a year and a half and a half ahead. Foresight is a product of a way of thinking, gathering information and updating perceptions. This habit can be learned and developed. You can test yourself: www.goodjudgement.com

INTELLIGENT, BUT NOT GENIUSES. The population that took the test had a higher score on intelligence and knowledge tests than 70% of the total population. The SF’s had higher than 80% of the population, well above average, but most of them well below the so-called “ingenious territory” (often arbitrarily defined as the best 1%, or IQ of +135).

THINK PROBABILISTICALLY. The best forecasters, according to Philip Tetlock, are good at numbers but rarely use mathematical models or formulas. They think in terms of probability. They know that the difference between the good and the amateurs is that the former know the difference between a 60/40 bet and a 40/60 bet (in poker, investing and management).

PROBABILITY IS AN INTUITION THAT REQUIRES EXPERTISE. Bridge players can develop well-calibrated judgment, but research shows that what is calibrated in one context is rarely transferred well to another. To get better at a certain type of forecast, that specific one should be repeated with good feedback.

DEFECTIVE IMPULSES. System 1 is fast and always running in the background. If you get a question and you immediately know the answer, it comes from system 1. It is designed to jump to conclusions based on some evidence. System 2 is more logical and challenges that answer – is it fact-based? This takes time and energy and sometimes it is not activated. Researchers have come to the conclusion that people who assume that their initial assessment is wrong, and then make another estimate in combination with the first, often improve the accuracy. The same effect applies if there are a few weeks between the first and second estimate.

Beliefs are hypotheses to be tested, not treasures to be guarded.

HAPPINESS IS GIVEN TO THE PREPARED. Randomized controlled trials have shown that mastering the contents of a small booklet can improve your accuracy by about 10%. It shows that “Fortunes favors the prepared mind”. But there is much we do not know. There is a three-stage way to quickly separate what is “knowable” and what is “unknowable”.

1. FERMI-IZE. Physicist Enrico Fermi conducted an experiment with his students, who did not have Google or other aids, if they could guess the number of piano tuners in Chicago. To break down the answer, we need to guess (1) the number of pianos in Chicago, (2) how often the piano is tuned each year, (3) how long it takes to tune a piano and (4) how many hours per year a piano tuner works.

2. BASE RATE – OUTSIDE IN. The base rate is how common something is within a broader class. To answer how likely a family is to have a pet, one must start with how likely an American household is to have a pet.

3. DIRECTED AND PURPOSEFUL INVESTIGATION. The SF’s often use several different analytical tools and seek information from several sources to later synthesize it into a single conclusion.

PERPETUAL UPDATE. A forecast that is updated to reflect the latest available information is likely to be closer to the truth than a forecast that is less based on current data. The Bayesian theorem states that your new belief should depend on your previous beliefs (and all the knowledge that informed about it) multiplied by the “diagnostic value” of the new information.

SUPERFORECASTERS IN A NUTSHELL. A superforecaster is cautious (nothing is certain), humble (reality is infinitely complex) and non-deterministic (there are several potential outcomes). In their abilities and ways of thinking, they tend to be open-minded (perceptions are hypotheses to test), intelligent and knowledgeable (curious) and reflective (introspective and self-critical). They are comfortable with numbers, pragmatic in their methods and analytical. In addition, they are “dragonfly-eyed” (values different views and synthesizes them into their own), probabilistic (many layers of “maybe”), well-thought-out updaters (when facts change, they change their view) and good intuitive psychologists (test thinking for biases).

Leave a Reply