Expert Political Judgment

In a comprehensive 1982 analysis, economists reviewing over 100 studies on professional forecasting—covering everything from population growth to energy prices—found a startling pattern: adding more complexity to a forecasting model, beyond a few key variables, didn't just fail to improve accuracy, it often made the predictions worse. The same held true for expert opinion; a group's consensus forecast was consistently more reliable than the single most confident, high-status expert in that group. Across domains, the data showed that our reverence for elaborate, confident predictions is deeply misplaced. So, what happens when this same analytical rigor is applied to the high-stakes world of political and economic punditry, where confident forecasts are a daily currency?

This very question obsessed a young political psychologist named Philip Tetlock. He watched commentators on television speak with absolute certainty about the future of nations and markets, yet he rarely saw anyone held accountable when those futures failed to materialize. He wondered if the confident pronouncements of experts were any better than random chance. This curiosity launched an audacious 20-year research project, involving 284 experts making over 28,000 specific, testable predictions about the future. Tetlock, a professor at the University of Pennsylvania and UC Berkeley, was trying to decode the very nature of good judgment. His goal was to move forecasting from an art of confident storytelling to a science of measurable skill, and the results of that massive study form the foundation of this book.

Module 1: The Great Divide — Foxes vs. Hedgehogs

Tetlock's first major discovery is a simple but powerful framework for understanding how experts think. He borrows a metaphor from the philosopher Isaiah Berlin. It divides thinkers into two categories: hedgehogs and foxes.

The hedgehog knows one big thing. It's an expert with a grand, overarching theory. Think of a die-hard Marxist who sees everything through the lens of class struggle, or a free-market fundamentalist who believes deregulation is the answer to every problem. Hedgehogs apply a single, powerful idea to every situation. They are confident. They are decisive. They make for great television because they offer simple, compelling narratives. When they explain the world, they do it with force and clarity. They seek to fit new facts into their existing theory. If the facts don't fit, they often dismiss the facts, not the theory.

So what's the alternative? The fox. The fox knows many little things. It's a scavenger of ideas, borrowing from different disciplines and traditions. Foxes are not committed to a single grand theory. Foxes use a variety of mental models to understand the world. They are more cautious. They are comfortable with nuance, complexity, and contradiction. Instead of forcing the world into a neat box, they see it as a messy, probabilistic system. When confronted with new information that contradicts their view, they are more likely to adjust their thinking. They see the world in shades of gray.

Now, let's turn to the core question. Who is better at predicting the future? Tetlock's twenty-year study provides a clear, data-driven answer. Across thousands of forecasts, on topics ranging from economic growth to military conflict, the foxes consistently outperformed the hedgehogs. It wasn't even close. Hedgehogs were more overconfident. Their long-range forecasts, in particular, were often wildly inaccurate. The more famous an expert was, the more likely they were to be a hedgehog and the less accurate their forecasts became. The media, it turns out, rewards the confident, simple story of the hedgehog over the cautious, probabilistic accuracy of the fox. This insight is crucial. Good judgment is about having a toolkit of different perspectives and knowing which one to apply.

Module 2: The Science of Keeping Score

A key problem in public discourse is the lack of accountability. An expert predicts a market crash. It doesn't happen. A year later, they are back on TV predicting the next one. No one ever checks the receipts. Tetlock argues that this has to change. If we want to improve judgment, we first need to measure it.

His research project was built on this foundation. He forced experts to move beyond vague language. Words like "likely," "might," or "a real possibility" are useless for accountability. One person hears "likely" and thinks 90% probability; another hears 60%. Instead, experts had to assign specific numerical probabilities to their forecasts. For example: "There is a 70% probability that the ruling party will lose the next election." This simple step makes predictions testable.

From this foundation, Tetlock introduces two key metrics for scoring judgment. The first is calibration. Good forecasters are well-calibrated; their confidence matches their accuracy. If a forecaster says something is 80% likely, it should happen about 80% of the time. Tetlock found that most experts, especially hedgehogs, are horribly calibrated. They are systematically overconfident. When they said something was 100% certain, it only happened about 80% of the time. When they said it was 80% certain, it happened about 65% of the time. Their confidence was a poor guide to reality.

But calibration isn't enough. The second metric is discrimination. Good forecasters have high discrimination; they assign high probabilities to things that happen and low probabilities to things that don't. It's easy to be well-calibrated by always predicting a 50% chance for everything. But that's useless. Discrimination separates the insightful from the indecisive. It's the ability to see the signal through the noise. Tetlock found that foxes were not only better calibrated, but they also had higher discrimination scores. Their probabilistic thinking allowed them to more accurately sort possible futures.

So here's what that means for us. When you hear an expert, ask yourself: Are they speaking in vague terms or in testable probabilities? Are they offering a single, confident story, or are they acknowledging uncertainty and alternative scenarios? By applying these simple scoring rules, you can start to distinguish real expertise from confident storytelling.