Expert Political Judgment: How Good Is It? How Can We Know?
Format: PDF / Kindle (mobi) / ePub
The intelligence failures surrounding the invasion of Iraq dramatically illustrate the necessity of developing standards for evaluating expert opinion. This book fills that need. Here, Philip E. Tetlock explores what constitutes good judgment in predicting future events, and looks at why experts are often wrong in their forecasts.
Tetlock first discusses arguments about whether the world is too complex for people to find the tools to understand political phenomena, let alone predict the future. He evaluates predictions from experts in different fields, comparing them to predictions by well-informed laity or those based on simple extrapolation from current trends. He goes on to analyze which styles of thinking are more successful in forecasting. Classifying thinking styles using Isaiah Berlin's prototypes of the fox and the hedgehog, Tetlock contends that the fox--the thinker who knows many little things, draws from an eclectic array of traditions, and is better able to improvise in response to changing events--is more successful in predicting the future than the hedgehog, who knows one big thing, toils devotedly within one tradition, and imposes formulaic solutions on ill-defined problems. He notes a perversely inverse relationship between the best scientific indicators of good judgement and the qualities that the media most prizes in pundits--the single-minded determination required to prevail in ideological combat.
Clearly written and impeccably researched, the book fills a huge void in the literature on evaluating expert opinion. It will appeal across many academic disciplines as well as to corporations seeking to develop standards for judging expert decision-making.
it means to fail coherence/process tests but divisions on where to locate the pass-fail cutoffs. The prototypic tests involve breaches of rules of fair play in the honoring of reputational bets and in the evenhanded treatment of evidence in turnabout thought experiments. To qualify as a good judge within a Bayesian framework—and many students of human decision making as well as high-IQ public ﬁgures such as Bill Gates and Robert Rubin think of themselves as Bayesians— one must own up to one’s
happen about 70 percent of the time, and so on. The discrimination index taps forecasters’ ability to do better than a simple predict-thebase-rate strategy. Observers get perfect discrimination scores when they infallibly assign probabilities of 1.0 to things that happen and probabilities of zero to things that do not. To maximize calibration, it often pays to be cautious and assign probabilities close to the base rates; to maximize discrimination, it often pays to be bold and assign extreme
presumptuous for nonexperts to tell experts how “diagnostic” particular evidence is with respect to particular causal hypotheses. It is possible, however, to design a better mousetrap for documenting the impact of theory-driven thinking about counterfactual history. Imagine that we transform our thought experiment into an actual experiment that holds evidence constant—say, documents recently discovered in Kremlin archives—but manipulates ﬁndings—say, whether the documents contain revelations
bets, to resist hindsight bias, and to be better Bayesians. This chapter rounds out the normative indictment. Since we introduced the hedgehog-fox distinction in chapter 3, hedgehogs have repeatedly emerged as the higher-risk candidates for becoming “prisoners of their preconceptions.” Figure 5.1 integrates the key ﬁndings from chapter 5 into the conceptual model of good forecasting judgment that has been evolving through the last three chapters. In this revised scheme, moderate foxes have a new
the defense. CHAPTER 6 The Hedgehogs Strike Back There are two sides to every argument, including this one. —Anonymous It requires better defense counsel than the author to get the hedgehogs acquitted of all the charges against them. Too many lines of evidence converge: hedgehogs are poor forecasters who refuse to acknowledge mistakes, dismiss dissonant evidence, and warm to the possibility that things could have worked out differently only if doing so rescues favorite theories of history