Review of: Philip E. Tetlock. 2005. Expert Political Judgment: How Good is it? How Can We Know?
4 Pages Posted: 1 Jul 2008
Date Written: June 29, 2008
Abstract
Typically, researchers report new findings in scholarly journals and Tetlock (1998, 1999) has done so for of some part of the findings of his study. Still, Tetlock has gone beyond journal articles, turning to a book to report on his large-scale and important study. Publishing a book has allowed him to deal with issues in detail and provide a full report, to reach a larger audience, and to have a stronger impact one hopes on experts and the consumers of their services, including political leaders and the voting public.
The book assaults common sense with evidence. In order to mount his assault on accepted wisdom, Tetlock spends more some 238 pages of text explaining his methods and findings, and considering and refuting many alternative explanations, and adds some 75 pages of technical appendices. The downside of this approach is that some readers may find the book too demanding. That would be a pity as his findings are important.
Tetlock's book reports the results of a two-decade long study of expert predictions. He recruited 284 people whose professions included "commenting or offering advice on political and economic trends." He asked them to forecast the probability that various situations would or would not occur, picking areas (geographic and substantive) within and outside their areas of expertise. In addition to eliciting forecasts, Tetlock also asked questions aimed at understanding how the forecasters came to formulate their forecasts, how they dealt with the failure of their forecasts, how they responded to contradictory information, and how they evaluated the probable accuracy of others' theories and predictions. By 2003, he had accumulated 82,361 forecasts, which provided him a database to evaluate. He then evaluated experts' predictions against outcomes, and against various alternate predictions that he derived from simple statistical procedures, from uninformed non-experts, and from well-informed non-experts.
Suggested Citation: Suggested Citation