Published research findings are sometimes refuted by subsequent evidence, says Ioannidis, with ensuing confusion and disappointment.

Philip E. Tetlock and I agree that forecasting tools are best evaluated in peer-reviewed settings and in comparison not only to expert judgments, but also to alternative modeling strategies. Applying his suggested standards of assessment, however, certain forecasting models not only outperform expert judgments, but also have gone head-to-head with alternative models and outperformed them. This track record demonstrates the capability to make significant, reliable predictions of difficult, complex events. The record has unfolded, contrary to Tetlock's contention, not only in (...) government and business applications, but also in numerous peer-reviewed publications containing hundreds of real-time forecasts. Moreover, reliable prediction is achieved while avoiding significant false-positive or false-negative rates. (shrink)

Philip E. Tetlock's finding that "hedgehog" experts are worse predictors than "foxes" offers fertile ground for future research. Are experts as likely to exhibit hedgehog- or fox-like tendencies in areas that call for explanatory, diagnostic, and skill-based expertise-as they did when Tetlock called on experts to make predictions? Do particular domains of expertise curtail or encourage different styles of expertise? Can we trace these different styles to childhood? Finally, can we nudge hedgehogs to be more like foxes? Current research can (...) only grope at the answers to these questions, but they are essential to gauging the health of expert political judgment. (shrink)

In five works spanning a decade, Philip E. Tetlock's interest in counterfactuals has changed. He began with an optimistic desire to make social science more rigorous by identifying best practices in the absence of non-imagined controls for experimentation. Soon, however, he adopted a more pessimistic analysis of the cognitive and psychological barriers facing experts. This shift was brought on by an awareness that experts are not rational Bayesians who continually update their theories to keep up with new information; but instead (...) are affected by political, cognitive, and psychological heuristics, including hindsight bias, cognitive conservatism, and the fundamental attribution error. But techniques of computational simulation-involving the rigorous production of large numbers of counterfactual worlds-make it possible to mitigate both problems that Tetlock identifies: that history, produced only once, is a lousy teacher; and that humans, with their collection of non-Bayesian heuristics, are lousy pupils. Tetlock was wrong to reject this approach as theoretically promising but rhetorically and practically impractical. (shrink)

In his justly celebrated Expert Political Judgment, Philip E. Tetlock evaluates the judgment of economic and political experts by rigorously testing their ability to make accurate predictions. He finds that ability profoundly limited, implying that expert judgment is virtually useless, if not worse. He concludes by proposing a project that would seek to improve experts' performance by holding them publicly accountable for their claims. But Tetlock's methods severely underestimate the value of expert opinion. Despite their notorious disagreements, experts have highly (...) pertinent information and advice in many areas of public policy. The main problem with experts is that political actors seize on their most extreme views in one direction or the other, and overlook their areas of general agreement and the core of relatively reliable information that they provide. (shrink)

Philip Tetlock underestimates the import of his own Expert Political Judgment. It is much more than a critical scientific evaluation of the accuracy and consistency of political pundits. It also offers a blueprint for challenging expertise more generally-in the name of scientific advancement. “Thinking the unthinkable”-a strategy Tetlock employs when he gets experts to consider counterfactual scenarios that are far from their epistemic comfort zones-has had explosive consequences historically for both knowledge and morality by extending our sense of what is (...) possible. Thinking the unthinkable might also put into question contemporary assumptions about the legitimacy of science. (shrink)

ABSTRACT Tetlock's Expert Political Judgment is a creative, careful, and mostly convincing study of the predictive accuracy of political experts. My only major complaints are that Tetlock (1) understates the predictive accuracy of experts, and (2) does too little to discourage demagogues from misinterpreting his work as a vindication of the wisdom of the average citizen. Experts have much to learn from Tetlock's epistemological audit, but there is still ample evidence that, compared to laymen, experts are very good.

In his remarkably meticulous and even-handed 2005 book, Expert Political Judgment, Philip E. Tetlock establishes that the only thing we can count on in the political experts' predictions is that they will underperform-in some cases significantly-the predictions made by mechanical statistical procedures, including random chance. Experts have many uses and Tetlock does not claim that they have no value. However, Tetlock zeroes in on experts' important political role-as prognosticators. Tetlock does not attempt the impossible by trying to judge experts on (...) their ability to prognosticate the effects of public policies; this would require that Tetlock himself could know what those effects were, and even in retrospect, it isn't that simple. Experts' failure to predict much simpler matters, however, such as GDP growth within a broad range, strongly suggests that technocracy is worse than putting dart-throwing chimps in charge of public policy. (shrink)