Not Entirely Believable

If this article (seen via Daniel Drezner) is to be believed, Bruce Bueno de Mesquita has just invented psychohistory:

To verify the accuracy of his model, the CIA set up a kind of forecasting face-off that pit predictions from his model against those of Langley’s more traditional in-house intelligence analysts and area specialists. “We tested Bueno de Mesquita’s model on scores of issues that were conducted in real time—that is, the forecasts were made before the events actually happened,” says Stanley Feder, a former high-level CIA analyst. “We found the model to be accurate 90 percent of the time,” he wrote. Another study evaluating Bueno de Mesquita’s real-time forecasts of 21 policy decisions in the European community concluded that “the probability that the predicted outcome was what indeed occurred was an astounding 97 percent.” What’s more, Bueno de Mesquita’s forecasts were much more detailed than those of the more traditional analysts. “The real issue is the specificity of the accuracy,” says Feder. “We found that DI (Directorate of National Intelligence) analyses, even when they were right, were vague compared to the model’s forecasts. To use an archery metaphor, if you hit the target, that’s great. But if you hit the bull’s eye—that’s amazing.”

Saaay what? If they claimed 75% accuracy, I would applaud a model that can achieve that much in such a fuzzy subject. At 90% accuracy, I wonder if anybody has actually looked at the data. At 97% accuracy, I begin to smell a disinformation campaign.

On the other hand, anybody opposed by Mearsheimer and Walt can't be all bad.

For more information on the reliability of expert predictions, see this article (also seen via Daniel Drezner).