Explaining the Wisdom of the Crowd

I'm sure my fellow anti-social malcontents have heard of “the Wisdom of the Crowd” and that the first reaction of most of us was skepticism. (How can you prevent groupthink? Aren't speculative bubbles a counterexample?) I'd like to give a mathematical explanation of this.

Let us assume we have a continuous distribution of estimates in which the median and the Truth differ (see figure below). We are certainly not assuming that the median is infallible. In this figure, the dark areas represent the opinions of people who are further away from the Truth than the median. Half of those opinions are on the other side of the median from the Truth. (For every contrarian, there is an opposite contrarian.) In addition, there are opinions that go too far in the direction of the Truth. (You can usually find people who make your opinions seem moderate, no matter how cracked you may seem.) What this means is that at least half the population will have opinions that are more wrong than the median … and you are probably one of them.

As a specific example, I'm dubious about the Linear No-Threshhold theory of radiation damage. The opposite contrarians would be the people who think the Petkau Effect implies low doses of radiation are more dangerous than the linear theory predicts. The contrarians who might be going too far are the people who think that there's proof that radiation hormesis implies low doses of radiation are good for you.

Apparent counterexamples

Speculative bubbles aren't a counterexample. There is no reason to buy when you agree with the median opinion, as represented by the market price. On the other hand, the people buying during a bubble might think they're agreeing with the median opinion (“everybody knows the market's going up”) but are mistaken. It's also worth noting that selling short during a bubble is also risky since “the market can stay irrational longer than you can stay solvent.”

Another apparent counterexample is from the late Hal Finney: an opinion cascade. In an opinion cascade, everybody adjusts their opinion to that of the apparent majority, producing a situation in which the first few people to express an opinion have a disproportionate influence. One possible way to stop this:

It might be best to express what your personal analysis says while simultaneously adjusting your private opinions (which might be revealed in your actions) closer to the opinions of the majority.

There's another reason to be dubious about following the median estimate. The median opinion (at least in this society) disagrees with it. That might have enabled Western Civilization to outcompete more conformist civilizations. On the other hand, that might be a matter of contrarianism being a public-goods problem (also from Hal Finney):

Most people would become more accurate by shifting their opinions towards the consensus average. But this probably would cause social harm, in that we all benefit from having a wide variation of opinion and debate in society.

It may be one reason why society encourages individualism and thinking for yourself. It is harmful to individuals but beneficial for society.

This is a classic public goods problem. Are you going to do what is good for yourself, suppress your individualism and accept the group consensus? Or are you going to accept the social propaganda to think for yourself, even though you will be less accurate?

Again the best solution might be express what your personal analysis says while simultaneously adjusting your private opinions closer to the opinions of the majority. Just don't let anybody catch you applying for tenure while decrying insulation from the market … or dodging a draft while defending an unpopular war … or criticizing racism while moving to an all-white neighborhood … or flying around the world to global-warming conferences … etc. (I had an additional discussion of that type of hypocrisy here.)

Digression: the trouble with tradition

Opinion cascades are a major problem with tradition. If the people in each generation adjust their opinions to the average of earlier generations, the first few will have a disproportionate effect. On the other hand, rejecting tradition entirely means getting your opinions from just one generation in the present.

On the gripping hand, every time there's an improvement in meme storage, the generation after also has a chance to get its say. Let's take Judaism as a typical tradition: The invention of the alphabet was followed by the Torah; the invention of postal systems (which made collaboration between distant scholars possible) was followed by the rest of the Old Testament; the invention of books divided into pages (which made large books usable) was followed by the Talmud; the invention of paper was followed by Maimonides organizing Jewish law; the invention of printing was followed by the Shuchan Aruch; the invention of the Internet …

The difference between median and mode

Some types of contrarianism might be worthwhile. For example, a large fraction (I hope it's still a minority) of the public believes that Columbus discovered the world is round in 1492. In the real world, the fact that the Earth is round was discovered around 2000 years earlier. If most people don't believe that, the median estimate for “when did people discover the world is round?” will be less than 1492 but the mode is likely to be 1492. If contrarianism is a matter of being skeptical about the accuracy of the mode instead of being skeptical about the accuracy of the median, it's likely to be worthwhile.

On the other hand, sometimes the Official Truth really is true and the contrarians are wrong. For example, the number of deaths caused by the Fukushima meltdown is likely to be at or close to zero. The mode estimate probably agrees with that but the median might be much higher. You can think of the mode as the estimate by the “sheeple” (who are sometimes right).

Cross-posted from my Netcom/Earthlink site.