The Battle Lines in the CRU Data Release Debate

On one side, the claim is that the data butchery and opinion suppression revealed make the standard claims about global warming look dubious.

On the other side, the claim is that this is standard scientific procedure. It's apparently normal to find hacked statistics and narrow mindedness in scientific research.

On the gripping hand, according to the well-known paper Why Most Published Research Findings Are False:

There is increasing concern that most current published research findings are false. The probability that a research claim is true may depend on study power and bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientific field. In this framework, a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance. Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias. In this essay, I discuss the implications of these problems for the conduct and interpretation of research.

Of course, the above finding might be one of the erroneous results. We also see at Less Wrong:

Parapsychology, the control group for science, would seem to be a thriving field with "statistically significant" results aplenty.  Oh, sure, the effect sizes are minor.  Sure, the effect sizes get even smaller (though still "statistically significant") as they collect more data.  Sure, if you find that people can telekinetically influence the future, a similar experimental protocol is likely to produce equally good results for telekinetically influencing the past.  Of which I am less tempted to say, "How amazing!  The power of the mind is not bound by time or causality!" and more inclined to say, "Bad statistics are time-symmetrical."  But here's the thing:  Parapsychologists are constantly protesting that they are playing by all the standard scientific rules, and yet their results are being ignored - that they are unfairly being held to higher standards than everyone else.  I'm willing to believe that.  It just means that the standard statistical methods of science are so weak and flawed as to permit a field of study to sustain itself in the complete absence of any subject matter.  With two-thirds of medical studies in prestigious journals failing to replicate, getting rid of the entire actual subject matter would shrink the field by only 33%. We have to raise the bar high enough to exclude the results claimed by parapsychology under classical frequentist statistics, and then fairly and evenhandedly apply the same bar to the rest of science.

Hmmm…

In that case, should any scientific results be trusted at all? According to Orac, a substantial fraction of kooks have used the CRU release as an excuse for general skepticism. I can think of two criteria for trustworthy scientific results.

• First, a scientific theory that has been accepted for a while is usually only replaced by a theory which contains the earlier theory as a special case. For example, modern astronomical theories can explain why Ptolemaic astronomy produced reasonable results.
• Second, a scientific theory that has been linked to a large number of types of facts (especially if those facts come from outside the theory and have led to working devices) is more likely to be right. For example, someone skeptical of Einstein's Theory of Relativity has to explain how those bombs explode. Someone skeptical of quantum mechanics has to explain how semiconductors work.

Let's analyze the Global Warming controversy using the above. The theoretical results that indicated that sufficient accumulation of CO₂ might lead to an uncomfortable climates have been around for almost a century and the general outlines have been rather well checked, even if some of the exact details are disputed. On the other hand, it's hard to use them to get hysterical since they indicate it's a long-term problem and won't get serious until long after the last internal-combustion engine is in a museum. On the gripping hand, the experimental results that are used as the basis for hysterical regulations (aka the “hockey stick”) have not been around long enough to be properly checked. In addition, we only have to explain away one time series (which might be due to solar changes).

Now let's analyze evolution (as an example of something criticized by cranks) using the above. Darwin's explanation has not only been around for over a century but it even outlasted two temporary fads that might have counteracted it. (Socialist economics might have meant that order can only be imposed by central planning and Freud's psychology might have meant that large parts of human psychology are non-adaptive.) In addition, someone trying to explain away evolution has to explain away fossil evidence, biochemical evidence, radioactive-dating evidence, and the theoretical results that indicated genetic algorithms can produce non-trivial results.