Consensus Bashing

The Science and Public Policy Institute certainly does provide a lot of hilariously twisted commentary on climate change.

Two years ago (January 2009), Doran and Zimmerman (D&Z) published a paper based on Zimmerman’s masters thesis. Unsurprisingly, they found that the vast majority (97%) of climate scientists think climate change is real and human-induced. This kind of thing really, really irritates climate change denialists, and so we have Dennis Ambler from SPPI launching into a blisteringly woeful attack on the survey.

He sets the tone with this:

[The survey] was roundly de-bunked at the time by several commentators and it would have been forgotten and consigned to its proper place in the dustbin, if it hadn’t been continually
quoted by activists as fact.

If you’re going to claim that a paper has been “roundly debunked”, a little elaboration would not go astray. Some of us might just be curious about just what arguments were put forth, and you’re not giving us much to go on. Also, if we’re going to lend it such credence, I would expect some sort of expertise to be involved in this debunking, not just a vague reference to unspecified “commentators”.

He then bemoans the surveying of experts as a means of assessing scientific opinion:

This is not arcane knowledge for the select priesthood, this is science and we can read scientific papers and apply quality judgements to them, whether we be specialists or not.

No, Ambler, you really can’t. I know this for two reasons:

  1. Those who’ve genuinely tried to read and understand technical papers in a field they don’t work in will know just how much of an uphill battle it can be. There’s unfamiliar jargon, horrendous equations, often enormous amounts of assumed background knowledge, and frequently little attention paid to overall readability. These papers are written for a very narrow audience, and you can’t just plant the flag of egalitarianism and ignore all the hard work that goes into building the necessary expertise.
  2. Even if you were equipped to read and understand technical papers from any discipline, the sheer quantity of them would make the task logistically impossible. They don’t just dribble out one or two at a time every news cycle. There’s countless thousands (possibly millions) of them, and nobody (scientists included) can ever hope to read them all. That’s why we have surveys. Even researchers themselves rely on survey papers, for instance, to make sense of their own fields.

The remainder of Ambler’s article demonstrates his unique inability to “read scientific papers and apply quality judgments”. I say “unique” because D&Z’s paper is actually quite short and accessible. Given a modicum of education and common sense, there really isn’t much of an excuse for not understanding it.

We are also told that only 5% of the original sample responses were climate scientists, so if we pragmatically apply those proportions we end up with just 141 from the US, 9 from Canada and just 6 from 21 countries around the world, hardly a global consensus.

Is there some significance to the 5% figure? The survey was a broad look at the opinions of Earth scientists. Climate scientists just form an important subset of that population, and it’s hardly the fault of the authors or anyone else if the proportion happens to be 5%.

Moreover, Ambler knows he can estimate the number of respondents from each country, but he seems not to understand that the very same mathematical device is the reason you don’t need to ask everyone in the world. So long as you have a representative sample (and consulting a database of Earth scientists, as D&Z did, would seem to be perfectly acceptable), you can generalise your findings. If 97% of your sample believes X, and your sample is representative of a given group (e.g. climate scientists), then you infer that about  97% of the overall group believes X as well. This is the entire basis of surveys. If this statistical logic did not hold, surveys would not exist.

Why so few non-American climate scientists? That’s just a result of the database used by D&Z, coming as it did from the American Geological Institute. There’s no reason to think that American and non-American climate scientists are likely to have any specific, major points of professional disagreement, so this shouldn’t be a problem.

We find that they originally contacted 10,257 scientists, of whom 3,146 responded, less than a 31% response rate. “Impending Planetary Doom” was obviously not uppermost in the minds of over two thirds of their target population.

31% is a very good response rate, in my experience. I would not have raised an eyebrow if it was only 10% (except that the authors would then have been less well-equipped to draw conclusions). It’s silly to start attributing reasons for non-response, because by definition you don’t have the data. It’s certainly very silly to suggest that 7111 scientists don’t care about the issue merely because they failed to fill out a questionnaire on it. Perhaps they were too busy actually working on the problem!

Ambler does us a service by linking to D&Z’s summary paper, but he’s a bit of a cheapskate:

The paper is behind a pay wall but there is a comprehensive summary here.

It’s obvious from reading Ambler’s article that his own investigative skills cannot penetrate this “pay wall”. Despite describing the summary as “comprehensive”, he repeatedly complains about missing details. For example:

There is little detail of how many peer reviewed papers are needed to qualify as a specialist, it could by their definition be just two papers, one of which needs to be on climate change. What a poor example of scientific enquiry this survey really is.

The one-and-a-bit-page summary paper does not, of course, include all the information from the 141-page thesis. The price for an electronic copy of Zimmerman’s thesis is only $US 2, hardly a prohibitive sum. I bought a copy myself just so that I could write this post. Zimmerman provides an extensive explanation of the process of verifying whether survey respondents are, in fact, active publishers on climate science (page 16). However, this kind of nit-picking was never going to undo the rather stark results.

There were supposed to have been nine questions asked, but we are only given sight of two of them.

Again, this is what you get if you only read a summary. For those interested, the full set of nine questions consisted of four opinion-related questions and five demographic questions:
  1. When compared with pre-1800’s levels, do you think that mean global temperatures have generally risen, fallen, or remained relatively constant?
  2. Do you think human activity is a significant contributing factor in changing mean global temperatures?
  3. What do you consider to be the most compelling argument that supports your previous answer?
  4. Please estimate the percentage of your fellow geoscientists who think human activity is a contributing factor to global climate change.
  5. Which percentage of your papers published in peer-reviewed journals in the last 5 years have been on the subject of climate change?
  6. Age
  7. Gender
  8. What is the highest level of education you have attained?
  9. Which category best describes your area of expertise?

Ambler of course takes issue with the first two questions. For question 1:

Has it got warmer since pre-1800 levels? This really depends on the time period referred to. Do they mean the Little Ice Age, when disastrously cold temperatures caused massive loss of life and untold hardship? Of course temperatures are now warmer than that desperate period in climate history. Is that what they would wish to regard as normal?

Climate denialists often have a mild obsession with two proposed fluctuations in global temperature over the last few hundred years. They explain away the current warming trend by saying that we’re merely coming out of a cold period (the Little Ice Age), and that temperatures have been warmer in the past (the Medieval Warm Period), but the evidence for either of these is rather limited. This is related to another pet denialist obsession: the “hockey stick” graph, which shows that the current warm temperatures are unprecedented over at least the last millennium. It is essential denialist lore that the hockey stick has been discredited. In reality, it has numerous independent replications.

For question 2, on whether human activity is a factor:

This is the classic closed question, in that it implies mean global temperatures are being changed and someone must be responsible.

First, respondents are not asked this question if they previously said that temperatures remained relatively constant; so no, the question does not assume temperatures are being changed. It certainly does not assume that “someone must be responsible” – I have no idea how Ambler could have read that into it.

About half-way through his article, Ambler makes his biggest departure from reality, and one that cannot be excused by lack of information. This divergence begins as follows:

Of [the 3146 respondents], only 5% described themselves as climate scientists, numbering 157. The authors reduce that by half by only counting those who they classed as “specialists”.

The authors do no such thing. They categorise their 3146 respondents by field (climatology, geology, etc.) and whether more than 50% of their recent published papers were related to climate science. In the media, the most widely-reported statistics are, appropriately enough, for actively-publishing climatologists. However, this categorisation does not omit anyone, but merely provides more detailed information.

It is disingenuous to now use the “climate scientists” as a new population sample size. The response figure of 3,146 is the figure against which the 75 out of 77 should be compared and in this case we get not 97% but just 2.38%.

Ambler appears not to notice that there are statistics on the complete set of 3146 respondents, not just the 77 who happen to be actively-publishing climate scientists. Ambler’s 2.38% is the proportion of respondents who agree that humanity has an influence on climate and who also happen to be climate specialists. If you think that climate change is real but you’re not a specialist, Ambler is counting you in the total population but not in the “yes” pile (and so by implication in the “no” pile).

That’s dishonesty if ever I’ve seen it. D&Z expressly state in their summary paper that 90% of respondents overall agreed that temperatures have risen, and 82% agreed that humanity was a factor. Ambler expressly ignores these statistics and then tries to reverse engineer them using profoundly broken mathematics.

The original number contacted was 10,157 [sic] and of those, 69% decided they didn’t want any part of it, but they were the original target population. When the figure of 75 believers is set against that number, we get a mere 0.73% of the scientists they contacted who agreed with their loaded questions.

Ambler now wants to count non-respondents in the total as well, making the unsupportable implication that they would have said “no”. This is utter nonsense and is a complete corruption of general survey methodology. You use the data you have – that’s how science works – not by making assumptions about the data you don’t have.

Ambler ends by ridiculing media reports that quite fairly echo D&Z’s findings. Apparently they’re not privy to his powers of deduction, and neither am I.

One thought on “Consensus Bashing

Comments are closed.