Is falsifiability a scientific idea due for retirement?

Sean Carroll argues that it is.

He characterises the belief that "theories should be falsifiable" as a "fortune-cookie-sized motto"; it's a position adopted only by "armchair theorizers" and "amateur philosophers", and people who have no idea how science really works. He thinks we need to move beyond the idea that scientific theories need to be falsifiable; this appears to be because he wants to argue that string theory and the idea of the multiverse are not falsifiable ideas, but are still scientific.

This position is not just wrong, it's ludicrous. 

What's more, I think deep down Sean – who is normally a clear, precise thinker – realises that it is ludicrous. Midway through his essay, therefore, he flaps around trying to square the circle and get out of the corner he has painted himself into: a scientific theory must, apparently, still be "judged on its ability to account for the data", and it's still true that "nature is the ultimate guide". But somehow it isn't necessary for a theory to be falsifiable to be scientific.

Now, I'm not a philosopher by training. Therefore what follows could certainly be dismissed as "amateur philosophising". I'm almost certain that what I say has been said before, and said better, by other people in other places. Nevertheless, as a practising scientist with an argumentative tendency, I'm going to have to rise to the challenge of defending the idea of falsifiability as the essence of science. Let's start by dismantling the alternatives.

The nature of publications

A paper in the journal of Genome Biology and Evolution has been doing the rounds on the internet recently and was shown to me by a friend. It is titled "On the immortality of television sets: “function” in the human genome according to the evolution-free gospel of ENCODE", by Graur et al. The title is blunt enough, but the abstract is extraordinarily so. Let me quote the entire thing here:

A recent slew of ENCODE Consortium publications, specifically the article signed by all Consortium members, put forward the idea that more than 80% of the human genome is functional. This claim flies in the face of current estimates according to which the fraction of the genome that is evolutionarily conserved through purifying selection is under 10%. Thus, according to the ENCODE Consortium, a biological function can be maintained indefinitely without selection, which implies that at least 80 – 10 = 70% of the genome is perfectly invulnerable to deleterious mutations, either because no mutation can ever occur in these “functional” regions, or because no mutation in these regions can ever be deleterious. This absurd conclusion was reached through various means, chiefly (1) by employing the seldom used “causal role” definition of biological function and then applying it inconsistently to different biochemical properties, (2) by committing a logical fallacy known as “affirming the consequent,” (3) by failing to appreciate the crucial difference between “junk DNA” and “garbage DNA,” (4) by using analytical methods that yield biased errors and inflate estimates of functionality, (5) by favoring statistical sensitivity over specificity, and (6) by emphasizing statistical significance rather than the magnitude of the effect. Here, we detail the many logical and methodological transgressions involved in assigning functionality to almost every nucleotide in the human genome. The ENCODE results were predicted by one of its authors to necessitate the rewriting of textbooks. We agree, many textbooks dealing with marketing, mass-media hype, and public relations may well have to be rewritten.

Ouch.

The paper that Graur et al. implicitly deride as "marketing, mass-media hype and public relations" is one of series of publications in Nature (link here for those interested) by the ENCODE consortium. I'm not going to claim any expertise in genetics, though the arguments put forward by Graur appear sensible and convincing.¹ But I do think it is interesting that the ENCODE papers were published in Nature.

Nature is of course a very prestigious journal to publish in. In some fields, the presence or lack of a Nature article on a young researcher's CV can make or break their career chances. It is very selective in accepting articles: not only must contributions meet all the usual requirements of peer-review, they should also be judged to be in "the five most significant papers" published in that discipline that year. It has a very high Impact Factor rating, probably one of the highest of all science journals. In fact it is apparently one of the very few journals that does better on citation counts than the arXiv, which accepts everything.

But among some cosmologists, Nature has a reputation for often publishing claims that are over-exaggerated, describe dramatic results that turn out to be less dramatic in subsequent experiments, or are just plain wrong.² One professor even once told me – and he was only half-joking – that he wouldn't believe a particular result because it had been published in Nature.

It is easy to see how such things can happen. The immense benefit of a high-profile Nature publication to a scientist's career leads to a pressure to find results that are dramatic enough to pass the "significance test" imposed by the journal, or to exaggerate the interpretation of results that are not quite dramatic enough. On the other hand, if a particular result does start to look interesting enough for Nature, the authors may be – perhaps unwittingly – less likely to subject it to the same level of close scrutiny they would otherwise give it. The journal then is more reliant on its referee's to provide the scrutiny to weed out the hype from the substance, but even with the most efficient refereeing system in the world given enough submitted papers full of earth-shattering results, some amount of rubbish will always slip through.

I was thinking along these lines after seeing Graur et al.'s paper, and I was reminded of a post by Sabine Hossenfelder at the Backreaction blog, which linked to this recent pre-print on the arXiv titled "Deep Impact: Unintended Consequences of Journal Rank". As Sabine discusses, the authors point to quite a few undesirable aspects of the ranking of journals according to "impact factor", and the consequent rush to try to publish in the top-ranked journals. The publication bias effect (and in some cases, the subsequent retractions that follow) appear to be influenced to a degree by the impact factor of the journal in which the study is published. Another thing that might be interesting (though probably hard to check) is the link between the likelihood of scientists holding a press conference or issuing a press release to announce a result, and the likelihood of that result being wrong. I'd guess the correlation is quite high!

Of course the only real reason that the impact factor of the journal in which your paper is published matters is that it can be used a proxy indication of the quality of your work for the benefit of people who can't be bothered, or are unable, to read the original work and judge it on merit.

The other yardstick by which researchers are often judged is the number of citations their papers receive, which at least has the (relative) merit of being based on those papers alone, rather than other people's papers. Combining impact factor and citation count is even sillier – unless they are counted in opposition, so that a paper that is highly cited despite being in a low-impact journal gets more credit, and a moderately cited one in a high-impact journal gets less!

Anyway, bear these things in mind if you ever find yourself making a reflexive judgement about the quality of a paper you haven't read based on where it was published.

¹The paper includes a quote which pretty well sums up the problem for ENCODE:

"The onion test is a simple reality check for anyone who thinks they can assign a function to every nucleotide in the human genome. Whatever your proposed functions are, ask yourself this question: Why does an onion need a genome that is about five times larger than ours?"

²Cosmologists (the theoretical ones, at any rate) actually hardly ever publish in Nature. Even observational cosmology is rarely included. So you might regard this as a bit a of a case of sour grapes. I don't think that is the case, simply because it isn't really relevant to us. Not having a Nature publication is not a career-defining gap for a cosmologist: it's just normal.

Things to Read, 2nd December

Time for another collection of worthwhile links from elsewhere on the internet.

Science links:

• Jester at Résonaances has been producing a series of very good informative posts from the frontlines of particle physics: particularly worth reading are this summary of the state of play with the "detection" of dark matter by the Fermi telescope, this explanation on what is and isn't contained in the new Higgs analyses from the LHC, including intriguing information on why some data hasn't yet been updated, and this take on the relevance of the LHCb measurements of Bs meson decay for supersymmetry (which has been the subject of a lot of hype in the popular press).
• Speaking of the popular press, Time magazine has seen fit to nominate the Higgs boson as one of the candidates for "person of the year" (you can vote for it if you like). As if that weren't ridiculous enough, the accompanying description must surely qualify as one of the worst pieces of science journalism ever: literally every single sentence is wrong. Worth a look — even if you're not a physicist, you might be able to spot the mistakes.
• I've recently discovered a series of super-slow-motion videos of lightning strikes knocking around on the internet. These are shot at several thousand frames per second, and really show the details of how the charge leaders meander towards the ground before the main secondary stroke follows. Even more interesting are some videos showing lightning travelling upwards, from ground to cloud. Here's a video:
  
  
  and here's a popular-level description of the phenomenon.
• Another particularly cool video I saw some time ago was this one, showing the synchronisation of 32 metronomes placed on a flexible platform:
  
  

Other links:

• This one is part science, part history and part politics. Have you ever wondered what the connection is between prehistoric plankton from the Cretaceous era, and the distribution of votes for Obama? Of course you haven't, but now you can find out anyway. 
• On a more serious note: one of the striking things about the US election was how completely wrong Republicans and the right-wing were in their predictions of the outcome. Especially so since there were so many people who were able, by simple analysis of the available facts, to arrive at predictions that were much better (Nate Silver, for one). There is lots to be said about this, I suppose. One striking observation, which maybe hasn't been made enough of, is the hope that people realise that if the right-wing media can spout such rubbish (not to say lies) about the polls, perhaps the rest of what they say is rubbish too. 
• And following on from that, an interesting article from someone on the right of American politics: The Revenge of the Reality-Based Community.
• It's not just in the US that facts have a well-known anti-right-wing bias of course. And it's not just US right-wing politicians who dislike them. When economist Jonathan Portes of the National Institute for Economic and Social Research appeared before the UK parliament and explained some fairly elementary economics that happens to contradict the ideology of the Conservative Party, he received thinly veiled threats from MP Jesse Norman (watch; transcript). This drives other economists to express their equally thinly veiled contempt. Norman responds, and Wren-Lewis dismisses him again.
• And finally, something more cheery: some time ago I posted the story about the dog who climbed a mountain with me in the Himalayas. I thought that dog was pretty awesome, but then I saw this dog, who seems to have done an even harder climb!
  
  
  
  (She does seem to have a protective harness though, which probably gives her an unfair advantage ...)

Extortion in Prisoner's Dilemma

This week I stumbled across a fascinating paper by William H. Press and Freeman J. Dyson titled "Iterated Prisoner's Dilemma contains strategies that dominate any evolutionary opponent," published in the Proceedings of the National Academy of Sciences. I think it is fair to say it is the most interesting thing I have read this year. Press and Dyson have somehow discovered something completely new and profound in a field that has already been studied extensively for decades, and have written a beautifully clear and concise paper that you require only some elementary linear algebra to understand. This is the kind of paper we would all love to write, and I've found it hard to stop thinking about it since I first read it a few days ago.

(Incidentally, there is also a physics connection here. Although Press is a computational biologist at the University of Texas he happens to have written several papers in theoretical astrophysics and cosmology, including a famous one with Paul Schechter in 1974, which is one of the most highly cited theoretical papers in all of cosmology. He is also one of the authors of the Numerical Recipes books, which anyone who has ever done any programming has probably used at some point. Dyson is best known to physicists for his work on the theory of quantum electrodynamics, in particular for being the first person — other than Feynman — to recognise the importance of the Feynman diagram. According to Steven Weinberg he was "fleeced" of a Nobel. He's also a remarkable 88 years old. You can read a beautiful character portrait of this complicated man here.)

Why are plants green?

Yes, I know, because chlorophyll is green. But why is that? This is a genuine question, so if any readers of this blog can answer it for me, I'd be grateful.

Let me explain a bit more. Yesterday I stumbled across an old post by Peter Coles asking why the Sun isn't green. If you haven't wondered this before, you should have a read. The reason for asking is that the intensity of light emitted by the Sun — which is very similar to a blackbody spectrum at a temperature of close to 6,000 Kelvin — peaks at the wavelengths associated with green.

Some example blackbody spectra. The Sun's surface temperature is about 5,800 K.

Naively therefore one might expect the Sun to appear green, rather than the colour it does, which is ... white (depending a little on where it is in the sky).

I won't explain here why the Sun doesn't appear green, or indeed why we don't see any green stars at all, because that's been done very well elsewhere on the internet. In particular you might want to check out this video explanation. As perhaps you might have suspected, the answer is less to do with physics than with biology of the human eye.

In fact, it's quite easy to come up with a simple ex post facto rationalisation of why we should have evolved so as to perceive sunlight as neutral white colour. 'White', after all, is a somewhat vague and flexible concept, as anyone who has operated a digital camera would know.

But in light of this, what I don't have such a simple explanation for is the fact that plants are green. Chlorophyll appears green because it uses light from the red and blue parts of the spectrum to power photosynthesis, and reflects green wavelengths. But why has it evolved in such a way as to reject precisely that part of the solar spectrum where the intensity of light is highest? Would it not have conferred an evolutionary advantage to make better use of this incident energy?

As I said, I don't know the answer to this little puzzle. Perhaps the explanation comes from chemistry — maybe there is simply some restriction on the possible chemical pathways by which CO₂ and H₂O can be converted into organic molecules, which all evolutionary variants would have to respect, and which renders light of certain frequencies unsuitable. Or perhaps it comes from some quirk of evolutionary history — one of those little inefficiencies that occur due to historical accident, and which are themselves evidence for the action of evolution, like the panda's thumb.

Either way, if you know the answer or can hazard a guess at it, please do tell me.