Is falsifiability a scientific idea due for retirement?

Sean Carroll argues that it is.

He characterises the belief that "theories should be falsifiable" as a "fortune-cookie-sized motto"; it's a position adopted only by "armchair theorizers" and "amateur philosophers", and people who have no idea how science really works. He thinks we need to move beyond the idea that scientific theories need to be falsifiable; this appears to be because he wants to argue that string theory and the idea of the multiverse are not falsifiable ideas, but are still scientific.

This position is not just wrong, it's ludicrous. 

What's more, I think deep down Sean – who is normally a clear, precise thinker – realises that it is ludicrous. Midway through his essay, therefore, he flaps around trying to square the circle and get out of the corner he has painted himself into: a scientific theory must, apparently, still be "judged on its ability to account for the data", and it's still true that "nature is the ultimate guide". But somehow it isn't necessary for a theory to be falsifiable to be scientific.

Now, I'm not a philosopher by training. Therefore what follows could certainly be dismissed as "amateur philosophising". I'm almost certain that what I say has been said before, and said better, by other people in other places. Nevertheless, as a practising scientist with an argumentative tendency, I'm going to have to rise to the challenge of defending the idea of falsifiability as the essence of science. Let's start by dismantling the alternatives.

Can we use some other words instead?

Sean tries to substitute "falsifiability" with two of his own criteria: according to him, to be judged to be scientific a theory must be "definite", and it must be "empirical". 

Apparently a theory is definite so long as it makes some clear and unambiguous statement about how reality works, but this is such a broad definition as to include basically any idea about the world, from Plato to creationism to string theory and everything in between. Nor is "empirical" a better starting point, because this apparently merely means that theories should "account for the data" in some way. 

To see what's wrong with this, let's look at a concrete example. In 1857, Philip Henry Gosse wrote a book called Omphalos: An Attempt to Untie the Geological Knot, in which he put forward his theory of how to reconcile the existing evidence of fossilised animals (on which he, as a geologist, was an acknowledged expert), and his belief in the literal accuracy of the Biblical story of creation.

According to Gosse, the fossils that were being uncovered from geological strata were not evidence of creatures that had existed many millions of years ago, but rather had been put there by God when he created the world, in order to convey the impression that they were the remains of animals that had lived many millions of years ago. Other people might claim that the fossilized remains of partially digested food that was sometimes found in the stomachs of these animals indicated that they had actually once lived, and eaten, and digested food. But for Gosse this was just another part of God's master plan of deception: misleading evidence cleverly placed to further our mistaken impression. And indeed, we would expect God to not overlook even the tiniest detail, wouldn't we?

Now, clearly Gosse's theory was completely unscientific tosh, and no one has ever taken it seriously.¹ But what exactly is wrong with it? After all, it is "definite": it certainly makes perfectly unambiguous statements about reality. It is also "empirical" in the sense that Sean uses the word. The theory is constructed precisely in order to explain data (the fossils), and it does so in a perfectly coherent manner. If it meets both of these new criteria for scientific validity, does that mean we must treat it as scientific?

Obviously not. As Stephen Jay Gould argued beautifully in his essay Adam's Navel, the reason that Gosse's theory is unscientific tosh is because it is not falsifiable. It is a Theory of Anything. There is no possible fossil data that it could fail to explain. It is based on some assumptions, but it is impossible to use evidence to decide whether these assumptions are true or not. Therefore it is not science.

The point, clearly, is that the criterion of falsifiability is still a useful and necessary one in distinguishing theories that are scientific, and theories that are not. You can't go around arguing that we should do away with falsifiability as a standard criterion for scientific acceptability when your suggested alternatives would pass Gosse's theory as scientific as well.

Does the meaning of the word 'falsifiability' need to be updated?

Perhaps the problem is that our notion of what makes a theory "falsifiable" is just too simplistic, too Popperian. Maybe if we had a more nuanced view of what the word "falsifiable" should mean, we would be able to sort the theories and ideas worth pursuing from their unscientific relatives.

This is an idea I actually have a bit of sympathy with. Popper's standard picture The armchair philosopher's view of how falsification works in science – you come up with a theory, and use it to make a definite prediction for what should be seen in an experiment; if you see something different, you kill the theory – is far too simplistic. [Edit: correction inserted here after being called out in the comments for misrepresenting Karl Popper!] That's simply not how science works, or even how it should work.

For a start, experiments are often wrong, so a single experimental result is rarely enough to overturn a theory. Take for instance the case of the neutrinos that apparently travelled faster than light. Almost no serious physicist believed the theory needed to be changed, despite the very impressive quoted statistical significance of 6 standard deviations (i.e., a chance of roughly one in ten billion that it was due to a fluke).² As it turned out, quite rightly so.

Even in other less extreme circumstances, the simple picture of falsification doesn't work because scientists are all Bayesians in practice. What happens is that we start with a prior belief in the truth or otherwise of a theory (informed partly by prejudice and partly by previous experimental results), and we update this assigned probability in light of each new piece of evidence. Given a sufficiently strong prior, even conflicting empirical evidence may not be enough to cause us to reject a theory, but only slightly reduce the posterior probability we assign to it.

However – and this is where I'm going to volunteer my own definition – to be scientific a theory must admit the possibility that experimental evidence will reduce the strength of our belief in it. It may never be possible to rule a theory out, but if experiments can never make it less believable then it's not scientific. We will have stretch the meaning of the word "falsifiability" to include such cases where experimental data only allow us to shift the balance of probabilities rather than make definite yes or no statements, but that's not unreasonable. It is then still OK to insist that to be scientific, a theory must be falsifiable. If there is no earthly event that could ever occur that would make you reconsider your belief in some idea, it's not scientific, as simple as that.

Note that it isn't enough to claim that an idea is falsifiable in principle, but just not in any practically accessible realm of experiment. By that standard, Gosse's Omphalos would still be scientific, because in principle it could be falsified if God were to make himself visible to us and explain that he didn't put the fossils there to fool us but that evolution really happened. A theory is only scientific if there is any realistic possibility of experimental results that might cause us to alter the probability we assign to it being true.

This doesn't necessarily mean that those experiments must be possible to perform today. It isn't even necessary that we should be able to immediately imagine the form of the experiments that would test the theory. Because of the difficult nature of theorizing and the long time it may take to make progress in a field, or for technological advances to catch up with the state of the art, we should be willing to grant each idea a grace period: we accept that it is not yet clear whether it is falsifiable, but that it might turn out to be, and in the meantime it may be considered worthy of scientific endeavour. The length of the grace period we are willing to grant is influenced by our prior level of belief in the theory, which will in turn depend on such factors as its internal logical coherence, the number of new assumptions it requires, and the rate of progress towards establishing falsifying tests. But no grace period is infinitely long.

So what about the multiverse and string theory? 

Of course we were always going to end up here, because the whole point of Sean's piece was to argue that although the multiverse and string theory are not falsifiable ideas, they should still count as science. I hope I've convinced you that this is rubbish. I'm not necessarily saying that the multiverse idea isn't falsifiable in the slightly broader sense defined above; but if it isn't, it certainly should not be considered a scientific theory.

One line of defence that people sometimes produce in favour of the multiverse – Sean's original piece does this, but so does Shaun Hotchkiss here – is that it may not be falsifiable, but it still could be true. [Edit: Shaun states that this is not quite his position, see the clarification in this comment below.] I think this misses the point. Of course it could be true. So could Omphalos. If you think about it, there is no way to prove that God did not cunningly create the universe just to appear as if inflation and the Big Bang and evolution and all the rest happened, therefore the theory that he/she/it did so could be true. But this does not make it scientific, nor does it make it worthy of study. In fact we would almost certainly still gain more concrete knowledge of the nature of reality by spending our time studying other, falsifiable, theories, even if they are subsequently falsified.

On the other hand, some ideas are worthy of study irrespective of whether they can be tested, and irrespective of whether they are "true" or not. For instance, proofs of theorems that the require the axiom of choice to be false could be just as mathematically interesting as those which require it to be true. A thorough evaluation of the logical consequences of a clearly stated set of untestable assumptions is a fine intellectual pursuit. But it is then mathematics we are doing, not science.

If the assumptions underlying string theory or the multiverse cannot ever be tested by experiment, then that is what they should be classified as – mathematics. We need the clear demarcation between what is science and what isn't in order to keep out intruders like Omphalos. As scientists, we face a constant struggle against unscientific nonsense in the world at large. Falsifiability is our great weapon in this struggle, and we shouldn't give up that discriminatory power in order to fit someone's favourite square peg into a round hole.

---

That said, there is a way in which the multiverse idea that arises from string theory could potentially be falsifiable. This is through its "prediction" that the parameters that appear in the laws of physics should lie close to what are called "catastrophic boundaries". For a good introduction to how this reasoning works, you could read Shaun's posts on the multiverse here and here. If these parameters could be shown to lie close to catastrophic boundaries, this could tip the probabilities in favour of the multiverse. Or if they did not, it would strongly argue against it.

In such a sense, the multiverse would be falsifiable and therefore scientific, even if we could never conclusively say if it were true or not. (It seems to me that this is the sensible approach for defenders of the multiverse to take, rather than ridiculously glorifying in its lack of falsifiability, as Andrei Linde does here.)

In fact it is often argued (as Shaun does) that the strongest point in favour of the multiverse is that it explains the extremely small value of the cosmological constant we observe. This would indeed be a very strong point in its favour! However, I don't think this issue is quite so simple as is made out. In my next blog post, I will try to explain why.

¹ Gosse's contemporaries rejected his ideas out of hand, though they did so not necessarily because they understood evolution, or the scientific method, but because they – perhaps understandably! – could not bring themselves to believe in a God who would be so deceitful.

² The fact that they didn't believe the theory needed to be changed didn't of course stop them from immediately speculating on ways in which it could be changed. This is because theoretical physicists are opportunists, who will take any chance to write a newsworthy paper. The fact that they expected the result to go away soon just added an urgency to try to publish quickly.