Part of the reason this blog has been quiet recently is that I decided at the start of this year to try to avoid — as far as possible — purely negative comments on incorrect, overhyped papers, and focus only on positive developments. (The other part of the reason is that I am working too hard on other things.)
There are only three things in these headlines that I disagree with: that this thing is a "structure", that it is the largest in the Universe, and that it solves the Cold Spot mystery.
Let's focus on the last of these. Readers of this blog may remember that I wrote about the Cold Spot mystery in August last year, referring to a paper my collaborators and I had written which conclusively showed that this very same supervoid could not explain the mystery. Our paper was published back in November in Phys. Rev. D (journal link, arXiv link). And yet here we are six months later, with the same claims being repeated!
Does the paper by Szapudi et al. (journal link, arXiv link) refute the analysis in our paper? Does it even acknowledge the results in our paper? No, it pretends this analysis does not exist and makes the claims anyway.
Just to be clear, it's possible that Szapudi's team are unaware of our paper and the fact that it directly challenged their conclusions several months before their own paper was published, even though Phys. Rev. D is a very high profile journal. This is sad and would reflect a serious failure on their part and that of the referees. The only alternative explanation would be that they were aware of it but chose not to even acknowledge it, let alone attempt to address the argument within it. This would be so ethically inexcusable that I am sure it cannot be correct.
I am also frankly amazed at the standard of refereeing which I'm afraid reflects extremely poorly on the journal, MNRAS.
Coming to the details. In our paper last year, we made the following points:
If this counts as a "solution to a cosmic mystery" then I'm Stephen Hawking.
The supervoid can only account for less than 10% of the total temperature decrement at the centre of the Cold Spot (angle of $0^\circ$). At other angles it does worse, failing to even predict the correct sign! And remember, this prediction only assumes that our current understanding of cosmology is not completely, drastically wrong in some way that has somehow escaped our attention until now.
You'll also notice that if the entire red line is somehow magically (through hypothetical "modified gravity effects") scaled down to match the blue line at the centre, it remains wildly, wildly wrong at every other angle. This is a direct consequence of the fact that the supervoid is very large, but really not very empty at all.
By contrast, the simple fact that the Cold Spot is chosen to be the coldest spot in the entire CMB already accounts for 100% of the cold temperature at the centre:
Similarly, the fact that Mt. Everest is much higher than sea level is not at all surprising. The highest mountains on other planets (Mars, for instance) can be a lot higher still.
But how to explain the fact that a large void does appear to lie in the same direction as the Cold Spot? Is this not a huge coincidence that should be telling us something?
Let's try the following calculation. Take the hypothesis that this particular void is causing the Cold Spot, let's call it hypothesis H1. Denote the probability that this void exists by $p_\mathrm{void}$, and the probability that all of GR is wrong and that some unknown physics leads to a causal relationship as $p_\mathrm{noGR}$. Then
$$p_\mathrm{H1}=p_\mathrm{void}p_\mathrm{noGR}$$On the other hand, let H2 be the hypothesis that the void and the Cold Spot are separate rare occurrences that happen by chance to be aligned on the sky. This gives
$$p_\mathrm{H2}=p_\mathrm{void}p_\mathrm{CS}p_\mathrm{align},$$where $p_\mathrm{CS}$ is the probability that the Cold Spot is a random fluctuation on the last scattering surface, and $p_\mathrm{align}$ the probability that two are aligned.
The relative likelihood of the two rival hypotheses is given by the ratio of the probabilities:
$$\frac{p_\mathrm{H1}}{p_\mathrm{H2}}=\frac{p_\mathrm{noGR}}{p_\mathrm{CS}p_\mathrm{align}}.$$Suppose we assume that $p_\mathrm{CS}=0.05$, and that the chance of alignment at random is $p_\mathrm{align}=0.001$.[1] Then the likelihood we should assign to "supervoid-caused-the-Cold-Spot" hypothesis depends on whether we think $p_\mathrm{noGR}$ is more or less than 1 in 20,000.
This exact calculation appears in Szapudi et al's paper, except that they mysteriously leave out the numerator on the right hand side. This means that they assume, with probability 1, that general relativity is wrong and that some unknown cause exists which makes a void with only a 20% deficit of matter create a massive temperature effect. In other words, they've effectively assumed their conclusion in advance.
Well, call me old-fashioned, but I don't think that makes any sense. We have a vast abundance of evidence, gathered over the last 100 years, which show that if indeed GR is not the correct theory of gravity it is still pretty damn close to it. What's more, we have lots of cosmological evidence — from the Planck CMB data, from cross-correlation measurements of the ISW effect, as well as from weak lensing — that gravity behaves very much as we think it does on cosmological scales. Looking at the figure above, for the supervoid to explain the Cold Spot requires at least a factor of 10 increase in the ISW effect at the void centre, as well as a dramatic effect on the shape of the temperature profile. And all this for a void with only a 20% deficit of matter! If the ISW effect truly behaved like this we would have seen evidence of it in other data.
For my money, I would put $p_\mathrm{noGR}$ at no higher than $2.9\times10^{-7}$, i.e. I would rule out the possibility at $5\sigma$ confidence. This is a lot less than 1:20,000, so I would say chance alignment is strongly favoured. Of course you should feel free to put your own weight on the validity of all of the foundations of modern cosmology, but I suggest you would be very foolish indeed to think, as Szapudi et al. seem to do, that it is absolutely certain that these foundations are wrong.
So much for the science, such as it is. The sociological observation that this episode brings me back to is that, almost without exception, whenever a paper on astronomy or cosmology is accompanied by a big press release, either the science is flawed, or the claims in the press release bear no relation to the contents of the paper. This is a particularly blatant example, where the authors have generated a big splash by ignoring (or being unaware of) existing scientific literature that runs contrary to their argument. But the phenomenon is much more ubiquitous than this.
I find this deeply depressing. Like most other young researchers (I hope), I entered science with the naive impression that what counted in this business was the accuracy and quality of research, the presentation of evidence, and — in short — facts. I thought the scientific method would ensure that papers would be rigorously peer-reviewed. I did not expect that how seriously different results are taken would instead depend on the seniority of the lead author and the slickness of their PR machine. Do we now need to hold press conferences every time we publish a paper just to get our colleagues to cite our work?
One possible response to this is that I was hopelessly naive, so more fool me. Another, which I still hope is closer to the truth, is that, in the long run, the crap gets weeded out and that truth eventually prevails. But in an era when public "impact" of scientific research is an important criterion for career advancement, and such impact can be simply achieved by getting the media to hype up nonsense papers [2], I am sadly rather more skeptical of the integrity of scientists [3].
[1] This probability for alignment is the number quoted by Szapudi's team, based on the assumption that there is only one such supervoid, which could be anywhere in the sky. In fact, as I've already said, theory and simulation suggest there should be as many as 20 supervoids, and several have already been seen elsewhere in the sky (including one other by Szapudi's team themselves!). The probability that any one supervoid should be aligned with the Cold Spot should therefore be roughly 20 times larger, or 0.02.
[2] Not everything in Szapudi's paper is nonsense, of course. For instance, it seems quite likely that there is indeed a large underdensity where they say. But there is still a deal of nonsense (described above) in the actual paper, and vastly more in the press releases, especially the one from the Institute for Astronomy in Hawaii.
[3] On the whole, given the circumstances I though journalists handled the hype quite well, especially Hannah Devlin in the Guardian, who included a skeptical take from Carlos Frenk. (I suspect Carlos at least was aware of our paper!)
Unfortunately, last week a cosmology story hit the headlines that is so blatantly incorrect and yet so unashamedly marketed to the press that I'm afraid I am going to have to change that stance. This is the story that a team of astronomers led by Istvan Szapudi of the University of Hawaii have found "the largest structure in the Universe", which is a "huge hole" or "supervoid" that "solves the cosmic mystery" of the CMB Cold Spot. This story was covered by all the major UK daily news outlets last week, from the Guardian to the Daily Mail to the BBC, and has been reproduced in various forms in all sorts of science blogs around the world.
There are only three things in these headlines that I disagree with: that this thing is a "structure", that it is the largest in the Universe, and that it solves the Cold Spot mystery.
Let's focus on the last of these. Readers of this blog may remember that I wrote about the Cold Spot mystery in August last year, referring to a paper my collaborators and I had written which conclusively showed that this very same supervoid could not explain the mystery. Our paper was published back in November in Phys. Rev. D (journal link, arXiv link). And yet here we are six months later, with the same claims being repeated!
Does the paper by Szapudi et al. (journal link, arXiv link) refute the analysis in our paper? Does it even acknowledge the results in our paper? No, it pretends this analysis does not exist and makes the claims anyway.
Just to be clear, it's possible that Szapudi's team are unaware of our paper and the fact that it directly challenged their conclusions several months before their own paper was published, even though Phys. Rev. D is a very high profile journal. This is sad and would reflect a serious failure on their part and that of the referees. The only alternative explanation would be that they were aware of it but chose not to even acknowledge it, let alone attempt to address the argument within it. This would be so ethically inexcusable that I am sure it cannot be correct.
I am also frankly amazed at the standard of refereeing which I'm afraid reflects extremely poorly on the journal, MNRAS.
Coming to the details. In our paper last year, we made the following points:
- Unless our understanding of general relativity in general, and the $\Lambda$CDM cosmological model in particular, is completely wrong, this particular supervoid, which is large but only has at most 20% less matter than average, is completely incapable of explaining the temperature profile of the Cold Spot.
- Unless our understanding is completely wrong as above, the kind of supervoid that could begin to explain the Cold Spot is incredibly unlikely to exist — the chances are about 1:1,000,000!
- The corresponding chances that the Cold Spot is simply a random fluctuation that requires no special explanation are at worst 1:1000, and, depending on how you analyse the question, probably a lot better.
- This particular supervoid is big and rare, but not extremely so. In fact several voids that are as big or bigger, and as much as 4 times emptier, have already been seen elsewhere in the sky, and theory and simulation both suggest there could be as many as 20 of them.
To illustrate point 1 graphically, I made the following figure showing the actual averaged temperature profile of the Cold Spot versus the prediction from this supervoid:
 |
| Image made by me. |
If this counts as a "solution to a cosmic mystery" then I'm Stephen Hawking.
The supervoid can only account for less than 10% of the total temperature decrement at the centre of the Cold Spot (angle of $0^\circ$). At other angles it does worse, failing to even predict the correct sign! And remember, this prediction only assumes that our current understanding of cosmology is not completely, drastically wrong in some way that has somehow escaped our attention until now.
You'll also notice that if the entire red line is somehow magically (through hypothetical "modified gravity effects") scaled down to match the blue line at the centre, it remains wildly, wildly wrong at every other angle. This is a direct consequence of the fact that the supervoid is very large, but really not very empty at all.
By contrast, the simple fact that the Cold Spot is chosen to be the coldest spot in the entire CMB already accounts for 100% of the cold temperature at the centre:
 |
| The red line is the observed Cold Spot temperature profile. |
Similarly, the fact that Mt. Everest is much higher than sea level is not at all surprising. The highest mountains on other planets (Mars, for instance) can be a lot higher still.
But how to explain the fact that a large void does appear to lie in the same direction as the Cold Spot? Is this not a huge coincidence that should be telling us something?
Let's try the following calculation. Take the hypothesis that this particular void is causing the Cold Spot, let's call it hypothesis H1. Denote the probability that this void exists by $p_\mathrm{void}$, and the probability that all of GR is wrong and that some unknown physics leads to a causal relationship as $p_\mathrm{noGR}$. Then
$$p_\mathrm{H1}=p_\mathrm{void}p_\mathrm{noGR}$$On the other hand, let H2 be the hypothesis that the void and the Cold Spot are separate rare occurrences that happen by chance to be aligned on the sky. This gives
$$p_\mathrm{H2}=p_\mathrm{void}p_\mathrm{CS}p_\mathrm{align},$$where $p_\mathrm{CS}$ is the probability that the Cold Spot is a random fluctuation on the last scattering surface, and $p_\mathrm{align}$ the probability that two are aligned.
The relative likelihood of the two rival hypotheses is given by the ratio of the probabilities:
$$\frac{p_\mathrm{H1}}{p_\mathrm{H2}}=\frac{p_\mathrm{noGR}}{p_\mathrm{CS}p_\mathrm{align}}.$$Suppose we assume that $p_\mathrm{CS}=0.05$, and that the chance of alignment at random is $p_\mathrm{align}=0.001$.[1] Then the likelihood we should assign to "supervoid-caused-the-Cold-Spot" hypothesis depends on whether we think $p_\mathrm{noGR}$ is more or less than 1 in 20,000.
This exact calculation appears in Szapudi et al's paper, except that they mysteriously leave out the numerator on the right hand side. This means that they assume, with probability 1, that general relativity is wrong and that some unknown cause exists which makes a void with only a 20% deficit of matter create a massive temperature effect. In other words, they've effectively assumed their conclusion in advance.
Well, call me old-fashioned, but I don't think that makes any sense. We have a vast abundance of evidence, gathered over the last 100 years, which show that if indeed GR is not the correct theory of gravity it is still pretty damn close to it. What's more, we have lots of cosmological evidence — from the Planck CMB data, from cross-correlation measurements of the ISW effect, as well as from weak lensing — that gravity behaves very much as we think it does on cosmological scales. Looking at the figure above, for the supervoid to explain the Cold Spot requires at least a factor of 10 increase in the ISW effect at the void centre, as well as a dramatic effect on the shape of the temperature profile. And all this for a void with only a 20% deficit of matter! If the ISW effect truly behaved like this we would have seen evidence of it in other data.
For my money, I would put $p_\mathrm{noGR}$ at no higher than $2.9\times10^{-7}$, i.e. I would rule out the possibility at $5\sigma$ confidence. This is a lot less than 1:20,000, so I would say chance alignment is strongly favoured. Of course you should feel free to put your own weight on the validity of all of the foundations of modern cosmology, but I suggest you would be very foolish indeed to think, as Szapudi et al. seem to do, that it is absolutely certain that these foundations are wrong.
So much for the science, such as it is. The sociological observation that this episode brings me back to is that, almost without exception, whenever a paper on astronomy or cosmology is accompanied by a big press release, either the science is flawed, or the claims in the press release bear no relation to the contents of the paper. This is a particularly blatant example, where the authors have generated a big splash by ignoring (or being unaware of) existing scientific literature that runs contrary to their argument. But the phenomenon is much more ubiquitous than this.
I find this deeply depressing. Like most other young researchers (I hope), I entered science with the naive impression that what counted in this business was the accuracy and quality of research, the presentation of evidence, and — in short — facts. I thought the scientific method would ensure that papers would be rigorously peer-reviewed. I did not expect that how seriously different results are taken would instead depend on the seniority of the lead author and the slickness of their PR machine. Do we now need to hold press conferences every time we publish a paper just to get our colleagues to cite our work?
One possible response to this is that I was hopelessly naive, so more fool me. Another, which I still hope is closer to the truth, is that, in the long run, the crap gets weeded out and that truth eventually prevails. But in an era when public "impact" of scientific research is an important criterion for career advancement, and such impact can be simply achieved by getting the media to hype up nonsense papers [2], I am sadly rather more skeptical of the integrity of scientists [3].
----
[1] This probability for alignment is the number quoted by Szapudi's team, based on the assumption that there is only one such supervoid, which could be anywhere in the sky. In fact, as I've already said, theory and simulation suggest there should be as many as 20 supervoids, and several have already been seen elsewhere in the sky (including one other by Szapudi's team themselves!). The probability that any one supervoid should be aligned with the Cold Spot should therefore be roughly 20 times larger, or 0.02.
[2] Not everything in Szapudi's paper is nonsense, of course. For instance, it seems quite likely that there is indeed a large underdensity where they say. But there is still a deal of nonsense (described above) in the actual paper, and vastly more in the press releases, especially the one from the Institute for Astronomy in Hawaii.
[3] On the whole, given the circumstances I though journalists handled the hype quite well, especially Hannah Devlin in the Guardian, who included a skeptical take from Carlos Frenk. (I suspect Carlos at least was aware of our paper!)
