A Supervoid cannot explain the Cold Spot

In my last post, I mentioned the claim that the Cold Spot in the cosmic microwave background is caused by a very large void — a "supervoid" — lying between us and the last scattering surface, distorting our vision of the CMB, and I promised to say a bit more about it soon. Well, my colleagues (Mikko, Shaun and Syksy) and I have just written a paper about this idea which came out on the arXiv last week, and in this post I'll try to describe the main ideas in it.

First, a little bit of background. When we look at sky maps of the CMB such as those produced by WMAP or Planck, obviously they're littered with very many hot and cold spots on angular scales of about one degree, and a few larger apparent "structures" that are discernible to the naked eye or human imagination. However, as I've blogged about before, the human imagination is an extremely poor guide to deciding whether a particular feature we see on the sky is real, or important: for instance, Stephen Hawking's initials are quite easy to see in the WMAP CMB maps, but this doesn't mean that Stephen Hawking secretly created the universe.

So to discover whether any particular unusual features are actually significant or not we need a well-defined statistical procedure for evaluating them. The statistical procedure used to find the Cold Spot involved filtering the CMB map with a special wavelet (a spherical Mexican hat wavelet, or SMHW), of a particular width (in this case $6^\circ$), and identifying the pixel direction with the coldest filtered temperature with the direction of the Cold Spot. Because of the nature of the wavelet used, this ensures that the Cold Spot is actually a reasonably sizable spot on the sky, as you can see in the image below:

The Cold Spot in the CMB sky. Image credit: WMAP/NASA.

Well, so we've found a cold spot. To elevate it to the status of "Cold Spot" in capitals and worry about how to explain it, we first need to quantify how unusual it is. Obviously it is unusual compared to other spots on our observed CMB, but this is true by construction and not very informative. Instead the usual procedure quite rightly compares the properties of the cold spots found in random Gaussian maps using exactly the same SMHW technique to the properties of the Cold Spot in our CMB. It is this procedure which results in the conclusion that our Cold Spot is statistically significant at roughly the "3-sigma level", i.e. only about 1 in every 1000 random maps has a coldest spot that is as "cold" as* our Cold Spot.** (The reason why I'm putting scare quotes around everything should become clear soon!)

So there appears to be a need to explain the existence of the Cold Spot using additional new physics of some kind. One such idea that that of the supervoid: a giant region hundreds of millions of light years across which is substantially emptier than the rest of the universe and lies between us and the Cold Spot. The emptiness of this region has a gravitational effect on the CMB photons that pass through it on their way to us, making them look colder (this is called the integrated Sachs-Wolfe or ISW effect) — hence the Cold Spot.

Now this is a nice idea in principle. In practice, unfortunately, it suffers from a problem: the ISW effect is very weak, so to produce an effect capable of "explaining" the Cold Spot the supervoid would need to be truly super — incredibly large and incredibly empty. And no such void has actually been seen in the distribution of galaxies (a previous claim to have seen it turned out to not be backed up by further analysis).

It was therefore quite exciting when in May a group of astronomers, led by Istvan Szapudi of the Institute for Astronomy in Hawaii, announced that they had found evidence for the existence of a large void in the right part of the sky. Even more excitingly, in a separate theoretical paper, Finelli et al. claimed to have modeled the effect of this void on the CMB and proven that it exactly fit the observations, and that therefore the question had been effectively settled: the Cold Spot was caused by a supervoid.

Except ... things aren't quite that simple. For a start, the void they claimed to have found doesn't actually have a large ISW effect — in terms of central temperature, less than one-seventh what would be needed to explain the Cold Spot. So Finelli et al. relied on a rather curious argument: that the second-order effect (in perturbation theory terms) of this void on CMB photons was somehow much larger than the first-order (i.e. ISW) effect. A puzzling inversion of our understanding of perturbation theory, then!

In fact there were a number of other reasons to be a bit suspicious of the claim, among which were that N-body simulations don't show this kind of unusual effect, and that several other larger and deeper voids have already been found that aren't aligned with Cold Spot-like CMB features. In our paper we provide a fuller list of these reasons to be skeptical before diving into the details of the calculation, where one might get lost in the fog of equations.

At the end of the day we were able to make several substantive points about the Cold Spot-as-a-supervoid hypothesis:

1. Contrary to the claim by Finelli et al., the void that has been found is neither large enough nor deep enough to leave a large effect on the CMB, either through the ISW effect or its second-order counterpart — in simple terms, it is not a super enough supervoid.
2. In order to explain the Cold Spot one needs to postulate a supervoid that is so large and so deep that the probability of its existence is essentially zero; if such a supervoid did exist it would be more difficult to explain that the Cold Spot currently is!
3. The possible ISW effect of any kind of void that could reasonably exist in our universe is already sufficiently accounted for in the analysis using random maps that I described above.
4. There's actually very little need to postulate a supervoid to explain the central temperature of the Cold Spot — the fact that we chose the coldest spot in our CMB maps already does that!

Point number 1 requires a fair bit of effort and a lot of equations to prove (and coincidentally it was also shown in an independent paper by Jim Zibin that appeared just a day before ours), but in the grand scheme of things it is probably not a supremely interesting one. It's nice to know that our perturbation theory intuition is correct after all, of course, but mistakes happen to the best of us, so the fact that one paper on the arXiv contains a mistake somewhere is not tremendously important.

On the other hand, point 2 is actually a fairly broad and important one. It is a result that cosmologists with a good intuition would perhaps have guessed already, but that we are able to quantify in a useful way: to be able to produce even half the temperature effect actually seen in the Cold Spot would require a hypothetical supervoid almost twice as large and twice as empty as the one seen by Szapudi's team, and the odds of such a void existing in our universe would be something like a one-in-a-million or one-in-a-billion (whereas the Cold Spot itself is at most a one-in-a-thousand anomaly in random CMB maps). A supervoid therefore cannot help to explain the Cold Spot.***

Point 3 is again something that many people probably already knew, but equally many seem to have forgotten or ignored, and something that has not (to my knowledge) been stated explicitly in any paper. My particular favourite though is point 4, which I could — with just a tiny bit of poetic licence — reword as the statement that

"the Cold Spot is not unusually cold; if anything, what's odd about it is only that it is surrounded by a hot ring"

I won't try to explain the second part of that statement here, but the details are in our paper (in particular Figure 7, in case you are interested). Instead what I will do is to justify the first part by reproducing Figure 6 of our paper here:

The averaged temperature anisotropy profile at angle $\theta$ from the centre of the Cold Spot (in red),  and the corresponding 1 and $2\sigma$ contours from the coldest spots in 10,000 random CMB maps (blue). Figure from arXiv:1408.4720.

What the blue shaded regions show is the confidence limits on the expected temperature anisotropy $\Delta T$ at angles $\theta$ from the direction of the coldest spots found in random CMB maps using exactly the SMHW selection procedure. The red line, which is the measured temperature for our actual Cold Spot, never goes outside the $2\sigma$ equivalent confidence region. In particular, at the centre of the Cold Spot the red line is pretty much exactly where we would expect it to be. The Cold Spot is not actually unusually cold.

Just before ending, I thought I'd also mention that Syksy has written about this subject on his own blog (in Finnish only): as I understand it, one of the points he makes is that this form of peer review on the arXiv is actually more efficient than the traditional one that takes place in journals.

Update: You might also want to have a look at Shaun's take on the same topic, which covers the things I left out here ...

* People often compare other properties of the Cold Spot to those in random maps, for instance its kurtosis or other higher-order moments, but for our purposes here the total filtered temperature will suffice.

** Although as Zhang and Huterer pointed out a few years ago, this analysis doesn't account for the particular choice of the SMHW filter or the particular choice of $6^\circ$ width — in other words, that it doesn't account for what particle physicists call the "look-elsewhere effect". Which means it is actually much less impressive.

*** If we'd actually seen a supervoid which had the required properties, we'd have a proximate cause for the Cold Spot, but also a new and even bigger anomaly that required an explanation. But as we haven't, the point is moot.

Is falsifiability a scientific idea due for retirement?

Sean Carroll argues that it is.

He characterises the belief that "theories should be falsifiable" as a "fortune-cookie-sized motto"; it's a position adopted only by "armchair theorizers" and "amateur philosophers", and people who have no idea how science really works. He thinks we need to move beyond the idea that scientific theories need to be falsifiable; this appears to be because he wants to argue that string theory and the idea of the multiverse are not falsifiable ideas, but are still scientific.

This position is not just wrong, it's ludicrous. 

What's more, I think deep down Sean – who is normally a clear, precise thinker – realises that it is ludicrous. Midway through his essay, therefore, he flaps around trying to square the circle and get out of the corner he has painted himself into: a scientific theory must, apparently, still be "judged on its ability to account for the data", and it's still true that "nature is the ultimate guide". But somehow it isn't necessary for a theory to be falsifiable to be scientific.

Now, I'm not a philosopher by training. Therefore what follows could certainly be dismissed as "amateur philosophising". I'm almost certain that what I say has been said before, and said better, by other people in other places. Nevertheless, as a practising scientist with an argumentative tendency, I'm going to have to rise to the challenge of defending the idea of falsifiability as the essence of science. Let's start by dismantling the alternatives.

Quasars, homogeneity and Einstein

[A little note: This post, like many others on this blog, contains a few mathematical symbols which are displayed using MathJax. If you are reading this using an RSS reader such as Feedly and you see a lot of $ signs floating around, you may need to click through to the blog to see the proper symbols.]

People following the reporting of physics in the popular press might remember having come across a paper earlier this year that claimed to have detected the "largest structure in the Universe" in the distribution of quasars, that "challenged the Cosmological Principle". This was work done by Roger Clowes of the University of Central Lancashire and collaborators, and their paper was published in the Monthly Notices of the Royal Astronomical Society back in March (though it was available online from late last year). 

The reason I suspect people might have come across it is that it was accompanied by a pretty extraordinary amount of publicity, starting from this press release on the Royal Astronomical Society website. This was then taken up by Reuters, and featured on various popular science websites and news outlets, including New Scientist, The Atlantic, National Geographic, Space.com, The Daily Galaxy, Phys.org, Gizmodo, and many more. The structure they claimed to have found even has its own Wikipedia entry.

Obligatory artist's impression of a quasar.

One thing that you notice in a lot of these reports is the statement that the discovery of this structure violates Einstein's theory of gravity, which is nonsense. This is sloppy reporting, sure, but the RAS press release is also partly to blame here, since it includes a somewhat gratuitous mention of Einstein, and this is exactly the kind of thing that non-expert journalists are likely to pick up on. Mentioning Einstein probably helps generate more traffic after all, which is why I've put him in the title as well.

But aside from the name-dropping, what about the main point about the violation of the cosmological principle? As a quick reminder, the cosmological principle is sometimes taken to be the assumption that, on large scales, the Universe is well-described as homogeneous and isotropic. 

The question of what constitutes "large scales" is sometimes not very well-defined: we know that on the scale of the Solar System the matter distribution is very definitely not homogeneous, and we believe that on the scale of size of the observable Universe it is. Generally speaking, people assume that on scales larger than about $100$ Megaparsecs, homogeneity is a fair assumption. A paper by Yadav, Bagla and Khandai from 2010 showed that if the standard $\Lambda$CDM cosmological model is correct, the scale of homogeneity must be less than at most $370$ Mpc. 

On the other hand, this quasar structure that Clowes et al. found is absolutely enormous: over 4 billion light years, or more than 1000 Mpc, long. Does the existence of such a large structure mean that the Universe is not homogeneous, the cosmological principle is not true, and the foundation on which all of modern cosmology is based is shaky?

Well actually, no. 

Unfortunately Clowes' paper is wrong, on several counts. In fact, I have recently published a paper myself (journal version here, free arXiv version here) which points out that it is wrong. And, on the principle that if I don't talk about my own work, no one else will, I'm going to try explaining some of the ideas involved here.

The first reason it is wrong is something that a lot of people who should know better don't seem to realise: there is no reason that structures should not exist which are larger than the homogeneity scale of $\Lambda$CDM. You may think that this doesn't make sense, because homogeneity precludes the existence of structures, so no structure can be larger than the homogeneity scale. Nevertheless, it does and they can.

Let me explain a little more. The point here is that the Universe is not homogeneous, at any scale. What is homogeneous and isotropic is simply the background model we use the describe its behaviour. In the real Universe, there are always fluctuations away from homogeneity at all scales – in fact the theory of inflation basically guarantees this, since the power spectrum of potential fluctuations is close to scale-invariant. The assumption that all cosmological theory really rests on is that these fluctuations can be treated as perturbations about a homogeneous background – so that a perturbation theory approach to cosmology is valid.

Given this knowledge that the Universe is never exactly homogeneous, the question of what the "homogeneity scale" actually means, and how to define it, takes on a different light. (Before you ask, yes it is still a useful concept!) One possible way to define it is as that scale above which density fluctuations $\delta$ generally become small compared to the homogeneous background density. In technical terms, this means the scale at which the two-point correlation function for the fluctuations, $\xi(r)$, (of which the power spectrum $P(k)$ is the Fourier transform) becomes less than $1$. Based on this definition, the homogeneity scale would be around $10$ Mpc.

It turns out that this definition, and the direct measurement of $\xi(r)$ itself, is not very good for determining whether or not the Universe is a fractal, which is a question that several researchers decided was an important one to answer a few years ago. This question can instead be answered by a different analysis, which I explained once before here: essentially, given a catalogue with the positions of many galaxies (or quasars, or whatever), draw a sphere of radius $R$ around each galaxy, and count how many other galaxies lie within this sphere, and how this number changes with $R$. The scale above which the average of this number for all galaxies starts scaling as the cube of the radius, $$N(<R)\propto R^3,$$ (within measurement error) is then the homogeneity scale (if it starts scaling as some other constant power of $R$, the Universe has a fractal nature). This is the definition of the homogeneity scale used by Yadav et al. and it is related to an integral of $\xi(r)$; typically measurements of the homogeneity scale using this definition come up with values of around $100-150$ Mpc.

The figure that proves that the distribution of quasars is in fact homogeneous on the expected scales. For details, see  arXiv:1306.1700. 

To get back to the original point, neither of these definitions of the homogeneity scale makes any claim about the existence of structures that are larger than that. In fact, in the $\Lambda$CDM model, the correlation function for matter density fluctuations is expected to be small but positive out to scales larger than either of the two homogeneity scales defined above (though not as large as Yadav et al.'s generous upper limit). The correlation function that can actually be measured using any given population of galaxies or quasars will extend out even further. So we already expect correlations to exist beyond the homogeneity scale – this means that, for some definitions of what constitutes a "structure", we expect to see large "structures" on these scales too.

The second reason that the claim by Clowes et al. is wrong is however less subtle. Given the particular definition of a "structure" they use, one would expect to find very large structures even if density correlations were exactly zero on all scales.

Yes, you read that right. It's worth going over how they define a "structure", just to make this absolutely clear. About the position of each quasar in the catalogue they draw a sphere of radius $L$. If any other quasars at all happen to lie within this sphere, they are classified as part of the same "structure", which can now be extended in other directions by repeating the procedure about each of the newly added member quasars. After repeating this procedure over all $18,722$ quasars in the catalogue, the largest such group of quasars identified becomes the "largest structure in the Universe".

It should be pretty obvious now that the radius $L$ chosen for these spheres, while chosen rather arbitrarily, is crucial to the end result. If it is too large, all quasars in the catalogue end up classified as part of the same truly ginormous "structure", but this is not very helpful. This is known as "percolation" and the critical percolation threshold has been thoroughly studied for Poisson point sets – which are by definition random distributions of points with no correlation at all. The value of $L$ that  Clowes et al. chose to use, for no apparent reason other than that it gave them a dramatic result, was $100$ Mpc – far too large to be justified on any theoretical grounds, but slightly lower than the critical percolation threshold would be if the quasar distribution was similar to that of a Poisson set. On the other hand, the "largest structure in the Universe" only consists of $73$ quasars out of $18,722$, so it could be entirely explained as a result of the poor definition ...

Now I'll spare you all the details of how to test whether, using this definition of a "structure", one would expect to find "structures" extending over more than $1000$ Mpc in length or with more than $73$ members or whatever, even in a purely random distribution of points, which are by definition homogeneous. Suffice it to say that it turns out one would. This plot shows the maximum extent of such "structures" found in $10,000$ simulations of completely uncorrelated distributions of points, compared to the maximum extent of the "structure" found in the real quasar catalogue.

The probability distribution of extents of largest "structures" found in 10,000 random point sets for two different choices of $L$. Vertical lines show the actual values found for "structures" in the quasar catalogue. The actual values are not very unusual. Figure from arXiv:1306.1700. 

To summarise then: finding a "structure" larger than the homogeneity scale does not violate the cosmological principle, because of correlations; on top of that, the "largest structure in the Universe" is actually not really a "structure" in any meaningful sense. In my professional opinion, Clowes' paper and all the hype surrounding it in the press is nothing more than that – hype. Unfortunately, this is another verification of my maxim that if a paper to do with cosmology is accompanied by a big press release, it is odds-on to turn out to be wrong.

Finally, before I leave the topic, I'll make a comment about the presentation of results by Clowes et al. Here, for instance, is an image they presented showing their "structure", which they call the 'Huge-LQG', with a second "structure" called the 'CCLQG' towards the bottom left:

3D representation of the Huge-LQG and CCLQG. From arXiv:1211.6256.

Looks impressive! Until you start digging a bit deeper, anyway. Firstly, they've only shown the quasars that form part of the "structure", not all the others around it. Secondly, they've drawn enormous spheres (of radius $33$ Mpc) at the position of each quasar to make it look more dramatic. In actual fact the quasars are way smaller than that. The combined effect of these two presentational choices is to make the 'Huge-LQG' look far more plausible as a structure than it really is. Here's a representation of the exact same region of space that I made myself, which rectifies both problems:

Quasar positions around the "structures" claimed by Clowes et al.

Do you still see the "structures"?

One year on

It has been a long month-and-a-bit since I last had the time to write a proper post here. Primarily this is because I am not very good at doing more than one thing at a time – at least while attempting to do those things properly – and I have been working on some real research papers, which I thought I should try to do properly. As a result science communication, and blogging in general, has had to wait. But I will get back in the swing of things very soon: there are some interesting new results, some interesting rehashes of old arguments about inflation, and one of my own papers that I will write about over the next couple of weeks.

One of the many things that I omitted to mention during this little break was the fact that Blank On The Map had its first anniversary a couple of weeks ago. All in all, I would say it has been quite a satisfying year of blogging, and, at least relative to my prior expectations, a reasonably successful one too. Blogger doesn't provide me with many detailed statistics, but it does tell me that despite the low recent rate of new postings, there have been roughly 23,000 clicks over the last twelve months. A sizeable portion of of this traffic came from one link on Peter Woit's blog (boy he must have a huge number of readers!) – though the majority appear to arrive via Google search, either by accident or design.

Anyway, to all those who have arrived here at some point over the last year, welcome! I hope you found what you were looking for, and enjoyed what you read. This explains what this blog is about. In case you haven't worked your way through the archives, here is a short selection of some highlights from the last year:

• Steven Weinberg and the 'Crisis of Big Science'
• An explanation of a dark matter controversy
• A question of probability
• The dog who climbed a mountain
• The Structure of a Scientific Revolution
• Explaining Planck by analogy

I should also use this post to try to get some feedback from (all three) regular readers. Do you think I should post more often? Less often? More posts about cosmology and fewer about mountaineering and other stuff in general? Or the other way round? Let me know through the comments box. Constructive criticism of the writing style or any other aspect of this blog will also be well received, though I can't promise to change ...

A real puzzle in cosmology: part II

(This post continues the discussion of the very puzzling observation of the integrated Sachs-Wolfe effect, the first part of which is here. Part II is a bit more detailed: many of these questions are real ones that have been put to me in seminars and other discussions.)

Update: I've been informed that one of the papers mentioned in this discussion has just been withdrawn by the authors pending some re-investigation of the results. I'll leave the original text of the post up here for comparison, but add some new material in the relevant sections.

Last time you told me about what you said was an unusual observation of the ISW effect made in 2008.

Yes. Granett, Neyrinck and Szapudi looked at the average CMB temperature anisotropies along directions on the sky where they had previously identified large structures in the galaxy distribution. For 100 structures (50 "supervoids" and 50 "superclusters"), after applying a specific aperture photometry method, they found an average temperature shift of almost 10 micro Kelvin, which was more than 4 standard deviations away from zero.

Then you claimed that this observed value was five times too large. That if our understanding of the universe was correct, the value they should have seen should not have been definitely bigger than zero. Theory and observation grossly disagree.

Right again. Our theoretical calculation showed the signal should have been at most around 2 micro Kelvin, which is pretty much the same size as the contamination from random noise.

But you used a simple theoretical model for your calculations. I don't like your model. I think it is too simple. That's the answer to your problem – your calculation is wrong.

That could be true – thought I don't think so. Why don't we go over your objections one by one?

Does Nate Silver get his maths wrong?

Although I haven't had much time to write anything new for this blog recently, like most people I know I have of course been paying close attention to the unfolding circus of the US Presidential election. Now that all the action is done and dusted, there is a lot of discussion on the internet about the predictions made by poll aggregators, the best-known of whom is the New York Times blogger and statistician Nate Silver. And although I'm not going to talk about politics, I do want to make a comment about probabilities and the statistics of poll aggregation.

The reason Silver has been in the news so much is because even when a lot of innumerate and possibly partisan journalists were making a huge amount of noise about "Romneymentum" and predicting a landslide Republican victory, Silver calmly analysed the actual polls and stated that the odds continued to favour an Obama victory. For this he attracted an enormous amount of flak in the right-wing press and blogs, and a substantial section of the mainstream media refused to believe him. (See here if you'd like a frequently-updated list of everyone who attacked Silver and ended up looking stupid.) Even on election eve, when Silver gave Obama more than a 90% chance of victory, such impartial websites as the BBC continued to describe the result as being on a "knife-edge".

Of course now that Obama has won, the pendulum has turned the other way, and Silver is being hailed as a mathematical genius, a wizard, a guru and much else besides. This adulation rather ignores the fact that there were several other bloggers and poll aggregators who also predicted an Obama win. Some of these are Sam Wang at the Princeton Election Consortium, Drew Linzer at Votamatic, Pollster at the Huffington Post, Simon Jackman and so on. (To be fair, Silver does have a much more prominent position than the others, being at the New York Times. He did also provide very insightful regular explanations and updates, and some nice interactive graphics to help explore the data, so one can forgive the extra attention given to him.)

But the interesting question is not whether Nate Silver does a better job of predicting elections than, say, Donald Trump. How does he compare with other, equally numerate and mathematically minded poll-aggregating bloggers? And there is really a difference between them: for a large part of October, Silver rated Obama's chances of re-election at only around 65-75%, whereas Sam Wang regularly claimed it was more like a 99% chance. Or have a look at Votamatic, where the predicted median of electoral college votes for Obama has been fairly constant at 332 (the correct number) since July, compared to Silver's prediction of closer to 290 at the same time. You'll also notice, if you play with the graphics at FiveThirtyEight, that the margin of error Silver quoted on his prediction was significantly larger than that the other forecasters gave.

So in summary, Nate Silver made more conservative predictions and was more pessimistic about Obama's re-election chances than his direct competitors. (Why so? As I understand it, Wang and Linzer only include information from published poll data, appropriately weighted for freshness and sample size. Silver included also what he calls "fundamentals" — his estimate of other underlying factors, such as economic data and so on, which might influence the final outcome. Silver's exact model is proprietary, but for an educated guess at how it works, explained in layman's terms, see here.) In the end though, Obama was re-elected, as all the forecasters said he would be. Since the only differences between them lay in exactly how confident they were in their prediction (and also whether they thought he would win Florida ... ), and the election can only be run once, how do you decide whose model is better?

A common argument you may have heard in favour of Silver before Obama's most recent victory is that he correctly predicted 49 out of 50 state results in 2008. Add to that his predictions this year and his tally is now 99 correct out of 100. This is obviously impressive, and clearly you should trust Silver and his maths more than basically every conservative hack who attacked him.

But the problem is that Silver may be too accurate. To put it bluntly, if you say the probability of your prediction being correct is around 70%, you should expect to be wrong 30% of the time, not 1% of the time. So if you predict the right result 99% of the time but quote probabilities of only 70%, you are probably overstating the error bars on your prediction. On the other hand, if someone else says he is 99% sure and is right 99% of the time, his model might be better.

OK, some obvious caveats. It is misleading to count all 50 state predictions as successes for Silver's model. I doubt even Donald Trump would have had trouble predicting Utah would go Republican and New York would go Democrat. In reality, there were only around 10 or 11 states that were at all competitive in each election cycle, so adjusting for that would mean lowering Silver's actual success percentage. Also, his model didn't predict all results with equal confidence — for instance, maybe he gave only a 75% probability for Obama to win Ohio, but a 99% probability for him to win Minnesota.

How should one account for this in quantitatively comparing models? Unsurprisingly, Sam Wang himself suggests a method, which is to assign each prediction a Brier score. This takes the square of the difference between the predicted probability of an event occurring — e.g., 0.75 for a prediction of 75% probability — and the post facto probability (i.e. 1 if it did happen, 0 if it didn't). In the case of multiple predictions (for multiple states), you then average the Brier score for each prediction. Thus the Brier score punishes you for getting a prediction wrong, but it also punishes you for hedging your bets too much, even if you got the end result right. Getting a Brier score of exactly 0 would mean you knew the results in advance, getting a score of 0.25 is equivalent to just guessing randomly (if there are only two possible outcomes).

Have a look at Wang's result tables: it looks as though he did slightly better than Silver in predicting the Presidential race, and much better in predicting the Senate results. As he rather pithily put it,

additional factors used by FiveThirtyEight – “fundamentals” – may have actively hurt the prediction. This suggests that fundamentals are helpful mainly when polls are not available.

Anyway, the point of this post, despite the deliberately provocative title, was not really to attack Nate Silver. Even if he did hedge his bets and overstate his error bars a little, given his very public platform and the consequent stakes, I think that's understandable. The Princeton Election Consortium and Votamatic really don't have the same exposure, so Wang and Linzer would have faced less general opprobrium if they had made a mistake. Also, perhaps more statistics are needed to really make a judgement — though let's not have another election for some time please!

But I think this does illustrate a subtle point about probabilistic predictions that most of the media seem to have missed, so it is worth pointing out. And I feel a little personal guilt for being too pessimistic to believe Sam Wang before last Wednesday!

How to update your beliefs in light of new evidence

This post is the second in a series on probability and statistical inference, and follows this one, in which I asked readers a question that I had once put to applicants to Oxford in physics interviews. If you have not yet answered the question in the poll, you should click on the link above and try it first! The poll will remain open for anyone who would like to try answering the question before reading the answer below.

A question of probability

In my post today, I'd like to ask readers of this blog a little question on probability that we* used a few years ago as an interview question for Oxford physics admissions.

The Oxford interview process is quite famous — or notorious, depending on your point of view — in the UK, and every year, come admissions time, newspapers run stories containing collections of "impossible" or eccentric questions that have supposedly been asked of candidates by malicious interviewers. In some newspapers the slant then given to the story is that the entire interview process is, by design or by coincidence, weighted against applicants from state schools, who are less well prepared for such shocks and are reduced to nervous wrecks.

I don't wish to enter into such a political discussion here, but in case any prospective interview candidates were to read this blog post I'd like to assure them that the interviews — at least in physics and at least in my experience — are certainly not like this. Interviewers realise that nervous applicants do not reflect their true potential, so they do not deliberately add to the tension and will in most cases attempt to create as relaxed an atmosphere as possible. They also realise that interviews, because of their highly subjective nature, are statistically not very good indicators of the true physics ability of the student. It is well-known that performance in the physics aptitude test applicants are required to take is a better predictor of their subsequent performance when at Oxford, so test results are substantially weighted up relative to interview scores when making decisions on who to admit. (You're also probably better off reading Oxford's own guide to sample interview questions than anything in the press!)

Anyway, I digress. The point of this post was the question of probability, not a discussion of admissions procedures. I thought the question was quite a nice one, but as it turned out no interviewees answered it correctly when it was first put to them, and after being given a gentle hint or two, almost all of them got to the right answer. So it was pretty useless at distinguishing between applicants and as a result not a very successful interview question. On top of that, I've also recently discovered that it is explained in great detail in a best-selling popular book on probability, so it probably won't be used again — which is why I feel it is safe to disclose it here! So here it is, slightly reworded:

You learn about the existence of some rare disease, X, which is known to affect 1 in every 10,000 people. Being a bit of a hypochondriac, you are afraid that you may have disease X, so you go to your doctor for a blood test. The doctor tells you that the test for this disease is accurate 99.9% of the time. To your horror, the test result comes back positive. What is the probability you have disease X?

I won't give the answer immediately (update: the answer is now explained here), because I'd first really like to know how readers of this blog would answer it. So I'd like to encourage everyone who reads this post to vote in the following poll -

What is the likelihood you have disease X?

Please don't be shy about voting! If you get it wrong, well, you've done no worse than some of the brightest A-level students in the UK. On the other hand, you could have the satisfaction of getting it right and displaying your knowledge to the world (the poll's completely anonymous, but still). Feel free also to comment on this post, but please don't give the answer away if you know it.

The reason that I've posted this question and the reason I'd like to know people's answers is that although it wasn't a good interview question, I think it is still a good way to get people to think about probability, and the question of how to deduce information from limited evidence. These are topics which I would like to discuss in a series of further posts over the next few weeks, so I'll use this as a starting point.

* I took part in admissions interviews between 2007 and 2010 as part of a tutoring job while finishing my doctorate. This particular question was one suggested by my (senior) colleague for use in one of those years.