More Thoughts on Peer Review

This morning I saw a new blog post by Peter Coles dealing with a topic I have written about before: the problems with the current system of scientific peer review. He too has strong opinions on the subject, which he has expressed several times on his blog. For the most part, I think our shared disgruntlement with the system stems from the same source — academic publishers trumpeting Peer Review (capitals intentional) as some sort of absolute certificate of quality that only they are able to provide, and provide at an exorbitant cost at that, whereas in fact not only it can probably be organised far more cheaply, the current system isn't all that hot at sorting out the wheat from the chaff anyway.

Coles takes a particular example of a paper recently published in the Astrophysical Journal in order to illustrate his point. The authors make a strong claim — the sort of strong claim that requires watertight evidence to back it up — and yet a key figure in the paper relies on interpreting a curve, which is supposed to be a cubic spline interpolation of the data, that passes through only two data points. Where are the other data points? Presumably they exist, but they aren't displayed in the figure. At the very least this is sloppy and unhelpful to anyone trying to reproduce the results, and the referee should have asked for a modification. But referees are often sloppy and distracted themselves; when there is only one referee to pass judgement on the merits of a paper the outcome of Peer Review should really be taken with a giant pinch of salt.

(I should point out that I have no idea whether this paper was reviewed by only one referee; most of the journals I am familiar with tend to run a default single-referee policy, but the Astrophysical Journal may be different. However, I have noticed several papers accepted to the Astrophysical Journal recently that in my opinion suffer from similar sloppy faults, one of which I blogged about here.)

Anyway, if the system of peer review is faulty in this manner, what should be done about it? I argued that at a minimum, journals should publish the correspondence between the referee and the editor, and the authors' responses to criticisms. This would help to highlight the cases of sloppy negative reviews, such as the instances, pointed out in a comment by Sarah, of referees simply requesting changes that are already included in the manuscript (sadly this happens quite a lot, in my experience). It would put some pressure on journal editors to improve the standard of refereeing in order to protect reputations, and make it obvious when a paper has only been reviewed by one individual. It wouldn't do a great deal to deal with sloppy positive reviews, leading to the acceptance of papers of the sort that Coles is complaining about today.

Coles' own solution is to do away with Peer Review by one or two individuals altogether, and instead have a sort of community rating system on the arXiv itself, along the lines of reddit:

I’m not saying the arXiv is perfect but, unlike traditional journals, it is, in my field anyway, indispensable. A little more investment, adding a comment facilities or a rating system along the lines of, e.g. reddit, and it would be better than anything we get academic publishers at a fraction of the cost. Reddit, in case you don’t know the site, allows readers to vote articles up or down according to their reaction to it. Restrict voting to registered users only and you have the core of a peer review system that involves en entire community rather than relying on the whim of one or two referees. Citations provide another measure in the longer term. Nowadays astronomical papers attract citations on the arXiv even before they appear in journals, but it still takes time for new research to incorporate older ideas.

I agree with the sentiment, but in the interest of fairness I should point out a couple of objections. Firstly, clearly only some papers will get reviewed this way. Many perfectly correct papers will be ignored simply because they are not on currently fashionable topics. Although writing papers that are of current interest to your fellow scientists is of some importance, it shouldn't be the sole criterion for judging merit!

Secondly, simple votes up or down are not particularly useful to a reader: if I am to trust the judgement of the people voting on the quality of a paper I'd really need to read a justification from them of why they voted in a particular way. This would amount to a mini-review of the paper and its strong (or weak) points. Such reviews would need to be provided anonymously — by the registered users of the site — in order ensure that people felt secure enough to express honest opinions. Non-anonymous review systems would I think inevitably dwindle into relative irrelevance like Cosmo Coffee has done, but completely anonymous systems would probably develop into flame wars in the classic tradition of the internet (at least with the current system the identity of the referee is known to the journal editor if no one else!).

I'd like to hear further comments from interested and informed readers. Perhaps a perfect system isn't possible, but I'm sure there are improvements which can be made to the current model. And even if that isn't the case, I think the moral you should draw from the story is that not all that is published is gold, not all those who aren't are lost. Or something more poetically phrased than that!

LftW: 25th June

Sorry about the relative lack of posting, this will be rectified soon. Here are some of the things I noticed last week.

Physics links:

• As you will no doubt already have heard from elsewhere, there is a press conference scheduled at CERN for the 4th of July, in which updates on the status of the searches for the Higgs boson at the ATLAS and CMS detectors will be provided. You've probably also heard the rumours that the results will feature higher statistical significance than the previous results from last December, and possibly even an official announcement of "discovery".
  
  I believe these rumours originated from Peter Woit's blog (see here and here), and they have caused a bit of a kerfuffle in the blogosphere. Matt Strassler rather bad-temperedly complained about "non-particle-physicist bloggers" spreading rumours, and Jon Butterworth also seemed a little upset, claiming spoilers would be "bad for science". Personally, I don't see how that argument works. Sure, I understand any scientist only wants to make an official statement in a professional capacity when they're sure they've got the analysis correct. And a large collaboration has to have rules to govern behaviour of individuals, so they're well within their rights to want to avoid leaks. But I don't see how, if the leak is made, publishing a blog post clearly marked as "rumours" is bad for science. If anything, it simply ensures that more people tune in to the press conference to check whether the rumours are actually true. (I suppose this makes it slightly more likely that the servers providing live streams from CERN will fail.)

• Still on the topic of the Higgs, Matt Strassler also got rather annoyed with a New York Times article on the topic: here and followed up here. Again, it's not something that bothered me too much, but at least in this case I do recognise the feeling of seeing some arcane aspect of your field slightly misrepresented in a summary in the popular press and getting worked up about it. (If the rumours above turn out to be true then this principled objection will be even less relevant!)

• On a completely different note, here is an amusing little game that incorporates special relativity (hat tip to Shaun). 

Other links:

• In the UK, the Education Secretary Michael Gove apparently wants to do away with GCSE qualifications for 14-16 year-old students and go back to the old O-level exams. I don't have any sensible comments to make about the advisability or otherwise of such a plan, but I did see Peter Coles had posted a link to a GCSE Science paper from 2006. It is absolutely astonishing. Question 3 is a particular classic.

• Two weeks ago Jonathan Portes caught George Osborne implicitly admitting his economic policy had been wrong for the last two years. Last week he caught the G20 admitting they'd been wrong too.

• I recently learned about the confluence of the Rio Negro and the Solimões river (which later becomes the Amazon) in Brazil, where the waters of the two rivers, which are very different colours, flow side-by-side for nearly 6 kilometres without mixing (because they are at different temperatures, and flow at different speeds). This makes for nice dramatic photographs, and I couldn't resist including one here:

Confluence of Rio Negro and the Solimoes.

LftW: 18th June

Not many physics links this week, but plenty of other interesting stuff to make up for it.

• Like most people I know, I have been following the football games at the current European Championship in Poland and Ukraine quite closely. Generally speaking the football has been very exciting, but there has also been some scrutiny of the venues for the games, and quite a bit of discussion about racism and anti-Semitism in both countries. A lot of this possibly stems from the history of the region; in The Observer, Michael Goldfarb wrote about the Ukrainian city of Lviv, formerly Polish Lwów, Russian Львов and German Lemsberg. A few days before that, The New York Review of Books had carried a review of a film about Lviv by Polish director Agnieszka Holland. Well worth reading both for some historical perspective.
• On the topic of European political geography, there was a great YouTube video which went viral a couple of weeks ago, showing a time lapse of the changing political map of Europe from AD 1000 to 2006 in about 11 minutes, with notes about the main events and battles included. If, like me, you didn't know a great deal about the history of Europe but were fascinated by little tidbits such as the Spanish and Austrian dynasties being related, it was very educational. Unfortunately it has since been taken down by the copyright holders, so I can't link it here. Instead, you can have a look at the maps on this website, showing the boundaries at intervals of every hundred years, from AD 1  to the present. A poor substitute for the video, but pretty good nevertheless.
• The British government's self-imposed austerity policy was always an unnecessary, ideologically motivated gamble, which several respectable economists opposed from the start. As it happens, events have proven their fears right, but the Conservatives have been reluctant to change course, because to do so would be openly admit their initial mistake. Nevertheless, George Osborne has been giving a big economic policy speech this week, in which Jonathan Portes catches him implicitly admitting that he was wrong and the Keynesians were right all along. There's more on this from the polite-but-scathing Simon Wren-Lewis, and the simply scathing Paul Krugman.
• Germany will be playing Greece in the quarter-finals at the Euros in a few days, so perhaps it is worth recalling a previous game between these two nations (hat tip to Elton):
• And finally, James Felce's blog post The War of The Immune Worlds at partner blog The Trenches of Discovery has made it to the semi-finals of 3 Quarks Daily's science writing competition. Worth a read!

Talent vs Hard Work or Nature vs Nurture

Listening to back episodes of the Four Thought series I highlighted last week, I found a talk given by Matthew Syed on the 9th of May this year which I thought was well worth listening to. Syed is a former international table tennis player and now a sports journalist at The Times. His thesis was that "talent" and "natural ability" are overrated and provide no guide to future success in most endeavours. Instead it is repeated and focussed practice of complex skills that actually enables people to master them, so everything is really down to hard work rather than some innate ability. According to him, even child geniuses will, on inspection, actually turn out to have simply compressed their hundreds or thousands of hours of practice into a relatively short period of time.

The reason this point is important, he argues, is that we too often take the opposite view, which inevitably leads people to not practise enough and consequently underachieve. A relevant example he cites is of students (particularly at state schools in the UK) just accepting as a fait accompli that they "just don't have the brains to go to university" or that they "just don't have the ability to do maths" or "don't have a talent for languages". Equally dangerous is the fact that we glamourise "effortless" achievement. He cites some studies which appear to show that young students praised for their natural ability or talent in some field will tend to avoid stretching themselves in future assignments in order to maintain that impression, whereas those who are praised for their effort are more willing to take on tougher challenges. (During the few years that I tutored undergraduates beginning their physics degrees at Oxford, I encountered all of these examples!)

To some extent obviously nature must play a role, as Syed admits. Sprinting and basketball are two examples he gives of sports where a genetic advantage is very important. But the more complex the activity that is to be performed, the more time must be spent to train muscles and brain to achieve success, and the greater the relevance of hard work over innate "talent". So activities such as playing the piano, hitting a cricket ball, or doing physics are all sufficiently complex to reward more dedicated practice.

While I found the talk itself interesting, I'm not sure I am completely convinced. Perhaps this is simply years of received wisdom speaking, but I find it hard to believe that enough training of muscles and neural networks can render all genetic pre-dispositions irrelevant. Especially at the elite level in any activity, where surely all participants have put in similar levels of training. I am sure that a person's natural mental state — encompassing such things as boldness in unfamiliar situations — must also be very important.

Having said that, however, almost none of us ever operate at an elite level in any sphere. Below this rarefied level, efficient training is almost certainly more important than anything else. Mental states can also — to an extent — be learned. I certainly agree that there is no justification for school pupils to ever consider themselves incapable of learning any particular skill, or for teachers to ever encourage that belief. That just sells young people short, and is a disservice to those regarded as talented as well.

I should add that I am also convinced this is one of the main reasons for the disproportionate under-representation of women in science, and in physics in particular.

The further this message is disseminated, the better.

Grexit followup

The Greek election looks like it will result in a victory for the current ruling coalition of New Democracy and Pasok, rather than for the more radical left-wing party Syriza. I get the feeling from reading various newspapers that there is a feeling of relief around Europe: the idea is that these parties will keep Greece following the austerity path and somehow avoid exit from the Euro, subsequent bank runs in Spain and then Italy, general chaos, global financial meltdown and so on and so forth.

Well, good luck of course. But informed economic opinion seems to be fairly unanimous that Greece will leave the Euro as a matter of necessity no matter whether they remain signed up to austerity or not. At most they will buying themselves a few months, but either way chaos is bound to ensue in Greece.

So this result is not really bad for Syriza at all. Euro exit, when it occurs, will now not be blamed on them, but on the austerity advocates. It is bad news for mainstream political parties, and undoubtedly we have not yet heard the last of the rise of the extremist parties in Greece.

The slow motion train crash continues in slow motion.

(Edit: After writing this I noticed that Paul Krugman says much the same thing on his blog. Probably not a complete coincidence!)

(New edit: So does Tim Duy.)

LftW: 11th June

Hello again — normal service is now resumed. Obviously having been away I have missed out on lots of interesting stuff, but I did see a few items that could be included in this week's links.

Physics links:

• The big astronomy event of last week was of course the transit of Venus. I entirely missed seeing it, mostly because I was lying in a tent listening to the rain at the time, but the wonderful thing about the modern world is that you need never really miss an event. So if like me you didn't see the real thing and don't think you'll be around in 2117, have a look at NASA's time-lapse HD video of it:

• There are also many great photographs of the event all over the internet. One of the nicest ones I saw was actually not of this transit but of the previous one, in 2004, taken by David Cortner.
• Sean Carroll: Does This Ontological Commitment Make Me Look Fat? Nice title, worthwhile discussion about objections to multiverse ideas.

Other links:

• In my opinon, BBC Radio 4 is one of the best things about the UK. Luckily, you can also listen to it from outside the UK! Last week Kamin Mohammadi gave a very interesting personal talk about the social lives of young people in Iran, as part of the Four Thought series. Her main thesis was that life under an authoritarian regime is more creative — but even if you don't agree with that, you should listen to her simply for the stories of how young Iranians use rides in shared taxis to share some physical intimacy, or use mobile phones with Bluetooth technology to chat under their parents' noses! (Incidentally, if you download the podcast version of Four Thought from here, you can also hear the questions the audience ask after the talks, which isn't included in the first link.)
• Simon Wren-Lewis: The Euro: an alternative moral tale
• Francine Prose: Getting Them Dead. On the Obama administration's tactics of assassination of terrorist suspects (and the use of the English language).
• Three Englishmen Saved from Boiling Pot by Cannibal Chief, Who Was Friend At Oxford

Interregnum

I'm going on holiday for a week, and will be avoiding the internet as much as possible. So no new blog postings until I'm back. Instead I'll leave you with a fun CERN video from 1967, pointed out to me in the comments on this post by my friend Rob. As he noted, the bit around 9 minutes in when the scientists attempt to explain on a blackboard what they are trying to do at CERN is quite funny!

An explanation of the recent dark matter controversy

About two weeks ago, I mentioned a recently published paper by Chilean authors Moni Bidin et al., claiming to have found no evidence for dark matter around our local neighbourhood in the Milky Way. Very soon afterwards, I pointed out a new pre-print by Bovy and Tremaine that rebutted this claim. Although I suppose I was fairly quickly on the scene, I wasn't particularly happy with the way I had reported this argument: just mentioning that some people disagree with each other without any explanation of the reasons is a bit closer to mere journalism than I was intending.

Since then, there have been several blog posts on this issue elsewhere — see Résonaances, Peter Coles, Matt Strassler and Sean Carroll among others — but to be honest I didn't think they actually did much more than report that people disagreed with each other either (although many of them did add some sociological comments). Which is fine as far as it goes, but if you were looking for someone to explain the science to you, you might have been disappointed. Jon Butterworth at least made a commendable effort in his Guardian blog, but he was perhaps limited by space and an inability to include any mathematics. In the end he simply said that he believed the Bovy and Tremaine result because Peter Coles believed it (and Peter Coles essentially told us to believe it because Scott Tremaine is an acknowledged world expert in galactic dynamics). Which is all very good and probably correct, but not very enlightening!¹

So eventually I've decided to fill this gap in the market myself: this post will attempt to summarise and explain both papers to a non-expert (i.e. anyone who doesn't actually work in the field of galactic dynamics). A major caveat before I start: I don't work in galactic dynamics either. I don't own, and had never read, the main reference work on this subject (by Binney and Tremaine; you see why people might trust Tremaine to be right on this one!). In preparation for writing this post I borrowed a copy and read a small amount of it — enough to feel I understood what was going on — but even so much of what I am about to write might be over-simplified, misrepresented or simply wrong. If you happen to spot any mistakes, please let me know.

Now that we've got that out of the way, let's start.