A long summer

Indeed it has been a long summer, though the good weather appears to be drawing to a close. Over the last few months, I have attended three cosmology conferences or workshops and also been on a two-week holiday in the Dolomites, where I occupied my time by doing things like this:

La Guglia Edmondo de Amicis, near the Misurina lake.

and enjoying views like this:

Cima Piccola di Lavaredo, from the Dibona route on Cima Grande.

This explains the lack of activity here in recent times.

Returning home a couple of weeks ago, I was full of ideas for several exciting blog posts, including a summary of all the hottest topics in cosmology that were discussed at the conferences I attended, and perhaps an account of my argument stimulating discussion with Uros Seljak. However, it has come to my attention that there are other physicists in other parts of the world who happen to be working on the exact same topic that my collaborators and I have been investigating for the last few months. The rule in the research world is of course "publish or perish" (though some wit has suggested that "publish and perish" is more accurate) – so most of my time now will be spent on avoiding being scooped, and the current hiatus on this blog will continue for a short period. Looking on the bright side, once normal service resumes, I hope to have some interesting science results to describe!

In the meantime, I can only direct you to other blogs for your entertainment and enlightenment. Those of you who like physics discussions and have not already read Sean Carroll's blog (a vanishingly small number perhaps?) might enjoy this post about Boltzmann brains. I personally also enjoyed this argument against philosopher Tom Nagel.

For people interested in climbing news, I can report that my friends on the Oxford Greenland Expedition that I mentioned once here have returned safely after a successful series of very impressive climbs. I found their regular reports of their activities in the expedition diary well-written and rather thrilling – not just the climbing, but also the account of the journey to Greenland by sea in the face of seemingly never-ending gales! Well worth a read, as is this.

The nature of publications

A paper in the journal of Genome Biology and Evolution has been doing the rounds on the internet recently and was shown to me by a friend. It is titled "On the immortality of television sets: “function” in the human genome according to the evolution-free gospel of ENCODE", by Graur et al. The title is blunt enough, but the abstract is extraordinarily so. Let me quote the entire thing here:

A recent slew of ENCODE Consortium publications, specifically the article signed by all Consortium members, put forward the idea that more than 80% of the human genome is functional. This claim flies in the face of current estimates according to which the fraction of the genome that is evolutionarily conserved through purifying selection is under 10%. Thus, according to the ENCODE Consortium, a biological function can be maintained indefinitely without selection, which implies that at least 80 – 10 = 70% of the genome is perfectly invulnerable to deleterious mutations, either because no mutation can ever occur in these “functional” regions, or because no mutation in these regions can ever be deleterious. This absurd conclusion was reached through various means, chiefly (1) by employing the seldom used “causal role” definition of biological function and then applying it inconsistently to different biochemical properties, (2) by committing a logical fallacy known as “affirming the consequent,” (3) by failing to appreciate the crucial difference between “junk DNA” and “garbage DNA,” (4) by using analytical methods that yield biased errors and inflate estimates of functionality, (5) by favoring statistical sensitivity over specificity, and (6) by emphasizing statistical significance rather than the magnitude of the effect. Here, we detail the many logical and methodological transgressions involved in assigning functionality to almost every nucleotide in the human genome. The ENCODE results were predicted by one of its authors to necessitate the rewriting of textbooks. We agree, many textbooks dealing with marketing, mass-media hype, and public relations may well have to be rewritten.

Ouch.

The paper that Graur et al. implicitly deride as "marketing, mass-media hype and public relations" is one of series of publications in Nature (link here for those interested) by the ENCODE consortium. I'm not going to claim any expertise in genetics, though the arguments put forward by Graur appear sensible and convincing.¹ But I do think it is interesting that the ENCODE papers were published in Nature.

Nature is of course a very prestigious journal to publish in. In some fields, the presence or lack of a Nature article on a young researcher's CV can make or break their career chances. It is very selective in accepting articles: not only must contributions meet all the usual requirements of peer-review, they should also be judged to be in "the five most significant papers" published in that discipline that year. It has a very high Impact Factor rating, probably one of the highest of all science journals. In fact it is apparently one of the very few journals that does better on citation counts than the arXiv, which accepts everything.

But among some cosmologists, Nature has a reputation for often publishing claims that are over-exaggerated, describe dramatic results that turn out to be less dramatic in subsequent experiments, or are just plain wrong.² One professor even once told me – and he was only half-joking – that he wouldn't believe a particular result because it had been published in Nature.

It is easy to see how such things can happen. The immense benefit of a high-profile Nature publication to a scientist's career leads to a pressure to find results that are dramatic enough to pass the "significance test" imposed by the journal, or to exaggerate the interpretation of results that are not quite dramatic enough. On the other hand, if a particular result does start to look interesting enough for Nature, the authors may be – perhaps unwittingly – less likely to subject it to the same level of close scrutiny they would otherwise give it. The journal then is more reliant on its referee's to provide the scrutiny to weed out the hype from the substance, but even with the most efficient refereeing system in the world given enough submitted papers full of earth-shattering results, some amount of rubbish will always slip through.

I was thinking along these lines after seeing Graur et al.'s paper, and I was reminded of a post by Sabine Hossenfelder at the Backreaction blog, which linked to this recent pre-print on the arXiv titled "Deep Impact: Unintended Consequences of Journal Rank". As Sabine discusses, the authors point to quite a few undesirable aspects of the ranking of journals according to "impact factor", and the consequent rush to try to publish in the top-ranked journals. The publication bias effect (and in some cases, the subsequent retractions that follow) appear to be influenced to a degree by the impact factor of the journal in which the study is published. Another thing that might be interesting (though probably hard to check) is the link between the likelihood of scientists holding a press conference or issuing a press release to announce a result, and the likelihood of that result being wrong. I'd guess the correlation is quite high!

Of course the only real reason that the impact factor of the journal in which your paper is published matters is that it can be used a proxy indication of the quality of your work for the benefit of people who can't be bothered, or are unable, to read the original work and judge it on merit.

The other yardstick by which researchers are often judged is the number of citations their papers receive, which at least has the (relative) merit of being based on those papers alone, rather than other people's papers. Combining impact factor and citation count is even sillier – unless they are counted in opposition, so that a paper that is highly cited despite being in a low-impact journal gets more credit, and a moderately cited one in a high-impact journal gets less!

Anyway, bear these things in mind if you ever find yourself making a reflexive judgement about the quality of a paper you haven't read based on where it was published.

¹The paper includes a quote which pretty well sums up the problem for ENCODE:

"The onion test is a simple reality check for anyone who thinks they can assign a function to every nucleotide in the human genome. Whatever your proposed functions are, ask yourself this question: Why does an onion need a genome that is about five times larger than ours?"

²Cosmologists (the theoretical ones, at any rate) actually hardly ever publish in Nature. Even observational cosmology is rarely included. So you might regard this as a bit a of a case of sour grapes. I don't think that is the case, simply because it isn't really relevant to us. Not having a Nature publication is not a career-defining gap for a cosmologist: it's just normal.

Things to Read, 8th September

I have been neglecting this series over the summer, so there are rather a lot of things to highlight today.

Science items:

• The most interesting piece of physics news over the last few months has been that the wealthy Russian investor and physics fan, Yuri Milner, decided to award a 'Fundamental Physics Prize' to 9 theorists he chose himself. The amount of the award is a staggering $3 million each, wired overnight into the physicist's bank accounts — as many people have pointed out, this makes the prize worth far far more than the Nobel Prize. Suddenly being a physicist almost appears to be a lucrative career choice. The lucky physicists were Nima Arkani-Hamed, Alan Guth, Alexei Kitaev, Maxim Kontsevich, Andrei Linde, Juan Maldacena, Nathan Seiberg, Ashoke Sen and Ed Witten. All big names in theoretical physics.
• Unsurprisingly, the award of this prize generated a lot of debate. For one thing, it is a huge amount of money. For another, the prizes were awarded basically on the whim of one man, who is not a physicist himself. But perhaps the most controversial thing was reported in the New York Times:
  

  Unlike the Nobel in physics, the Fundamental Physics Prize can be awarded to scientists whose ideas have not yet been verified by experiments, which often occurs decades later.

  This was picked up on by Peter Woit, who is of course well-known for his dislike of some of the work on string theory that the said physicists are famous for, and was highly critical of the prize, including in an article for an Italian newspaper (English transcript from his blog). I guess most physicists would share his unease with the prize, if perhaps not with string theory itself. It was quite amusing to read Matt Strassler finding himself in a position of agreement with Woit but managing to attack him nevertheless (I get the feeling he really dislikes Woit).
• Motivated by this discussion, Strassler did put up a very interesting and nuanced personal defence of string theory. Whether you have an opinion on 'The String Wars' or not, you will probably have heard strong criticisms of string theory. This piece will provide some balance.
• Among the other worthwhile physics discussions were two guest posts at Cosmic Variance: by Terry Rudolph and Douglas Finkbeiner. Both discuss recent research work (in quantum mechanics and searches for dark matter, respectively) and both include interesting little insights into the world of academic publishing.
• Outside physics, I saw a piece by Dr. Dave Hone in the Guardian science section that includes the lines:

  [...] we have dinosaurs everywhere around us. We have dinosaurs nesting in trees in our gardens, tiny dinosaurs that can hover and fly backwards and feed on nectar, aquatic dinosaurs that live in Antarctica. There are dinosaurs that we eat, we have dinosaurs that can circle the globe without landing, intelligent and puzzle-solving dinosaurs, tall and flightless dinosaurs that can sprint at over 40 miles and hour, and brightly coloured and beautiful dinosaurs almost too numerous to mention.

Other items:

• Of course the big news over the summer has been the Olympics. It's been impossible to miss the outpourings of national pride in British newspapers at Team GB's medal tally and I have heard also of despondency in countries such as Russia and Australia. But what is the benefit to a nation from this arms race for medals? This was discussed, rather provocatively perhaps given the timing, by Ian Johnson in the New York Review of Books. 
• There's been a lot of discussion in the US about the Republican presidential ticket, and in the UK about the dire state of the national economy, now in a longer slump than during the Great Depression. Paul Krugman made the obvious connection, and labelled Chancellor of the Exchequer George Osborne "Britain's Paul Ryan." That post includes a graph that should be shown to everyone who still believes austerity in the recession is the answer to anything:
  
  
  
  One could forgive Krugman for feeling rather frustrated (and perhaps a little smug), because this is pretty much exactly what he predicted back in 2010:
  

  The British government’s plan is bold, say the pundits — and so it is. But it boldly goes in exactly the wrong direction.

  See also here.
• Of course some people might remember the 20 "leading economists" who wrote an open letter to The Sunday Times supporting George Osborne's fiscal plans back before the 2010 election. Well, some time ago the New Statesman had the bright idea to go back and ask them if they still stuck by their advice: 9 of the 20 had changed their minds, 10 refused to comment, and only one still agreed with himself.
• You might wonder what a poor politician is to do: if you have only a degree in History and an ideology as substitutes for genuine competence, you naturally rely on the advice of professional macroeconomists. But when the economists disagree with each other, which ones should you listen to? Jonathan Portes had a post laying out some sensible criteria for judging who to take seriously. Unsurprisingly, people like Krugman, Simon Wren-Lewis and Martin Wolf passed the test, but none of the Sunday Times economists did. Portes notes that not all those who pass his test always agree with each other — but he still always takes them seriously.
• The New York Review of Books carries a review of two books on the current global and American economic mess by Paul Krugman (End This Depression Now!) and Joseph Stiglitz (The Price of Inequality: How Today's Divided Society Endangers Our Future). What particularly interested me here was learning just how radical Stiglitz appears to be.
• Incidentally, for those who were interested in the Niall Ferguson episode featured here, John Cassidy in the New Yorker has a longer look at the arguments, and comes to the same conclusions: Ferguson is a lightweight who is being repeatedly knocked to the canvas, and this "isn't a fair fight".
• And finally, since I'd like to end on an uplifting note, here's a video to cheer us all up.

More Thoughts on Peer Review

This morning I saw a new blog post by Peter Coles dealing with a topic I have written about before: the problems with the current system of scientific peer review. He too has strong opinions on the subject, which he has expressed several times on his blog. For the most part, I think our shared disgruntlement with the system stems from the same source — academic publishers trumpeting Peer Review (capitals intentional) as some sort of absolute certificate of quality that only they are able to provide, and provide at an exorbitant cost at that, whereas in fact not only it can probably be organised far more cheaply, the current system isn't all that hot at sorting out the wheat from the chaff anyway.

Coles takes a particular example of a paper recently published in the Astrophysical Journal in order to illustrate his point. The authors make a strong claim — the sort of strong claim that requires watertight evidence to back it up — and yet a key figure in the paper relies on interpreting a curve, which is supposed to be a cubic spline interpolation of the data, that passes through only two data points. Where are the other data points? Presumably they exist, but they aren't displayed in the figure. At the very least this is sloppy and unhelpful to anyone trying to reproduce the results, and the referee should have asked for a modification. But referees are often sloppy and distracted themselves; when there is only one referee to pass judgement on the merits of a paper the outcome of Peer Review should really be taken with a giant pinch of salt.

(I should point out that I have no idea whether this paper was reviewed by only one referee; most of the journals I am familiar with tend to run a default single-referee policy, but the Astrophysical Journal may be different. However, I have noticed several papers accepted to the Astrophysical Journal recently that in my opinion suffer from similar sloppy faults, one of which I blogged about here.)

Anyway, if the system of peer review is faulty in this manner, what should be done about it? I argued that at a minimum, journals should publish the correspondence between the referee and the editor, and the authors' responses to criticisms. This would help to highlight the cases of sloppy negative reviews, such as the instances, pointed out in a comment by Sarah, of referees simply requesting changes that are already included in the manuscript (sadly this happens quite a lot, in my experience). It would put some pressure on journal editors to improve the standard of refereeing in order to protect reputations, and make it obvious when a paper has only been reviewed by one individual. It wouldn't do a great deal to deal with sloppy positive reviews, leading to the acceptance of papers of the sort that Coles is complaining about today.

Coles' own solution is to do away with Peer Review by one or two individuals altogether, and instead have a sort of community rating system on the arXiv itself, along the lines of reddit:

I’m not saying the arXiv is perfect but, unlike traditional journals, it is, in my field anyway, indispensable. A little more investment, adding a comment facilities or a rating system along the lines of, e.g. reddit, and it would be better than anything we get academic publishers at a fraction of the cost. Reddit, in case you don’t know the site, allows readers to vote articles up or down according to their reaction to it. Restrict voting to registered users only and you have the core of a peer review system that involves en entire community rather than relying on the whim of one or two referees. Citations provide another measure in the longer term. Nowadays astronomical papers attract citations on the arXiv even before they appear in journals, but it still takes time for new research to incorporate older ideas.

I agree with the sentiment, but in the interest of fairness I should point out a couple of objections. Firstly, clearly only some papers will get reviewed this way. Many perfectly correct papers will be ignored simply because they are not on currently fashionable topics. Although writing papers that are of current interest to your fellow scientists is of some importance, it shouldn't be the sole criterion for judging merit!

Secondly, simple votes up or down are not particularly useful to a reader: if I am to trust the judgement of the people voting on the quality of a paper I'd really need to read a justification from them of why they voted in a particular way. This would amount to a mini-review of the paper and its strong (or weak) points. Such reviews would need to be provided anonymously — by the registered users of the site — in order ensure that people felt secure enough to express honest opinions. Non-anonymous review systems would I think inevitably dwindle into relative irrelevance like Cosmo Coffee has done, but completely anonymous systems would probably develop into flame wars in the classic tradition of the internet (at least with the current system the identity of the referee is known to the journal editor if no one else!).

I'd like to hear further comments from interested and informed readers. Perhaps a perfect system isn't possible, but I'm sure there are improvements which can be made to the current model. And even if that isn't the case, I think the moral you should draw from the story is that not all that is published is gold, not all those who aren't are lost. Or something more poetically phrased than that!

Can peer review be improved?

Last week I was back in Oxford briefly and visited my DPhil supervisor Subir Sarkar, who, in the course of a long conversation about various things, told me about a discussion he had apparently had with Professor Anton Zeilinger after a recent colloquium there. Zeilinger had apparently mentioned the academic peer review process and had some ideas about how the process might be made to serve science better by being more transparent. I found these ideas (as relayed to me) rather thought-provoking, and I'd like to promote a bit of discussion about them on this blog.

To paraphrase the argument: an important problem with the peer review system as it stands is that in many (or even most) cases there is only one referee who decides whether the submitted paper meets the scientific standards of the journal in question, this referee is completely anonymous, and his or her report on the paper remains entirely confidential, seen only by the journal editor and the author(s) of the paper in question.

Guaranteed anonymity is well known to alter people's behaviour on the internet, in general making people more opinionated, far ruder, and less receptive to reasoned discussion about alternative points of view. Unfortunately, scientists being only human, this is almost certainly also true to some extent in the peer review process. So removing the referee's anonymity might well improve civility in some cases, and you can see why senior academics might be in favour of it — though I notice that my feeling that even anonymous reviewers are more polite when the corresponding author on a paper is a senior figure rather than a graduate student or young postdoc seems to be shared by some others, so senior academics should have the least to complain about.

Removing referee anonymity is not a completely crazy idea: I'm told it is the norm in Scandinavian countries for the identity of the panels reviewing grant proposals to be known to applicants — though here there is some aspect of collective responsibility, within the panel there may only be a single expert on a particular field, so in practice this doesn't offer a lot of cover. If all authors and reviewers were perfectly objective, with no tendency to personal animosity or desire for retribution, and all had the same permanent job security, it might work. But they aren't, and they don't, so it can't. In particular, much reviewing is actually done by younger scientists, postdocs at the start of their careers, who may well one day be applying for a job to the author whose paper they are reviewing. The potential conflicts of interest don't bear thinking about. I'm pretty sure referees must remain anonymous.

Similarly, it is impractical to demand several referees for each submitted paper. In physics a few, very high profile, journals do use multiple referees, but usually only for short papers in a Letter format (i.e., less than about 5 pages long). Having more than one referee for each full-length paper at each journal would presumably increase the time taken to accept or reject submitted papers to unacceptable levels.

What about the third option? Why can't journals put a link next to the online version of accepted papers, that allows those interested to read the correspondence between the authors and the (still completely anonymous) referee? Knowing this correspondence was to be made public might cause referees (and, equally importantly, authors replying to the referee's criticisms!) to be far more civil. It would certainly help put an end to the unfortunate but not-entirely-unknown incidents of referees holding up publication on specious grounds until certain citations are added to the manuscript. I think it would also raise the level of debate. Journals of course would only provide such a summary of correspondence for accepted papers, but authors could perhaps, if they felt their paper was unfairly rejected, also put the correspondence online, say on the arXiv.

But as I see it, the most important argument in favour of such a model is that it increases the information available to the readers of the papers. I personally often discover some interesting preprints on the arXiv from fields somewhat beyond my expertise. When looking at the published version of the paper on the journal website, I'd like to know what changes the referee requested, and how the authors responded to the request: this would give me an idea of what the contentious issues are in that field, and help me to decide how much weight to assign to claims made in the paper. This sort of information ultimately surely helps the advance of scientific knowledge.

For instance, in the case of the recent dark matter argument I blogged about here, I'd like to know whether the referee for the first paper just didn't realise there was a problem with the authors' assumptions, knew about the issue but didn't think it was a problem, or knew that it was contentious but decided on balance the paper was interesting enough to be published anyway? If the referee simply missed it, does that tell us something about the standards of refereeing at the journal in question (a well-respected one in this case), or did the authors of the second paper have some unique insight?

At the moment I can see lots of positives but no drawbacks to this idea of providing more information about the process of peer review. Other people might have other ideas, and I'd like to hear from you in the comments — I'm happy to be convinced either way!

Links for the week: 27th May

Physics links:

• At partner blog The Trenches of Discovery, my colleague and regular co-author Shaun has put up the 4th part of a series of blog postings on "The ISW Mystery". The earlier ones are here, here and here. This is an ambitious attempt by him to describe some research we have recently done in a manner accessible to the layman. Definitely worth a little investment of time.
• Sean Carroll draws attention to some nice "back of the envelope" physics problems by Edward Purcell, and points out that quantum mechanics is needed to explain how our eyes work.
• In the UK, the STFC have decided that the gigantic Square Kilometre Array (SKA) radio telescope will be built in both South Africa and Australia and not in either country exclusively. Andy Lawrence evaluates the decision. (Incidentally, SKA is not known as SKA because the radio dishes will be placed over an area of a square kilometre on the ground -- that would actually be rather small for a radio telescope. Instead it is because the total area of the dishes themselves will add up to a square kilometre, which really is enormous.)

Other links:

• The New York Review of Books features a journalistic report drawn in comic form, by artist and journalist Joe Sacco, about his visit to Kushinagar district in Uttar Pradesh, India. The novel presentation is worth a look in itself, but the report also deals with a very serious and important issue: the continuing caste-based discrimination faced by the Dalits in rural India, more than 60 years after such practices were officially outlawed in the constitution.
• Rhys has a very justified rant about the mindless abuse of scientific terms in art; he describes a particularly bad example of this "drivel".
• Peter Coles has an equally justified rant about a ridiculous article in The Guardian attempting to justify the astronomical profits made by academic publishers. See also here for a similar angry rebuttal by Mike Taylor.