Last week I was back in Oxford briefly and visited my DPhil supervisor
Subir Sarkar, who, in the course of a long conversation about various things, told me about a discussion he had apparently had with
Professor Anton Zeilinger after a recent colloquium there. Zeilinger had apparently mentioned the academic peer review process and had some ideas about how the process might be made to serve science better by being more transparent. I found these ideas (as relayed to me) rather thought-provoking, and I'd like to promote a bit of discussion about them on this blog.
To paraphrase the argument: an important problem with the peer review system as it stands is that in many (or even most) cases there is only one referee who decides whether the submitted paper meets the scientific standards of the journal in question, this referee is completely anonymous, and his or her report on the paper remains entirely confidential, seen only by the journal editor and the author(s) of the paper in question.
Guaranteed anonymity is well known to alter people's behaviour on the internet, in general making people more opinionated, far ruder, and less receptive to reasoned discussion about alternative points of view. Unfortunately, scientists being only human, this is almost certainly also true to some extent in the peer review process. So removing the referee's anonymity might well improve civility in some cases, and you can see why senior academics might be in favour of it — though I notice that my feeling that even anonymous reviewers are more polite when the corresponding author on a paper is a senior figure rather than a graduate student or young postdoc seems to be shared by
some others, so senior academics should have the least to complain about.
Removing referee anonymity is not a
completely crazy idea: I'm told it is the norm in Scandinavian countries for the identity of the panels reviewing grant proposals to be known to applicants — though here there is some aspect of collective responsibility, within the panel there may only be a single expert on a particular field, so in practice this doesn't offer a lot of cover. If all authors and reviewers were perfectly objective, with no tendency to personal animosity or desire for retribution, and all had the same permanent job security, it might work. But they aren't, and they don't, so it can't. In particular, much reviewing is actually done by younger scientists, postdocs at the start of their careers, who may well one day be applying for a job to the author whose paper they are reviewing. The potential conflicts of interest don't bear thinking about. I'm pretty sure referees must remain anonymous.
Similarly, it is impractical to demand several referees for each submitted paper. In physics a few, very high profile, journals do use multiple referees, but usually only for short papers in a Letter format (i.e., less than about 5 pages long). Having more than one referee for each full-length paper at each journal would presumably increase the time taken to accept or reject submitted papers to unacceptable levels.
What about the third option? Why can't journals put a link next to the online version of accepted papers, that allows those interested to read the correspondence between the authors and the (still completely anonymous) referee? Knowing this correspondence was to be made public might cause referees (and, equally importantly, authors replying to the referee's criticisms!) to be far more civil. It would certainly help put an end to the unfortunate but not-entirely-unknown incidents of referees holding up publication on specious grounds until certain citations are added to the manuscript. I think it would also raise the level of debate. Journals of course would only provide such a summary of correspondence for accepted papers, but authors could perhaps, if they felt their paper was unfairly rejected, also put the correspondence online, say on the
arXiv.
But as I see it, the most important argument in favour of such a model is that it increases the information available to the readers of the papers. I personally often discover some interesting preprints on the arXiv from fields somewhat beyond my expertise. When looking at the published version of the paper on the journal website, I'd like to know what changes the referee requested, and how the authors responded to the request: this would give me an idea of what the contentious issues are in that field, and help me to decide how much weight to assign to claims made in the paper. This sort of information ultimately surely helps the advance of scientific knowledge.
For instance, in the case of the recent
dark matter argument I blogged about
here, I'd like to know whether the referee for the first paper just didn't realise there was a problem with the authors' assumptions, knew about the issue but didn't think it was a problem, or knew that it was contentious but decided on balance the paper was interesting enough to be published anyway? If the referee simply missed it, does that tell us something about the standards of refereeing at the journal in question (a well-respected one in this case), or did the authors of the second paper have some unique insight?
At the moment I can see lots of positives but no drawbacks to this idea of providing more information about the process of peer review. Other people might have other ideas, and I'd like to hear from you in the comments — I'm happy to be convinced either way!