Peer review

Posted by Jean Adams

“Dear Dr Adams - Given your extensive expertise in the area, we would be grateful if you would be able to review this paper.” 

Peer review by Josh

Another day, another flattering invitation to peer-review a manuscript submitted to a journal. Sometimes they even call me Prof. Adams. Time was when I thought that it was pretty cool to be mistaken for Prof. Adams. These days I just think they were probably too lazy to check.

Peer-review is self-regulation for scientists. You do your research. You write it up and send it to a journal. The journal editor asks someone else an independent expert to check that the research and write-up are kosher. Depending on the peer-review report, your manuscript is either accepted, returned to you for corrections, or rejected out of hand.

The idea of peer-review is that mistakes will be identified, flaws in an argument spotted, and over-enthusiastic interpretation of results calmed before a manuscript makes it to print. But like self-regulation, peer-reviewed is notoriously flawed. Not only do genuine mistakes make it past peer-review, but so too do serious errors with important and ongoing implications.

Despite this, peer-review is an academic’s life blood. Unless my papers are peer-reviewed, they will not get published. So I feel an obligation to contribute to the peer-review roundabout and to review papers that are sent for my ‘expert’ assessment. Even when the abstract sent with the invitation to review looks like the paper is likely to be nonsense, I routinely accept: because I submitted a paper this week too, and because if I don’t, who will?

So what am I to do when a 5000 word paper promises to tell me about the relationship between variable x and “human behaviour”? Should I send the paper back with a brief note pointing out that no 5000 word paper could possible ever hope to explore the relationship between anything and all human behaviour? Or should I painstakingly go through the paper and identify each over-simplification, and each wild assumption, line-by-line, point-by-point?

One of the problems with peer-review is that you are rarely given any instructions. I genuinely don’t know what I am supposed to do in this situation. Mostly I am super-conscientious and end up doing the point-by-point response, but at the same time feeling like I have been duped into providing supervisory feedback without having agreed to supervise.

I recently started following @richardhorton1, the editor of the Lancet. He is full of hilariously bombastic wisdom:

“The mistake, of course, is to have thought that peer review was any more than a crude means of discovering the acceptability—not the validity—of a new finding. Editors and scientists alike insist on the pivotal importance of peer review. We portray peer review to the public as a quasi-sacred process that helps to make science our most objective truth teller. But we know that the system of peer review is biased, unjust, unaccountable, incomplete, often insulting, usually ignorant, occasionally foolish, and frequently wrong.”

So maybe I should just stop doing peer-review at all? Then what would happen?