But Who Reviews the Reviewer?

Some time ago, at a friend’s party, a conversation came up about the peer-reviewing process. I know what you might be thinking, my dear reader. “Wow, this guy must be fun at parties”. But really, I am. I promise I was not enforcing an academic topic on a non-academic audience.

I think… Crap, now I’m not sure.

This friend of mine – it is important to mention – is not an academic. And I, from the top of my “I am a proud scientist” pedestal, was very confidently explaining how well thought the peer-reviewing process is (read this with moderate arrogance, please):

You see, our work as scientists is only officially considered new knowledge to the field after it goes through a rigorous process of reviewing by no one other than our own peers. The people with the most similar competence as us! Isn’t it such a democratic and open process?

And to this, he asked, with genuine curiosity:

And does someone review the reviewers?

The idea of the peer review process is that I - a researcher who has prepared their work - am a flawed human being, with biases and prejudices, who needs their research work to go through an independent party (or two or more) – the reviewer(s) – in order to have some extra pairs of eyes to highlight its positives and point out the negatives.

But the reviewers themselves are - and this is probably not going to shock anyone - human beings with equal biases and prejudices. Their interests and novelty conflicts might very easily come up. Of course, there’s an underlying ethical duty to refrain from one’s own biases when acting as a reviewer. But… we also know that that is many times poorly achieved.

It is not possible to get rid of these issues 100%. But we can make an effort to dilute it to their minimum. Maybe a second layer of revision would act as a good solvent (Yes, yes. I’m enjoying this dilution metaphor too much. Leave me alone.).

It's true, we can’t just keep reviewing the reviews of the reviewers of the reviewers in an endless cycle. And yes, for the sake of the timing and feasibility of the process, we have to draw the line somewhere.

I will risk sounding too bold here, but I would say we can apply to the peer review process the same perspective Winston Churchill shared about democracy: It is not perfect, but it’s the best system we have*.

Yes, it has its flaws. Its intricacies make it sometimes frustrating at the service of society. But it’s the best system we have. And, while it is like that, it doesn’t mean we cannot keep improving it and making it – although never perfect – better and better every day.

Am I talking about democracy or the peer-reviewing process? Yes, exactly.

*The actual quote is something along the lines of “Democracy is the worst form of government except for all the others”.