The disinhibited reviewer

Just like the anonymous commenter on a reddit thread or a social media post, I think the anonymous reviewer suffers from the “disinhibition effect” that comes with knowing that they cannot be identified.

A reviewer “persona” is created and suddenly gives them the option to be especially harsh (sometimes cruel…). Would they have made the same comment face to face, at a conference, for example?

Peer reviewing an article for an academic journal is a time-consuming, difficult and thankless job. At the same time, the anonymity and lack of regulation completely strips the reviewer from any sort of liability for their comments. Usually, reviewers know who the authors are (unless it’s double-blind). This one-way transparency may make things worse: an established pioneer undermining the new researcher, an early career researcher feeling empowered (or reluctant!) to “take down” the big name, other personal/non-EDI opinions…

Is the secrecy justified?

Peer review reports are usually only visible to the authors-reviewers-editors involved. So any “mean comment” disappears after the peer review process, never to be seen by anyone.

And there are even more secretive practices in peer review, such as “confidential comments to the editor”. In most journals, there is no explanation about the purpose of this little box. So, we can regularly find reviewers asking “What should I write here?” to other internauts. Witness a heated debate here. 

Some very questionable (in my opinion) content examples for the box that were discussed:

- Conflicts of interest. Shouldn’t this be declared already before accepting to review?

- Not in the scope of the journal. It’s not the reviewer’s job to decide that.

- My honest opinion. So, what was all that text you wrote to the authors?

- Not novel. Just tell the authors!

- Being more blunt. Again, why not just tell the authors!

- Concise summary of the report. Why, though?

- I will not do any more peer reviews for your journal. Email the editor, if you want.

- I suspect misconduct. Email the editor immediately.

- I can’t decide whether to reject or ask for major revisions. Come on.

Some other uses that are fine:

- Leaving it empty.

Why is it a big deal?

I think it leaves the door open for unethical uses that will never be known by the authors. Here is a great analogy for confidential comments to the editor that explains it better:

Similar to a trial, the editor acts as the judge, entrusted with an unbiased review of an author's manuscript (the defendant). The “jury” of reviewers is tasked with giving a recommendation to accept, reject, or (frequently) recommend a revision. [...] In a court of law, it is a violation of due process for a judge to visit and exchange information with one party without the other party being present. Such one-sided conversations are called “ex parte.” They are prohibited because one party has the opportunity to influence the judge without the other party being present to rebut any accusations or even to be aware of them. Violations of this principle are taken seriously. 

Extracted from: Swanson et al, 2023.

Thankfully, some journals are starting to publish the peer review reports alongside the articles. This, in my opinion, is key and enforces reviewer accountability. 

It should be mandatory for every publication.

Reviewer #3: “None of the claims made in this study are valid.” 

This is an actual comment I received from a reviewer. Sure, you might be thinking “Maybe they are right, revise your paper!”. My aim is not to defend the paper, even though this reviewer’s opinion was an outlier out of five. My point here is that such generalization and absolutist phrasing (“none of the claims”) points to an emotional, rather than scientific criticism. A good review must be specific, and not lazily dismiss entire studies.

In choosing demolition over constructive criticism, this reviewer #3 modeled exactly what academic mentorship should never be. Their comment serves as a perfect case study in academic failure. And I don’t mean the research, but the fundamental responsibility to nurture the next generation of scientists.

It shows everything that is wrong with traditional peer review, while simultaneously proving why we so desperately need to change it.