Is Reviewer #2 really the harshest? A pseudo-scientific answer.

Nowadays, many journals are openly sharing the peer review reports for the articles they publish. If the report is there, it will be the first thing I read, even before looking at the actual results in the paper. It clearly shows why some of the figure panels are there, what was the message before the review/what it ended up becoming.

And also, it’s fun. 

It’s interesting to see how aggressive, condescending or nice people can be when they don’t have to share their names. Even with an omniscient editor (or two!). Plus, the experience is painless since the criticisms are not directed at you, and you are just witnessing someone else getting grilled. 

So, I decided to look for some peer review reports to read. You know, just for fun. Obviously, accessing papers and downloading them one by one would have been tedious and I would have probably given up after 10 or 15. 

But we are in 2026. And people are openly sharing all kinds of data out there, including a virtuous soul who decided one day to download ALL the peer review reports from several BMC journals and upload them to Zenodo. Why? Honestly, I did not look into it and I don’t know. But thank you, sir, for your service. 

I went for the BMC Genomics dataset because it’s fundamental basic science and I relate to that more. In this file, there were 3,836 manuscripts with peer review reports, with a total of 16,594 reviews. Yep. From this point on, I onboarded Claude to be my code-writing, graph-making, tone-analyzing assistant. What a time to be alive.

Disclaimer from the beginning: take all this with a grain of salt. I’m not pretending to do a thorough analysis of these reports and there is most definitely nothing scientific about this post. Okay, now that we got that out of the way, we can go to the entertaining part.

Most papers have 2 reviewers and go through 1-2 rounds of review

Looking at the dataset, the file names were enough to extract some basic information, like the number of reviewers and the number versions the review report had, which could give a clue on the rounds of reviews (although for some reports, some versions looked exactly the same). Overall, the file names and version tracking was not very efficient/representative in this dataset, so I would not fully trust it. But it still gives a general idea.

62% of manuscripts had 2 reviewers and 24% had 3. I don’t know what was happening with those 50 manuscripts that had 6+ reviewers, I really hope that was a file version problem. (Although I did have 5 reviewers for a manuscript in the past.)

But the point here is that most manuscripts had two reviewers. We can spend hours discussing whether that is enough or appropriate to peer review a research paper and take a decision on it. The reality though is that it is getting increasingly difficult to find reviewers for papers. And some manuscripts are sitting for months and months in the editor’s “looking for reviewers” folder with no resolution in sight. 

Remember that all the reports we are talking about here are from papers that were accepted and published. There is a gargantuan portion of desk-rejected and reviewed-then-rejected manuscripts that we have no idea about. 

According to file names, the reports usually had 3, 4 or 5 versions. Considering the back and forth between reviewers and authors, that would roughly mean 1 or 2 rounds of review in most cases. This is perfectly normal and reassuring in my opinion. 

However, let’s not overlook the considerable number of manuscripts that went through up to 5-6 rounds of review, that could mean more than a year+ of exchanging and waiting…

The answers you have been waiting for

First, looking at all reports in general (reviewers + authors comments), Claude focused on positive vs negative words of its own choosing and concluded that review reports have 3.33 times more negative words than positive words. Are we surprised? 

Yet you can see that it did not really find too many of these keywords, probably because it is not so easy to extract the tone based on specific words in this special case of peer review reports. The painful comments are usually not that direct, but can nonetheless, and all the more so, stab your soul.

Obviously, I then asked Claude to read all these reports (RIP my API credits) and tell me about the tone of the texts. Here I only took the first versions of the reports, meaning only the reviewers’ comments were there (no author comments), and I only focused on reviewers #1, 2 and 3 (since the majority of manuscripts had 2-3 reviewers, see Figure 1. No, no, no; don’t look at it. Stay right here. Let's not pretend this is scientific).

And of course, the question on everyone’s mind:

Is it really always Reviewer #2 that is the worst? Are they out to get us? 

We finally have an answer (at least for BMC Genomics). 

Yes, Reviewer #2 is slightly harsher than the others. 

Very slightly. But apparently that small increase in harshness is enough to fuel all those memes.

To be specific, Reviewer #2 seems to be a little bit more commanding, less warm and also dismissive:

At the same time though, they are somewhat more constructive. In fact, according to this graph, in the first round of review, reviewers are mostly warm and constructive.

Does this restore a little bit of faith in the system?

Maybe.

Will we ever reconcile with criticism?

I'm sure you realized this had very little to do with reviewers and the peer review reports.

We are faced with criticism constantly, manuscript after manuscript, grant after grant, we receive these heavy comments that feel entirely too personal. I think this is why there is a strange (almost guilty) comfort in analyzing criticism..especially when it is directed to someone else. It makes all of it feel a little less personal.

We are not done here. Tune in soon for another post that will go over a selection of the most unexpected (for lack of a better word) quotes from reviewers.