Selecting the best..from the best - Part II

This is Part II of the series. To read Part I, click here.

For over a decade, organizations such as DORA (San Francisco Declaration on Research Assessment) or, more recently, CoARA (Coalition for Advancing Research Assessment) have been promoting a change in the way scientists are evaluated. They propose to remove the focus from the most obvious indicators based on simple metrics such as publications and grants, and instead include a much wider view of the activities of researchers in the evaluation process. Initiatives such as the use of Narrative CVs, replacing the traditional category based, bullet point CVs, move in this direction and are becoming increasingly popular among (mainly) European funding agencies and research institutions. They do not guarantee a more complete analysis of the quality of a researcher, that’s fair, but at least the relevant information is available, and one can evaluate motivations, interests or non-conventional career paths, information difficult to extract from bullet point CVs.

But even with these initiatives in mind, we need to admit that the problems of subjectivity and bias combined with the need to make decisions among candidates that are very similar in quality will persist as long as success rates remain low. Peer review is not an adequate tool for the fine tuning required in the “grey zone” where decisions are made these days. And we will never have a better one, let’s face it.

So, why not be honest about it? Why not admit that while we can be reasonably confident that the candidates at the very top of the ranking are clearly better than the candidates at the very bottom, we cannot possibly distinguish between the candidates within ten positions above or below the cut off?

One obvious consequence is the potential for simplification of the selection process. How do you choose among equally deserving candidates? Lottery. Or in more technical terms, focal randomization (focal, because it applies to a preselected group of candidates, those in the grey zone). This method of selection is not a modern invention. In fact, civic lottery or sortition was already applied in ancient Athens about 2500 years ago to choose the members of the Council of Five Hundred, among preselected individuals. Later, it was used, for instance, to select public officers in Italian republics such as Venice or Florence from the 12^th century until the 1700s. And for the last decade or so, some brave funders have been experimenting with it, although to be fair no major international program has ventured into lotteries so far.

A second consequence of this is that we can stop telling candidates we know are good “you were rejected because there were better candidates”, which we know is not really true (or at least, let’s admit we don’t really know). Instead, we can tell them, “your application was considered fundable but unfortunately we do not have the resources to fund all meritorious applications”. In one case you are worse than others. On the other, you are as good as them. You could even be awarded a non-stipendiary fellowship and be considered a fellow for all intents and purposes except the money. This may have unintended positive consequences in the future: “For unto every one that hath shall be given, and he shall have abundance: but from him that hath not shall be taken away even that which he hath”, Matthew 25:29. This Bible verse, which in layman’s terms means “those who already have, will have more, and from those who don’t we will take the little they have away” or even simpler “the rich will get richer and the poor, poorer”, inspires what is usually known as the Matthew effect. Applied to scientific grants, the simple fact of having been awarded grants or fellowships in the past, increases the chances of getting new ones in the future. Being able to demonstrate that you have been competitive in the past, for instance being awarded a non-stipendiary fellowship of the kind described above, increases your chances to be competitive  in the future.

Bottomline, lottery, randomization, sortition or whatever name we use (some people are offended by the use of the word “lottery” in this context) is probably the fairest way to distribute a scarce resource among equally deserving individuals. And that is exactly the problem we have here.

And like many other problems in human history, the Greeks invented the solution 25 centuries ago.