So, it is common to hear scientists complain about peer review, about how it is “broken,” and there is probably something to that. Over at Backreaction, a blog by theoretical physicists at The Economist, Sabine Hossenfelder argues that the future of peer review, on that will fix its problems, is already here, in the form of what she calls “pre-print peer review.”

The idea is to separate the peer review process from the journals, and attach it to the manuscript. So, if I write a manuscript, I would send it out, for a fee, to a peer review service, which might be run by a publishing company, or by some other entity. According to Hossenfelder, once you got back the review,

This report you could then use together with submission of your paper to a journal, but you could also use it with open access databases. You could even use it in company with your grant proposals if that seems suitable.

Okay, so maybe Hossenfelder has a very different perception of what is wrong with peer review than I do. If your ultimate goal is to submit the manuscript for traditional publication, this seems problematic and, ultimately, unsustainable.

Just think for a moment about the dynamics and market pressures. First of all, if authors have control over the reviews that they purchase, one might expect that they will only attach these reviews to their papers when those reviews are positive. Furthermore, if there are multiple peer-review services, the market pressures would presumably drive them all towards more and more positive reviews. Basically, it sets up a system that will be unraveled by “review inflation.” Thinking as a journal editor or grant reviewer, I suspect that I would quickly become very skeptical of these reviews. And I certainly would not be willing to substitute their recommendations for my own judgment and the opinions of referees I selected.

You can imagine ways to address this problem. For instance, certain peer-review services could build reputations as tough reviewers, so that their “seal of approval” meant more. At this point, however, you’ve merely layered on another set of reputations and rankings that must be kept track of. While this approach is billed as a way to simplify the peer review process and make it cheaper and more efficient, I have difficulty imagining that it would not do just the opposite.

Hossenfelder argues that this new model of peer review is not just desirable, but inevitable

irrespective of what you think about this, it’s going to happen. You just have to extrapolate the present situation: There is a lot of anger among scientists about publishers who charge high subscription fees. And while I know some tenured people who simply don’t bother with journal publication any more and just upload their papers to the arXiv, most scientists need the approval stamp that a journal publication presently provides: it shows that peer review has taken place. The easiest way to break this dependence on journals is to offer peer review by other means. This will make the peer review process more to the point and more effective.

First, in what way does this have anything to do with high subscription fees? Most open access journals have pretty much the same peer-review structure that subscription journals have. There are legitimate problems with the current dominance of scientific publishing by for-profit corporations that use free labor to evaluate publicly funded science, and then turn around and charge people a lot of money to access that science. However, given the expanding number of high-quality open-access journals that use the traditional peer review system, it seems like peer review is orthogonal to this issue.

Second, yes, there are many people who feel that they need the peer-review stamp of approval. The potential benefit here is that an author could pay for peer review and then post their work on the arXiv, thereby circumventing journals altogether, and allowing more junior researchers to pursue this publishing model. It just seems to me that an author-funded system that is so easily gamed is unlikely to provide any real sense of legitimacy to anyone with this specific concern.

Third, when she says that this will make the process “more to the point and more effective,” I honestly can’t imagine what mechanism she has in mind. Given that it is published in The Economist, my suspicion is that this claim is based on some sort of invisible hand argument — that if we just free peer review from its shackles, it will become efficient and beautiful. But maybe that’s unfair on my part.

The post goes on to point to two outfits that are already working to implement this model: Peerage of Science (which is up and running) and Rubriq (which is getting started). Rubriq seems focused on the author-pay model, creating a standard review format that could travel from journal to journal. Peerage provides reviews free to authors, and it paid by journals when they use a review and then publish a paper. I’ve not seen anything that addresses the problem of review inflation.

I don’t know. Maybe there’s something I’m missing here. What do you guys think?