Suzana Herculano-Houzel

NOC014_Peer review sucks

 Dear scientist trying to publish your findings,

Yes, Peer review sucks. I’m with you. I too have had my share of discontent and disappointment and disparaging remarks, and even insults. I have been gratuitously accused of not knowing the first thing about my own field; of not knowing that I should use the method that my peers prefer even if I state the reasons why it is inappropriate for my questions; of cherry-picking data to make a point. I’ve been told “she should know better” (does anybody ever write “he should know better”, I wonder?). I have every reason to suspect that a reviewer once stalled my work and then quickly came out with their own, and I am sure that I am not alone in my suspicion.

Just like democracy, which Winston Churchill famously, and cynically, called “the worst form of government—except for all the others”, Peer review sucks—but it beats the alternative of not having our research go through Peer review as part of the publishing process. So let us do our part to make it better, and I propose we start with the simplest question: Why even bother with Peer review?

The point of the Peer review process depends on your role in it: whether you are the author, the editor, or that most hated but necessary actor, the reviewer.

The point of peer review for the author

Authors have much to gain by considering that the point of Peer review is to serve as a kind of soft opening for a show that they have put in a lot of effort to prepare and that they really want to see succeed. Authors should think of the Peer review process as a test run with a carefully hand-picked, select public who is given the privilege of sitting there and reading the first version or watching the first screening of their work. And if reviewers are the privileged first audience, then the best thing that can happen to the author is to get feedback, friendly yet critical feedback, from the reviewers—because you want your test audience to tell you behind closed doors if you got anything wrong before you mess up in public!

Take my case: last year, I published a paper on dinosaur brains in this very journal (and no, of course, I did not serve as editor for my own paper). Am I a paleontologist? Absolutely not. Did I get things wrong? Oh yes! When you step outside of your scientific comfort zone, either you have a coauthor—which wasn’t my case—or you count on having really helpful reviewers, and I lucked out that I had them. I also benefitted from another kind of soft opening that more and more authors in the biomedical sciences adopt, and we at JCN heartily encourage: depositing my work in draft form on bioRxiv and talking about it on social media to invite the experts to chime in. Both formal and informal Peer reviews gave me a soft-opening opportunity to fix silly and serious mistakes and do better so that my final, published work was something I could fully stand by and be really proud of. Let us not fool ourselves: the reviewers are just the skeptics that we hear from. Once your paper is published, it is out there for everyone to sneer and point fingers at and think that they would have done a better job of it—you just don’t hear those remarks. So better embrace the Peer review process already as a somewhat controlled environment, and one in which you have someone to arbitrate when reviewers cross the line: the Editor.

The point of peer review for the editor

The editor is the person in charge of making publication decisions, which they do based on the input they receive from associate editors and reviewers. Associate editors are the handful of experts who serve their field as the direct handlers of authors, papers, and reviewers: they match submitted papers with experts who are invited to serve as the test audience for the work’s soft opening and provide feedback, and once those reports are in, associate editors are in charge of advising the editor-in-chief by communicating their recommendations about the manuscripts and the comments made by the reviewers.

And here the problems begin, because it is on the editors to deal with reviewers who overstep their role as test audience and act like they are the arts critics: The gatekeepers of what is worthy to be on the stage that is the journal publishing the work. The input that a test audience gives to the play director is completely different from the input that an arts critic gives to the public. A test audience wants to be helpful and make your work the best possible version of itself, for their own sake and that of their fellow citizens who will be in the audience when it is prime time. The arts critic, however, wants to show how much they know about the subject, how knowledgeable they are about the field and previous works, and wants to boast about whether they think that your work is good or bad or worthy, or how it compares to what they think is worthwhile.

But being the arts critic is not even for the editors to do; the public will be the harshest of critics. The role of the editors is to decide what should be on their stage, yes; and the reviewers, as the test audience for the soft opening, provide invaluable input for the editors to do their job. If and when a reviewer oversteps their role as test audience, it is on the editors to intervene.

Unfortunately, there are associate editors out there who just rubberstamp whatever the reviewers say, and editors who then rubberstamp the recommendation they receive from the associate editor. Whether it is because the publisher’s platform is so fully automated that it invites rubberstamping, or because editors are overworked and underpaid, or just don’t care, rubberstamping is unfortunate, and is unprofessional—but it is a reality of the Peer review process, and you as an author should be prepared to voice your discontent directly to the editor whenever that happens. And if all the editor does is send your appeal back to the reviewer you are disputing… well, then you want another stage for your work, don’t you? One where editors actually do their job—which includes building a healthy culture with their authors and reviewers.

The point of peer review for the reviewer

Editors need solid grounding and information to make an informed decision on whether a submitted work is appropriate for the journal’s stage. A submission only proceeds to Peer review once the editor-in-chief consults with the associate editors and together they consider that the work will withstand the quality check process that is the soft opening, so that reviewers’ precious (and unpaid!) time is not wasted. No matter how knowledgeable the editor and how specialized the journal is, any field in modern science has become too large for any single person to judge what is new and solid. The associate editor is the first person who can provide that expert feedback and pass the work on to reviewers that they will hand-pick, where appropriate. And then… the associate editors hope for the better: that the reviewer does a good job of being a test public.

And why should the reviewer do that? What is the point of Peer review for the reviewer? In the current reality, reviewers are not paid or compensated in any form other than the good feeling of doing their part, especially when the time comes when they will need a test audience for their own work, and they will want someone to agree to volunteer for the job. If that is not enough, invited reviewers could also be reminded that they are being given an opportunity to sneak a peek behind the curtains and find out firsthand about what is soon to become news in their field. Acting as a reviewer is also a great opportunity to actually read a paper in full (honestly, when was the last time you did that?) and brush up on recent work in the field that you may have missed but will most likely be cited in the Introduction or in the Discussion.

For the young, getting invited to review a paper is also a sign that you made it in your field: somebody thinks that you are an expert! Isn’t the best that you can hope for, when you do your Ph.D., exactly that there should be that one little part of the world of science that nobody knows better than you do—or at least, that nobody has thought of as much as you have? Being asked to Peer review is the recognition that you have become that person, so put that expertise that you built to use and do the job already.

Plus, reviewers get to be good citizens and do that thing of caring for the good of their own field. You know, being proud that you helped get another beautiful paper out there, advancing knowledge—the reasons we do science, to begin with. If we want a culture of good, constructive, helpful, positive-meaning Peer review, we need to build that culture—and the way to do that is by being that kind of reviewer when it is our turn: a solidary test audience, not the arts critic.

And being a good test audience for your peers opens doors. Maybe you don’t want to build a career as an editor, but it doesn’t matter: the reputation that you build as a good reviewer contributes to your standing in your field. You become that person who is met by their editor colleagues at conferences not with “Ugh, there they come, that person who writes those single-paragraph reviews that are just disparaging and useless” but with “Oh cool, there’s that person who is so knowledgeable and writes those beautiful reviews and puts all that effort into really understanding the paper and looking up the background and being helpful—I want to work with that person, I want to talk to them, ask them what they think about this, that and the other,” And that is how good, solid, thorough, deep networking really happens. So do it because you think that the golden rule is the best thing in the world, or because you think that it will be good for you—but should you choose to step up to the challenge of being a reviewer, the point is volunteering to be a test audience for your colleagues’ soft opening.

What can we do to make it better?

Like everything else, being a good reviewer is something that one learns by doing, but some key pointers definitely help—and that will be a topic for another time. For now, here is the pledge from an optimistic editor-in-chief who aims to make the publishing process something that her authors can look forward to, rather than dread as a necessary evil: I will work to provide feedback to reviewers and authors alike, and, together with the associate editors, we will offer mediation to build a Peer reviewing culture that ensures that all parties feel appreciated. There are no automated publishing decisions made in our journal. We care about what we publish, and we care about how it gets to be published. Because we, the editors and authors at The Journal of Comparative Neurology, are also the audience of what we put on our stage, and so we need to be our own reviewers, too. The point is not to keep science from the stage but to make sure that the show is the best that it can be. Join us, will you?

Suzana Herculano-Houzel


Originally published in the Journal of Comparative Neurology on February 9th, 2024; available here

More Posts

pt_BRPortuguês do Brasil