If there is a single academic craft that is most sorely neglected in doctoral programs, most infrequently honed over the course of one's career, and most inconsistently exhibited at the top ranks of the academy, it is the practice of reviewing an article. And yet, for so many academics around the world, the fate of their work is decided by their colleagues’ aptitude in exactly this area. At their most severe, poorly conducted reviews can lead to the suppression of pathbreaking work, or the elevation of sub-par scholarship. More commonly, they tend to lead to the overextension of the editorial process, the drafting of additional reviewers, and delays to manuscript publication. This collective wastage of hours of authors’, reviewers’, and editors’ time can severely impact scholarly productivity at the macro-level.
All this is to say that we often decline to lend serious attention to the practice of how one should review a manuscript, even though the review process is so central to academic scholarship. Reflecting on conversations with editorial colleagues at Contention and other broad-scope journals, I have drawn together some brief guidelines on how best to compose the three most basic components of any academic review: criticism, praise, and recommendations to the editor. These guidelines are written with the interdisciplinary or generalist editor's needs in mind, and are furnished with some illustrative excerpts from hypothetical reviews.
Offer Precise, Constructive Criticism
Editors of broad-scope and interdisciplinary journals depend on reviewers for their expertise and explanatory acumen regarding an area (or set of methods) with which the editor may only be somewhat familiar. We rely on our reviewers for their sensitivity to the kinds of small but important things that a nonspecialist might miss and their ability to explain how these flaws might be resolved. We are looking to identify what exactly is wrong with a manuscript and how it might be improved.
One of the least helpful forms of criticism often exhibited by reviewers often comprises a forthright rejection of some element of an article followed by a circular explication or justification of that criticism:
Review 1:
I did not find the author's empirical argument compelling. The author needs to do more to convince the reader of the veracity of their argument. Furthermore, the author's ideas do not strike me as original. I believe that I have seen the same argument made by many other scholars in the past.
Neither of these critiques meet the essential criteria that editors are looking for in critical comments: precision and potential for improvement. An ameliorated version of these comments might read as follows:
Review 2:
The author's empirical argument missed out several important details about their case, such as event A or B, which were pivotal because of C. The author should explain why they consider these events to be irrelevant, or otherwise incorporate them into the manuscript. Furthermore, the ideas in the article clearly resemble a prior argument made by Author(Date), which similarly contended that D. Despite this, the author presents their argument as original. The author needs to either more clearly distinguish their argument from D, or to resituate their argument as applying D to their case.
Another common source of poor-quality criticism arises from reviewers’ desire to have their own work—or that of close colleagues—discussed or credited in the manuscript. While this is not in itself a problem, it is imperative that reviewers clearly explain to editors what the value of including this work would be and how the work is particularly relevant to the question at hand. Stating, for example, that “the author neglects the work of Authors E, F and G, who have all made significant contributions in this area” is of little help to the editor, who is constantly balancing the cost of extending an article's word count with the benefit of including additional material. Instead, reviewers need to advocate for these kinds of additions and justify where, how, and why these potential inclusions are important. The same principle applies to asking an author to include new tables, appendixes, or large additional written sections.
Avoid Giving “Bad” Praise
One of the most frustrating things to come across an editor's desk is a positive, yet poorly explicated review. It is quite common for reviewers to write much shorter, less detailed reviews for pieces that they wish to see published than they do for those that they wish to see rejected. While these kinds of reviews offer suboptimal material for journal editors of any kind, they are particularly problematic for broad-scope and interdisciplinary editors, whose knowledge of the particular subfield that the article speaks to is much more limited than those of narrow-focus journals.
Take, for example, the following hypothetical “extremely positive” review:
Review 3:
I thought that this article was superb. A profound contribution to the literature, its empirical content was simply excellent, the methods were clearly justified, and I saw that it was very well written. It is a wonderful piece of scholarship, and I recommend it unreservedly.
A review of this sort is perhaps the least editorially helpful review a positive evaluator could have written about such a piece, and certainly the greatest disservice to its author. Let us imagine that Review 3 was paired with a poorly written but totally negative review in the style of Review 1. In a context where an editor is not highly familiar with the subject matter of the article, they are left with two options: either extend the review process by requesting new or supplementary reviews, or offer a cautionary reject of the manuscript. At Contention, we pride ourselves on recommissioning any reviews that we deem to be of unsatisfactory quality, but this is a luxury of a leaner, fresher journal that is often not available to our colleagues working on the larger generalist journals in the social sciences and the humanities, where only a minute proportion of submitted manuscripts can make it all the way to publication. The ultimate consequence of writing unjustified praise in the style of Review 3 is to harm the article's prospects for publication. This is even more likely when other reviewers offer high-quality, well-justified criticism of the type seen in Review 2.
So, how might Review 3 have been better written? By applying the same principles that we apply to critique to praise. Explain exactly what is good about the article and how it might be best amplified:
Review 4:
I thought that this article was superb for a number of reasons. First, it offered a profound contribution to the literature. The problem of D is something that those of us in the field have grappled with for a long time and to which we have never found a satisfactory conclusion. This article does exactly that, moving beyond the existing literature and producing new, clearly durable findings with clearly explained theoretical consequences.
Second, the article's empirical content was simply excellent. I was particularly impressed by the use of novel sources of data and the high-resolution explanation of how the complex theoretical dynamics the author discussed manifested in real-world situations. To make this even better, the author could offer a footnote emphasizing the novelty of their sources and offer some commentary in the conclusion on how other scholars might use them.
Furthermore, the methods were clearly justified, and they utilized techniques from the cutting edge of the discipline such as those by authors E and F. Recently, however, some scholars have been critical of earlier versions of these methods, and the author might benefit from explaining how and why their new approach does not suffer from these flaws.
Finally, I must remark that the article was very well written, with no unclear sentences or hard-to-follow sections. I could only wish that this was replicated in the Abstract, which was comparatively hard to read. Likewise, I did not feel that the piece's title reflected the ambition and impact of its content. It would be a good idea to mention the name of the author's new theoretical model in the title.
All in all, this is a wonderful piece of scholarship, and I recommend it unreservedly.
A positive review written in the style of Review 4 would be highly useful for an editor, telling them how to get the most out of the author and make their work even better. Furthermore, remarking on the article's positive and distinctive qualities in a clear, crisp fashion will help the editor contextualize any negative criticism that other reviewers might have.
Recommendations to the Editor
In reality, few reviews are actually wholly positive or negative. Academic articles are complex efforts, and usually draw both criticism and praise from their readers. Navigating a reviewer's sentiments (generally communicated in their comments to the author) can be difficult for an editor, particularly when they are not highly familiar with the central topic of the article. Despite this, it is surprisingly rare that reviewers will write more than one or two sentences in their comments to the editor: usually the recommendation for an editorial decision or an alternative outlet.
It is a great shame that reviewers so often neglect this element of the review: confidential comments to the editor are an excellent place to be perfectly frank about the contents of a review. They offer an avenue for reviewers to disclose the limits of their knowledge, estimate how much work it might take the author to respond to any proposed revisions, or draw attention to any more substantial suspicions about the article's contents. Some of the most helpful editorial comments I have come across have used the comments to the editor as an opportunity to contextualize their comments to the author. An illustrative example of useful comments can be found in the below example:
Review 5, Confidential Comments to Editor:
I thought this was an excellent article, but I am not an expert in the methods, which seem quite ontologically complex. While I would suggest accepting it with minor revisions, I would also recommend consulting a specialist in this area before going to publication.
I stated that the author could have spent more time reviewing classic works in the field, and I do think that this would strengthen the article. Nonetheless, I understand that this would take the article over the word limit, so feel free to ignore this suggestion. Finally, you might want to ask the author a bit about their interviewees. They claimed that the interviews are anonymized, but I googled one of their pseudonyms and found the name of an activist being interviewed in the news who had a remarkably similar experience.
If you end up not taking this piece, I think that the author might want to consult a more specialized journal such as H or J.
These kinds of comments are of particular assistance to editors, who work with a panel of reviews that are almost always dissenting (and sometimes even contradictory) in their evaluations of the strengths and weaknesses of articles. Letting an editor know the parts of a review's assessment in which the reviewer is most confident and those where they are more broadly passing comment helps generalist and interdisciplinary editors weight reviewer evaluations and resolve inter-reviewer disagreements. Making suggestions for additional reviewers or alternative outlets helps editors give higher-quality feedback to authors and facilitate a smoother review process.
All in all, there is much more to the review process than discussed herein, but I hope that this brief intervention has offered something in the way of guidance to those who are more used to narrow-focused journals, or otherwise not in the practice of regularly reviewing articles for interdisciplinary, broad-focus and generalist outlets. Ultimately, a good review for journals like Contention is one which not only helps the author—who generally knows more about the article's topic than the reviewer—improve their work, but also helps the editor—who generally knows less about the article's topic than the reviewer— carefully evaluate the article's merits, demerits, and scholarly potential.