How to do a decent peer review for an academic journal is one that is part of the craft of being an academic. But, there is a gap in the literature and thus on guidance as to undertaking peer review. There is a large literature on the effectiveness and usefulness of the peer review process – See early work by Chubin and Hackett (1990), Burnham (1990), Peters and Ceci (1982), through to more recent work by Sugimoto and Cronin (2013), Ma et al. (2013), Ren and Gong (2012), Bornmann (2012). However, there is surprisingly little, it seems, on how to do it. Doing a good journal review is a bit of a black box.
As to what editors perceive as a good review, there is even less. Caligiuri and Thomas (2013) provide some perspective based on the Journal of International Business Studies rate the reviewer process. This suggests some useful elements, particularly the need for adequate length. The interplay between reviewers and papers and editors is well discussed in Harvey (2012), particularly the need at times for editors to adjust the actual or implied recommendation of the referee. Kohli (2011) also provides reflections on the interplay between reviewer and paper. However, the actual "how to do it" seems to be assumed. Publisher websites contain some useful advice. Elsevier in particular provide useful advice on structure (see http://www.elsevier.com/reviewers/reviewer-guidelines#youve-been-asked-to-review). Nonetheless, some tips might prove useful.
1. Be professional. It's called peer review for a reason. You, the putative reviewer, are the peer. Peers are a college and you need to be collegiate. If you don't do it for them why should they do it for you? Although frequently under rewarded, this is a core part of your job as an academic. It shows both that you are part of the academy; collegiate and willing to engage in the interplay that makes the profession work. Beyond that, reviewing is an excellent way to keep up with the literature and a superb way to sharpen your own writing.
2. Be pleasant. Don't be cruel. If the paper is truly awful, suggest a reject (see below) but don't engage in ad hominem remarks. Rejection should be a positive experience for all. The idea here is to advance knowledge, not to show how smart you are. Don't say things in a peer review that you would not say to the persons face in a presentation or in a bar after a conference.
3. Read the invite. Most journals now have in their email inviting you to review the paper a link to accept and one to reject. Don't respond to the editor with a long apology about how you would love to do it but your cat has had kittens and you have a paper yourself to do plus a class to teach and anyhow wouldn't Prof von Juntz at Miskatonic be better? Click. The. Link. The invite tells you when it is due. Use. A. Calendar. It may give you specific instructions. Follow. These. Editors are busy people and even those with editorial assistants find it hard to keep up. So, don't feel bad about rejecting an invite, be positive and use the process to suggest others. Editors keep track – if you keep rejecting invitations then they will cease, but this is not spam. Getting no invitations to review is much worse than too many.
4. Be helpful. The aim of a review is to make suggestions to the authors as to how to overcome the shortcomings you identify. It is the easiest thing in the world to poke holes in something. It is usually orders of magnitude harder to suggest how to fix them. A good review is more than a suggestion to revise or to reject or to accept. It should be meaningful. It should guide the author on what is good and what is not so good as you see it. If it's too short then it probably isn't going to do that. So be loquacious. Explain what is going on in your thinking. Suggest alternative approaches.
5. Be scientific. Your role is that of a scientific peer. It is not that of an editor in either the proofreading or decision-making sense. Don't fall back on filling a review with editorial and typographic issues. If the paper is rife with errors, tell the editor and give examples. Concentrate rather on showing the added value of your scientific knowledge and not so much on missing commas etc. If as part of your revision you think that the paper should be professionally proof edited then say so. If it is so poorly constructed as to fail in its communication role then tell me that. Remember that in the end the paper is not about style but substance until and unless the style gets in the way. In that case, then it's failing on the most basic level.
6. Be timely and swift. There is no point complaining about how slow is the paper publication process if you are part of the problem. When you agree to review a paper with a timeline given, unless there is a really good reason, you should stick to it. Believe it or not editors do track who is reviewing what when. We have to balance the natural tendency to give more reviews to those that do most with a realization that people are doing this essentially pro bono and have limited time. So the timeframe we give are designed to be timely but mildly pressurizing. Deadlines are good. Stick to them. A review is not a phd. It should not take you a month of work to do. It should take as long as it takes to read the paper once to get a sense of it, then work through the detail noting issues as you go along, then checking the issues are not your lack of understanding, and writing the review. That's not a month.
7. Be realistic. Be realistic about the work presented, changes you suggest and your role. You as a reviewer are part of the process. You don't have final say on the determination of the manuscript, I, as editor, have that. Sometimes editors override the suggestions of reviewers (hopefully with good reasons). You can and should in that case engage in a dialog with the editor as to why – ideally this is a learning opportunity for all. Sometimes this overriding is because the bar being set by the reviewer is too high for that paper. Data may not be available, a paradigm suggested not appropriate. These may be useful suggestions for another paper but each paper is and should be one main idea. The referee may be wrong. Sometimes the work to undertake the reviewers suggestions is simply too much for the author bearing in mind the journal it is being pitched at, the main idea of the paper or the timelines and conditions under which someone works. A paper is the start of a conversation on an issue or a contribution to it. It is rarely the final word. Don't expect closure in social sciences, or indeed in any.
8. Be empathetic. Yes, academia is competitive. But it doesn't have to be rude as well as brutal. Think of the best review you have gotten, in terms of guiding a paper forward. Then think of the worst. Which would you rather get on average? Then put yourself into the shoes of the author whose paper you are reviewing. Where along the scale will your review fall? What goes around comes around and therefore ensuring that your reviews are scientific, helpful and courteous is a good idea. As you review, so shall you be reviewed, on average.
9. Be open. Unless it is a review for the Journal of Incredible Specialization, specialists and generalists both have a role to play. Anyhow, specialization is for ants. Editors, especially of general interest journals, will try to get both specialized and more general reviewers. Saying "it's not my area" is rarely an excuse unless it truly is not. But, it is especially not a good one when you have recently published a very closely related paper. Saying "I'm only one of the authors" in response? That doesn't cut it. Editors try to balance reviewers. That is why we ask a number of reviews. We may want a generalist, a subject specialist, someone with experience in the methodology and someone whose work is being critiqued. If we ask you then unless it is totally out of your field (there would be no point in giving me theoretical papers to read for example but most empirical papers I should be able to add SOME useful comments) assume you have a valid and useful role to play. Again, a paper is communication. Be open to it. In your occasional browsing through journals you most probably do not single-mindedly follow an ever narrower track. These reviews in areas that are cognate but not 100% on point are useful interesting diversions from the main route.
10. Be organized. A review is, like a paper, communication. It therefore requires structure and a logical flow. It is not possible to critique a paper for logical holes, grammatical howlers, poor structure etc. if your critique is itself rife with these flaws. Draft the review as you go along, then redraft. Very few of us can submit a first draft paper with any hope that it will not be laughed off the editors desk – why would a review be any different? Most publishers, see above, provide on their websites short guides on structuring a peer review. Read some of these and follow the main principles. Ask senior colleagues for examples of reviews they have undertaken. As a suggestion, here is my ideal review (as both an editor, reviewer and author); At the start give a brief one or two sentence overview of your review. Then give feedback on paper structure, the quality of data sources and methods of investigation used, specific issues on the methods and methodologies used (yes, there is a difference), discuss the logical flow of mind, then some comment on style voice and lexical concerns and choices. Give suggestions on how to improve. That is the main point and should be emphasized strongly. Write the review as a short note – unless the system doesn't allow uploaded attachments, do it that way. It's nice as an author to see that someone has taken the time and care to rip your ideas to shreds…
Brian Lucey is Professor of Finance at Trinity College Dublin. He is editor of International Review of Financial Analysis and of Journal of Behavioral and Experimental Finance, both published by Elsevier. He is associate editor of Journal of Banking and Finance and sits on several other editorial boards. He has published over 80 papers and reviewed hundreds. He blogs at brianmlucey.wordpress.comwhere an earlier and much more scattered version of this note can be found. His twitter handle is @brianmlucey. He has probably broken every one of these ideal suggestions over the decades.
Bibliography
Bornmann, L, 2012, The hawthorne effect in journal peer review, Scientometrics 91, 857–862.
Burnham, J C, 1990, The evolution of editorial peer review, Journal of the American Medical Association.
Caligiuri, P, and D C Thomas, 2013, From the editors: how to write a high-quality review, Journal of International Business Studies 44, 547–553.
Chubin, D E, and E J Hackett, 1990, Peerless Science: Peer Review and US Science Policy (State University of New York Press,Albany).
Harvey, Campbell R., 2012, Reflections on editing the journal of finance, 2006-2012, SSRN Electronic Journal.
Kohli, A K, 2011, From the editor: reflections on the review process, Journal of Marketing 75, 1–4.
Ma, Z, Y Pan, Z Yu, J Wang, J Jia, and Y Wu, 2013, A quantitative study on the effectiveness of peer review for academic journals,Scientometrics 95, 1–13.
Peters, D P, and S J Ceci, 1982, Peer-review practices of psychological journals: the fate of published articles, submitted again,Behavioral and Brain Sciences.
Ren, Q, and X Gong, 2012, Evaluation index system for academic papers of humanities and social sciences, Scientometrics 93, 1047–1060.
Sugimoto, C R, and B Cronin, 2013, Citation gamesmanship: testing for evidence of ego biasin peer review, Scientometrics 95, 851–862.
This article was first published on SSRN (Social Science Research Network) September 26 (2013)
Some more helpful tips may be obtained from
Bourne PE, Korngreen A (2006) Ten Simple Rules for Reviewers. PLoS Comput Biol 2(9): e110