Authors: Alexander J Fowler (1,2), Kiron Koshy (1,3), Buket Gundogan (1,4), Riaz A Agha (1,5)
Contact person: Kiron Koshy
- Academic Surgical Collaborative, UK
- Department of Emergency Medicine, Guys and St Thomas NHS Foundation Trust, London, UK
- Brighton and Sussex University Hospitals, Brighton, UK
- UCL Medical School, University College London, London, UK
- Balliol College, University of Oxford, Oxford, UK
Peer review is a key part of the scientific process, whereby a body of experts review a manuscript and provide feedback. Here, we discuss the steps typically undertaken in completing a peer review and key questions to be asked of the manuscript while undertaking a review.
To learn more about why you should peer review, check out this article (1).
To read a more extensive guide on how to peer review here’s the published article of this guide (2).
The first thing to consider when reviewing a paper is the context, if you’ve been invited, you are likely an expert in the field, but it’s still worth double checking recent papers or reviews to ensure you’re completely up to speed on the topic.
At this point, it is also worth pulling up any reporting guidelines that are relevant to the paper to have alongside you as you work through the paper, as these provide very useful pointers as to what should be included and how the information should be presented. For example the PRISMA guidelines for systematic reviews and the SCARE guidelines for surgical case reports (Table 1).
Table 1: PICOS Framework
The next thing to do is flick through the paper, this allows you to get a rough overview of what you’re dealing with. For example, has someone submitted their whole 50,000 word PhD thesis with no apparent headings, or is it a well laid out manuscript, clearly headed and well formatted. While this isn’t a key part of peer reviewing, it certainly enables you to quickly appreciate the calibre of the work.
There are key questions that you must aim to answer in your review:
- What is the research question (or hypothesis) the authors are trying to address? Is it clearly defined?
- Is the research question worth answering?
- Are the methods appropriate to answer the research question? Are they sufficiently robust?
- Have the authors provided answers to the research question(s)?
- Are their conclusions justified by the data and the methods?
- What does this study add to the scientific literature?
Now let’s cover each section of a scientific paper.
The introduction is where authors describe the background to the topic and make a persuasive argument for what is currently unknown or uncertain
The key questions the reviewer should ask are:
- Is the central objective of the present work clear?
- Does this work follow clearly from previous work?
- What is the importance of this new objective?
- Is this study design appropriate for the question posed? Is it methodologically robust enough?
- Could this design be reproduced from the details provided in the methods?
- Are there any potential grey areas that could become confounders or bias?
The methods are a critical section of the paper. The following components should be addressed:
- Registered protocol – is there a publicly registered and published ‘methods’ for the study which describes the research questions and the methods?
- Ethical Approval – Ethical approval is required prospectively for all research studies involving human participants.
- Study Design – What is the design of the study that the authors have selected to answer the above question? Is it appropriate? The PICOS framework is very useful for author and peer review to evaluate (table 2).
- Participant selection -This should include explicit inclusion and exclusion criteria that were applied in the cohort selection and the dates of the study.
- Adequate description of technical procedures – Any technical procedures used in the study (e.g. how a drug was made up, how a certain surgical procedure was performed) should be referenced if standard, or described in a way such that a peer could repeat the activity.
- Clear definition of standardised outcomes – Outcomes are the ways that Outcomes should be defined properly with a standard definition. So this is especially important with multi-site studies, where the local level for a particular marker of disease may be different from other centres.
- Data collection – The method of data collection should be described, including if this was done prospectively or retrospectively and if standardised forms were used to collect the data.
- Follow-up – The type and length of follow-up should be reported as this can affect the data collection and introduce specific biases.
- Accounting for non-follow up – This is particularly important in large cohort and randomised studies and a plan should be in place for dealing with loss to follow up.
- Standard definitions or scales – Standard definitions should be provided for all comorbid diseases and for any other important covariates.
- Statistical methods – For all outcomes of interest or for all questions, there should be a clearly defined statistical method.
Table 2: PICOS Framework (3)
Results present the findings of the experiment or study.Key questions:
- Are full data and results provided, is anything missing?
- Is the layout clear and concise, with standard reporting of statistical measures?
- Are there any results with missing methods, have they cherry picked a result with a good p value?
There are a number of ways that discussions are presented, but here are some key sub-sections in order:
- A summary of the research findings
- Discussion of the research findings in the context of the literature (ideally including both index papers and recent literature)
- Implications of the work for clinical practice and/or research
- Future research questions generated or now needing to be answered
- Strengths, weaknesses and limitations of the work
These are the same appraisals that you will make about the paper.
The conclusion should be the takeaway message(s) of the paper. It shouldn’t include numbers, unless necessary to summarise a point. It should end with implications and direction of the work for the wider clinical and academic community. They key thing to note is whether the conclusions can be justified by the data and the methods.
Other important points to note
- Funding – who paid for the study? Who provided equipment?
- Authorship – who did what in the study? Criteria for authorship have been outlined by ICMJE.20 Some journals also like to identify a guarantor. The guarantor accepts full responsibility for the work and/or the conduct of the study, had access to the data, and controlled the decision to publish.
- References – are these up to date? Are the citations appropriate to the journal style?
- Conflicts of interest – what are the conflicts of interests of the respective authors?
- Images – is appropriate consent provided for all images if used? Are they of sufficient quality? Print journals will often require images captured at higher resolutions.
- Reporting guidelines – does the manuscript fulfil relevant reporting standards (Table1).
A checklist for reviewers is provided in Table 3.
Table 3: Peer Reviewer Checklist
Decision/Providing feedback – Structuring your peer review report
Feedback typically consists of an opening few sentences that briefly outline the work and an opinion on whether the paper is of significant interest/quality and should be published (with or without revisions). This is followed by a point by point critique. This is often easiest if laid out in bullet points per each section, rather than as a single block of text. For example, revisions categorised into introduction, methods, results etc.
These can also be divided into major and minor points. The ‘key points’ in the overview section at the start of this article are a useful framework when structuring a peer-review.
Journals differ on the overall decision they can provide, but these may include:
- Accept – if you feel the paper is ready for publication in its current form.
- Accept with minor revisions – if some small, minor changes are required e.g. spelling/grammar/reference issues. The decision implies that if these changes are made, the paper would be accepted.
- Minor revision – a small or modest revision is required to improve the paper e.g. small additional details needed to any of the major sections of the paper, perhaps certain discussion points not elaborated fully or a recent, relevant paper is not discussed and referenced.
- Major revision – major ‘surgery’ of the paper is required e.g. significant gaps in the methods or discussion, poorly presented results. This often requires more data to be collected for the study, or additional outcomes to be assessed.
- Reject and invite resubmission – the topic or research question is interesting, but the wrong methods or insufficiently robust methods are used and hence the data is not reliable. Authors are being asked to do it again differently. This decision may also be used where a paper requires enormous amounts of changes, or where an author repeatedly fails to respond to calls for revision.
- Reject – The paper is currently unacceptable (please ensure you submit comments alongside a reject decision, as these can be very useful for an author to improve their work prior to their next submission).
Typically, this decision is based on a combination of your point by point feedback, the overall clarity of the paper and the importance of the question being answered. It should be noted that it is not the reviewer’s responsibility to do a proof read and point out all the grammatical, spelling and sentence structure issues if there are many. A reviewer is within their right to state that a paper is simply unintelligible if the English language is so poor that the paper cannot be understood. Authors can then submit a revised version or the editor can reject it on that basis. There are numerous English language services that authors can be pointed to that will perform such a task for a fee. Large international institutions may even such people in-house.
Summary of guide
Peer reviewing is central to the scholarly publication process. Peer-reviewers, along with editors are guardians of the scholarly record and have an important role in the advancement of science. We have outlined a detailed a step-wise guide in how to effectively peer review a paper. We have outlined key points that should be addressed in any peer review, which should provide a manual, or how to guide for any peer review.
1. Koshy K, Fowler AJ, Gundogan B, Agha RA. Peer review in scholarly publishing part A: why do it? Int J Surg Oncol. 2018 Feb;3(2):e56.
2. Fowler AJ, Koshy K, Gundogan B, Agha RA. Peer review in scholarly publishing part B: how to do it? Int J Surg Oncol. 2018 Feb;3(2):e55.
3. Methley AM, Campbell S, Chew-Graham C, McNally R, Cheraghi-Sohi S. PICO, PICOS and SPIDER: a comparison study of specificity and sensitivity in three search tools for qualitative systematic reviews. BMC Health Serv Res. 2014 Nov 21;14(1):579.