Peer review is central to the scientific process and involves the review of scientific output by a body of peers. Experts review a manuscript and provide feedback on the work, offering a recommendation of whether the work is worth publishing. Publishing enters an article into the greater body of scientific output and is a step towards improving outcomes and scientific understanding.
The first thing to consider when reviewing a paper is the context – if you’ve been invited you’re likely an expert in the field, but it’s still worth double checking recent papers or reviews that ensure you’re totally up to speed on the topic. At this point, it is worth pulling up any reporting guidelines that are relevant to the paper to have alongside you as you go over the paper – these provide very useful pointers as to what should be included. Similarly, if this is a research paper including human subjects you should obtain the protocol for the study, if available.
- Flick through – format/issues
The next thing to do is flick through the paper, this allows you to get a rough oversight of what you’re dealing with (for example, has someone submitted their whole 50,000 word PhD thesis with no apparent headings, or is it a well laid out manuscript, clearly headed and well formatted). While this isn’t a key part of peer reviewing, it certainly enables you to quickly appreciate the calibre of the work.
The introduction is where authors describe the background to the topic, and make a persuasive argument for what is currently unknown or uncertain. This should be followed by a single line outlining the central aim or hypothesis of the work. The introduction should be concise, but provide a clear explanation that is appropriate for the level of readership of the journal. The key questions are:
- Is the central objective of the work clear?
- Does this work follow clearly from previous work?
- What is the importance of this new objective?
The methods are a critical section of the paper; this is where peer review becomes very important. Assuming an important; or valid question is being posed by the authors at the end of the introduction, the methods are how the authors have decided to go about answering that question. A few key components to be addressed:
1. Study design
What is the design of the study that they have selected to answer the above question? Is it appropriate? Are there any particular reason they outline for deciding to follow this design?
2. Participant selectionIdeally, this should include explicit inclusion and exclusion criteria that were applied in the cohort selection and the dates of the study. If this is a randomised controlled trial, then the method of randomisation should be stated, and power calculations should be provided. Similarly, the location and population the cohort is being drawn from should be described.
Ethical approval and registration is required for all studies involving human research participants. The reference number and ethical review board providing approval should be clearly stated, with dates of approval also provided. Protocol registration or how the study was planned should also be stated here, if there is an available copy of the protocol, ensure you have a copy to hand to identify if the methods outlined there are the same and see if there are any discrepancies between the two.
4. Data Collection
The method of data collection should be described, including if this was done prospectively or retrospectively and if standardised forms were used to collect the data.
5. Statistical Methods
For all outcomes of interest or for all questions, there should be a clearly defined statistical method. This may range from using simple comparisons of means (Student’s T test etc) up to multinomial logistic regression, where statistical methods must include how variables were selected for. Here, the package used should also be stated.
6. Adequate description of technical procedures
Any technical procedures used (e.g. how a drug was made up, how a certain surgical procedure was performed) should be referenced if standard, or described in a way such that an expert could re-create the activity. If there are any differences in how two groups were treated, this should be made explicitly clear.
7. Accounting for non-follow up
This is particularly important in large cohort and randomised studies – a plan should be in place for dealing with loss to follow up for their patient group. This may include how the data that was already collected would be dealt with statistically, or what level of data was appropriate for inclusion of the patient within the cohort. Figures for loss of follow up should be presented in results.
8. Standard definitions or scales
Standard definitions should be provided, for example in hospital mortality may be standardised to “any mortality prior to hospital discharge” where as early mortality may be defined as “death within thirty days of an operative procedure”. These would both include similar patients, but the overall groups would be different. Similarly, when talking about the definition of disease or morbidity, standardised, and ideally recognised, definitions should be used and referenced.
9. Clear definition of outcomes and standardisation of these
Outcomes should be defined apriori and should have a standard definition, this is especially important with multi-site studies, where the local level for a particular marker of disease may be different from other centres, so a centralised system or definition is required. Complications should use standard scales, such as Clavien-Dindo for post-operative complications or postoperative morbidity surveys. Standardisation of outcomes allows clearer comparisons to be drawn, increased accuracy of data and future inclusion in meta-analytic techniques.
- Is this study design appropriate for the question posed?
- Could this design be reproduced from the details provided in the methods?
- Are there any potential grey areas that could become confounders or bias?
Results present the findings of the experiment or study. How this is presented will depend upon the study design, but it should broadly include: Demographic information to highlight the comparability of any groups, results for each statistical method and for each outcome. Are the data appropriately presented, using tables, charts and images where necessary (pie charts are rarely ever the appropriate presentation…). It is worth going through the results alongside the methods, specifically seeing if there any results with omitted methods or any methods where the results are not available? Are statistical measures of uncertainty reported?
- Is full data and results provided, is anything missing?
- Is the layout clear and concise, with standard reporting of statistical measures?
- Are there any results with missing methods, have they cherry picked a result with a good p value?
There are a number of ways that discussions are presented. Ideally, this opens with the key finding of the paper, ideally this should directly answer the question posed at the end of the introduction. This should then be followed by appropriate contextual discussion – this is where your previous reading comes in handy, have the authors missed a key paper that is crucial to their conclusions? If there is a surprising finding, then efforts should be made to account for this. Here, make sure that the authors are making statements that their results can back up, if you read it and think there has been a quantum leap from data to discussion, say so!
The implications of the work for patients, clinicians and researchers should then be outlined, will this paper change how we practice? Or does it point to a small step towards improved outcomes, does it highlight an area that is hugely under research and requires significant input? Often a paper will bring up a mixture of such implications.
Strengths and limitations must be addressed – any issues you identified while reviewing the methods section should be included here. Limitations of the study are a key area of reflection for the authors, and a paper missing this section is likely to have some serious problems.
Future research areas is the final portion of the discussion, and this should be where hypotheses may be generated, or where a future direction for the work to move is described.
The conclusion should be the takeaway message of the paper. It shouldn’t include numbers unless absolutely necessary and should be the key points of the paper, ending with implications and direction of the work.
- Are the results adequately discussed?
- Are there any quantum leaps?
- Are limitations and strengths discussed?
- What is the future direction and implications of this work?
- Other points to note
Funding – who paid for the study? What was their role?
Authorship – who did what in the study?
References – are these up to date?
Conflicts of interest – what are the conflicts of interests of the respective authors?
Images – is appropriate consent provided for all images if used?
- Decision/Providing feedback
Feedback typically consists of an opening few sentences that briefly outline the work, followed by a point by point critique. This is often easier for everyone if laid out in bullet points per each heading, rather than as a single block of text. Many reviewers like to go through section by section as above. Journals differ on the recommendation you can provide, but these are typically:
Accept – if you feel the paper is ready for publishing
Accept with minor revisions – some small, minor changes are required, but the vast majority of the paper is acceptable
Minor revision – a revision is required to improve the paper
Major revision – an overhaul of the paper is required
Reject – The paper is currently unacceptable (please ensure you submit comments alongside a reject decision, as these can be very useful for an author to improve their work prior to their next submission).
PS. It is not your job to correct all grammar and spelling mistakes. If you spot a typo, by all means report it, if it is covered in errors, then an opening comment regarding the poor quality of the language/English is often all that is required.
The Annals of Medicine and Surgery is an open access journal of international scope that focuses specifically on the development of juniors and students. There is the opportunity to become involved in peer reviewing as a junior peer reviewer, giving you excellent exposure to the other side of publishing. Peer reviewers who provide excellent reviews for the journal are considered for Editorial Board positions.
To sign up, please email: [email protected]