What is critical appraisal, why is it important and what can you do to develop your appraisal skills?

Julian Kurz - medical student, University of Sheffield
Adam G Hague – Specialist Registrar, General Surgery, Yorkshire and Humber Deanery
Giordano Perin - Specialist Registrar, General Surgery, Yorkshrie and Humber Deanery
Saba P Balasubramanian – Consultant Surgeon and Honorary Professor, Sheffield

In this short blog entry - and related podcast ( ) - Julian Kurz, Adam Hague, Gio Perin and Prof Saba Balasubramanian discuss why critical appraisal is important for both medical students and trainees at all levels!

What is critical appraisal?

A formal definition of ‘critical appraisal’ from Wikipedia is as follows: “The use of explicit and transparent methods to assess the data in published research, applying the rules of evidence to factors such as internal validity, adherence to reporting standards, conclusions and generalisability.” In other words, a critical appraisal is the systematic examination of the research question, methodology and results of a scientific study. It aims to assess factors such as study design, confounding variables and other sources of bias that may influence the trustworthiness, value and relevance of the study to patients and clinical practice.
In a research study, to better understand disease, determine optimum practice and improve clinical outcomes; doctors and clinical scientists make observations of patients and disease states and trial the effects of interventions on patients’ health and clinical outcomes. Critical appraisal allow us to determine the extent to which a study’s results are accurate and not significantly influenced by bias or error. This concept is also known as internal validity, and can help the reader judge if the methodology employed is reasonable, the presented results are acceptable and the conclusions are justified.
In addition to demonstrating that the study has been performed with ‘scientific rigour’, it is important to establish if the question addressed in the study is relevant, the methods are feasible and reproducible, the cohort and setting resembles everyday practice, and the results are generalisable. If this is the case, the study is deemed to be externally valid and the results could have a significant impact on your clinical practice.
Some questions you would ask yourself when critically appraising a study would be:
  • Is the research question important and relevant to my setting?
  • Is there a hypothesis and is it a valid one?
  • What was the nature of the population studied?
    • Are the inclusion and exclusion criteria justified?
    • Does the included cohort resemble that seen in clinical practice?
  • How was the outcome of the study measured?
    • Is that a reliable way to measure?
    • Was there a signifant difference in the outcome between groups?
  • Are the results generalisable?
    • What implications does this study have for your practice?
  • Is further work required to answer the research question?
    • How can this work be done?
This is by no means an exhaustive list of questions. However, for the busy clinician who may simply skim through several papers in a short period of time, the questions relating to a specific paper could be summarised as – what, so what, how and what next?
  1. What – is the paper about, what have the authors done or attempted to do, and what does it say (with particular reference to results)?
  2. So, what – is it important and is it relevant to me or my practice?
  3. How – have they done the study? How are the methods justified and validated?
  4. What next – are the implications to me and my patients? Do I change practice or is further research necessary?
Prof Ioannidis from Stanford University published a series of papers (you can google him) and made the case that most published research findings are false and that many of the conclusions in research studies may not be valid. He also demonstrated that research findings are likely to be exaggerated in many instances. This exaggeration apparently particularly occured if the research question was in a ‘hot’ topic, if the number of studies was small, if the ‘effect size’ was small, if the study tested many associations or relationships, if the study involved methods with potential for significant variation or flexibility and if the study was on a topic with commercial interests.

Why is critical appraisal important to clinicians and trainee (junior) doctors?

Understanding the basis of clinical practice - Clinicians and trainees (and to a certain extent students) need to be up to date in their field of practice and understand the basis of decision making and also the changes to decision making that occur over the years. Medicine is constantly changing; an intervention which is considered essential or ‘standard’ in today’s practice, may be shown to be completely redundant; or in some cases, harmful; in as little as 5 years. Alternatively, a treatment considered first line for ‘disease X’ may become second line when another more effective alternative takes over. Not only, do the clinicians need to implement the alternative in new patients coming to their practice, they may need to also consider changing current prescriptions for patients already established on the older treatment. This includes consideration of the risks and benefits of both treatments and consideration of the risks and benefits of the change in treatment. These considerations are based not only on the knowledge of the theory of medicine, experience and judgment but also understanding of empiric evidence from published papers.
Communicating science and research - Communicating the science and research findings to students, trainees, peers and/or patients is an essential part of most clinicians’ working life. But, this requires a good understanding of the terms and concepts used in clinical research. Typically, this occurs in the assessment and communication of measures of risk. Consider a simple example - an increase in success rate of a new intervention from 98% (with the standard intervention) to 99% could be expressed either as an absolute increase in benefit (success) of 1% or a relative decrease in failure rate of 50%! If you are confused, don’t worry! Many will be, but get involved in improving your ‘critical appraisal’ skills… Getting involved in clinical research - Many clinicians do get involved or have the opportunity of getting involved in clinical research at some stage in their careers. Designing clinical studies well and putting the results of the studies in the context of other published literature requires critical appraisal skills. A lack of understanding of methodology and a lack of confidence in dealing with numbers, charts and analytical methods is often the cause of reluctance (and failure) to engage with clinical research. A typical reason often given by clinicians and trainees for not partaking in research is “I am not good at statistics”! Most of us are not… however (and fortunately for most of us!), understanding, conducting and explaining clinical research only requires a basic understanding of methodology and research concepts (which can be gained with interest, time spent and patience) and a careful consideration of the study methods and results. This is what a lot of appraisal is about…

What about medical students?

Medical schools struggle to teach the entirety of the medical curriculum over the 5 or 6 years of the course, and rely on medical students’ ability to self-learn and decide on what may or may not be important or relevant to them. In this sense, students often rely on information presented in publications and online resources as a source of up to date knowledge. These sources may present conflicting information and on occasion, misinterpret or misreport findings from research studies. Critical appraisal skills may be invaluable in judging the quality of the published material and deciding upon whether to accept published material at face value.
For example, students researching effective drug treatments for a specific condition may read a published review of the condition by author(s) with a commercial interest. Awareness of the influence of commercial interest and critical appraisal skills may help in detecting sources of bias that may have influenced the findings presented in the review. Bias may be in the form of the methods employed in doing the review, outcomes and measures used in presenting the results and the interpretation of the results. Developing critical appraisal skills is therefore important for medical students, as it would enhance their ability to efficiently sieve through information and judge how reliable and applicable these resources may be in relation to their curriculum.
Critical analysis skills also provide a concrete foundation for students’ future role as a junior doctor, as they will be expected to recognize limitations of research and how studies may impact their practice in a clinical setting. With a greater understanding of the applicability of research to clinical practice, junior doctors will be able to explain the pros and cons of their proposed management strategies as well as better inform their patients of the potential benefits or risks of an intervention. The expectation for students and junior doctors to understand and get involved in clinical research is increasing. Research is vital to the continuous advancement in the understanding of disease states, the benefits of new technologies and interventions and the risks posed by these advances. However, due to a growing strain on NHS hospitals and healthcare, there are less opportunities for clinicians to support and supervise junior doctors in their research; and formal research training is limited. This increases the need for students and junior doctors to engage with clinical research at an early stage and develop their skills on a regular ongoing basis.

A (UK) surgical trainee’s perspective (Adam Hague)

During my time in medical school, I was involved in several journal clubs and research initiatives, which enabled me to frequently review and critically appraise research. This sparked my interest to intercalate (i.e. take time off medical school to do a research degree) late, in my 5th year(!), when I undertook a research project aimed at developing a risk scoring system for cardiac surgery. It was during this time where I completed an intensive 12 week course that provided me with a basic understanding of research methodology. Fast forward 12 months to FY1 (first year of supervised clincial practice): no journal club, no research initiatives, no involvement in appraising research. Like all skills, without regular practice, my ability to pick up a paper and meaningfully critically appraise it diminished alongside my confidence to do so. Fast forward through FY2, CT1, CT2, LAS and ST3 (early training years for a UK surgical trainee): no journal clubs, few opportunities to get involved in research initiatives and minimal involvement in appraising surgical research. Speaking to my peers, this is by no means a unique situation, but the mainstay of the experience within the region and elsewhere. This may not be the same experience everywhere - there are some journal clubs elsewhere that seem to be well led, organised and often well funded. Anecdotally, however, this is by no means common.
There are many criticisms to the abovesaid: why didn’t I set up a journal club? Why didn’t I design and carry out my own research? Why didn’t I find clinicians interested in research and stick to them like glue, with the hope that presentations and publications would somehow ‘filter down’ to me? The short answer to this critcism is that I did try, but it was neither fruitful nor easy - tens if not hundreds of unaswered emails, statistical stumbling blocks and the waxing and waning of numerous projects which did not get beyond some enthusiastic discussions in the pub or coffee shop!
Critical appraisal of research, and even the written word, is a fundamental part of any clinician’s career. Being involved in good quality critical appraisal on a regular basis is not about reinventing the wheel. A regular, well organised and led journal club on ‘high impact’ papers on ‘clinically relevant’ topics can keep attendees up to date, provide the stimulus to doing research, develop and maintain appraisal skills to improve understanding of research methods and enable good quality and confident critical appraisal.

How can you develop your critical appraisal skills?

Many medical schools in the UK struggle to fit a critical appraisal course into their already stretched course, with students usually having to re-develop this skill later on in the course. Meats et al. (2009) determined that, out of a cohort of 20 medical schools, the average lecture time spent teaching an Evidence Based Medicine curriculum was around 8 hours, and ranged from 1 to 36 hours; indicating a lack of opportunity for students to learn and develop their appraisal skills. As illustrated above, the situation is the same (if not worse) for doctors in training. One way forward for students and trainees is to use resources available (both online and from textbooks) to learn and understand the principles of methodology and appraisal. There are several online resources which both teach and develop your understanding of critical appraisals. Some of these are listed below: In addition to learning these skills, it is important to maintain these skills by putting them to use in a regular manner. This will also help you build on the knowledge you have gained and develop your appraisal and your communication skills. This has traditionally been done in journal clubs that used to be run on a regular basis in many clinical departments. Unfortunately, these seem to be on the wane for a variety of reasons. An alternative is taking part in online critical appraisal and journal club sessions.
CRAM-Surg is an Online Journal Club, which is ran by a group of students, trainee and consultant doctors interested in the application of critical appraisal and methodology skills to surgical research. This group meets regularly online for live sessions, and welcomes surgical trainees and students in the ‘Yorkshire and Humber’ deanery and beyond to attend and to also take part in presenting a study. If you are interested in presenting, please visit https://cramsurg.org/presentwithus/index.html, or if you are just interested in attending a live session, please feel free to follow our social media sites listed below to keep up to date with when the next session will be held.