Critical appraisal: how to evaluate research for use in clinical practice

Not all data in healthcare research are of equal quality​[1]​. To incorporate evidence-based medicine (EBM) into practice, pharmacists must be able to assess the quality and reliability of evidence​[2]​. This requires the development of critical appraisal skills.

Critical appraisal

The critical appraisal of health-related literature by healthcare professionals is a multi-step process that requires​[2]​:

1. Formulation of a question that is important for improving patient health while advancing scientific and medical knowledge;
2. Searching the relevant literature to find the best available evidence;
3. Appraising research critically to evaluate quality and reliability, as well as applicability to the formulated question;
4. Applying the evidence to practice;
5. Monitoring the interventions to ensure the outcomes are reproducible and effective. 

Assessment and evaluation of publications can be daunting. However, this article aims to assist pharmacists when critically reviewing a research paper to support clinical decision making and evidence-based practice. 

This article focuses on the theory behind critical appraisal.

Types of studies in health research 

• Cohort studies
• Case-control studies
• Cross-sectional studies
• Randomised clinical trials
• Systematic reviews
• Others

To undertake critical analysis, it is important to first understand the types of studies that are used to generate evidence, and how the data are analysed to provide standardised measurements of outcomes​[3]​. These can then be compared to evaluate whether an intervention is effective. A summary of the main research studies used in healthcare research, including their advantages and limitations, can be found in Table 1.

The most common types of studies used to report healthcare research include:

Cohort studies

These are observational studies that can either be retrospective (e.g examine historical records) or prospective. Here a group of people are selected for inclusion who do not have the outcome of interest (e.g. exploring the association between major depression and increased risk of advanced complications in type 2 diabetes)​[4,5]​. Over a period, they are observed to see if they develop the outcome of interest and, therefore, the relative risk can be determined when compared with a control group​[6]​. One of the biggest problems with cohort studies is the loss of participants (e.g. owing to personal reasons, or their condition not improving post-treatment). This can significantly affect the results and outcomes​[7]​. Most importantly, cohort studies are the best way to test a hypothesis, without experimental intervention​[8]​. 

Case-control studies

A type of observational study and typically retrospective, where patients in a group with a particular outcome of interest are compared with another group that does not have the outcome, but the same degree of exposure as the test group​[6,9]​. Case-control studies determine the relative importance of a predictor variable in relation to the presence or absence of the disease​[6,9]​. An example of a case-control study is investigating the association of low serum vitamin D levels with migraine​[10]​.

Cross-sectional studies 

These studies commonly employ interviews, questionnaires and surveys to collect data​[10]​. Although not rigorous enough to assess and measure clinical and medical interventions, they can be used to determine attitudes of a cross-section of the population that is representative of the outcome of interest​[11]​. For example, one cross-sectional study aimed to identify the main competencies and training needs of primary care pharmacists to inform a National Health Service Executive training programme​[11]​.

Randomised clinical trials (RCT)

The most rigorous and robust research methods for determining whether a cause–effect relationship exists between a new treatment or intervention and its outcome​[12]​. Although no study alone is likely to prove causality, randomisation reduces bias and the studies are often blinded, so the clinicians, patients and researchers do not know whether patients are in the control or intervention groups​[12,13]​. RCTs are considered the gold standard in clinical research studies and are positioned at the top of the evidence pyramid​[14]​ (see Figure).

The greatest advantage of RCTs is the minimisation of bias providing strong clinical evidence, which is favoured by healthcare professionals; however, there are some limitations to this type of study​[15]​ (see Table 1).

Systematic reviews

These studies are robust, thorough and comprehensive. They obtain a more accurate and evidence-based assessment of a research question​[16,17]​. By comparing a large body of data, from a wide range of sources from primary literature, the results are analysed collectively (e.g. meta-analysis) to assess for consistency and reproducibility​[16,17]​. Study inclusion is set by an explicit selection criterion and reviews are typically, although not always, quantitatively analysed for statistical significance​[17]​. Systematic reviews are useful to obtain current, updated information regarding contemporary topics in healthcare. For example, in one review on the safety and efficacy of COVID-19 vaccines, data from several RCTs were analysed and the results were compared to obtain a more justified argument for vaccine use. 

Others

Other studies used to gather evidence in healthcare research include:

• i) case studies and case series — focusing on individuals or a collection of cases that are of interest to the author, but does not involve trying to find the answer to a hypothesis;
• ii) qualitative studies — well-suited for investigating the meanings, interpretations, social and cultural norms and perceptions that impact health-related practice and behaviour;
• iii) diagnostic tests — investigate the accuracy of a diagnostic test; it is common to compare to a ‘gold standard’ and measure either the specificity or sensitivity​[17–20]​.

Steps to follow when reviewing an article

Once an article has been identified as relevant to the topic of interest, it is essential to first determine the quality of the study by assessing its appropriateness, including whether the study design was able to answer the hypothesis/research question. 

The following steps outline the main considerations when validating a study and are summarised in Table 2.

1. Determine whether the study addressed a clearly focused issue

The introduction of the article should clearly state the aims and objectives of the research being undertaken, and background information should be provided so the reader understands the reasons why this research is needed, and how the research findings will contribute to advancing clinical and scientific knowledge.

Most research studies will evaluate one of the following:

• Therapy — efficacy of a drug treatment, surgical procedure or other intervention;
• Causation — if a suspected risk factor is related to a particular disease;
• Prognosis — outcome of a disease following treatment/diagnosis;
• Diagnosis — the validity and reliability of a new diagnostic test;
• Screening — test applied to a population to detect disease.

2. Identify the study population

Particular attention must be given to the selection criteria used for RCTs. Exclusion of groups of patient populations can lead to impaired generalisability of results and over-inflation of the outcomes of the study​[29]​. Women, children, older people and people with medical conditions are often excluded from these studies, so caution must be applied when interpreting the results​[30]​.

Crucial to the selection criteria is that all study participants share common aspects other than the variable being studied so comparisons can be made​[23]​. For observational studies, such as cohort and cross-sectional, the individuals selected should be an accurate representation of a defined population​[31]​.

3. Interpret the results

Assessing the appropriateness of statistical analysis can be tricky, but for evidence-based practice it is necessary to have a basic understanding of statistics since errors have been known to occur in published manuscripts​[28]​. The ‘method’ section of the paper should be clear about the rationale for the approach and how the outcomes and results were obtained. The language used should be understandable to the journal’s readership.

There are two main uses for statistics in research. These are to provide general observations and to allow comparisons or conclusions to be made​[32,33]​. A previous article from The Pharmaceutical Journal offers a basic introduction to statistics, providing a practical overview of differential/inferential statistics and significance testing. These will not be discussed in detail in this article.

4) Assess for bias 

Bias can occur at any stage within a research study, and the ability to identify bias is an important skill in critical appraisal because it can lead to inaccurate results. Bias is the systematic (non-random) error in design, conduct or analysis of a study resulting in mistaken estimates. Different study designs require different steps to reduce bias. Bias can occur because of the way populations are sampled, or the way in which data are collected or analysed. Unlike random error, increasing the sample size will not decrease systematic bias​[31]​.

There are many types of bias, but they can be considered under three main categories:

• Selection bias is when the composition of the study subjects or participants in a research project systematically differs from the source population. A simple example would be during recruitment of participants for an influenza vaccine trial, where the participants are healthy adults. However, the sample population is not representative of a cross-section of the general population — missing out children, older people and adults with comorbidities; 
• Information bias, or ‘misclassification’, occurs when outcomes, exposures of interest (factors measured) or other data are incorrectly classified or measured. This is particularly problematic in observational studies (cross-sectional, case or cohort studies) where data are gathered using questionnaires, surveys and interviews. The method of data collection is argued as unreliable; 
• Confounding is often referred to as a ‘mixing of effects’, where the effects of the exposure under study on a given outcome are mixed in with the effects of an additional factor (or set of factors), resulting in a distortion of the true relationship. Confounding factors may mask an actual association or, more commonly, falsely demonstrate an apparent association between the treatment and outcome when no real association between them exists​[34]​. For example, alcohol intake has been identified as a cause of increased coronary heart disease​[35]​. However, there are many confounding factors that ‘blur’ the facts, such as differences in socio-economic and lifestyle characteristics, the type of drink consumed (beer, wine), and the fact that smokers are more likely to drink alcohol than non-smokers. These factors will confound the observed relationship between the amount of alcohol consumed and risk​[36]​. 

5) Determine whether the study can be applied to practice

Pharmacy professionals can determine the applicability of study results to clinical practice by:

• Comparing research results to relevant guidelines (e.g. National Institute for Health and Care Excellence);
• Identifying whether local or national clinical policies exist that are supported by EBM;
• Discussing recommendations and the applicability of research findings with colleagues and peers;
• Summarising and critically appraising the various interventions studied in relevant clinical trials and studies; 
• Evaluating the cost-effectiveness of the interventions​[37]​.

Critical appraisal skills are necessary to extract the most relevant and useful information from published literature and it is the duty of all healthcare professionals to keep up to date with current research to identify gaps in knowledge and to ensure optimal patient outcomes. It is also particularly beneficial for pharmacists, as demand for such skills increases with the rise in opportunities to deliver advanced clinical services.

Additional resources — critical appraisal tools

Several user-friendly tools are available to assist individuals with developing critical appraisal skills. Table 3 summarises a selection of useful websites that provide checklists and guidance on critical appraisal skills.

References