From paper to practice: a pharmacy professional’s guide to critical appraisal tools

Introduction

Critical appraisal is an essential skill for pharmacy professionals, allowing them to evaluate the quality, relevance and applicability of research findings to ensure clinical decisions are evidence-based^​1​. In today’s fast-paced academic and scientific environment, the ability to critically evaluate research is crucial as it ensures that clinical decisions are guided by the best available evidence. Critical appraisal goes beyond merely reading a paper; it involves a detailed analysis of the research methodology, data analysis and the conclusions drawn by the authors.

By developing these skills, pharmacists can effectively engage with the literature, identify meaningful findings and contribute to informed decision-making processes.

This article is a step-by-step guide of how to select and make use of suitable appraisal tools when systematically assessing research papers. 

Critical appraisal tools 

A wide range of different appraisal tools have been developed to assist readers when critically evaluating a research paper. Tools can help to identify sources of bias or errors and provide a structured means of assessing the reliability, importance and relevance of the results. Appraisal tools vary considerably and choosing between them will depend on the study type and evaluation criteria needed. 

We highlight why different appraisal tools are suited to various research methodologies. For instance, the ‘AXIS’ tool is particularly effective for evaluating cross-sectional studies, addressing aspects such as study design, sampling techniques and statistical methods.

In contrast, the ‘PRISMA’ checklist is designed for systematic reviews and meta-analyses, ensuring that these reviews are conducted and reported with transparency and rigour. PRISMA helps researchers not only structure their review process, but also write up their research by providing a comprehensive framework for reporting all essential elements of the review.

Understanding the appropriate application of these tools enables pharmacy professionals to critically appraise literature more effectively, leading to more informed and evidence-based clinical decisions^​2,3​.

Two commonly used examples are:

• Joanna Briggs Institute (JBI) tool^​4​;
• Critical Appraisal Skills Programme (CASP) tool^​5​.

JBI is an international research organisation based at the University of Adelaide, south Australia, which develops and delivers evidence-based information, software, education and training^​4​. 

CASP is a widely recognised training provider that specialises in offering critical appraisal tools and workshops tailored to healthcare professionals. It provides structured guidance for assessing the trustworthiness, relevance and results of published research, aiming to help healthcare professionals improve their ability to interpret evidence-based studies. The CASP tool is versatile and suitable for various study designs, offering a structured approach to assess validity, results and applicability^​5​. 

Other tools have been developed for more specific purposes, including: 

• Evaluation Tool for Qualitative Studies (ETQS)^​6​;
• Appraisal Tool for Cross-Sectional Studies (AXIS)^​7​;
• Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA)^​8,9​.

The ETQS focuses on qualitative research, emphasising credibility, transferability, dependability and confirmability, making it ideal for in-depth qualitative data analysis^​6​.

The AXIS tool was developed to specifically evaluate cross-sectional studies, providing a structured framework to assess study design, sample selection and data analysis^​7​. It allows healthcare professionals to critically appraise the methodological quality of observational research, ensuring that key elements, such as bias and validity, are thoroughly considered.

The PRISMA checklist is an internationally recognised tool designed to improve the transparency and completeness of reporting in systematic reviews^​8,9​. By guiding researchers through rigorous standards, PRISMA ensures that evidence is synthesised in a reliable and replicable manner, which is essential for informing healthcare decisions based on comprehensive literature reviews.

Selecting between tools will depend on many factors and they all have relative strengths and weaknesses. For instance, many reviewers use the CASP tool for critical appraisal; however, it is less sensitive to validity aspects compared with the ETQS and JBI tools^​2,3​. The ETQS offers detailed instructions for interpreting criteria, while the JBI tool focuses on congruity and provides the most coherent framework for appraisal^​2,3​.

Critically appraising different types of studies

To understand how different appraisal tools can be selected and used, it is helpful to consider a cross section of research methodologies and look at how these can support your critique of evidence, highlighting some of the key questions that reviewers should ask as they read the paper. Links to the tools mentioned are included and it is recommended that you access and consult these alongside the examples provided.

Randomised controlled trials

Randomised controlled trials (RCTs) are considered the gold standard for evaluating the effectiveness of interventions owing to their ability to minimise bias^​10,11​. 

Critical appraisal tools that could be used to critique a RCT include:

• CASP’s RCT checklist^​5​; 
• Revised JBI critical appraisal tool for the assessment of risk of bias for randomised controlled trials^​4​.

Key questions

The CASP tool consists of 11 questions^​5​. The first 3 questions (Section A) focus on the validity of the basic study design and can be answered quickly (see Figure 1). If, considering your responses to Section A, you think the study design is valid, continue to Section B to assess whether the study was methodologically sound. If so, continue with the appraisal by answering the questions in Sections C and D.

Cohort studies

Cohort studies follow a group of people over time to see how certain exposures affect outcomes^​12​. They are useful for studying the causes of disease and the long-term effects of interventions.

Critical appraisal tools: 

• CASP cohort study checklist^​5​;
• JBI critical appraisal checklist for cohort studies (see Table 1)^​4​.

Worked example

In 2017, Jeganathan et al. conducted a prospective study in patients with inflammatory bowel diseases (IBDs) to investigate medication adherence rates^​13​. They compared the adherence rate in paediatric-to-adult transitioned participants with young adults diagnosed in adult services. The transition cohort (13.2%) did not show significantly lower adherence compared with the young adult group (24.4%) (P = 0.203)^​13​. Of 17 patients reporting non-adherence, 13 received pharmacist counselling, which significantly decreased non-adherence rates: by 60% at a mean follow-up of 240 days (SD 85) (P = 0.004)^​13​.

The quality of the study was assessed using the JBI critical appraisal checklist for cohort studies (see Table 1)^​4​.

Case-control studies

Case–control studies compare patients with a disease (cases) to those without it (controls) to identify factors that may contribute to the disease^​14​. 

Critical appraisal tools:

• CASP case-control study checklist^​5​;
• JBI checklist for case-control studies^​4​.

The CASP Case–Control Study Checklist consists of 11 questions^​5​. The first three questions are screening questions and can be answered quickly. If the answer to all three is ‘yes’, it is worth proceeding with the remaining questions.

Key questions

1. Are the results of the study valid? (Section A)
2. What are the results? (Section B)
3. Will the results help locally? (Section C)

Cross-sectional studies

Cross-sectional studies are observational studies that look at data from a specific group of people at one point in time^​15​. These studies are useful for understanding how common a certain condition or characteristic is within a group^​15​. For example, a cross-sectional study might measure how many people in a community have high blood pressure at a certain time. While these studies can provide important insights, they cannot show cause and effect because they do not track changes over time^​15​.

Critical appraisal tools: 

• AXIS appraisal tool for cross-sectional studies^​7​.

The AXIS tool assesses the quality of cross-sectional studies by asking key questions about the study’s purpose, design and whether the participants represent the larger group^​7​.

Key questions

1. Were the aims/objectives of the study clear?
2. Was the study design appropriate for the research question?
3. Were the participants representative of the target population?

Meta-analyses

Meta-analyses are studies that combine data from several different studies to get a clearer answer to a research question^​16​. By pooling results from multiple studies, meta-analyses can provide stronger evidence than any single study alone^​16​. This is particularly helpful when individual studies have conflicting results, as combining them can help clarify the overall findings.

Critical appraisal tools: 

• PRISMA^​8,9​. 

PRISMA is a tool that guides how to report and evaluate systematic reviews and meta-analyses, ensuring that the process is transparent and rigorous^​8,9​.

Key questions

1. Did the review address a clearly focused question?
2. Were the criteria for selecting articles appropriate?
3. Were the included studies sufficiently similar?

Qualitative studies

Qualitative research focuses on understanding people’s beliefs, experiences, attitudes, behaviours and interactions, and generates non-numerical data through a humanistic or subjective approach^​17​. It often involves methods such as interviews and focus groups, in contrast to quantitative research, which relies on numeric data and objective measurements^​17​.

Critical appraisal tools: 

• CASP qualitative sudies checklist^​5​;
• JBI checklist for qualitative research^​4​;
• ETQS^​6​.

Worked example

In 2004, Por et al. designed a study to identify healthcare professionals’ opinion of the main issues regarding the transition of care for young adults with chronic conditions^​18​. The authors used a survey design to allow for the collection of descriptive data. Healthcare professionals involved in transition from one NHS trust were invited to take part. The ETQS contains 38 questions and is therefore more time-intensive than other tools and requires qualitative expert knowledge to complete it^​6​. For the purposes of this article, the assessment of the quality of the study was performed using the CASP checklist for qualitative studies (see Table 2)^​5​.

Conclusion

Developing critical appraisal skills is crucial for pharmacy professionals, enabling them to assess the quality of research effectively. Understanding the strengths and limitations of different study designs, applying appropriate critical appraisal tools, and interpreting the findings in the context of clinical practice, is vital for ensuring evidence-based decision-making.

This article has highlighted the significance of critical appraisal, discussed essential tools and frameworks, and addressed the challenges encountered in clinical practice. Practical strategies have been outlined to help healthcare professionals improve their appraisal skills and integrate these practices into their daily routines.

As healthcare continues to evolve, so too must the approach to critical appraisal, ensuring it remains a dynamic and integral part of clinical practice. The ongoing development of appraisal skills among pharmacy professionals is essential for maintaining high standards of care and improving patient outcomes.