What we found when we went looking for hidden bias in the registration assessment

The GPhC analysed more than 1,000 Common Registration Assessment questions for signs that some trainees were unfairly disadvantaged. Meaningful differences were rare, but the exceptions are shaping the assessment, writes Niall Stewart-Kelcher.
Grey background, diverse people are on different orange levels

When a trainee pharmacist sits the Common Registration Assessment (CRA), they should only have to focus on one thing: whether they have the knowledge and skills necessary for safe and effective pharmacy practice. They should not have to wonder whether the assessment itself is putting them at a disadvantage.

Fairness in assessment is not something we can assume. It must be tested and reviewed regularly.

At the General Pharmaceutical Council (GPhC), we are continually looking at how we can improve the CRA. This includes exploring questions relating to equality, diversity and inclusion (EDI). This work supports the GPhC’s ‘Strategic plan 2025–2030’, in which EDI is not a standalone initiative but a principle that informs how we regulate, improve and assure quality across our work.

We know outcomes vary across different groups of trainee pharmacists. The GPhC is also committed to working with partners to address differential attainment and awarding gaps in pharmacy education. The CRA is one example of how we turn this commitment into action.

EDI in assessment is linked to assessment quality: both are about ensuring the assessment measures the knowledge, judgement and skills required for safe, effective pharmacy practice.

Recently, we have approached this from two different perspectives:

  • Are our questions performing fairly for groups of trainees?
  • Does the assessment reflect the diversity of the patients that pharmacists will care for?

Looking for hidden differences

One part of this work involved differential item functioning (DIF) analysis. DIF occurs when trainees from different demographic groups — but with the same underlying ability — have different probabilities of answering a question correctly.

We wanted to know whether there were questions where trainees might be unfairly advantaged or disadvantaged because of characteristics unrelated to what the question was intended to assess.

We analysed 12,500 assessment attempts from 2021 to 2023, covering more than 1,000 questions across Part 1 (calculations) and Part 2 (clinical knowledge and its application) of the CRA. Six variables were considered: ethnicity, age, sex, sector of foundation training, school of pharmacy and country of foundation training.

There was a pattern where hospital-sector trainees generally performed better than community-sector trainees on questions showing differential item functioning

The results were reassuring. Meaningful DIF was uncommon. Only 27 Part 1 questions and 29 Part 2 questions — fewer than 6% of those analysed — showed medium or large effects, which is the level we considered potentially meaningful.

Most of the DIF identified was associated with school of pharmacy in Part 1 and sector of foundation training in Part 2. Few were associated with a protected characteristic. There was a pattern where hospital-sector trainees generally performed better than community-sector trainees on questions showing DIF.

This may suggest that differences in educational or training experiences contribute to some variation in performance, rather than personal characteristics.

All 56 questions with meaningful DIF were reviewed by panels involving members of the Board of Assessors — whose core role is to protect the public by maintaining the assessment at the appropriate standard — and pharmacist contributors from across the UK. The aim was to understand whether there was something about the content, language or style of these questions that may explain the differences.

This review has informed improvements to question style, language and content in the updated CRA framework.

This is one of the most valuable aspects of the analysis. Finding little evidence of meaningful DIF is reassuring. However, the greatest value came from reviewing the exceptions and asking what they could tell us about the assessment in order to identify opportunities for improvement.

Fairness is not only about the trainee

An assessment can be statistically robust while failing to reflect the diversity of the patients that pharmacists encounter.

This led us to look at another aspect of the CRA: the representation of skin tones in clinical images. Under-representation of darker skin tones is recognised as a potential source of bias in healthcare and can contribute to disparities in patient care​1​.

If pharmacists are expected to recognise and respond to clinical presentations in diverse patient populations, the assessment should support that expectation

The CRA has included 116 skin images since 2016. Since July 2021, we have worked to diversify the range of those images.

The change over time has been encouraging. Earlier assessments predominantly featured light skin tones, while more recent assessments include a much broader range of medium and dark skin tones (see Figure 1).

This was a deliberate change, which was made to reflect the patients that pharmacists will care for in practice. If pharmacists are expected to recognise and respond to clinical presentations in diverse patient populations, the assessment should support that expectation.

While this work focuses on the CRA, it also sits alongside wider efforts across the GPhC to consider how EDI can inform regulatory practice. In assessment, that may involve reviewing images, language and question performance. The underlying principle is the same: using evidence to identify opportunities for improvement.

Figure 1: Skin tone images

Why this matters to me

I have worked both in clinical practice and in professional regulation, and that experience has shaped how I think about assessment. Each assessment question has a practical purpose: helping determine whether a trainee is ready to join the profession and care for patients safely.

The focus is on validity. The assessment should minimise the influence of factors unrelated to professional competence. The more confidently we can do that, the more confidence we can have in the decisions it informs.

One aspect of this work that particularly appeals is its emphasis on evidence. We collect data, explore potential issues, test assumptions and make changes where appropriate.

Fairness is not a finished product

Fairness is not a one-off achievement. The CRA is continually evolving, as is the pharmacy profession. That means continuing to ask difficult questions: are there unintended barriers? What does the data tell us?

As we look towards 2027, we will continue to explore how fairness, representation and assessment quality can be reflected in the ongoing development and delivery of the CRA

The findings from these two pieces of work have been reassuring, but they highlight areas for further exploration. As we look towards 2027, we will continue to explore how fairness, representation and assessment quality can be reflected in the ongoing development and delivery of the CRA.

This work is about maintaining confidence that the assessment is measuring the knowledge, judgement and skills required for safe and effective pharmacy practice. It also provides an example of how EDI considerations can inform practical improvements in assessment design and delivery.

Acknowledgments

Niall Stewart-Kelcher is the specialist assessment lead at the General Pharmaceutical Council (GPhC), leading the quality assurance, governance and development of the Common Registration Assessment for trainee pharmacists. He is a registered pharmacist practising in respiratory medicine, bringing clinical experience to his regulatory role.

The author would like to thank the Board of Assessors and AlphaPlus Consultancy for their valuable contributions to this work. Particular thanks are extended to Lisa Smith, postgraduate pharmacy dean, Northern Ireland Centre for Pharmacy Learning and Development (formerly professional assessment manager at the GPhC) for their support with statistical analysis, critical review and helpful discussions that informed the work and implementation of improvements outlined in this article.


  1. 1.
    Kaundinya T, Kundu RV. Diversity of Skin Images in Medical Texts: Recommendations for Student Advocacy in Medical Education. Journal of Medical Education and Curricular Development. 2021;8. doi:10.1177/23821205211025855
Last updated
Citation
The Pharmaceutical Journal, PJ November 2026, Vol 317, No 8015;317(8015)::DOI:10.1211/PJ.2026.1.435220

    Please leave a comment 

    You may also be interested in