How to evaluate the impact of a quality improvement intervention

Introduction

Medical science is constantly evolving, and health services are facing significant pressure, so it is no surprise that more and more pharmacy professionals are becoming involved in quality improvement (QI). The terms ‘QI’, ‘service development’ and ‘performance improvement’ are often used interchangeably, with the common goal of developing some element of the healthcare system. In this article, we will use the term ‘QI’ to describe the use of a defined method to implement solutions to issues, with the goal of creating quantifiable change^​1​. Some of the elements that QI aims to improve include efficiency, productivity, safety and both staff and service-user experience^​2​.

For QI to be successful, several factors should be considered. Arguably, the most important factor is to ensure that a method is applied to making the improvement^​2​. Some of the tools used in QI have been developed specifically for healthcare — such as the Institute for Healthcare Improvement’s ‘Model for Improvement’ framework — but others have been adopted from other industries, such as Lean and Six Sigma^​3​. Common to all QI methodologies, however, is the importance of measuring the impact of the change being made^​4​. Measurement is a large topic, and this article looks to introduce the concept and some of the common tools used.

Importance of measurement when evaluating a QI intervention

Measurement serves several functions when it comes to QI. In the first instance, before any change is made, it allows an assessment of whether there is actually a problem that needs to be solved at all^​5​. To give an example, a pharmacist working in a community pharmacy thinks that they have seen an increase in the number of prescriptions that they are being asked to check, and this is resulting in increased wait times for patients. So, they keep a record of the number checked each day, over a couple of weeks, and see that there is, indeed, an upward trend in both prescriptions received and the wait time for patients. 

Starting this measurement before making any change also provides baseline data to assess any change against. Establishing a baseline is vital to being able to assess if an intervention has had an impact on a process^​6​. If the pharmacist had decided to use a locum to help before measuring the number of prescriptions checked, there would be no way of reliably telling if any reduction owed to the extra resource or a reduction in the number of prescriptions being checked. Measurement is, therefore, essential in assessing the impact of QI interventions^​5​. 

This is a very simple example that assumes that there are no other ways of collecting the data required to assess change. In today’s healthcare system, which relies heavily on the use of technology, there is usually a range of systems that can provide sources of data. 

What to measure

Choosing what to measure can be surprisingly complex with larger projects, and it really depends on what the intervention is trying to achieve. There are some essential concepts that apply to all measurement: don’t over-complicate your measurement strategy and don’t try to measure too many variables or variables that do not add to the evaluation of your intervention^​7​. Measuring too many or irrelevant variables will only add to the burden of analysis and interpretation. The right level of data will vary, depending on the project you are undertaking, but the aim is to spend the right amount of time measuring and evaluating — not to waste time looking at data that does not add value.

Types of measures

There are three main categories of measurement that should be considered as part of any QI project: outcome, process and balance measures^​5​. Outcome measures quantify the effect that you want to see from making your intervention^​8​. In the example already given of patients waiting longer because of an increase in the number of prescriptions needing to be checked, the outcome measure would be the time that patients spend waiting for their prescription. Put simply, it is a direct measure of what we are trying to achieve.

Process measures quantify how you are trying to achieve your outcome^​9​. If, for example, a hospital pharmacist wanted to increase the number of patients they can see in a warfarin clinic, process measures could be how long each patient spends in clinic, the downtime between patients in clinic or the number of people that did not show up. The outcome measure in this example would be the number of patients seen in each clinic.

Finally, balance measures look for any consequence of your intervention that goes beyond the outcome you are trying to achieve^​10​. In trying to reduce the amount of time that patients wait for their medicines, the pharmacist may find that other services or tasks are delayed owing to a shift in focus. In this example, balance measures could include time to access other services provided by the pharmacy, the cost of running the pharmacy if extra help has been hired or staff burnout from having more work to do.

These are some of the core concepts to consider when measuring a QI intervention. The number of variables that can be recorded is vast, so the take-home message from this section is to make a clear plan, ensuring that your measures clearly quantify the aim of the intervention.

Measure data over time

There are many different tools and techniques for measuring data, and there is no one correct approach. The appropriate choice will depend on your project and your confidence in using the tool that you select. However, one essential factor is to collect data over a period of time^​11​.

Imagine that several batches of manufactured medicines were to fail quality control, based on the results of microbial testing. It is discovered that the correct disinfection process is not being followed, so the technician responsible for manufacturing decides to retrain the operatives responsible for producing the medicines in this process. The technician then conducts a single audit a week later, which shows staff are following the correct procedure. Consequently, they close the QI project, reporting to their seniors that it has been successful.

In the following weeks, however, the number of failed batches does not fall. When the technician reaudits, they find that the correct disinfection procedure is still not being followed. The initial audit showing improvement could have been a spurious result, caused by the individual staff members involved rather than an overall improvement. Had the technician assessed the disinfection technique over a period of time, they would have discovered that the retraining had not been effective in getting all of the operatives to follow the procedure. Additionally, they could have decided that the training should be altered or that another intervention was required.

There are several tools that are useful for measuring data over time. They can be relatively simple, such as a tally chart, bar chart or histogram, or more complex, with the use of run charts and statistical process control (SPC) charts. Run charts are relatively simple and involve plotting data over time, alongside the median of the values you are plotting^​12​ (see Figure 1). A run is one or more data points in succession on the same side of the median line. However, data points falling on the median line are not counted^​13​.

The Pharmaceutical Journal

Run charts aim to determine if the observed data is a result of normal fluctuation in a process, also known as common cause variation, or a particular event (e.g. a QI intervention) — known as special cause variation^​12​. We can apply a set of rules to the data to determine whether change has been observed. These rules are described below (see Table^​13,14​).

SPC charts are more complex than run charts and more commonly used in larger QI projects. SPC charts still plot data against time, but they plot the mean, along with upper and lower control limits, which indicate statistically determined boundaries of expected variation in a process^​15​. An SPC chart is more sensitive to special cause variation than a run chart, but requires more data points to be statistically accurate^​16​. An example of an SPC chart is illustrated in Figure 2. Owing to their complexity, constructing and interpreting these charts is beyond the scope of this article. However, templates and explanations are available online, such as this one provided by NHS England^​17​.

The Pharmaceutical Journal

Data visualisation

Make your data as easy to interpret as possible^​18​. For example, if a pharmacy was trying to target a specific age group with a service, they would want to measure the number of people accessing the service and their ages. It would be better to display this data graphically, such as in a bar chart or pie chart, rather than as a long table (see Figure 3). 

Where to get help with measurement

Measurement and data handling can seem daunting, but there are plenty of sources of help. If you work for the NHS, many organisations have an information or data team who have specialist skills in data management. They are normally willing to give you pointers and may even have courses that you can attend or tools and templates available to use. 

For private providers and those working in other sectors, there is normally someone in your organisation who can help. Sometimes, these teams are located within centralised corporate services. 

There are also tools and templates available online. When choosing a template online, be sure to use a credible source to ensure the accuracy of the tool you are using. Credible sources include NHS England, AQUA and the Institute for Healthcare Improvement.

Conclusion

This brief introduction to the topic of measuring a QI intervention has hopefully given you an insight into some of the main concepts. Measurement is crucial to assess the impact of an intervention. However, it does not have to be complicated and should be tailored to the project that you are undertaking. An essential concept that should be adhered to is the measurement of data over a period of time. This is critical to being able to determine if your project has made an impact and could be done using a run chart or SPC chart. Other ways to promote successful improvement include choosing the clearest and most effective way to display your data visually and not burdening yourself with unnecessary analysis.

Whatever you decide to measure, by planning, following the principles in this article and seeking help when required, you’ll have the best chance of success.