Reaching statistical significance: lessons from our first salary survey

Graphs showing data and people, statistical significance concept

The 2017 Pharmaceutical Journal Salary Survey was the journal’s first attempt at performing a workforce salary and satisfaction survey.

The task at first appeared simple — prepare a list of questions to capture as much data as possible, recruit participants, analyse the dataset for reportable results, and employ the skills of a biostatistician to confirm statistical significance. 

However, five months and 1924 respondents later, the biostatistician deemed the results statistically… insignificant, or rather statistical significance was unable to be determined owing to factors discussed below. This was a sobering and unexpected finding despite the seemingly endless data that had been collected and analysed. It was only after a lengthy discussion with our biostatistician, Megala, that we discovered the flaws in our methodology. Regardless, by publishing our results we hope to increase awareness and add to the discussion about issues surrounding pharmacy, and how this influences job satisfaction and loyalty to the profession.

Below is a summary of the lessons we have learnt.

Sometimes less is more

Collecting as much data as possible without understanding what value it has, proved to be impractical. The sheer volume of data generated was overwhelming. The results published in the 2017 Salary Survey Infographic represent less than a quarter of the data that was actually collected.

Although we set out to answer a number of questions by collecting extensive data, including geographical information, sector-specific details as well as first and second pre-registration placements, it has been hard to objectively address the results in light of questions owing to the relationship between differing factors.

In 2018, the aim is to develop questions which are clear and concise and reflect the results being collected. Information that is useful and usable. It should be noted, however, that without this initial mass collection of data, identifying relevant themes to focus on for this year’s survey would have been a much more onerous task.

It is not just about what you ask, but how you ask it

The manner in which questions are worded and how responses are designed to be answered, influences how the data will be analysed. 

For example, by specifying ranges instead of exact figures, it is not possible to determine an average or median salary. In addition, free text answers provide respondents with more flexibility when answering questions, but make analysing the data extremely difficult.

Power in numbers

Statistical significance was not achieved in most cases owing to low numbers in certain categories, even after combining them. For example, with lower or higher salary bands. Although in some instances, this was unavoidable as the outcome became apparent only after completion of the analysis.

One example is the question: ‘Is the pay gap between employed male and female non-locum pharmacists within various age groups statistically significant?’ Although we were able to determine statistical significance (p<0.001) to support the hypothesis that there is a gender pay disparity within a certain age group, there was not enough data to ascertain which gender earns more or less, and within which salary band.

Know your limitations

Statistics is not my forte. Therefore, the input and advice from Megala was invaluable. An important lesson we learnt is that it is never too early to include a biostatistician in the process, especially when working remotely. This is to provide guidance and advice on the sample size needed for the survey to address specific research questions, study design, mock data analysis tables and detailed analysis plan.

Failure is an amazing data point which tells you which direction not to go

I would not consider the salary survey from last year a wasted effort or a failure; rather it was a necessary step for us to take in order to plan for this year.

I would like to thank everyone who took the time to complete the survey from 2017. The success of our survey is reliant on pharmacist participation; and therefore I would encourage readers to participate in the 2018 Pharmaceutical Journal Salary Survey, due to be rolled out in July 2018. It is our aim to build on the lessons we have learnt from last year, and by continuing to perform the survey we hope to report data which are not only robust, but also truly representative of the pharmacy workforce.

Last updated
The Pharmaceutical Journal, Reaching statistical significance: lessons from our first salary survey;Online:DOI:10.1211/PJ.2018.20204561

You may also be interested in