Vaccine evaluation process needs urgent reform

The lack of transparency surrounding decisions on which vaccines should be used in the UK must not continue. An urgent reform of the processes used to inform recommendations of vaccines by the Joint Committee on Vaccination and Immunisation (JCVI) is required to make it clear how decisions have been reached.

The JCVI, an independent expert committee, makes recommendations to the Department of Health (DH) on vaccines and the UK immunisation schedule. The JCVI operates in a different way to the National Institute for Health and Care Excellence (NICE), which is responsible for evaluating all other health technologies, including drugs, medicines, medical devices and other public health interventions.

The JCVI’s set of evaluative processes and procedures should be brought in line with the NICE model for health technology appraisal (HTA), which is widely recognised as an international benchmark, particularly for the level of transparency involved. This is exemplified by the creation of NICE International – the consulting wing of NICE that provides advice to other international HTA bodies – and other international equivalents that follow a similar model, such as the Swedish TLV and French Commission Évaluation Économique et de Santé Publique (CEESP).

Although the JCVI essentially adheres to the same cost-effectiveness rules as NICE, the mechanisms for reaching JCVI decisions are markedly different and decidedly less transparent. Transparency – an area the Cabinet Office has prioritised through various directives (for example, its Open Data Strategy) – is important, especially as the NHS Constitution specifically requires that JCVI recommendations are based on demonstrable evidence of cost-effectiveness (i.e. a cost per quality-adjusted life year (QALY) below the NHS threshold of £20,000 to £30,000).

Being open about how decisions are reached is fundamental to achieving public confidence in JCVI recommendations. Moreover, the appropriateness of current methods for determining cost-effectiveness cannot be sensibly understood without due consideration of the process involved.

Earlier in 2014, the JCVI recommended that the meningitis B vaccine, Bexsero, be added to the childhood immunisation schedule. This was not without controversy, since it was a reversal of a JCVI decision made a few months before. The committee initially rejected Bexsero on the grounds that it was not cost-effective, but this was reversed following a substantial review of cost-effectiveness assumptions.

Although it is positive to see an effective vaccine finally being recommended, the Bexsero decision is a case study in the JCVI process. The committee acknowledged in its minutes that important data were not originally reviewed and there was a lack of formal stakeholder engagement. It remains unclear how the JCVI adjusted the QALYs gained resulting from vaccination with Bexsero.

Another example of sub-optimal JCVI processes concerns the ongoing decision not to recommend the human papillomavirus (HPV) vaccine, Gardasil, for boys. Gardasil is currently given to girls aged 12-13 years to protect against cervical cancer and genital warts. In addition, evidence suggests a link between HPV and penile, anal, and head and neck cancers.

Although a JCVI subgroup is now looking at these issues, the committee’s existing position is based on a model that largely overlooks this broader burden of disease and it is arguably an unrealistic high estimate of herd immunity (indirect protection) resulting from routine vaccination of girls. Regardless, the original cost-effectiveness model has never been fully published, which makes it difficult to assess the robustness of this analysis.

A world of difference

Despite the JCVI declaring a commitment to follow “NICE-like processes” in its code of practice, its processes differ substantially.

Barring emergencies, JCVI meetings occur three times a year and are held in private, whereas NICE appraisals are held regularly and in public following a consultative approach that includes stakeholder representations from manufacturers, clinicians and patient groups. The JCVI publishes minutes following its meetings that summarise key decisions and recommendations, in contrast to NICE, which publishes full appraisal reports including accompanying documentation to justify HTA decisions. Stakeholder engagement lacks the full scoping phases of NICE with a comparatively rudimentary approach to product horizon scanning (treatments likely to become available to the NHS that may have significant implications for clinical practice, service design and finance) regarding new vaccines coming on to the market. In the case of NICE, horizon scanning is an extensive and formal process involving manufacturers at early stages of product licensing. Although the JCVI has recently introduced a ‘Call for evidence’ framework to gather data and selective presentations at sub-committee meetings, it is unclear whether submitted evidence is used in decision-making, raising questions about the comprehensiveness and application of evaluated data.

Furthermore, economic models and technical reports from which assessments of cost-effectiveness analysis are derived – and thus essentially decide whether a vaccine receives a recommendation or not – are currently not released to manufacturers and, in some cases, not even published through peer-review journals (for example, the cost-effectiveness analysis used to recommend the rotavirus vaccine). In contrast, NICE is legally required to release economic models to manufacturers and routinely publishes key findings, following the 2007 judicial review of the Alzheimer’s drug Aricept.

If transparency of this kind is expected for evaluation of all other health technologies, why not for vaccines? There is no cogent reason why the JCVI should be “immune” from greater transparency.

NICE has a Citizens Council that meets periodically to review elements of the institute’s methods, procedures and decision-making. The council provides a forum for eliciting public opinion on NICE’s work. The JCVI has no such public involvement.

The lack of transparency with JCVI processes has a knock-on effect when it comes to DH tendering for vaccine procurement. Following positive JCVI recommendations, the DH assumes responsibility for procuring most vaccines through national tenders. National tenders are based on assessment criteria derived from the cost-effectiveness models used by JCVI. Manufacturers then submit evidence dossiers, including tender prices, to the DH. Currently, tender criteria are not routinely published nor are the calculations and technical reports showing how tender criteria were formulated. On occasions when parliamentary questions have elicited their release, publication has been partial or included redactions, such as the 2008 HPV tender criteria.

Time for reform

Despite the difficult and expert job involved, current processes informing decisions around UK vaccination lack acceptable standards of transparency. This position is largely indefensible, given NICE’s approach, transparency directives and the seminal Aricept judicial review. Decisions surrounding which health technologies are funded by the NHS are inherently difficult, raising various political controversies, so broader stakeholder engagement is critical. Moreover, given that vaccines are usually administered to healthy people, the need to retain and build confidence is imperative. Keeping the rationale for decisions in the dark is contrary to this endeavour.

The JCVI should therefore adopt NICE-like processes, including a scoping phase, formal submissions and stakeholder representations at meetings. Although the NICE process may not be perfect, this adoption would enhance JCVI’s decision-making and mitigate criticism of its recommendations. The case for public meetings is less clear-cut because of the possible hijack of meetings by the anti-vaccines lobby. Nonetheless, greater openness is needed, with stakeholder representation a minimum requirement. Initially, Committee recommendations should be published and made available to the public, explaining the rationale behind each decision on a vaccine.

An appeals process should be established to prevent JCVI decisions being challenged in the courts. The existing state of affairs is incompatible with basic judicial principles, such as the right to appeal. Economic models and technical reports should be released to manufacturers. Since NICE is legally mandated to do this, there is no explicable reason to exempt vaccines. A JCVI Citizens Council should also be created to reflect public views and provide a forum for elicitation of societal value judgements.

Admittedly, transparency does not come for free. For the JCVI to be more “NICE-like”, it will require more resources. However, the business case to DH is compelling given the government’s policy around transparency and best practice standards (for example, NICE’s own standards). The other option is for the JCVI to be moved into NICE. That might improve the existing state of affairs, but it would not be the best solution because the evaluation of vaccines requires certain specific considerations and parameters to be taken into account. Nonetheless, a lack of reform is not an option. Without it, JCVI decisions will continue to be shrouded in mystery.