Beyond RCTs: the real-world evidence revolution

Randomised controlled trials (RCTs) have long been considered the gold standard for evidence-based medicine, but, in an increasingly digital world, there is a new contender vying for first place.

Real-world evidence (RWE) is derived from data sources, such as electronic health records, disease registries, patient-reported outcomes or even wearable devices, which researchers hope will provide information that is difficult to obtain using conventional trials (see Box).

According to a study, between 1998 and 2019, 17 US Food and Drug Administration (FDA) or European Medicines Agency (EMA) new drug applications used real-world data (RWD) in the fields of oncology and metabolism, and all of them received approval. There have been a further 10 applications for indication extensions based on RWD in the same period​[1]​.

However, not everyone wants to join the RWE revolution, with some experts saying we abandon RCTs at our peril.

Rationale for real-world evidence

The rationale for turning to RWE can vary. RCTs may not be suitable for rare diseases, innovative and complex technologies, or in certain populations. Medicinal cannabis, for example, has multiple active ingredients, at different ratios, and a challenging legal environment. Anne Katrin Schlag, head of research at Drug Science — an independent science-led charity founded in 2010 to inform and campaign for drug law reform — says running RCTs for all the conditions that medicinal cannabis is being used for is almost impossible.

“A lot of the medical cannabis patients that we see have a lot of comorbidities — up to ten in some cases,” she says, explaining that many trial protocols would exclude these patients. Conventional RCTs also often miss longitudinal outcomes, something she says is essential in assessing the value of medicinal cannabis.

Drug Science has set up its own real-world study — Project Twenty21 — a UK registry of more than 3,500 patients, covering different medicinal cannabis products and indications.

In other cases, the shift towards RWE is related to changes in the drug discovery ecosystem, with the age of the ‘blockbuster’ drug for large target populations largely over. “We are in the age of the ‘niche buster’, [and] that means new treatments for small or very small target populations. Once you find yourself in that kingdom, running large RCTs becomes an impossibility because there aren’t enough patients,” says Hans-Georg Eichler, consulting physician of the Association of Austrian Social Insurance Institutions and previously senior medical officer at the EMA.

Louise Bowman, professor of medicine and clinical trials at the University of Oxford, agrees that developing well-powered clinical trials has become harder as medicine has progressed. With most of the big wins achieved, “our margins of return have got smaller as we’ve got better at treating people”, she says, adding that this means larger clinical trials are needed to prove therapies are beneficial.

On top of this, the regulations in place, which Bowman says were well-intended, have become “over interpreted”, leading to “a point where the practicalities of running trials can become stifling”. 

Efficacy–effectiveness gap

Alongside these problems, we are now living in an “exploding digital world”, says Bowman. We have large amounts of patient data from everyday clinical practice, as well as patient registries and electronic records, all potentially available for analysis outside of the confines of conventional clinical trials. Having such large data sets might provide a substitute for RCTs, but how these data can be reliably used is still up for debate.

One way to harness RWD is through patient registries, which are now being created in many countries and regions. The ‘Data Analysis and Real World Interrogation Network’ (DARWIN EU) registry was established in 2020 by the EMA to generate RWE. The EMA has partnered with the UK, Belgium, France, Spain, Finland, Estonia and the Netherlands to create a web of registries that includes data from hospitals, primary care, health insurance, registries and biobanks. By 2025, it hopes to deliver 150 RWE studies annually; the first of these will be an analysis of alternatives to sodium valproate for female patients, in light of its potential harm to unborn babies, and a study characterising prescribing patterns for 141 antibiotics to assist in the fight against antimicrobial resistance.

For Eichler, RWE can add value beyond clinical trials. “There is a myriad of relevant research questions, and the scientific community is simply not in a position to answer all these questions by way of RCTs. Even if we have all the financial resources on the planet, we would probably still not have the time to do it,” he says.

Pall Jonsson, programme director for data and real-world evidence at the National Institute for Health and Care Excellence (NICE), says, rather than answering questions of efficacy, for which RCTs are best suited, NICE is looking to RWE to plug gaps in its knowledge of how medicines perform in the real world — what he calls the “efficacy-effectiveness gap”.

“Because of how trials [are] designed and the patient population [included in] trials, the effectiveness in the real world can be different to the efficacy in a single trial,” he explains. “We want to use RWE to address the evidence gaps that we see in our appraisals.”

In 2022, NICE launched a framework for using RWE, which aims to define best practice in terms of study design, execution, reporting and analysis​[2]​.

However, Jonsson acknowledges that, in some cases, RWE may be appropriate to assess efficacy. For trials of rare diseases or certain oncology drugs, he says, NICE is increasingly seeing single-arm trials and is starting to see RWD being used as an external comparator or control arm, rather than a randomly selected placebo or comparator group within the trial. For example, in January 2023, NICE recommended the kinase inhibitor mobocertinib for treating advanced non-small-cell lung cancers, based on a single-arm phase I and II trial that used RWD as an external comparator. “We’re on a learning curve around how to do this robustly,” he adds.

Abandoning randomised controlled trials

For Bowman, despite the real difficulties in RCTs, abandoning this gold standard is worrying. She sees the value of RWD in understanding rates of disease, or informing the design of clinical studies, but says “people have been lured by the appeal of digital healthcare and huge datasets and [are] thinking, we’ve got such big numbers, that it’ll be fine”.

But she warns, “if you don’t randomise, you could still be barking up the wrong tree”. Bowman points to the recent example of claims made for the use of hydroxychloroquine in treating COVID-19, where observational data pointed to efficacy. However, when hydroxychloroquine was tested in a large RCT — the RECOVERY trial — it was a different story. “It wasn’t until the RECOVERY trial actually did a proper randomised, well-powered assessment of it, that it was clear that actually there were no benefits of this drug​[3]​.”

Beate Wieseler, from the independent Institute for Quality and Efficiency in Health Care (IQWiG) in Köln, Germany, shares Bowman’s concerns. She says the important issue is not the source of the data per se, but the loss of randomisation. Without randomisation, there is no defense against bias and there is always the chance that confounders — factors that distort the significance of any results — are present, she says. In addition, without control or comparison arms, it is impossible to know the absolute efficacy of new treatments or how they relate to other available options. “If this continues, we will get less and less understanding of the treatment landscape,” she adds.

Others are convinced the RWE approach will add value and are looking at how potential biases can be reduced. “I’m very optimistic to see that more and more papers are coming out from people who are trying to validate methods,” says Eichler. One approach is target trial emulation, where a trial is designed in the same way as a RCT but then populated with observational data. By selecting data that follow the trials precise eligibility criteria, treatment strategy and outcomes, the hope is that biases can be avoided. Jonsson says NICE is keen to explore this approach.

Drug Science is shoring-up its RWE using Bayesian analysis — a statistical method that allows researchers to take into account data as well as prior beliefs to calculate probability — to predict the likelihood that correlations are real. “We have used [the approach] on [observational] data from our paediatric epilepsy patients to see whether the results we have seen in our relatively small cohort are likely to be replicated if we had a bigger cohort,” says Schlag​[4]​. As more RWD become available, the probabilities can be refined, says Schlag.

But from Wieseler’s perspective, it is dangerous to rely on new statistical approaches. She argues that, even though they can adjust for some population differences that could otherwise blight non-randomised trials, that will not always be enough. “The data sources which are used for these studies very often don’t have the information that is necessary to adjust for these differences,” she says, which may be subtle but important; for example, populations from different socio-economic groups with associated differences in health outcomes, which may not be picked up.

The problem of relying on evidence outside of RCTs, Wieseler says, is epitomised in the approval of six new therapies to treat relapsed or refractory B-cell lymphoma over the past decade, including several CAR-T cell therapies — four of which were based on single-arm studies. Analysis by Wieseler et al., published in 2023, showed that the single-arm studies in fact had a larger number of patients enrolled than the RCTs​[5]​. She says this suggests the decision to abandon RCTs is a deliberate one. “Now we have three or even four CAR-T cells on the market [and] have no idea how they compare, we have no idea which patients would benefit from which ones and, of course, we have no idea how to price them.”

Improving randomised controlled trials

The solution, according to Wieseler, is not RWD, but a better infrastructure for enrolling patients in RCTs and a concerted effort to decrease the administrative burden and non-stop ‘box-ticking’ that comes with running such trials. “There is work going on to review the regulations and, in particular, good clinical practice guidance,” says Bowman, pointing to the Good Clinical Trials Collaborative in the UK.

Bowman concedes the current guidance is “no longer fit for purpose”.

There is hope for improvements in the ease of conducting RCTs. In May 2023, following an independent review, the UK government announced an investment of £121m to improve the UK’s commercial clinical trials landscape, and commitments to approving and setting up trials quicker, and making it easier for people to access trials and clinicians to access patients who could benefit from ground-breaking treatments​[6]​.

Another approach is using the sorts of patient registries that are being developed to collect RWD to identify patients for RCTs. Frustratingly, Wieseler says the EMA has limited the DARWIN EU registry to observational data sources and non-interventional studies, which, by definition, cannot be randomised. “It’s such a missed opportunity,” she says.

Credibility is still a crucial issue if RWE is to play a bigger role in the decisions made by regulators and clinicians. NICE hopes its framework will provide this and includes a data suitability tool that gives guidelines for data quality, provenance and relevance. “We’re starting to see submissions that have taken account of some of the best practice elements in our framework and, as a result, we’re seeing better evidence starting to come through,” says Jonsson. He thinks there is now a need for a better understanding of the technical aspects of analysing RWD across the healthcare system, “even in industry, pharma companies haven’t necessarily got the full skill set that they need”.

Other countries are also moving forward. In May 2023, Canada’s Drug and Health Technology Agency announced a partnership with other stakeholders to advance the integration of RWE into decision making, and published guidance for reporting RWE to lay the foundation for its use in regulatory approval and health technology assessments​[7]​.

However, in the United States, the FDA has indicated a move away from single-arm trials for early approvals of cancer drugs. Its practice had been to accept preliminary single-arm trials to gain conditional approval, followed by a full RCT but, in March 2023, it announced that failures to spot rare adverse events and difficulties in predicting clinical benefit had led to a rethink. Its draft guidelines now favour an RCT in all but a few limited cases​[8]​.

But Bowman is still concerned that regulators are not being cautious enough. “There is so much enthusiasm about RWE that I almost feel like the regulators have jumped on the bandwagon without really thinking it through at times.” Schlag feels this is not the case, however, saying the lack of RCTs has prevented NICE recommending medicinal cannabis in most cases, resulting in many of those who find it beneficial resorting to private prescriptions.

“It’s understandable in some ways, but the reality is, in terms of medical cannabis, hundreds of thousands of patients globally, again and again, state that this works.”

Ultimately, she hopes RWD will contribute to improving patient access and claims Drug Science has had some indications the NHS is interested in reviewing its data.

For Eichler, it is crucial to remember that RWD is a supplement, not a replacement for RCTs. “We need both,” he says. He has no doubts that RWD will start to play a more important role in the evidence used to access clinical practice but expects progress to be made in “fits and starts”.

While the gold standard of RCTs will remain central to regulating new medicines, beyond this, Jonsson says, “there’s a huge amount of valuable information out there and it would be almost unethical not to learn from this evidence”.