Evaluating clinical trial eligibility criteria: recruitment methods, informed consent, control and blinding

This article is part of a comprehensive series exploring how to evaluate clinical studies when addressing information needs using a five-step process:

1. Identifying study type or design;
2. Appraising the journal, authors and study purpose;
3. Critiquing the methods used;
4. Understanding basic data or statistical analysis;
5. Analysing the study results, interpretation and conclusions.

This article explores the third step: critiquing the methods used.

This chapter will focus on several important considerations when examining the methods used in a study, including the study sample and sample size, controlled experimental designs, assignment to treatment groups and blinding. It is nearly impossible to design the ‘perfect’ study. Although study weaknesses and limitations might be present, one should differentiate those that could invalidate the findings versus those that simply limit the application of some results.

It is recommended that you read this article in conjunction with ‘Evaluating clinical study methods: treatment considerations, outcomes, variables and measurements’.

To access other articles in the collection visit the hub page on The Pharmaceutical Journal.

Eligibility (inclusion and exclusion) criteria

Eligibility criteria define the characteristics of the subjects to be enrolled in a study. Those characteristics that must be present for the subjects to be included are referred to as ‘inclusion criteria’ (e.g. to be enrolled in the study, patients need to be aged from 18 to 85 years and have chronic heart failure and hyperuricaemia, defined as a serum uric acid >7.0mg%, etc.). The characteristics that would prevent individuals from being enrolled in the study are called ‘exclusion criteria’ (e.g. patients were excluded if they had an acute myocardial infarction, renal dysfunction defined as a serum creatinine >2.0mg%, were currently receiving therapy for hyperuricaemia, etc.).

Always refer to the patients/subjects enrolled in a study as the study sample. This makes it less confusing when learning how to interpret confidence intervals, since confidence intervals apply to the population outside the study sample.

Why are inclusion and exclusion criteria important in a published study?

They tell us the population of interest that the study is targeting, meaning the type of individuals for whom the study’s findings might be applied (i.e. extrapolated). Since it is virtually impossible to study an entire population of individuals, each trial ultimately enrols a sample of the population that the investigators wish to examine. Ideally, the study sample should represent the target population as closely as possible so the study’s findings can be appropriately extrapolated to those outside the study.

Eligibility criteria should also prevent outside interferences with the study and its methods to the extent possible, such as excluding:

• Non-study (i.e. concurrent) medications that might interact with the study drugs or on their own affect the condition studied;
• Patients who have contraindications to the study drugs, such as allergies or impaired renal or hepatic function that could alter the drugs’ safety or efficacy;
• Patients who have other medical conditions that could interfere with the study findings or prevent them from successfully completing the study.

The investigators want their study sample to be as similar as possible to persons in the targeted population at large so the findings can be extrapolated to the population outside the study. One type of bias — selection bias — occurs when the study sample is chosen in a way that causes patients not to represent the desired population. This is of particular concern with observational studies, such as case-control or cohort studies, in which patients already either have or do not have an outcome or exposure.

Why?

The cases/study patients selected for enrolment in these studies should not have important differences — other than the outcomes or exposures of interest — from the control/comparison patients. For example, suppose patients of a certain ethnicity are more likely to develop the medical condition being studied. If the study enrolled comparison patients in a manner that underrepresented the at-risk ethnic group, selection bias could be present.

Sampling (enrolment) considerations

To obtain a study sample that best represents the targeted population, everyone in that population, in theory, should have the same chance of being selected for the study. The best way to sample a population is through random methods in which chance alone determines who is selected. However, random sampling is not usually possible for experimental studies, since it is difficult to access an entire population with the medical condition of interest. The term ‘sampling bias’ refers to when patients are enrolled in a study in a way in which some population characteristics are either under or overrepresented in the sample. For example, suppose the study sample ends up being younger or with milder illness than the population patients most likely to have the condition. These differences could affect the ability to use the study’s findings in practice. Sampling bias can be considered as either a distinct bias or as a type of selection bias. For our purposes, the term selection bias will be used to encompass both.

Types of sampling (i.e. enrolment) include:

Simple random

Everyone in the population is identified and a random procedure (e.g. computer generated) is used to identify the persons for study inclusion.

Stratified random

When it is important that a study enrol similar numbers of patients who have or do not have certain characteristics (e.g. smokers versus non-smokers, patients with diabetes versus without diabetes), the population is first divided into groups (i.e. strata) based upon the presence or absence of that characteristic, and then a random sample is taken from each group for study enrolment.

Cluster 

Individuals present in already existing ‘clusters’ or naturally occurring groupings in the population (e.g. those living in certain communities, states, regions, or cities, etc.) are used for study enrolment; this is generally done to make it easier to conduct the study when the target population is very large or widely distributed. Either all individuals in the identified clusters are included (e.g. all members of certain rural cities are surveyed for their opinions about quality of healthcare) or a random sample could be taken from each cluster for study inclusion.

Systematic

Selecting every nth (e.g. 4th, 5th, 10th) person for study enrolment; if everyone in the population is known and the starting point is randomly selected, this is a type of random sampling.

Convenience

Non-random sampling that enrols patients based upon advertisements or whether they are being treated in certain clinics, hospitals or outpatient settings that the investigators work at or are affiliated with. Since it is usually not practical or possible to identify or contact all members of the target population for study inclusion, it is the sampling used most commonly in experimental studies. Examples of this type of sampling include, studying asthmatic children from paediatric clinics who meet specific eligibility criteria, studying drugs used to treat mild hypertension in patients from outpatient practice sites, etc. This is acceptable for experimental studies if it is not known at the time of study enrolment which treatment/intervention group the patient will be assigned to, which would be true for studies using random assignment to groups^​1​.

Informed consent

Investigators need to ensure to the extent that is possible that the subjects in their studies are protected from harm. To do this, an institutional review board (IRB) is used along with the process of informed consent. An IRB is a group responsible for assuring that researchers in their organisation or institution take appropriate steps to protect their subjects’ rights and welfare, both before a study is initiated and throughout the study.

IRBs were established in the United States owing to federal regulations for the protection of human subjects that became effective in 1974. The responsibilities of IRBs and guidelines for IRB written procedures can be found at the U.S. Department of Health and Human Services, Office for Human Research Protections (OHRP) website^​2​. An important part of protecting a subject’s rights involves the need for investigators to obtain informed consent from each subject prior to their enrolment in the study (see Box 1).

Sample size

A common question when analysing a study is, ‘Did the study have enough patients?’ or ‘How many patients is ‘enough?’ A study should have a sample size large enough to identify a statistically significant difference among treatments when an actual treatment effect exists. That is, it should be able to reject the null hypothesis — no difference among treatments — when it is indeed false and there really is a treatment effect. The extent to which a statistical test can identify a significant difference when there is an actual treatment effect is referred to as ‘power’.

Sample size is an important factor affecting a study’s power. All else being equal, as sample size increases, statistical power for outcomes of interest increases. As the number of patients enrolled in a study decreases, the power decreases. This is important as if the statistical power is too low, a study could find a fairly large — and possibly clinically important — difference in outcome measures between treatment groups, but its analyses might not find this difference to be statistically significant. As a result, the concern is that a real treatment effect would be missed.

The study’s sample size, or number of patients to enrol, should ideally be calculated before the study begins. To do this, investigators first select the desired power for the outcome(s) of interest in the study. By convention, an acceptable degree of power is 80% or greater. Using the selected power value, they then calculate the appropriate sample size needed to achieve that power (e.g. using an equation for determining power and solving for sample size, which is one of the variables in that equation). The investigators should clearly state for readers the power calculated and the sample size needed to achieve that power. 

Controlled experimental designs

The controlled experiment is a strong design that can prove cause-and-effect relationships, unlike observational studies, and establish therapy efficacy. Controls (i.e. comparison groups) should best be used since they reduce the likelihood that factors that are not part of the study (e.g. environment, naturally occurring changes in the condition studied) might affect the study results.

The previous article ‘Identifying and evaluating research study types’ briefly defined the types of control groups: placebo, active, historical, no treatment. The first two controls are most used and preferred whenever possible. No treatment controls can be of concern for ethical or practical reasons. Three primary designs are used for controlled experiments: concurrent control (i.e. parallel); crossover; and time series (before and after; see Figure 1).

Figure 1: Illustration of controlled experimental designs

Concurrent control design

In the concurrent control — or parallel — design, patients are assigned to receive either a control or study treatment. Patients only receive one of the interventions in a parallel design. Results are then compared between/among groups to determine if the outcomes in one group are significantly different from the other. With the parallel design it is important that the patients in each group are as similar as possible to help ensure comparability of the results.

Crossover design

In a crossover design, the patients receive each of the interventions (i.e. control and treatment). They are initially assigned to either the control or experimental therapy. After completing that course of treatment, the patients are assigned to the other group(s), one at a time, so that each patient eventually receives each intervention by the study’s end.

The crossover design generally includes a wash-out period — an amount of time during which no therapy is given — between each intervention so that (in theory) any effects from one treatment can be eliminated or ‘washed out’ from the body prior to beginning the next intervention phase.

With a crossover design, treatment effects should be compared between and within periods to determine if the response to an intervention during one period is the same as the response to that same intervention during the next period. This involves comparing:

• The difference in response between the treatment and control in the first period with the difference in response between both in the second period (see Figure 2; A);
• The control effect during the first period with the control effect during the second period and treatment effect during first period with treatment effect during the second period to ensure there are no differences based upon the sequence of treatments (see Figure 2; B).  

If there are no carry-over or other period or sequencing effects, those comparisons should all be the same. If any differences are present, the results can be difficult to interpret because it is unknown how much of the effect was owed to the treatment versus carry-over or another non-drug influence.

Figure 2: Comparing treatment effects between groups in crossover design

Time series design

In the time series (or before-and-after) design, each patient also receives each study intervention. However, this design differs from the crossover design in that all patients receive the same type of intervention at the same time. The advantages, disadvantages and key points of each of the three controlled experimental designs are summarised in Table 1 and Box 2.

Table 1: Advantages and disadvantages of controlled experimental designs

Assignment to interventions

How should patients/subjects be assigned to a certain intervention group (e.g. control versus treatment)? If an investigator simply chooses on their own who receives each intervention, bias could be present (consciously or subconsciously). Sicker patients might be assigned to one group over another. Thus, patient assignment is best done through a process called ‘randomisation’, in which patients are randomly assigned (e.g. using computer-generated random numbers, a random numbers table).

Random assignment means that each patient has an equal chance of being in a particular study group. This is extremely important to help reduce bias and ensure study quality. Random assignment also reduces the chance that extraneous factors might affect the study results since differences in patient characteristics would be more likely to be ‘balanced’ between groups. In addition to simple randomisation, there is also block randomisation and stratified randomisation.

Types of randomisation

There are several commonly used types of random assignment:

Simple random assignment

Upon enrolment, subjects/patients receive a random number for group placement through computer generation or a random numbers table. For example, for a pool of 20 subjects planned to be enrolled, a computer or table can provide numbers in random order, with investigators identifying in advance that odd numbers will be assigned to group one and even numbers to group two. When the first subject is ready to be enrolled, they are given the first number from the computer or table list. If an odd number, they will be assigned to group one. If even, to group two. This process would continue for each subject.

Block random assignment

With simple randomisation, one group might end up with more subjects than another by chance. Also, if a study happens to end before the planned number of subjects are enrolled, there could be uneven group numbers using simple randomisation. Block randomisation can help ensure that treatment groups have similar numbers of subjects. With block randomisation, small ‘blocks’ or groups are formed at the study locations, with random assignment to groups used within each small block. For example, if 20 subjects will be enrolled, 5 blocks of 4 can be identified. For every 4 subjects enrolled, 2 would be randomly assigned to one group and 2 would be assigned to the other.  

Stratified random assignment

Stratified randomisation is used if investigators want to ensure that certain characteristics of a population (e.g. race, sex, geographical location, socio-economic status) that could affect a study are balanced among treatment groups. With this method, the target population is initially subdivided into groups (i.e. strata) based on the characteristic of interest. Random assignment is then used to assign enrolled subjects in each strata to one of the study groups. This method is often used in multi-site studies to ensure similar enrolment numbers across locations. Block and stratified random assignment can also be combined for use.

Important points of randomisation:

• Any time a study states it is ‘randomised’, this is referring to random assignment to treatment groups, not random selection (i.e. enrolment);
• With random assignment, patient characteristics (known and unknown) that might affect the study results are more likely to be similarly distributed between the study groups;
• Random assignment will not guarantee that patients in the different study groups will have identical characteristics. By chance, there still might be important study group differences that could influence the results;
• To determine the success of randomisation, investigators will usually include in a table the demographics (e.g. age, gender, ethnicity) and key baseline (prior to therapy) characteristics of the patients assigned to each study group. These findings should ideally be compared statistically to determine if the groups are indeed similar regarding factors that might alter therapy response.
• If important baseline differences exist, statistical methods might be able to take these into account when interpreting findings.

Blinding

Blinding, or masking, is when the patients/subjects and/or investigators do not know the intervention group that the patients were assigned to (see Box 3). As a result, it is not known during the study whether a patient is receiving the control or the study drug. Blinding should be used to reduce the risk of bias that might result from patients or investigators knowing who is receiving which therapy. Many outcome measures, particularly those that are subjective in nature, such as pain relief, changes in mood, or development of side effects, can be affected by a person’s belief (patient or investigator) that a therapy will work or have a certain effect. An investigator might be hopeful that a new drug will be efficacious and unintentionally skew the results in that direction by asking patients more questions about the drug or repeating tests if findings are unexpected. If patients believe that the drug they are taking will be beneficial, they might report changes they would otherwise overlook or fail to mention potential problems experienced.

Open-label studies are problematic since bias can be introduced by both patients and investigators. In a single-blind trial, the investigators might slant the study’s measures in a certain direction or lead patients to figure out the treatment they are receiving. As a result, double-blind trials are preferred to minimise the likelihood of bias and are an important part of the ‘gold standard’ randomised, controlled experimental study design. However, it is not always possible to double-blind certain interventions in a study. For example, one could not blind a comparison of the effectiveness of in-depth patient counselling to minimal patient counselling or of a drug to surgical therapy.

A study might also start as double-blind, but certain side effects (e.g. nitroglycerin-induced headache), unique drug or treatment smells/tastes (e.g. a study of garlic’s blood pressure effects), or characteristic lab test alterations would ‘clue in’ the patients or investigators to what the patient is actually receiving. ‘Unblinding’ (i.e. unmasking) occurs when the patients or investigators can successfully identify what the patient is receiving during a blinded study. If unblinding occurs to a significant extent, the potential benefits of a blinded study disappear.

How to tell if unblinding has occurred

In a study comparing two different treatments, at a minimum the treatments should look alike for the study to be double blind. When considering the possibility of unblinding, think of other possible reasons why it might occur, such as a drug’s distinctive taste, odour and side effects — would non-health professional patients know the possible side/adverse effects of their study drug? Yes. Remember that, through informed consent, patients are told the potential benefits and risks (including side/adverse effects that are more likely to occur or could be potentially serious) for each study treatment they might receive.

The simplest way to determine if unblinding occurred is to ask patients which therapy they received. If most patients can successfully guess their treatment, unblinding was likely. Keep in mind that if a drug is highly efficacious compared to a control such as placebo, patients and investigators might correctly guess their therapy simply by the patient having a good response. Thus, determining unblinding by asking patients is somewhat questionable at times even though it is generally the only available option.

Summary and key points

• The inclusion and exclusion criteria should accurately characterise the type of patients (i.e. population) that a study wishes to target;
• Apply the findings from a study’s sample to the population specified by the inclusion and exclusion criteria;
• Random sampling (i.e. selection) is the best method for enrolling subjects in a study, but this is not usually possible for experimental studies;
• The sample size should be large enough to have sufficient power to identify a difference in outcome measures as statistically significant if there is a real treatment effect.
• A study’s power should usually be at least 80%;
• Investigators are required to obtain IRB approval before initiating a study, which includes obtaining informed consent prior to participants’ enrolment in the study;
• The concurrent control (i.e. parallel) experimental design is best and is preferred; the crossover design is better than the time series design;
• Double-blinding (or triple-blinding, if needed) and randomisation should be used whenever possible;
• If an active control is used it should be an appropriate choice to compare to the therapy being studied;
• Were sufficiently long wash-out periods used between the interventions in a crossover study? The investigators should also have analysed the results for carry-over or other period or sequencing effects;
• Examine the baseline (i.e. post-randomisation) characteristics of the patients assigned to each study group to see if potentially important factors are evenly distributed among groups. If not, consider whether these differences might have altered therapy response in the groups;
• Were there important baseline comparisons that the investigators overlooked? Examples might include differences among groups in duration or severity of illness, previous therapies tried and those currently being taken, and underlying renal or hepatic function;
• Consider the possibility of unblinding in a blinded study.

Application in practice

When evaluating a study, be sure to check the inclusion and exclusion criteria to determine the population to whom a study’s results can be applied, and whether your patients (and the population at large) are comparable to those sampled in the study.

It is also important to check the number of study patients needed to reach a power of 80% or greater. Make sure that the number of patients analysed did not drop below that needed for adequate power. If the power was less than 80% or never stated, it is possible that a real treatment effect was missed for non-statistically significant findings, especially if the difference between groups appeared fairly large. If an observational study cannot include the entire population of interest, random sampling methods should be used if possible to minimise bias in subject selection. But recognise this is often not feasible.

Do not be concerned if an experimental study uses non-random convenience sampling. This is acceptable as long as the patients are randomly assigned to a treatment group following enrolment.

Understanding how to read the methods section is critical to evaluating a clinical study, because it contains valuable information about how the study was conducted. A study must use a trial design that is appropriate for its purpose. Design considerations that need to be considered include the use of control groups, random allocation and blinding. The treatments used and measurements will be discussed in the next article in this series.

Self-assessment questions

Question 1

An open-label study was performed to compare the effect of cyclobenzaprine with other muscle relaxant drugs on tinnitus severity in patients who have chronic tinnitus. Cyclobenzaprine, but not the other treatments, was found to improve tinnitus severity significantly. What does ‘open label’ refer to in this study, and can it have an effect on the study findings?

Question 2

A randomised, double-blind, controlled study was conducted to evaluate the efficacy of Viagra (sildenafil) compared to Lyrica (pregabalin) for treatment of neuropathic pain. A total of 150 patients who presented to one of three pain clinics affiliated with the University of Southern California from January 2003 to December 2004 and who met eligibility requirements were enrolled in the study. Would it be correct to state that this study used consecutive random sampling to enrol its patients? Yes or no?

Question 3

A study is performed to compare metformin to glipizide for diabetes therapy in type 2 diabetic patients. In total, 120 patients are randomly assigned to receive metformin or glipizide for 12 weeks. At the end of the 12 weeks, the patients are then switched to the other therapy for an additional 12 weeks. Which controlled experimental design was used in this study? 

A: Concurrent control (i.e. parallel)

B: Crossover

C: Time series (i.e. before and after)

Question 4

Given the scenario outlined in question 3 above, is this the preferred design for such a study? Yes or no?

Answer guidance

Question 1

‘Open-label’ means that the study was nonblinded – both investigators and patients were aware of the treatment the patients were receiving. An open-label study is subject to a significant amount of bias that could influence the study’s findings. Since tinnitus is a subjective symptom experienced by patients, the patients’ ratings of tinnitus might be influenced if they believed their assigned treatment was going to be efficacious. The investigators might also skew the findings by how and what they discuss with patients during study visits if they feel that one drug is going to more efficacious than another. Although an open-label study can provide interesting preliminary findings, the results should be confirmed in subsequent double-blind studies.

Question 2

No: The patients appeared to be enrolled into the study using a non-random convenience method —attending a pain clinic associated with a particular university. There is no indication that the 150 patients were randomly sampled from among all the patients who were seen at the three pain clinics. It is also not clear that every patient who met the eligibility requirements and who showed up at clinics between the dates listed was enrolled into the study. Since random assignment to treatment groups was used, a non-random sampling method is acceptable for clinical trials such as this one.

Question 3

B: Crossover. With this design, patients are initially assigned to receive one of the study treatments for a certain length of time, then they are switched over to receive the alternative therapy for the next treatment period. With this design, it is important to use a wash-out period (i.e. time between treatments when no drug is administered) to allow the effects from the first treatment to disappear before the next treatment is given. This study did not use a wash-out period.

Question 4

No: The parallel design is preferred over a crossover study. In a crossover study it is important to use a wash-out period (i.e. time between treatments when no drug is administered) to allow the effects from the first treatment to disappear before the next treatment is given. This study did not use a wash-out period. The parallel design is preferred over a crossover study. Even when a wash-out period is used in a crossover study, there might be some carryover effects from the first treatment. Also, the longer study duration needed for a crossover design compared to a parallel design might influence the medical condition being evaluated or otherwise affect the study results.