How do you tell if a drug treatment really works? The role of anecdotal reports & clinical trial results
One issue that has been thrown into the spotlight recently, is how we can tell if an intervention, such as a drug treatment, really works. We’ve all heard about clinical trials, but is anecdotal evidence enough during a pandemic? What do we have to lose by trying a new treatment if there aren't any existing ones to use?
Information about how effective an intervention is can be gathered in a number of ways including:
Through casual advice from people who have used it
By observing people who use it and documenting what happens
By actively allocating people to receive certain interventions and then measuring the outcome
Knowing how much we should trust the evidence in front of us is vital when making important decisions, especially when your health may be at stake.
Here we take a look at the different types of evidence and touch on what changes to the evidence gathering process may be appropriate during a pandemic. The types of evidence covered include:
Evidence from observational (epidemiological) studies - including cohort, case-control and cross-sectional studies
Evidence from interventional (experimental) studies - including community and controlled clinical trials
What to know about anecdotal evidence
Many times people rely on anecdotal evidence. This is the casual or informal advice you gather from family, friends, or nowadays even strangers on the Internet. A common example of this type of evidence is when we try a new vitamin or supplement because a friend told us it worked for them.
Anecdotal evidence is often accompanied by emotional information that our brains like to respond to. The person sharing it likely had a particularly good or bad experience, which makes the information more memorable for us.
In some situations we can benefit from a wealth of anecdotal evidence, such as in the case of an experienced healthcare professional who is able to draw on years of clinical practice treating many patients. The problem with this type of evidence, however, is that it is unreliable compared with clinical study data and we may place more importance on it than we should. This won’t necessarily do us any harm if we’re trying to pick a new leaf blower or pair of shoes, but it can if we’re trying to treat a harmful new disease.
That said, anecdotal evidence can play an important role during a pandemic. It may form the basis of a new hypothesis - point us in the direction of a potential new treatment - which can be investigated further.
What to know about evidence from studies
Clinical studies are part of evidence-based medicine. They help us to answer specific questions we have about drugs, diagnostics and diseases for example. There are a number of different types of study designs, which fall into one of two broad categories:
Observational studies (epidemiological studies), including cohort, case-control and cross-sectional studies among others
Interventional studies (experimental studies), including community and controlled clinical trials
Different study designs work better, or are more appropriate, in different circumstances.
As the name suggests, observational studies are those where researchers observe people to understand the effects of what it is they are studying, which could be a diagnostic test or risk factor for example. The researchers don’t manipulate or intervene in the situation. These types of studies are also called epidemiological studies.
Observational studies may have a retrospective design - that is to say they look back into the past via patient notes or by interviewing patients - but they can also be prospective and follow the patients going forward in time.
Observational studies can be used to look at disease prevalence, or the number of people with a disease, either at a set point in time or over a period of time. They can also be used to look at the incidence, or the frequency or rate at which a disease occurs, telling us how many new cases accumulate over a period of time.
Observational studies often measure the association or links between two factors and report the results in the form of an odds ratio. For example, a study might be used to measure the relationship between eating a particular food and having a certain disease.
Cohort, case-control and cross-sectional studies
Common observational study types include cohort studies, case control studies and cross sectional studies.
Cohort studies: This type of study looks at a cohort, or group of participants with a shared characteristic, and follows them over a period of time (longitudinally). When the study begins, none of the participants should have the outcome being measured, which might be heart disease for example. The presence or absence of risk factors (such as high body weight or blood pressure) is documented in each participant at the beginning and then they are all typically followed going forward (prospective). Cohort studies can be expensive and time consuming, but can also be useful in situations where it would be unethical to enroll people in a trial and tell them to do something harmful like smoking.
Case-control studies: Case control studies are different to cohort studies in that two groups of participants are recruited - one with a disease (or outcome) and one without. Researchers match the two groups so that they are the same in all other respects, such as age, sex and other factors, except for the disease being studied. These studies look backwards in time (retrospective) and can also be useful when looking for risk factors, but their results may be less reliable than other study types. This is because it can be difficult to establish whether it was the risk factor or the disease that occurred first (temporality), or because of other factors such as bias in the selection of the participants or extra variables that hadn't been accounted for (confounding factors). Case-control studies are quick and cheap to conduct and are useful when studying rare diseases.
Cross-sectional studies: Cross sectional studies are carried out over a short time period or at a set point in time. They look for links between risk factors and the development of a disease, by analyzing information from participants all at one point in time. They are also quick and cheap to conduct, but their results may not be as useful as those obtained from other observational study types.
Interventional studies, which are sometimes called experimental studies, are ones in which the researchers intervene, assigning study participants to a particular intervention and then watching for certain outcomes. Interventional studies have a prospective design, following the participants forward through time, collecting information as they go. This sort of study is similar to a cohort study with a prospective design, but differs in that the researchers actively assign participants to an intervention rather than just observing people.
Interventional studies often allocate participants randomly (randomize) to receive the intervention or to be part of a comparison or control group. Some interventional studies, however, are non-randomized and don’t carry out this process. Interventions can include all sorts of things, such as a drug therapy, vaccine, diagnostic test or educational program. The intervention can be used to prevent disease (preventative) or provide treatment (therapeutic).
Community and clinical trials
There are two broad groups of interventional studies, including community or field trials and clinical trials. Community trials involve assigning whole communities to receive different interventions, while clinical trials assign individuals to one of or more groups.
Clinical trials are further categorized by phase. For example, drug therapies start out in small phase I trials designed to determine the best dose, before moving through to phase II and larger phase III trials to confirm the efficacy and safety of the drug, before seeking marketing approval from the US Food and Drug Administration (FDA). Phase IV trials might also be conducted after the drug is marketed to continue to monitor the safety of the drug.
A well-designed trial can provide proof that an intervention really works and also inform us about its downsides. It can provide information that enables us to make informed decisions about the medications we take and decide if the benefits are worth the risks. For example, an experimental drug with unknown side effects might be an appropriate choice for a terminally or critically ill patient, but the same drug might be considered too big of a risk for someone with mild disease.
Clinical trials allow researchers to control the environment in which an intervention is being studied and minimize variation between the different intervention groups. This gives researchers more certainty that any differences observed between the groups are due to the intervention itself, rather than some bias, confounding factors or just plain old chance. They do all this with the help of statistics, a form of mathematical analysis.
When interventional studies measure risk or association they often report a hazard ratio, which is a measure of relative risk. A hazard ratio involves looking at the risk of an outcome in one group compared with the risk of an outcome in another group at a particular point in time. A clinical trial, for example, might report a hazard ratio that tells us that patients treated with a new drug were ‘x’ amount more likely to have recovered compared with those treated with an older drug.
Randomized controlled trials
Here we cover some basic features of randomized controlled trials (RCTs), which provide us with ‘gold standard’ evidence concerning the efficacy and safety of an intervention, and are commonly used to investigate new drug treatments or therapies.
RTCs typically enroll a select group of patients and randomly assign them to two different treatment groups to study how effective and safe a new drug treatment is. One group, called the control group, usually receives the current standard treatment or placebo and the other receives the drug being studied. As mentioned above, the two groups are intended to be comparable overall, with the only difference between them being the study drug. This is done to help ensure that any difference in the endpoint - the event or outcome being measured - is due to the effects of the drug and not chance or some other factor.
Basic Features of Randomized Controlled Trials
|Randomization||Refers to the process of randomly assigning participants to a particular intervention group. This could be achieved by a coin toss for example||Produces two comparable groups, which enables the effects of the intervention to be measured. Removes potential bias|
|Control group||A group to which the experimental group is compared. The control group may receive a placebo or sham, the current standard therapy, a combination of the two or something else||The control group provides a baseline that the effect of the study intervention can be compared against|
|These are predefined events that when they occur should show the intervention improves survival, results in a measurable benefit to a patient, or decreases the chance of developing a disease or complication. Trials can have multiple endpoints||A well chosen endpoint ensures that the trial will inform us of whether or not the new intervention will be of benefit to patients|
|Sample size||This is the number of participants required. It needs to be carefully chosen to ensure it’s not so big that the trial is too expensive or can’t enroll enough patients, or so small that the trial doesn’t have enough power to detect the effect of the treatment||Selecting an optimal sample size ensures that just the right number of participants are enrolled to give the study enough power. In clinical trial statistics, power is the probability of finding an effect from your intervention if there is one to be found|
|Blinding||Refers to the process by which one or more parties involved in a trial (for example the participants or doctors) are kept unaware of which intervention is being administered. A double-blind study is when both the doctors and participants are kept unaware of which group a participant is assigned to.||Blinding minimizes bias. The expectations of doctors and participants can not influence the results in the same way if they don’t know which intervention is being administered|
|This involves analyzing the data from all participants enrolled and randomly assigned to a group||This type of analysis helps prevent bias, gives a more reliable estimate of the interventions effectiveness, ensures sample size is maintained and more. For example, it means that participants who withdraw from a drug trial because of a poor outcome or side effects will still be included in the results, which helps inform us about the true effects of the drug|
Despite the pandemic, we still need clear proof that drugs work
Without unbiased assessment of the outcomes - without unbiased evidence from clinical trials - we really don’t know whether a drug treatment will work for a particular disease or not. While anecdotal evidence has an important role to play, relying on it too much may mean we’re not only wasting money on an ineffective treatment, but it could also mean that we suffer serious side effects unnecessarily. We potentially have a lot to lose.
In a podcast focused on clinical trials during a pandemic, Dr. Janet Wittes, president and founder of Statistics Collaborative, mentions that it’s really hard to conduct a clinical trial in this sort of situation and stresses the importance of protecting patients and healthcare workers. While some niceties, such as blinding, might need be be dropped, she highlights the importance of thinking about what evidence you need to reach a conclusion and the need to keep the fundamentals of clinical trial design including:
To randomize patients to the control and experimental groups
Enabling unbiased collection of outcome or endpoint data
To be clear on who is enrolled in the trial
Employing an adaptive trial design, which can be modified as the trial progresses, may also prove to be a more efficient, informative and ethical approach at this time.
- Limb CJ. The Need for Evidence in an Anecdotal World. Trends Amplif. 2011 Mar; 15(1-2): 3–4. doi: 10.1177/1084713811425751.
- InformedHealth.org. What is evidence-based medicine? Updated September 8, 2016. Available at: https://www.ncbi.nlm.nih.gov/books/NBK279348/. [Accessed May 4, 2020].
- PennState. Eberly College of Science. STAT 509. Design and Analysis of Clinical Trials. 1.1 What is the role of statistics in clinical research? Available at: https://online.stat.psu.edu/stat509/node/2/. [Accessed May 4, 2020].
- EUPATI. The role of statistics in clinical trials. Available at: https://www.eupati.eu/clinical-development-and-trials/role-statistics-clinical-trials/. [Accessed May 4 2020].
- Sheldon TA. Estimating treatment effects: real or the result of chance? Evidence-Based Nursing. April 2000. Volume 3, Issue 2. http://dx.doi.org/10.1136/ebn.3.2.36.
- Cancer Research UK. Types of clinical trials. Available at: https://www.cancerresearchuk.org/about-cancer/find-a-clinical-trial/what-clinical-trials-are/types-of-clinical-trials. [Accessed May 4, 2020].
- Thiese MS. Observational and interventional study design types; an overview. Biochem Med (Zagreb). 2014 Jun; 24(2): 199–210. doi: 10.11613/BM.2014.022.
- Centers for Disease Control and Prevention (CDC). Interpreting Results of Case-Control Studies. Available at: https://www.cdc.gov/training/SIC_CaseStudy/Interpreting_Odds_ptversion.pdf. [Accessed May 4, 2020].
- Spruance SL, Reod JE, Grace M, Samore M. Hazard Ratio in Clinical Trials. Antimicrob Agents Chemother. 2004 Aug; 48(8): 2787–2792. doi: 10.1128/AAC.48.8.2787-2792.2004.
- Ranganathan P, Aggarwal R. Study designs: Part 3 - Analytical observational studies. Perspect Clin Res. 2019 Apr-Jun; 10(2): 91–94. doi: 10.4103/picr.PICR_35_19.
- Ranganathan P, Aggarwal R. Study designs: Part 4 – Interventional studies. Perspect Clin Res. 2019 Jul-Sep; 10(3): 137–139. doi: 10.4103/picr.PICR_91_19.
- Food and Drug Administration (FDA). Sullivan EJ. Clinical Trial Endpoints. Available at: https://www.fda.gov/media/84987/download. [Accessed May 4, 2020].
- Pallmann P, Bedding AW, Choodari-Oskooei B, et al. Adaptive designs in clinical trials: why use them, and how to run and report them. BMC Med 16, 29 (2018). https://doi.org/10.1186/s12916-018-1017-7.
- The Lawfare Podcast. Mom and Dad Talk Clinical Trials in a Pandemic. April 21, 2020. Available at: https://www.lawfareblog.com/lawfare-podcast-mom-and-dad-talk-clinical-trials-pandemic. [Accessed May 4, 2020].