• Studies designed as randomized controlled trials (RCTs) are intended to produce high-quality evidence and are crucial tools in the assessment of medical cannabis and cannabinoid efficacy and safety. However, high risk of bias was demonstrated by the majority of medical cannabis and cannabinoid RCTs in a comprehensive meta-analysis. Risk of bias makes it difficult to interpret and apply findings from these studies.

  • Risk of bias introduced into and observed in cannabis RCTs warrants scrutiny, and standardized tools are recommended when reviewing RCT reports of findings. Clinical practice should ideally be altered only when sufficient evidence is available and an understanding of the “levels of scientific evidence” and common limitations to RCTs should be communicated to clinicians.

  • Patients should be informed that no RCT should influence their behaviors. Open lines of communication with their physicians and other healthcare providers can help facilitate more informed consumption of media coverage and other dissemination of research findings.

RCTs are often referred to as the “gold standard” of evidence and the impact of RCTs cannot be understated. Results from RCTs inform clinical guidelines, lead to approval of new drugs, and are often disseminated to the general public through media coverage. While the strength of RCTs is in randomization of subjects to treatment or control, which mitigates confounding concerns, there remain other threats to the validity of RCTs that often limit our interpretation of the findings.

Among the first and most impactful meta-analyses of cannabinoid clinical trials, Whiting et al. [1] conducted a systematic review and meta-analysis of RCTs that evaluated cannabinoids or cannabis for broad medical uses. While more recent and more focused meta-analyses exist for individual conditions, for the purposes of this discussion, this broadly focused article provides a striking metric by which to benchmark the quality and rigor of RCTs that employ cannabis or cannabinoids as medical treatments. The authors identified and evaluated 79 RCTs; however, only 4 (5%) of these were graded as “low risk of bias.”

RCTs are randomized to mitigate confounding, meaning that randomization ensures that factors potentially related to both exposures and outcomes are equally distributed between the treatment and control groups. Bias in RCTs is therefore a result of systematic error, intentional or not, that can be introduced at any point in a study. Biases may be introduced via the design, during data collection, analysis, and reporting. Each of these potential sources of bias can distort the treatment effects under study and make it difficult to interpret results.

The Cochrane Risk of Bias (RoB) tool [2], and recently updated RoB 2.0 [3] (Table 1), is a widely used instrument by which we can measure bias in a published RCT. The original RoB tool used to assess risk of bias in the Whiting et al. [1] meta-analysis included 6 domains: (1) random sequence generation, (2) allocation concealment, (3) participant and personnel blinding, (4) outcome assessor blinding, (5) incomplete outcome data, and (6) selective outcome reporting. Each domain is categorized by raters as low, high, or unclear risk of bias and together encompass the minimum procedures recommended by an international committee to avoid bias in RCTs.

Table 1.

Cochrane Risk of Bias Tool domains and examples of low and high risk approaches [2]

Cochrane Risk of Bias Tool domains and examples of low and high risk approaches [2]
Cochrane Risk of Bias Tool domains and examples of low and high risk approaches [2]

The rated RoB categories for the 79 studies reviewed by Whiting et al. [1] can be found in the appendix of that article. The authors noted that more that 50% of trials reported substantial withdrawal of patients from the study but did not account for this in the analysis. Many studies had incomplete methods reporting, notably missing; how treatment was blinded to participants or outcome assessors, as well as omission of outcomes prespecified in a study protocol but not reported in the publication. The Whiting et al. [1] review found several RCTs introduced new outcomes not specified in the study protocol, or deviated from the selection of primary outcome from the protocol. Other protocol deviations were frequently noted such deficiencies in trial design or methods or simply how these elements were reported in the article. For example, lack of a proper randomization could be attributable to poor planning and design of the trial (an irreversible decision) or not sufficiently reported. Such omissions at publication can easily be avoided by thorough reporting. Selective outcome reporting can similarly be avoided by including complete results of a trial rather than selecting only those of most interest or only those that are statistically significant.

The effect of these biases is challenging to interpret but contributes to diminished confidence in RCT results when present within a substantial portion of reviewed RCTs. The potential consequences of publishing poorly designed, conducted, or reported cannabis RCTs are dependent on the direction in which the results are biased. If a treatment is shown to be beneficial when in fact it is not, the long-term impact could contribute to mistrust among the medical community and general public in cannabis or cannabinoid therapies and possibly lead to patient harm. In the reverse situation, if a treatment is truly beneficial but is shown not to be, there could be delays in therapy and unmet medical need.

Cannabis research must be held to the same standards and rigor expected of other clinical areas. The responsibility of designing, conducting, analyzing, and reporting cannabis RCTs starts with the trialists and study authors. However, journals and peer reviewers also play an important role as gatekeepers in ensuring only the highest quality evidence is published. Tools such as the CONSORT Statement [4] are typically required at the time of RCT submission to most leading medical journals and should be a minimum standard to ensure adequate reporting. Registration of RCTs is typically a requirement and any deviations from registered protocols should be clearly noted in published articles and thoroughly vetted during peer review. While designed for assessing already published RCTs, Cochrane’s RoB 2.0 tool could also be used at the design and reporting stages to ensure common threats to validity are avoided prior to conducting and publishing the results from the RCT.

While RCTs garner significant attention, it is worth noting that the clinical evidence pyramid is not capped by RCTs (Fig. 1) nor should clinical decisions be made based solely on the results of a single RCT. Rather, the highest level of evidence is meta-analyses such as the one conducted by Whiting et al. [1]. Meta-analyses are informative for many reasons but primarily serve to combine evidence from multiple trials while providing an objective, systematic quality assessment of these studies. Meta-analyses are especially critical in cannabis research as small sample sizes often limit interpretation of single studies while pooled results help overcome these limitations.

Fig. 1.

Clinical evidence pyramid showing increasing quality of evidence. RCT, randomized control trial.

Fig. 1.

Clinical evidence pyramid showing increasing quality of evidence. RCT, randomized control trial.

Close modal

Meta-analyses, however, depend on high-quality evidence to generate high-quality summaries and extract definitive conclusions regarding treatment effects and adverse events. Authors of RCTs risk exclusion of their study when the quality of an RCT design, implementation, or the subsequent reporting is insufficient. Low quality RCTs and low quality reporting of RCT methods and results may also have far-reaching impacts on patients. Those conducting RCTs on cannabis or cannabinoids should ensure, at a minimum, there they include appropriate methods and reporting of randomization, allocation concealment, blinding of patients and assessors, planning for and evaluation of participant withdrawal and attrition, and outcome reporting. The cannabis clinical research field must continue making strides towards high-quality design, conduct, analysis, and reporting of cannabis RCTs.

The authors have no conflicts of interest to declare.

The authors are supported by State of Florida appropriations to the Consortium for Medical Marijuana Clinical Outcomes Research (mmjoutcomes.org).

J.B. conceptualized and drafted the work. A.G. drafted and revised. Both approved the final version.

Evidence in Context is part of the outreach effort of the Consortium for Medical Marijuana Clinical Outcomes Research to examine and discuss implications of research into cannabis and cannabinoids for clinical practice, thus providing a translational approach to these studies to make clear, concise, and actionable evidence available for clinicians and patients.

1.
Whiting
PF
,
Wolff
RF
,
Deshpande
S
,
Di Nisio
M
,
Duffy
S
,
Hernandez
AV
, et al
Cannabinoids for medical use: a systematic review and meta-analysis
.
JAMA
.
2015
;
313
(
24
):
2456
73
. .
2.
Higgins
JP
,
Altman
DG
,
Gøtzsche
PC
,
Jüni
P
,
Moher
D
,
Oxman
AD
, et al
The cochrane collaboration’s tool for assessing risk of bias in randomised trials
.
BMJ
.
2011
;
343
:
d5928
. .
3.
Sterne
JAC
,
Savović
J
,
Page
MJ
,
Elbers
RG
,
Blencowe
NS
,
Boutron
I
, et al
RoB 2: a revised tool for assessing risk of bias in randomised trials
.
BMJ
.
2019
;
366
:
l4898
. .
4.
Boutron
I
,
Altman
DG
,
Moher
D
,
Schulz
KF
,
Ravaud
P
,
Group
CN
.
CONSORT statement for randomized trials of nonpharmacologic treatments: a 2017 update and a CONSORT extension for nonpharmacologic trial abstracts
.
Ann Intern Med
.
2017
;
167
(
1
):
40
7
. .

Joshua D. Brown and Amie J. Goodin are members of the leadership team of the Consortium for Medical Marijuana Clinical Outcomes Research.

Open Access License / Drug Dosage / Disclaimer
This article is licensed under the Creative Commons Attribution-NonCommercial 4.0 International License (CC BY-NC). Usage and distribution for commercial purposes requires written permission. Drug Dosage: The authors and the publisher have exerted every effort to ensure that drug selection and dosage set forth in this text are in accord with current recommendations and practice at the time of publication. However, in view of ongoing research, changes in government regulations, and the constant flow of information relating to drug therapy and drug reactions, the reader is urged to check the package insert for each drug for any changes in indications and dosage and for added warnings and precautions. This is particularly important when the recommended agent is a new and/or infrequently employed drug. Disclaimer: The statements, opinions and data contained in this publication are solely those of the individual authors and contributors and not of the publishers and the editor(s). The appearance of advertisements or/and product references in the publication is not a warranty, endorsement, or approval of the products or services advertised or of their effectiveness, quality or safety. The publisher and the editor(s) disclaim responsibility for any injury to persons or property resulting from any ideas, methods, instructions or products referred to in the content or advertisements.