Alcohol and other drug use (AOD) disorders are associated with a significant public health burden, including mortality, economic burden, and adverse social- and individual-level consequences [1,2]. They are highly comorbid with other mental health disorders [3] and can have profound negative effects on psychosocial functioning, including overall mental health and quality of life [4]. Although vastly underused, there are a number of effective treatments available for AOD. Among them, cognitive behaviorally based interventions (CBIs) are the most widely researched and clinically practiced interventions for AOD. CBIs (ie, cognitive behavioral therapies or cognitive behavioral therapies with elements of other evidence-based modalities such as motivational interviewing) consist of a broad range of behavioral interventions, including those targeting operant learning processes, cognitive and motivational strategies, and skills training targeting interpersonal, emotion regulation, and organizational or problem-solving deficits [5]. Multiple meta-analyses have provided evidence supporting CBI efficacy in treating AOD [6], including when combined with pharmacotherapy [7], and for co-occurring disorders [8].

Despite its demonstrated efficacy, barriers such as limited access to trained providers, costs of training, geographic limitations, and stigma impede the widespread delivery and uptake of CBIs in traditional settings [9,10]. To overcome these challenges, digital platforms (ie, computer-delivered, web-based, or mobile platforms) have emerged as a promising modality for expanding access to evidence-based care. Given its structured, problem-focused, and time-limited format, CBIs are among the most common therapeutic approaches in digital mental health interventions [11]. These digitally delivered cognitive behaviorally based interventions (dCBIs) aim to replicate the core components of CBIs in a highly scalable and often self-guided format. A body of research has emerged over the past 2 decades indicating support for dCBIs (also referred to as internet-based cognitive behavioral therapy [CBT] and technology-delivered CBT) in treating AOD. Systematic reviews and meta-analyses indicate a small, significant effect of dCBIs on reducing substance use compared to control conditions [12-14]. However, the impact of these interventions on outcomes beyond substance use is relatively unknown. Of particular interest are indicators of cognitive, behavioral, psychological, and lifestyle changes that are targeted by dCBIs that may promote recovery from AOD [15]. Evidence of a positive effect on psychosocial outcomes (eg, mental health, emotional well-being, coping strategies, daily functioning, and quality of life) would enhance the clinical significance of changes in substance use.

The aim of this meta-analysis is to evaluate dCBI effects across 2 broad classes of outcomes for AOD: substance use consumption and psychosocial functioning. These outcome classes were the primary subgroup variable in this study. Because the experimental comparator is a common predictor of effect size variability in meta-analyses [16], the comparator type was the second primary subgroup variable of interest. As a follow-up and in cases of significant statistical heterogeneity within subgroups, regression-based moderator analyses were undertaken. Additional sensitivity analyses were conducted (eg, risk of bias assessment and publication bias assessment) and are reported following the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines.

A librarian-supported literature search was conducted through October 2022 (with an updated search to October 2024) to identify studies for a meta-analysis project on CBIs for AOD. Specifically, a title, abstract, and keyword search by treatment (eg, “cognitive behavioral therapy”), outcome (eg, “alcohol”), and methodological (eg, “randomized clinical trial”) terms was conducted in the PubMed database (sample search strategy can be found in Multimedia Appendix 1). Other databases that were searched included Cochrane CENTRAL and EBSCO (ie, CINAHL, APA PsycINFO, and SocINDEX). Abstract screening occurred in Covidence and was conducted by 4 raters, with 2 raters (OT and LW) for each study, and consensus decisions were made by the lead study investigator (MM). Because varying modes of CBI delivery (eg, delivered via technology, delivered in-person, and combined with other behavioral modalities) were of interest to the larger project, the literature search yielded 1387 primary and secondary clinical trial reports that were downloaded and reviewed for eligibility in 1 or more meta-analyses. Additional steps included a bibliographic review of relevant review papers [17-30] and a clinical trial number search to derive all available publications on a given primary study sample.

Study inclusion criteria were as follows: (1) English language publication, (2) peer-reviewed publication, (3) publication date between 1970 and 2024, (4) randomized controlled trial, (5) adult patient population (median age ≥18 years), (6) patient population meeting criteria for an alcohol or other substance use disorder (ie, Diagnostic and Statistical Manual [DSM] III-R through 5 [31-34]) or at-risk use based on an established criterion (eg, Alcohol Use Disorder Identification Test score >8 [35]), and (7) a cognitive behavioral or combined cognitive behavioral (ie, combined with elements of other evidence-based AOD treatments) intervention delivered in a digital and/or mobile health format (ie, via a computer software program, an internet site, or a mobile phone–based app; see Multimedia Appendix 2 [36-100] for details on dCBI interventions).

There were several study descriptors or moderator variables of interest to this meta-analysis. Because the type of comparator is often a primary driver of effect size magnitude [6,16], effect sizes were pooled overall and by the following 4 between- and 1 within-condition comparator type, respectively: (1) dCBI at follow-up compared to assessment only or minimal treatment (eg, a pamphlet or a noninteractive website with AOD information) at follow-up, (2) dCBI at follow-up compared to another treatment modality (eg, other online intervention and a control treatment) or treatment as usual at follow-up, (3) dCBI added to treatment as usual at follow-up compared to treatment as usual at follow-up, (4) dCBI at follow-up compared to CBI delivered by a counselor at follow-up, and (5) dCBI at follow-up compared to dCBI at baseline. Study characteristic variables were organized by demographic, clinical, treatment, and study method factors. Demographic variables included mean age of participants, percentage of female participants, percentage of participants belonging to 1 or more racial groups (eg, Asian, Black, and White), percentage of participants belonging to the Latine ethnic group, percentage of participants identifying a sexual orientation other than heterosexual, percentage of participants identifying as transgender or other non-cisgender identities, percentage of participants with some college education, and percentage of participants who are full- or part-time employed. Clinical variables included the primary drug targeted (ie, alcohol, cannabis, opioid, polydrug, stimulant), substance use severity (ie, inclusion of individuals with use disorder vs at-risk consumption), and 1 or more co-occurring conditions (eg, depression and pain) as a sample inclusion criterion (ie, yes vs no). Treatment variables were dCBI format (ie, CBI only vs CBI combined with other evidence-based treatment elements), dCBI combined with pharmacotherapy (ie, yes vs no), clinically supported dCBI (ie, yes vs no), and treatment length (ie, number of content modules). Study methodological factors included study sample size, publication year, context (ie, community sample, specialty substance use or mental health clinic, other setting; eg, primary care clinic and college campus), publication country (ie, United States and other countries), and study-level risk-of-bias score [101]. Data extraction guidelines were detailed in a study codebook available, upon request, from the last author (MM). Data were extracted using consensus methods between the third and fourth authors (OT and LW; agreement rate 79.9%). The last author resolved disagreements, with input from members of the full study team as needed.

Hedges g is a standardized mean difference with a correction for a slight upward bias in the estimated effect size when samples are small [102].

Effect sizes were pooled using inverse variance weighting and a random effects model. Inverse variance weighting allows larger sample studies more weight in the overall pooled effect estimate [104]. A random effects model assumes a distribution, rather than a single population effect size [105]. For hypothesis testing, sources of dependency from multiple study-level effect sizes (ie, multiple study comparators, outcomes, and/or time points) were handled via robust SEs. Specifically, an approximated covariance matrix is derived from the product of within-study residuals and used to calculate cluster-robust SEs and CIs under a t-distribution [106]. The within-study correlation between dependent effects was assumed to be constant (ie, default ρ of 0.8), and sensitivity analyses were conducted that varied the correlation value. Cluster-robust SEs also gain accuracy as the sample of primary studies increases, but a small-sample correction allows for stable estimates in samples as small as 10 primary studies [107]. Meta-analysis was performed in the robumeta package for R (R Foundation for Statistical Computing) [106].

The random effects and pooled effect sizes in this study were calculated for 2 outcome classes: consumption (ie, abstinence, use frequency, heavy use frequency, peak consumption, quantity, dependence severity, and other drug use) and psychosocial functioning (ie, consequences, cognitive outcomes, coping outcomes, mental health symptoms, general health, and quality of life). These outcome classes as well as comparator type were the primary subgroup moderators of interest. However, in cases of significant residual study heterogeneity (ie, ≥50% [105] of between-study variance relative to sampling variance measured via the I² estimate and k≥10 [108]), regression-based moderator analyses by time point (ie, posttreatment to 3 months vs 4 months or later) and primary drug targeted (ie, alcohol, polydrug, and other) were undertaken in attempt to explain additional systematic variation between studies. Prediction intervals were additionally calculated in the case of statistical heterogeneity to obtain a range of potential future effect sizes that incorporate the systematic variability in the data [109]. We considered these sensitivity analyses as a method for examining the validity of our a priori subgroups. Additional sensitivity analysis included effect size pooling for each of the 13 outcome types, analysis of influential studies (ie, studies that, if removed, would change the substantive interpretation of the effect size), and analyses using traditional meta-analytic methods (ie, average dependent effect sizes to the study level and then calculate random effects pooled effect sizes and corresponding hypothesis test statistics). To test for potential publication bias, the relationship between the inverse of the SE and effect size was examined with graphical plots and a rank-order correlation [110]. In this analysis, asymmetry within the plot (ie, the majority of studies located at the bottom right quadrant) and a negative correlation would suggest publication bias. Finally, the risk of bias results is summarized for the sample of primary studies.

All sample-level study characteristics are summarized in Table 1, while characteristics at the individual study level are in Table 2. The sample included 65 randomized trials (K=110 publications; 753 effect sizes [consumption: 475; psychosocial: 278]) of dCBI for adult AOD published between 1997 and 2024 [36-44,46-100,111-150]. The mean sample size was 371.46 (SD 997.65) participants with a minimum of 20 [36] and a maximum of 7935 [37]. For participant characteristics, the samples’ mean age was 37.69 (SD 8.88) years, and samples were 45.63% female (SD 19.89%; 100% for 2 studies [38,39]) on average. Among studies that reported racial and ethnic data, the mean samples were 21.77% Asian or Pacific Islander (SD 38.92%; 100% Asian in Liang et al [111]), 34.24% (SD 24.11%) Black, 15.49% (SD 28.67%) Native American or Indigenous person, 59.62% (SD 24.43%) White, and 6.99% (SD 5.84%) multiracial. The mean samples were 20.25% (SD 22.79%; 100% in Paris et al [40]) of Latine ethnicity. Additional sample characteristics included sexual orientation, sex identity, education, and employment status. While reporting on these factors was relatively infrequent across studies, these descriptive data are summarized in Table 1.

The primary substance targets were alcohol use (35/65, 53.85%), followed by polydrug use (13/65, 20%), with a majority of samples meeting criteria for an alcohol or substance use disorder (40/65, 61.54%). Further, 15.38% (10/65) of studies targeted a co-occurring condition, such as posttraumatic stress disorder [41] or depression [42]. While full details on study interventions can be found in Multimedia Appendix 2, the mean length of treatment was 12.20 (SD 15.17) modules or exercises, 36.92% (24/65) included some form of clinical support, such as online support [43] or feedback on homework [112], and over half of the interventions included elements of other evidence-based treatments (eg, motivational interviewing). For study context, participants were mostly recruited from the community via online advertising (30/65, 46.15%), followed by specialty mental health or addictions treatment facilities (17/65, 26.15%). A majority of studies took place outside of the United States (39/65, 60%). Finally, study-level risk of bias results showed that most studies had at least 1 high risk designation (which we then designated as high risk overall), and these were typically due to a lack of participant or personal blinding or incomplete outcome data (Multimedia Appendix 5).

This study considered outcome class and comparison type as subgroup moderators to derive pooled effect sizes describing dCBI for AOD. However, significant residual heterogeneity was observed when dCBI was compared to minimal treatment (consumption and psychosocial outcomes), another treatment (consumption outcomes), when CBI was added to usual care and compared to usual care alone (psychosocial outcomes), and when within-dCBI effect sizes were estimated (consumption and psychosocial outcomes). In these sensitivity analyses, outcome time point (ie, posttreatment to 3 months vs 4 months or later) and primary drug targeted (ie, alcohol, polydrug, and other) were selected as study-level moderators in multivariate, correlated effects meta-regression models. For dCBI consumption outcomes in contrast to a minimal control, early follow-up studies were associated with smaller effect sizes than later follow-up studies, and polydrug studies showed smaller effects than alcohol studies. For dCBI psychosocial outcomes in contrast to a minimal control, polydrug studies also showed smaller effects than alcohol studies. The remaining pooled effect estimates, time point, and primary drug targeted were not significantly associated with effect size magnitude (see Table 3 for details).

In this meta-analysis, 13 outcomes reported across 65 clinical trials of dCBI for AOD were aggregated into consumption and psychosocial outcome classes. In this final sensitivity analysis, we examined each outcome type individually to report pooled effect direction, magnitude, significance, and relative heterogeneity. The results of these analyses are separated by between- and within-group effect size types and are shown in Figures 2 and 3, respectively. Figure 2 shows between-group effects, where all pooled effect sizes were positive and mostly small in magnitude. Statistically significant pooled estimates were observed for abstinence, frequency, quantity, use severity, consequences, other drug use, and mental health, but not heavy frequency, peak consumption, cognitive outcomes, coping outcomes, general health, and quality of life. Estimates of between-study variance, relative to sampling error, showed that heterogeneity was still above moderate in most of these estimates. Figure 3 shows within-group effects, where most pooled effect sizes were positive and large in magnitude. Statistically significant pooled estimates were observed for frequency, heavy frequency, peak consumption, quantity, use severity, consequences, other drug use, mental health, and quality of life, but not abstinence, cognitive outcomes, coping outcomes, and general health. Estimates of heterogeneity were above moderate for most subgroup effect sizes.

This is the first meta-analysis to evaluate the effect of dCBIs for AOD that included measures of psychosocial functioning as well as substance use consumption. The results generated here from 110 publications, involving 65 randomized controlled trials with over 24,000 combined participants, support the efficacy of dCBI at improving substance use and psychosocial outcomes with pooled findings suggesting a small relative benefit. However, more nuanced findings for between-group effects were apparent when studies were separated according to the type of comparison condition, with dCBI showing limited efficacy when compared to another treatment, in-person CBI, and when added to usual care (ie, nonsignificant efficacy in relation to psychosocial outcomes only). In total, 6 of 8 pooled between-group effect sizes showed statistical heterogeneity that was not fully accounted for by the time point of follow-up or the primary drug targeted within the clinical trial. Here, prediction intervals contained a wide range of consumption and psychosocial effect sizes that could be observed for dCBI for AOD in reference to minimal treatment, another treatment, or when added to usual care. Together, these sensitivity analyses suggest that there are unknown methodological or patient population factors driving variability in effects that were not identified by this study. Finally, within-condition effects that provide a nonexperimental estimate of change from baseline indicated significant, heterogeneous, and moderate to large effects for dCBI on consumption and psychosocial outcomes. These effects suggest that, on average, 60% to 80% of individuals who received dCBI showed improvement from baseline across various measures of substance use and functioning.

As is often the case for meta-analyses of behavioral therapies for substance use disorders [151], including CBT [6], there is little evidence of superior efficacy of dCBI when directly compared with another therapy. Significant and small between-group effects of dCBI were observed for both consumption and psychosocial outcomes when compared to assessment only or minimal treatment, with evidence suggesting larger effects in alcohol studies and for consumption outcomes, at earlier follow-up time points. Yet, dCBIs were not efficacious when compared to another treatment such as treatment as usual or CBI delivered by a therapist. This pattern is consistent with findings of digital mental health interventions more broadly [45,152], such that they appear generally effective at improving symptoms of psychological conditions but are not necessarily more effective than in-person psychotherapeutic interventions. In terms of comparative treatment benefit (ie, between-group effects), when dCBI was provided as an adjunct to treatment as usual, there were significant and small effects on consumption outcomes when compared to treatment as usual alone. However, this significant effect did not extend to psychosocial outcomes. It may be that the beneficial effects of adding dCBI to standard care are most relevant for improving AOD and not general life functioning. It should be noted that fewer studies in this between-group comparison reported psychosocial outcomes compared to consumption outcomes, and the magnitude of the effect size for this between-group comparison was similar to the effect size when all comparisons were combined, which was significant. Thus, sample size may have impacted the statistical inference on psychosocial outcomes for studies evaluating dCBIs as an adjunct to standard care. Furthermore, when one influential study [112] was removed from this analysis, the between-group effect reached statistical significance.

The findings with respect to substance use consumption outcomes are largely consistent with other meta-analytic examinations of dCBI (or related terminologies) for AOD. For instance, our prior meta-analysis of 15 published clinical trials of technology-delivered CBT for alcohol use found a significant and small effect as a stand-alone treatment in contrast to minimal treatment (g=0.20), but the effect was nonsignificant when compared to a more active control [14]. Most recently, Gregory et al [13] reported a small, statistically significant effect (g=–0.23) that favored dCBI in reducing substance use (alcohol, drugs, or alcohol or drugs combined) relative to controls in a meta-analysis of 18 published studies. Of note, the current meta-analysis is more comprehensive than existing reports both in terms of the number of studies included and the type and number of outcomes reported.

This is the first report of the pooled effect of dCBI on various psychosocial outcomes, including consequences of AOD, mental health symptoms, general health, quality of life, cognitive outcomes, and coping outcomes. This provides a more robust evaluation of dCBI efficacy and effects in terms of expanded definitions of recovery beyond alcohol or drug abstinence [15,153,154]. Although significant between-group effects demonstrating the efficacy of dCBI on psychosocial outcomes were more limited in scope as compared to consumption outcomes, it is notable that pooled between-group effect sizes for consequences and mental health outcomes were significant, as problems in these areas are often among the reasons for seeking treatment for AOD [155]. The within-group findings (ie, not compared to a control condition), while statistically heterogeneous, suggest that dCBI is effective at reducing other drug use and consequences from AOD, as well as improving mental health and quality of life. Thus, dCBI as an intervention can have a beneficial effect on some psychosocial outcomes, but the benefit across multiple domains may not be superior to that achieved through another established treatment for AOD.

This meta-analysis provides the most comprehensive view of dCBI outcomes to date. Therefore, a key strength of this work was the capacity to extract all available data from primary study reports and to report on 2 broad outcome classes—consumption and psychosocial—as well as 13 different outcome types. The extent of clinical effect data provided is substantial. However, there are some limitations to consider. For example, we prioritized published clinical trials to facilitate consistency in methodology, but it is unknown whether the inclusion of gray literature such as dissertations and conference abstracts would have changed our study findings. While our sample of studies is relatively large for a meta-analysis and certainly larger than dCBI reviews to date, sample size still may have impacted our capacity to observe statistical significance for some pooled effect estimates. Study reporting, however, is transparent as to the number of studies and effect sizes that each pooled effect represents. Reporting of demographic characteristics, especially race or ethnicity, was incomplete across studies, which restricts conclusions regarding generalizability. Limitations in sample sizes for certain outcomes also led to the grouping of measures such as motivation, craving, and self-efficacy into a single outcome type (ie, cognitive outcomes) when there are important conceptual differences between these measures. This limitation underscores the importance of future studies not only reporting all outcome data in descriptive format but also reporting all primary and secondary outcomes. As a result, more stable and informative pooled effect sizes would be possible in the future. It is also the case that some pooled estimates contained significant residual heterogeneity, which suggests that effect estimates were variable within groupings beyond what could be attributed to sampling error. While the primary drug targeted and the time point of follow-up were drivers in some studies, undiscovered moderators are possible. For instance, the varied characteristics of dCBIs included here (eg, range of dose and duration and integration with other evidence-based approaches) may have contributed to effect heterogeneity. The question of what defines CBT for AOD has become more challenging due to its evolution and diffusion over the years, resulting in new forms of CBT that may not fit traditional definitions but that have shared theoretical underpinnings, processes of change, and techniques [156]. Finally, the impact of user engagement on outcomes remains an unanswered question. Low engagement, or the lack of uptake and/or poor adherence, is one of the most widely cited challenges and is considered the greatest barrier to progress in digital mental health [157,158]. The lack of consistent reporting of rates of engagement with the dCBIs in the studies included here precluded an empirical investigation of this issue.

With these limitations in mind, the results of this meta-analysis show a modest benefit for dCBI as a stand-alone therapy and as an addition to usual care. Importantly, stand-alone effects were observed for consumption outcomes as well as measures of psychosocial functioning. Moreover, the comprehensive approach to outcome extraction resulted in effect estimates for a range of functional end points and suggests that across comparison conditions and length of follow-up, small effects can be observed on various consumption measures, including abstinence and use severity, and on psychosocial measures, such as consequences and mental health symptoms. When change from baseline is examined, the effects are moderate to large and statistically significant for all outcomes except for abstinence, cognitive and coping outcomes, as well as general health. Overall, the results of this meta-analysis provide further evidence supporting the beneficial effect, albeit small, of dCBIs in reducing substance use and suggest that the benefits may extend to some psychosocial outcomes. This is critical because recovery from AOD is not confined to substance use outcomes and should be considered holistically.