Feasibility, Acceptability, and Potential Efficacy of a Self-Guided Internet-Delivered Dialectical Behavior Therapy Intervention for Substance Use Disorders: Randomized Controlled Trial

Introduction

Background

Alcohol and substance use disorders (SUDs) are the leading causes of death and disability worldwide [1,2]. These conditions are often chronic, leading to elevated risks of co-occurring medical and mental health conditions, involvement with the criminal justice system, and loss of workplace productivity [1-4]. In 2019, the past-year use of alcohol, cannabis, tobacco, and illicit substances was 77%, 21%, 14%, and 3.6%, respectively, in Canadians [5]. Increased consumption during the COVID-19 pandemic in Canada and around the world has been linked to greater substance-related harms and concurrent mental health symptoms, such as depression, anxiety, and hopelessness [6]. Various evidence-based psychological treatments are available for SUDs; however, the availability and demand for these services come at a time when internet and mobile delivery formats are being promoted in care pathways [7]. These formats hold considerable public health promise in reducing the burden associated with SUDs. For example, a recent systematic review highlighted that existing mobile interventions were effective and rated as acceptable by people with SUDs [8].

Psychological Treatments for SUDs

Although pharmacological treatments exist for some substances (eg, alcohol and opioids), they have mixed evidence in treating other SUDs (eg, cannabis and stimulants [9]). Thus, psychological treatments remain a necessary therapeutic avenue for SUDs and may be particularly promising for those with multiple substance use concerns. Although psychological treatments vary greatly in their approach and theoretical framework, they tend to produce moderate effect size reductions in substance dependence [10,11]. To date, the greatest evidence supports cognitive behavioral and motivational enhancement approaches for treating SUDs.

SUDs rarely occur in isolation and often co-occur with depressive, anxiety, bipolar, and traumatic stressor disorders [12]. Psychological treatments are well suited to treat multiple conditions simultaneously when they incorporate a transdiagnostic focus or approach. There is growing consensus that people with SUDs, regardless of a specific substance, report higher difficulties in regulating their emotions compared with control samples and often use alcohol or other substances to cope with negative emotions [13]. More broadly, difficulties in emotion regulation appear to be a transdiagnostic risk factor underlying not only the development and course of SUDs but also depressive, anxiety, bipolar, and traumatic stressor disorders [14,15]. They also represent a promising treatment target, as emotion regulation skills tend to improve during psychological treatments for SUDs, along with more general improvements in self-efficacy and coping [16,17]. One psychological intervention that may be of substantial interest is dialectical behavior therapy (DBT), which was developed to treat individuals with high emotion dysregulation and includes comprehensive skills training in the domains of mindfulness, distress tolerance, emotion regulation, and interpersonal effectiveness.

DBT is a third-wave psychological intervention designed for patients with complex and severe behavioral, emotional, and interpersonal dysfunction [18,19]. DBT was first developed and found to be effective for severe clinical presentations related to suicidal behavior, nonsuicidal self-injury, and borderline personality disorder in adolescents and adults (refer to the study by Neacsiu et al [20] for review). Over time, DBT was reconceptualized as a transdiagnostic intervention appropriate for other mental health conditions and now includes specific content relevant to SUDs as well as other addictive behaviors [16,21,22]. Nevertheless, outpatient programs offering DBT are often safeguarded for those with acute suicide risk and behavioral problems. Importantly, although DBT was originally developed as a year-long multimodal intervention, evidence suggests that relatively brief formats focusing on DBT skills training (eg, 8-32 wk) are effective in treating SUDs, either as a primary condition or a co-occurring presentation in numerous clinical trials [16,22-24]. Despite these promising results, further research is needed to support the potential benefits of digital formats of DBT skills training, particularly within inclusive samples that reflect those seeking support for SUD.

Internet-Delivered DBT

Another way to increase the availability of DBT is through internet and mobile delivery formats. Thus far, research on internet-delivered DBT (iDBT) has been promising. In a review of 11 studies, iDBT was feasible and effective, although these results were based on small sample sizes, and few studies adopted a more rigorous methodology (eg, randomized controlled trials [RCTs] [25]). Various methods have been used, such as therapist-led sessions delivered via web-based videoconferencing [26], asynchronous material delivered via email [27], self-guided stand-alone websites [28,29], and therapist-guided programs [30]. Studies that evaluated potential efficacy suggested that iDBT was at least as effective as control conditions (waitlist or face-to-face) and was accepted by users. However, web-based delivery is not without harm or adverse events. One large-scale trial comparing integrated care management and skills training (ie, 4 self-guided DBT skills) for those with suicidal ideation found that the latter condition led to an increased risk of self-harm [31]. A discussion of the study suggested that it faced, among other issues, an implementation failure [32]. Thus, these and other considerations should be incorporated in future work.

In a seminal study, Wilks et al [30] evaluated therapist-guided iDBT in a sample of participants who are suicidal and alcohol dependent in a completely remote manner. This 8-week waitlist-controlled RCT delivered video trainings on mindfulness (2 wk), addiction (2 wk), emotion regulation (3 wk), and distress tolerance (1 wk) using an e-learning web-based platform along with handouts and worksheets delivered via email. The content was developed in collaboration with the developer of DBT. The intervention produced significant reductions in suicidal ideation, alcohol consumption, and emotion dysregulation. Although the treatment was deemed safe and acceptable to participants, there was substantial dropout, and technical issues were reported as a barrier to adherence [33]. Nevertheless, those who remained in the study reported that it was useful.

Following this work, a more advanced iDBT intervention called Pocket Skills (version 1.0) was created to overcome the accessibility and engagement issues encountered previously [34]. It is available through an internet browser on any device (ie, computer, tablet, or smartphone) and offers an interactive experience by using a chatbot along with embedded video lessons and practice. Pocket Skills 1.0 was evaluated in a single-arm trial as an adjunct intervention in individuals with a range of mental disorders completing in-person DBT for 4 weeks. The results of the study were promising, with both quantitative and qualitative evidence for its feasibility, acceptability, and potential use as an adjunct. We developed this study based on these 2 previous studies.

Current Study

This study aims to evaluate version 2.0 of Pocket Skills and advance the literature in several ways. First, the current investigation evaluates Pocket Skills 2.0, which includes some of the content from version 1.0, as well as revised and novel materials that have not yet been evaluated. Second, the delivery of iDBT in this study was predominantly self-guided, with limited therapist guidance compared with the previous trial that used iDBT intervention as a therapeutic adjunct [34]. Third, this investigation represented a more controlled study of Pocket Skills 2.0 as a stand-alone treatment in a sample of treatment-seeking adults with SUDs who were not receiving any other forms of psychological treatments. Finally, this investigation randomized participants to immediate versus delayed access to advance the previous single-arm study. A 12-week single-blinded parallel-arm waitlist-controlled RCT was initiated, with participants randomized to receive immediate access to the intervention or delayed access after 4 weeks. The 4-week intervention and follow-up periods are in line with previous implementations of self-guided digital mental health interventions [35-37]. These studies have found that attrition rates start to increase steadily after 4 weeks and especially after 7 to 8 weeks (eg, >50%).

Specifically, we hypothesized that greater than 50% of participants would start the intervention (H1a); not drop out early (H1b); engage with the intervention at a recommended dose of twice a week (or 8 d) in the first 4 weeks (H1c); and would still be using the intervention after 4 weeks (H1d). We also hypothesized that participants would rate the intervention as credible and acceptable on established measures (H1e). Second, we hypothesized that (H2a) participants in the immediate versus delayed iDBT group would show significantly greater improvements in our primary outcome of substance use dependence at the acute (week 4) and follow-up periods (week 12) in the form of an interaction effect (group×time). In addition, we hypothesized (H2b) significantly greater improvements for the immediate versus delayed iDBT group for our secondary outcomes (ie, depression, anxiety, emotion dysregulation, suicidality, functional disability, dispositional mindfulness, DBT skills, risky behaviors, and frequency of alcohol and substance use). Third, we hypothesized that iDBT would (H3) produce significant main effect improvements in both groups in the acute (week 4) and follow-up phases (week 12) of the intervention for all outcome measures.

Methods

Study Design

A 2-arm, single-blinded, parallel-group, preregistered RCT design was implemented, comparing individuals who received iDBT immediately with those who were first wait-listed for 4 weeks and then offered the intervention (delayed iDBT group). Assessments were completed at baseline and at 4 weeks, with additional follow-ups at 8 and 12 weeks. A CONSORT-EHEALTH (Consolidated Standards of Reporting Trials of Electronic and Mobile Health Applications and Online Telehealth) checklist was completed with more detailed information on the study design (Multimedia Appendix 1).

Ethical Considerations

All study procedures were approved by the Centre for Addiction and Mental Health research ethics board (#016/2021), and this research complied with the Declaration of Helsinki of 1975, as revised in 2000. Data is stored in a de-identified format to safeguard participant information.

Participants and Recruitment

Enrollment ran from August 2022 to March 2023, and all follow-ups were completed by June 2023. Participants were recruited from psychiatric hospital clinician referrals, waitlists, and research registries and from the surrounding community through several methods of advertisement (eg, hospital and other websites, social media posts, private DBT clinics, and local community organizations). All advertisements sought individuals who wanted to reduce their alcohol or substance use and specifically stated that they would be offered an internet-delivered intervention.

All prospective participants were initially informed about the study and were prescreened for eligibility over the phone. Inclusion criteria were as follows: (1) aged 18 to 65 years; (2) fluent in English; (3) understanding and willingness to comply with study requirements; (4) referred to addictions programming at our hospital or seeking treatment from the community, but not currently receiving any CBT or DBT intervention (support groups and psychiatric services were allowed); (5) alcohol or SUD in the past year; (6) use of alcohol or substance in the past month; (7) access to the internet (and assumed literacy); and (8) at least contemplation levels of wanting to reduce alcohol or substance use on the Contemplation Ladder measure [38]. Exclusion criteria included (1) any known practical factors that would preclude participation, (2) acute psychiatric (ie, suicidality, psychotic disorder) or medical condition (ie, acute intoxication or withdrawal) requiring medical attention, and (3) participation in another psychological intervention or treatment study. We did not exclude participants based on whether they were taking psychotropic medications or not.

Registration

The trial was registered with the ClinicalTrials.gov database (NCT05094440) on October 14, 2021. A revised registration was published on September 6, 2022, in line with changes to our protocol between our pilot study and this study. In this study, our analysis focused on the measures included in registration. One modification of the registered protocol was made, that is, the addition of a DBT skills measure at all time points to permit the evaluation of how this intervention was linked to changes in this key treatment target. Feasibility, acceptability, and engagement metrics were decided a priori for study implementation and were included in our study-specific protocol.

Randomization and Blinding

Participants were randomized to immediate or delayed iDBT using a blinded envelope system to ensure allocation concealment, with 3 randomization blocks (4, 6, and 8 participants). The randomization procedure was blinded to the participants and the experimenter who ran all baseline sessions (ARD). Thus, neither party knew which group the participant would be allocated to until after the informed consent and baseline procedures were completed. None of the participants withdrew immediately following randomization. The experimenters were not blinded to the procedures following the baseline session, including the follow-up assessments and contact. All follow-up assessments were conducted remotely and consisted solely of self-report measures.

Procedure

Eligible participants attended a 45-minute baseline session via a videoconference, where they provided informed consent (electronically), completed a demographic questionnaire and semistructured diagnostic interview, and were randomized into either immediate or delayed access groups. At the end of the baseline session, those randomized to the immediate group were provided the iDBT website URL and an invitation code and completed the sign-in procedure (15 min) with the experimenter during the videoconference call. Those randomized to the delayed access group were scheduled for an additional appointment in 4 weeks, where they met with the experimenter again and completed the sign-in procedure (15 min). Thus, although the time spent with the experimenter was approximately the same, the delayed group met with the experimenter via videoconference twice. Each participant was sent a guide to the intervention via email, with a suggested 8-week protocol.

Follow-up questionnaires, completed via REDCap (Research Electronic Data Capture; Vanderbilt University), were automatically distributed via email or text every 4 weeks. Text and email reminders for the follow-up questionnaires were sent daily for up to 4 days until completed, starting 2 days before each assessment was due. To support engagement, additional text messages were sent to consenting participants (56/72, 78%) twice a week for the first 4 weeks following the start of iDBT in both groups (following this point, reminders were discontinued). These text messages contained a link to a short REDCap survey that encouraged use, queried whether participants wanted a follow-up call, and reported any technical issues. Participants could request additional calls or meetings with the experimenter (via REDCap survey or email) to troubleshoot or clarify different components of the website; however, <10 of these calls or meetings took place throughout the study. Participants were compensated up to CAD $70 (US $45.5) for the completion of these procedures (CAD $10 [US $6.5] for baseline and CAD $20 [US $13] each for the 4-, 8-, and 12-week assessments). On average, participants were compensated CAD $59 (US $38.35), including those who did not collect their final payment.

Intervention

Pocket Skills 2.0 is an iDBT intervention developed by author CRW in collaboration with Microsoft Research and Dr Marsha Linehan; it is built upon the most recent DBT manual available [18]. It uses a web-based portal built on the Microsoft Azure platform that is compatible with any internet browser in addition to the Android and iOS mobile operating systems. This iDBT intervention incorporates lessons following the core modules of DBT as well as a specific module focused on addiction (Table 1 provides more details, and Figure 1 provides the screenshots). Within each module, participants selected a specific skill and were presented with a brief video featuring Dr Linehan introducing the skill and its uses. A practice session then ensues with the rule-based chatbot, which allows for feedback through both open-ended text input and a closed selection of responses. The chatbot guides users on how to select skills to use in different situations that may arise as well as the ability to gain points and unlock additional content, which increases user engagement. After logging in for the first time, participants were prompted to complete an introductory module in which they entered a nickname and set personal goals. Following the completion of this module, participants were able to enter any of the 5 DBT modules offered freely, without the need to unlock any content. The procedure in which iDBT was delivered in this study differs from that in previous studies (refer to Multimedia Appendix 2 [30,34] for a comparison).

‎

Measures

Diagnostic Interviews

The Diagnostic Assessment and Research Tool version 4.0 [39] was used to assess depressive, anxiety, bipolar, obsessive-compulsive, trauma and stressor, alcohol, and SUDs according to the Diagnostic and Statistical Manual of Mental Disorders, Fifth edition [40]. We also screened the presence of psychotic disorders. All interviews were completed by the first author, who is a licensed clinical psychologist.

Feasibility and Credibility Measures

For feasibility, we calculated the proportion of randomized participants who started the intervention (by signing in on the first day of access). Of those who started the intervention, we calculated the proportion that (1) dropped out of the intervention after starting (indicated by not logging in after the first day), (2) recorded at least 1 activity after 4 weeks, and (3) completed a recommended dose of using the intervention twice per week for the first month (8 d total). Next, we administered the 6-item Credibility and Expectancy Scale [41] at baseline to assess whether participants had favorable opinions of the intervention and its potential effectiveness before starting treatment. In line with previous work [42], the first 3 items were used to evaluate credibility (using a 9-point Likert scale), whereas a single item (item 4) was used to evaluate expectancy of clinical improvement (using an 11-point Likert scale ranging from 0% to 100%).

Acceptability and Engagement Measures

We used the 6-item Treatment Acceptability Questionnaire [43], which was administered at weeks 4, 8, and 12 to assess ratings of acceptability, perceived effectiveness, and trustworthiness using a 7-point Likert scale. In this analysis, we used only the week 4 and 12 scores. From the intervention source, we examined several metrics tied to engagement or use: the total amount of time spent on the website, the number of interactions with the website (eg, clicks, page views, and text inputs), unique days of log-in, and days of use spread. We then recalculated these metrics for the first 4 weeks, consistent with the acute period of the intervention.

Primary Outcome

The Substance Dependence Scale [44] is a 5-item self-report scale used to assess the severity of alcohol or substance dependence at the baseline and follow-up assessments. Higher scores indicated a higher level of substance dependence. Participants were first asked to indicate which class of substance (including alcohol) they were experiencing the most difficulties abstaining from, even if they reported no use in the past month. The ω reliability coefficient in this study was 0.95. All primary and secondary outcome measures were administered at each assessment point.

Secondary Outcomes

The Patient Health Questionnaire, Depression subscale [45], is a 9-item self-report measure used to assess depressive symptoms over the past 2 weeks, with excellent internal reliability and clinical utility in predicting depression. The ω reliability coefficient was 0.93.

The Generalized Anxiety Disorder-7 Scale [46] is a 7-item self-report measure used to assess generalized anxiety symptoms over the past 2 weeks, with excellent internal reliability and clinical utility in predicting generalized anxiety disorder. The ω reliability coefficient was 0.95.

The Suicidal Behaviors Questionnaire-Revised [47] is a 4-item measure of suicidal thoughts and attempts as well as future intent over the past month, with evidence for its reliability and clinical utility. The total score ranges from 3 to 18, with scores ≥8 indicating significant suicidal risk within clinical samples. The ω reliability coefficient was 0.87.

The World Health Organization Disability Assessment Schedule 2.0 [48] is a 12-item self-report measure assessing functional disability over the past month in several domains (cognition, mobility, self-care, and getting along with others). Higher scores indicate greater functional disability. The ω reliability coefficient was 0.94.

The Difficulties in Emotion Regulation Scale, Short Form [49], is a 16-item self-report measure with excellent internal consistency, assessing emotion dysregulation based on a 6-facet model first described by Gratz and Roemer [50]. Higher scores suggest greater emotion dysregulation difficulties. The ω reliability coefficient was 0.83.

The Mindful Attention Awareness Scale [51] is a 15-item self-report measure of dispositional mindfulness in the form of open or receptive awareness and attention to what is taking place in the present over the past month. Higher scores, which were summed and then averaged, reflected higher levels of dispositional mindfulness. Owing to an administrative error, the anchors were reversed when presented to participants for the entire duration of the study. Therefore, we reversed all scores to ensure a standard interpretation as above. The ω reliability coefficient was 0.94.

The DBT Ways of Coping Checklist [52] is a 59-item self-report measure that assesses the frequency of maladaptive and adaptive skills used to manage difficult situations over the past month, with good internal consistency and test-retest reliability. In this study, we only used the 38-item adaptive skills subscale, which includes skillful behaviors often learned in DBT without using DBT-specific language. The ω reliability coefficient was 0.80.

The National Institutes of Drug Abuse–modified Alcohol, Smoking, and Substance Involvement Screening Test is an adaptation of the original measure [53] used to assess alcohol, smoking, and substance use involvement. This measure was used to assess tobacco, cannabis, cocaine, amphetamine-type stimulants, inhalants, sedatives or sleeping pills, hallucinogens, and opioids. Each class of substance was rated for frequency over the past month using an ordinal scale: 0=never; 1=once or twice; 2=3 or 4 times; 3=5, 6, or 7 times; 4=2 or 3 times a week; 5=4 or 5 times a week; and 6=daily or almost daily. The ω reliability coefficient was 0.23, likely because of the heterogeneity and range of substances used in our sample.

The Daily Drinking Questionnaire [54] was used to assess the frequency of alcohol use on each day of a typical week. Participants were asked how many standard drinks they had consumed on a typical Monday in the past month, with separate questions for each day of the week. Responses were recoded into an ordinal scale: 0=none, 1=1 to 2 standard drinks, 2=3 to 4 standard drinks, 3=5 to 7 standard drinks, 4=8 to 10 standard drinks, 5=11 to 14 standard drinks; and 6=≥15 standard drinks. The ω reliability coefficient was 0.96.

The Risky, Impulsive, and Self-Destructive Questionnaire [55] is an inventory of 38 risky, impulsive, and self-destructive behaviors that sometimes cause problems for people. For brevity and to avoid overlap with other measures, we only used the 4-item risky sexual behavior subscale and the 4-item reckless behavior subscale. We recoded the frequency of responses, which were evaluated over the past month, into an ordinal scale: 1=none, 2=once or twice, 3=3 to 4 times, 4=5 to 6 times, 5=7 to 9 times, and 6=≥10 times. The ω reliability coefficient was 0.86.

Statistical Analysis

Overview

No outcome measure data were missing from the baseline, and participants returned at least partially completed follow-up questionnaires at rates of 94% (68/72; week 4), 78% (56/72; week 8), and 81% (58/72; week 12). At follow-up, scores for outcome measures were only used if there were <10% of items missing, and we treated outcome measures with no data as missing. The frequency of nonalcoholic substance use, standard alcoholic drinks per day, and risky impulsive behaviors was first recoded using ordinal values to approximately equate each scale with respect to their frequency of occurrence. For each measure, we took the average of each ordinal item score and then rounded the average value to the nearest one to serve as the dependent variable. This rounding was required as an ordinal regression relies on categorizing each value of the ordinal dependent variable as a factor variable. There are several ways to analyze ordinal variables, and this procedure was supported by our biostatistical consultation team.

Descriptive statistics were used to evaluate treatment feasibility, acceptability, and engagement data. Chi-square test, Fisher exact test, and 2-tailed t test analyses were used to evaluate baseline differences. Engagement data consisted of time stamped logs of each interaction (ie, clicks, page views, and text inputs) with the website, organized hierarchically within persons, with a total of 39,884 observations. To capture the time spent on iDBT, we ordered the data in Excel (Microsoft Corporation) according to time within persons and calculated a difference score (delta time) between rows. This difference score assessed the time between one meaningful interaction and the next. We then applied a filter to remove any difference scores >30 minutes to account for participants taking breaks or not returning to the app until the next day, capturing 93% of the data. A 10-minute filter captured 92% of the data; however, we wanted to account for playing video content, which could run up to 10 minutes, and the potential of practicing skills live, while remaining on 1 of the web pages. Once the filter was applied, we also calculated the number (and spread) of dates the app was used as well as the number of observations per person, which we called the meaningful interactions calculation. Sensitivity analyses were then performed by examining the same metrics over the first 4 weeks and the time spent on each iDBT module.

All other statistics were run in the statistical program R (version 4.2.1; R Foundation for Statistical Computing). To evaluate the internal consistency of our measures over time, we calculated the between-person ω reliability coefficient [56] statistic using the omegaSEM function from the MultilevelTools package (version 0.1.1). To characterize changes over time for our continuous variables, we ran a series of linear mixed models with the lme4 package (version 1.1-26 [57]), with each primary and secondary outcome serving as a dependent variable in separate models. To characterize changes over time for our ordinal variables, we ran additional linear mixed cumulative link models using the ordinal package (version 2022.11-16) with separate models for each outcome. As per recommendations, we adjusted each model by incorporating the baseline dependent variable value for each person irrespective of whether the difference was significant between groups [58].

All models included a random intercept for a person and relied on restricted maximum likelihood estimation. We omitted any random slope effects throughout the analyses because all our independent variables were level-2 grouping variables.

Each primary and secondary outcome variable was assessed with models containing an interaction effect (group×time, as factor variables) and main effects only (group+time, as factor variables) along with a continuous covariate controlling for the baseline assessment of each outcome per person. The final model chosen for interpretation was the better fitting model based on lower Akaike information criterion and Bayes information criterion values. Therefore, if the model fit was improved by the inclusion of the interaction term, we report that model; otherwise, we removed the interaction term and report the model with the main effects only. All model comparisons were evaluated using maximum likelihood estimation with the lmerTest (version 3.1-3 [59]) package, which uses the Satterthwaite df method. To further reduce the number of statistical tests reported, we also opted to interpret only the week 4 and week 12 contrasts against baseline as these were the most pertinent time points to address our hypotheses. Our α significance level was P=.05, and all statistical tests were 2 tailed. The outputs of each final statistical model are provided in Multimedia Appendix 2 for full transparency.

Power

To achieve at least a medium effect size reduction in substance dependence, as suggested by Wilks et al [30], we would require a minimum sample of 60 as per G*Power (version 3.1.9.7; Cohen f=0.15; 2 groups; 4 measurements over 12 wk; power=0.95; α=.05; and correlation between measures of at least 0.70) [60]. With an expected attrition rate of approximately 20%, we aimed to recruit approximately 72 to 75 individuals in total.

Results

Hypothesis 1: Feasibility and Acceptability

Participant Enrollment and Demographic Characteristics

Initially, 116 individuals were assessed for eligibility, and 72 participants aged 18 to 64 years completed all baseline procedures and were randomized. Demographic and clinical characteristics of the sample are presented in Table 2; these characteristics did not differ between groups, suggesting that the randomization procedure was successful. Of the 72 participants, 9 (13%) met the full threshold criteria for >1 SUD. The primary nonalcohol substance disorder across the sample was cannabis (22/72, 31%); nicotine (10/72, 14%); stimulants (7/72, 10%); and sedative, hypnotic, or anxiolytic (1/72, 1%). None of the participants had a current opioid use disorder. Participants met the criteria for a median of 3 psychiatric diagnoses overall (mean 3.30, SD 1.69; range 1-7).

At baseline, 46% (33/72) of the participants reported taking psychotropic medications in the past month, 22% (16/72) of the participants reported seeing a psychiatrist in the past month, and 8% (6/72) of the participants reported attending a community resource (eg, Alcoholics Anonymous and peer support group) in the past month. These rates did not increase when the participants reported the same services at each follow-up. As we did not restrict new options for care following baseline, 4 participants reported having access to outpatient programing at week 4, but only 2 reported this at both weeks 8 and 12. Figure 2 summarizes the study flow of participants in a CONSORT (Consolidated Standards of Reporting Trials) diagram.

‎

Feasibility and Credibility

Moreover, 94% (68/72) of the randomized participants started the intervention. Three participants in the delayed iDBT group did not attend the follow-up session and never connected to the intervention following attempts to reschedule and instructions provided via email. One additional participant withdrew from the delayed group owing to technical issues and being unable to sign in and therefore did not access the intervention. One participant asked to withdraw from the immediate group because of lack of time to spend on the intervention. In addition, 87% (59/68) of those who started iDBT used it for longer than the initial sign-in day (early dropouts: 9/68, 13%), 50% (34/68) recorded at least 1 iDBT activity after 4 weeks, and 35% (24/68) used it according to the recommended dose of 8 days within the first month. Finally, 63% (43/68) of the participants used it for at least 1 hour in total.

Perceptions of intervention credibility at baseline were positive (mean 74%, SD 16%; range 44%-100%). Participants thought it was logical and it was likely to raise their quality of functioning, and they were confident in recommending it to another person. Participants also estimated a mean 58% (SD 21%; range 10%-90%) improvement in symptoms.

Acceptability and Engagement

Treatment acceptability was similar in the immediate iDBT group (mean 36.7, SD 3.8) compared with the delayed iDBT group (mean 35.9, SD 5.3) at week 12; there was no difference between groups (t₅₆=0.66; P=.51). A summary of the engagement metrics is provided in Table 3. On average, participants used the app for 2 hours and 24 minutes over the course of 43 days during the study, with 10 unique sign-in days. Participants also recorded an average of 543 meaningful interactions with the website. A breakdown of engagement by module is also provided in Table 3. All metrics tended to be higher on average in the immediate versus delayed iDBT group, consistent with the waitlist control design. Metrics improved further after removing 9 dropout participants who did not use the app after the first day. These individuals appeared to abide closer to the recommendation of using the resource for 8 days within the first month.

Hypothesis 2: Were Improvements Greater in the Immediate Versus Delayed iDBT Group?

Overview

The unadjusted means, SDs, and the number of participants for each continuous variable are presented for each group and assessment point in Table 4, with between- and within-group effect size estimates presented in Table 5. The unadjusted values for each ordinal variable are provided in Multimedia Appendix 2.

Primary Outcome

Contrary to our hypotheses, we did not find any significant group×time interactions for the severity of substance dependence at week 4 or week 12.

Secondary Outcomes

Consistent with the hypotheses, the results supported greater benefits for the immediate versus delayed iDBT group for several secondary outcomes. At week 4, there were significant group×time interactions for depression and anxiety, where the immediate access group reported fewer depressive (b=−2.46; SE 1.05; 95% CI −4.51 to −0.40; P=.02) and anxiety symptoms (b=−2.22; SE 0.96; 95% CI −4.09 to −0.34; P=.02) compared with the delayed iDBT group. At week 12, there were significant group×time interactions for standard alcoholic drinks per day (b=−2.00; SE 0.83; 95% CI −3.64 to −0.36; P=.02) and nonalcoholic substance use (b=−3.74; SE 1.47; 95% CI −6.63 to −0.85; P=.01), where the immediate group had lower frequencies of both over the course of the study compared with the delayed group. Contrary to the expectations, there were no significant group×time interactions for all other outcomes: emotion dysregulation, suicidality, DBT skills acquisition, dispositional mindfulness, functional disability, and risky impulsive behaviors.

Hypothesis 3: Did iDBT Produce Improvements Regardless of Group?

Primary Outcome

There were significant main effects of time at week 4 (b=−1.73; SE 0.34; 95% CI −2.40 to −1.07; P<.001) and week 12 (b=−2.09; SE 0.36; 95% CI −2.80 to −1.39; P<.001), indicating a significant decrease in substance dependence for both groups, with no differences between groups (P=.25).

Secondary Outcomes

There were several findings supporting the benefits of the intervention in the follow-up phase of the study (no other significant main effects emerged at week 4). At week 12, there were significant main effects of time for depression (b=−2.95; SE 0.79; 95% CI −4.50 to −1.39; P<.001), anxiety (b=−1.57; SE 0.73; 95% CI −2.99 to −0.14; P=.03), suicidality (b=−0.70; SE 0.24; 95% CI −1.12 to −0.28; P=.001), emotion dysregulation (b=−6.56; SE 1.20; 95% CI −8.90 to −4.21; P<.001), functional disability (b=−3.64; SE 0.77; 95% CI −5.15 to −2.14; P<.001), dispositional mindfulness (b=0.44; SE 0.09; 95% CI 0.27-0.62; P<.001), and DBT skill acquisition (b=0.14; SE=0.05; 95% CI 0.04-0.23; P=.005), indicating that both groups saw significant improvements from baseline over the study duration, with no differences between groups (all P>.25). There were no main effects of time for risky impulsive behaviors and no difference between groups (P>.23).

Discussion

Summary

This study is unique in that it delivered high-quality iDBT in a self-guided format that participants could use through any internet browser on a computer, tablet, or smartphone. Here, we evaluated the feasibility, acceptability, and potential efficacy of iDBT in a sample of treatment-seeking individuals with SUDs often presenting with additional mental health symptoms. In this study, Pocket Skills 2.0 garnered some meaningful support as a potential intervention for those with SUDs and other mental health concerns. We also discuss some caveats and limitations in the following sections.

Feasibility and Acceptability

The intervention was deemed credible and potentially helpful by participants. In terms of treatment initiation, we found that 94% of the randomized participants started Pocket Skills compared with 98% in a previous remote iDBT intervention study [30]. For reference, 88% of the participants started the intervention in the study by van Spijker et al [28] and only 39% of the participants started self-guided iDBT in the study by Simon et al [31]. In this study, not initiating iDBT was mostly because of participants not attending a follow-up session after 4 weeks of being on the waitlist, and in 1 case, owing to technical issues. Thus, the feasibility of deploying iDBT remains high with few technical compatibility issues. These results support the feasibility of adapting DBT for delivery in internet-delivered formats in this context [25,30,34].

Of those who started iDBT, 13% (9/68) were early dropouts, defined as those who did not attempt the intervention after the first day of use, and 50% (34/68) continued to use the app after 4 weeks. Comparatively, Wilks et al [30] recorded a dropout rate of 19%, although different dropout criteria were used (eg, stopped attempting or completing the intervention for 3 weeks in a row). This study differs in that our participants had unrestricted access to all content, whereas the previous study used a week-to-week module approach; thus, we had different definitions of dropout by virtue of study design. There was an overall dropout rate of approximately 10% in the intervention arm in the study by van Spijker et al [28], whereas <9% went beyond the introduction section in the study by Simon et al [31,32]. The rate can also be compared with internet-based psychological treatments more broadly (31%), in-person delivered DBT for different clinical conditions (28%), and in-person psychological treatments for SUDs (30%) [63-65]. Although definitions of dropout vary across trials, this study provides some promise regarding the potential uptake and adherence to iDBT by individuals with SUDs. An analysis of the predictors of dropout from the 2018 study [30] indicated that technological barriers and low perceptions of usefulness emerged as significant [33]. Although we focused on the current hypotheses, we intend to examine predictors of treatment outcomes (including dropout) in future research.

The time spent on the iDBT intervention varied widely. Only 35% (24/68) of the participants completed the recommended dose of spending 8 days in the first 4 weeks, which improved to 41% (24/59) when early dropouts were omitted. These findings can be contextualized by the limited support and self-guided nature of the intervention. Comparatively, 42% of the participants in the study by Wilks et al [30] completed half of the iDBT content in the same time frame (1 month), which included considerably more support, such as daily reminders, homework assignments, and phone calls regarding suicide risk. Approximately half of the participants in the intervention arm completed ≥3 of 6 sessions in the study by van Spijker et al [28], and a similar proportion finished a 15-session course of DBT delivered by email [27]. While we did offer text message reminders to facilitate encouragement, few participants asked for additional meetings or followed our suggested guide. However, even with limited support, a sizeable proportion of participants used the app over several days within the first month, totaling >1 hour of use. These findings suggest potential benefits of additional meetings or coaching sessions, which may improve adherence and engagement to the intervention and improve clinical outcomes overall.

There were relatively high ratings for the content, suitability, and trustworthiness of Pocket Skills using an established measure of treatment acceptability, extending 2 earlier studies [30,34]. Participants recorded most of their engagement within the first 4 weeks of the intervention and in that time, averaged 2 hours of interaction. In a previous 4-week trial of an earlier version of Pocket Skills, which was conducted in patients concurrently completing in-person DBT, participants used the app for 14 out of 28 days and spent 2.25 hours on the app during that time [34]. This equated to approximately 4 minutes of activity per person per day. Given that our study was largely self-guided without additional support or check-ins, this newer version of Pocket Skills saw comparative engagement with respect to total time. One self-guided skills training intervention reported 10.5 hours of use on average or approximately 15 minutes per day over 6 weeks [28]. Examining iDBT as an adjunct to in-person (or videoconference) DBT to improve engagement even further may be warranted.

Potential Efficacy

With regard to our second hypothesis evaluating the waitlist control design of the study, we found little evidence that the immediate iDBT group benefited more in the acute period of 4 weeks. We did not find a difference between groups in our primary outcome at this time point, where we expected it. We found interactions at week 4 for 2 secondary outcomes (ie, depression and anxiety) in favor of the hypothesis. We found additional interactions during the follow-up period (week 12) for decreased standard alcoholic drinks per day and overall substance use frequency in favor of the immediate iDBT group. Notably, we originally planned a 16-week trial with an equal immediate and waitlist period of 8 weeks. However, in a pilot study, we found attrition and lack of engagement to be greater than in this study, which contributed to the revised study design. Most studies on app- and internet-based interventions use one month as an acute test of the intervention with another month as a follow-up assessment period owing to increasing attrition after 7 to 8 weeks [35-37].

Our third hypothesis regarding overall improvement was more consistently supported, with favorable improvements in our primary and secondary outcome measures in both groups by week 12. There were medium to large effect size improvements in our primary outcome of substance dependence, both in the acute (week 4) and follow-up phase, consistent with the literature supporting in-person DBT for alcohol use disorders and SUDs [16,21-24]. There were many medium to large effect size improvements to our secondary outcomes by week 12, including depression, anxiety, suicidal behavior, emotion dysregulation, and functional disability and these effects did not differ significantly by group once accounting for baseline differences. These findings are consistent with in-person DBT improvements in suicidal behavior, depression, anxiety, and emotion dysregulation [23,66-69] and extend a previous stand-alone iDBT study [30]. The intervention also improved DBT skill acquisition and dispositional mindfulness in both groups by week 12, as seen by small-to-large positive effect size values and in line with the literature findings on face-to-face DBT [52,70,71]. Whether DBT skill acquisition mediated treatment outcomes in this study, as implied by previous research [68,69], is a hypothesis that could be examined in a follow-up analysis.

Limitations

Our ability to detect interactions during the acute phase of treatment was limited given that differences would have had to be medium or large to detect using our unbalanced waitlist design. The use of a waitlist control may have led some individuals to consider other treatment options or drop out of the study without connecting to the intervention, especially given its unblinded nature following randomization. Alternatively, the waitlist condition may have supported a nocebo effect as participants reported favorable changes in most outcomes despite a lack of treatment [72]. Although the randomization procedure was conducted in a blind manner, following the baseline session, all other follow-ups and contact with participants were unblinded, which may have introduced potential experimenter bias. The primary and most secondary outcomes were participant rated (ie, self-report), which is also less robust to bias than a blinded outcomes assessor.

In terms of feasibility metrics, although we saw that roughly half of the participants remained active on iDBT after 4 weeks, it is not clear how consistently active they were. Because we did not restrict access to other interventions, it remains unclear how specific the intervention effects were tied to iDBT in this study. We attempted to mitigate concerns about this by asking questions about different services that participants were receiving at each follow-up and found little to no increase in psychotropic medications, community support groups, and additional treatments. More analyses are needed to understand the dose-response relationship between the intervention and treatment outcome, which will be addressed in future work.

Pocket Skills 2.0 omits several aspects of in-person formats of DBT that may improve engagement and adherence, such as a significant group therapy component; more robust tracking of thoughts, emotions, and behaviors using diary cards; and handouts and worksheets for homework. These implementation differences compared with standard in-person DBT may have influenced treatment outcomes. As a technical limitation, most participants accessed iDBT on a home computer or laptop. Although we discussed the ability to sign in on mobile devices with participants, we typically asked them to sign in for the first time on a computer or laptop based on experiences during the piloting phase of this study (ie, there was more difficulty signing in on the mobile iOS platform). This procedural issue may have introduced a barrier to using iDBT on mobile devices; however, we could have spent more time ensuring that the intervention was working on both participants’ smartphones and computer devices. Future implementations could include subsequent meetings with participants (eg, check-ins) to address any technological and compatibility issues more quickly.

Despite efforts to recruit a diverse sample, our sample was predominantly White, female, heterosexual, and largely aged between 18 and 45 years. Future research should attempt to replicate the outcomes of iDBT across more diverse samples, such as sexual, gender, and ethnoracial minority groups (refer to the study by Harned et al [73] for review). Our sample was heterogeneous with respect to their endorsement of alcohol or nonalcoholic substance difficulties. Measuring the frequency and severity of multiple substances in an efficient way is challenging. Owing to the use of different measures and rating scales to assess alcohol and nonalcoholic substances (as well as risky and impulsive behaviors), our raw data required rescaling to create new ordinal scales that approximately modeled the same frequency and severity across multiple scales. Future research could adhere to more standardized approaches that allow for responses to be collected and then coded more reliability. For example, the Timeline Follow Back interview has been used to self-report alcohol and substance use as well as risky sexual behaviors [74,75].

Conclusions

Notwithstanding the limitations of this study, our iDBT intervention Pocket Skills 2.0 was supported as a feasible and acceptable intervention for those with SUDs and other mental health concerns. However, methods to improve engagement should be further evaluated. The intervention not only showed potential effectiveness for substance dependence but also demonstrated positive effects across various mental health symptoms, affirming its clinical utility. These findings add to the sparse literature on internet-based DBT and internet-delivered psychological interventions for SUDs. This format has the potential to increase accessibility and reduce the costs and resources required for in-person DBT. Several research priorities were identified to potentially improve engagement and optimize treatment outcomes as well as understand how our iDBT intervention can be integrated into the larger landscape of treatment options for SUDs and other conditions.