Appropriate Use and Operationalization of Adherence to Digital Cognitive Behavioral Therapy for Depression and Anxiety in Youth: Systematic Review

Introduction

Background

Cognitive behavioral therapy (CBT) is a structured, skills-based psychotherapy, typically delivered in-person by a trained clinician over a duration of 12 to 18 weeks [1,2]. It is the gold standard psychological treatment for anxiety and depression [3] in both adults and young people, defined here as people aged between 12 and 24 years, consistent with the Australian Institute of Health and Welfare [4-7]. To safeguard the integrity of CBT and ensure that efficacy is maintained [8], CBT manuals provide clinicians with explicit instructions on administration. These manuals detail indications of who will benefit, the treatment targets and goals, the number and duration of sessions required, sequence of content delivery, and between-session practice and real-world enactment of skills (ie, homework [9]). Clinicians also adapt various engagement strategies to promote adherence to CBT, including developing a therapeutic alliance through collaborative goal setting, communicating and enhancing treatment expectancies, and clarifying concepts to ensure that treatment processes and rationale are well understood [1,2,5-7,10,11]. Some CBT manuals also differentiate the critical, core treatment components from optional modules and strategies to measure symptoms and treatment outcomes [1]. To summarize, the components of CBT manuals that guide clinicians’ administration as intended are as follows: recipients, target condition, number and duration of sessions, sequence of content, homework activities, engagement-promoting strategies, core therapeutic components, assessment and monitoring, and crisis management.

Despite its efficacy, the uptake of in-person CBT is suboptimal across all ages [12-15]. In young people, this is due in part to the affordability and availability of trained practitioners, perceived stigma, poor mental health literacy, and a preference for self-reliance [16-18]. Among the young people who do seek help, most do not receive CBT [19,20]. Improving access and uptake of CBT is particularly critical for young people, as three-quarter of depression and anxiety cases emerge by late adolescence [21], and these disorders are a leading cause of disability in this age group [22,23]. The transition of CBT from in-person professional administration to digital, self-directed delivery was hypothesized to overcome many of the treatment barriers faced by young people [24]. A recent systematic review and meta-analysis of internet- and computer-delivered CBT for youth confirmed the positive effects of these interventions on symptom reduction [25]. However, adherence to many of these interventions is low despite the alignment of digital CBT with young people’s help-seeking preferences [24].

Although personal factors such as motivation may account for low adherence, reliance on the young person to appropriately self-administer digital CBT interventions with none to limited clinician guidance or supervision may also contribute. Thus, similar to any self-administered treatment, it is essential that the young person is provided with clear instructions to ensure appropriate use; that is, to ensure that the intervention is used in a way that generates optimal clinical benefit. Unlike in-person CBT, where instructions for engaging with the treatment are well documented within manuals and directly relayed to youth through their clinician, it is unclear whether instructions or recommendations for the appropriate use of digital CBT are provided to users or consistently documented in the literature. Instructions for use may be provided during intervention onboarding or guided via explicit or implicit software design features. For example, many digital CBT interventions have used the design feature, “tunnelling,” to replicate the structure of in-person CBT, whereby therapeutic modules are presented sequentially to guide users through the content in an appropriate sequence [26,27]. Other software design features, such as tailoring content in response to user input, automated feedback, rewards and encouragement, and reminders and notifications, have also been used to replicate clinician guidance and supervision [28]. An understanding of how digital CBT interventions define appropriate use and how instructions for use are relayed to youth requires examination to determine their adequacy in supporting appropriate self-administration.

In addition to ensuring optimal outcomes, clear instructions for the use of digital CBT interventions are required for researchers and clinicians to operationalize users’ adherence to these interventions. Adherence is defined as a meaningful measure of the extent to which individuals’ intervention use corresponds with creators’ recommendations, instructions, and expectations of appropriate use [29,30]. For example, if the recommendation is to complete all modules within a digital intervention, adherence is measured by determining the number of modules completed. Alternatively, if the recommendation is to engage with a particular intervention activity on a specified occasion (eg, complete a mindfulness mediation upon waking), adherence is measured by determining the frequency of this event. In this way, adherence provides a measure of the validity of treatment administration. Thus, clear instructions for appropriate use and corresponding adherence measures are crucial for ensuring that the necessary standards of quality, safety, and efficacy are met. Despite the importance of this, in a review of the literature, Christ et al [25] found that many studies on digital CBT for youth generically operationalized adherence as program completion, heterogeneously reported as the proportion of participants who completed all treatment modules or the average number of modules completed across the sample. Several studies failed to report any adherence data, and definitions of appropriate use were not examined in the study by Christ [25].

Similar variability in the reporting of adherence has been found in reviews of adult digital health interventions. One systematic review found that almost three-quarter of studies (45/62, 73%) operationalized adherence as the degree of intervention use, reported as the number of completed modules and activities, log-ins, or time spent in the intervention [31]. Furthermore, statements on instructions for use were only reported in one-third (23/62, 37%) of studies, giving little indication of how interventions defined appropriate use [31]. Inadequate reporting of instructions for use and significant heterogeneity in reporting of adherence are further demonstrated in at least 5 systematic and narrative reviews of adherence to digital psychological interventions in adults [32-36]. Collectively, adherence appears to be operationalized in the absence of adequately reported definitions of appropriate use or is generically operationalized as adherence to total intervention exposure, based on the notion of “the more, the better” rather than explicitly measuring the accordance between recommended use and actual use. Studies have provided little justification for this approach [31]. Furthermore, studies of adults and youth have not consistently supported a linear relationship between adherence and outcomes [25,29,37-39].

Despite the prevalence of digital CBT interventions for youth [25], little is known about how young people have been instructed to use these interventions, how adherence to interventions has been operationalized, or whether measures of adherence adequately determine differences between actual and recommended use. It is important to examine this, as young people have different patterns of engagement with technology, preferences, and expectations compared with adults [40-42]. Clearer expectations of use may not only improve the effectiveness of digital CBT but also assist young people in selecting interventions most suited to their circumstances, while also facilitating greater endorsement and dissemination by mental health professionals. Establishing protocols for the disclosure of appropriate use and the operationalization of adherence is imperative for improving the uptake, adherence, and effectiveness of digital CBT among youth.

Objectives

The primary aim of this systematic review was to synthesize the published literature on digital CBT interventions for depression and anxiety in young people to investigate the definitions of appropriate use and the disclosure of instructions for use to users. To this end, descriptions, recommendations or instructions of appropriate use, or intervention features guiding use were extracted from the included studies and mapped onto the 10 components within CBT manuals that typically guide administration as a means of determining definitions of appropriate use. A specific focus on digital CBT differentiated our review from recently published reviews on engagement in digital health interventions that incorporated CBT and non-CBT interventions. This review also aimed to examine how adherence to recommended use has been operationalized and measured and to determine the associations between digital CBT intervention use and outcomes among young people. This information will improve our understanding of how young people have been instructed to self-administer digital CBT interventions and will determine whether the measures of adherence used by researchers have accurately captured the degree to which young people complied with instructions for use.

Methods

Protocol and Registration

PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) was used to ensure the quality and consistency of the procedure and reporting [43]. The review protocol was registered with PROSPERO (CRD42020208668). We deviated from the original protocol to exclude cognitive- and behavioral-only interventions and mindfulness-based and acceptance and commitment therapy interventions to confine the focus to digital CBT. In addition, we found that the study manuscripts rarely reported on the instructions of appropriate use. We acknowledge that this information may have been provided to users within the intervention itself, but this is not clear. As such, we derived the information on appropriate use from the study protocols and intervention features.

Ethical Considerations

Conducting a systematic review or peer-reviewed literature is not listed as a research activity that requires human ethics approval by the UNSW Research Ethics Board. As such, ethics approval was not applied for.

Eligibility Criteria

Participants

Participants were young people aged between 12 and 24 years. This age range was selected to cover the full spectrum of youth, as defined by the Australian Institute of Health and Welfare [4]. Where the age range of participants extended 12 to 24 years, studies were excluded if the total sample contained <80% of participants in the target range. In the absence of age range data, a judgment was made based on the recruitment setting (eg, secondary school) and the reported mean age. Diagnostic status was not used as an eligibility criterion for samples.

Interventions

Eligible interventions included those directly targeting anxiety or depression (including transdiagnostic interventions) via a predominantly CBT-based psychological treatment delivered by a computer, smartphone, or internet platform. Interventions were also required to be used on more than one occasion. Purely cognitive or behavioral interventions alone were not included, nor were treatments aimed solely at problem-solving. Interventions that were not predominantly CBT, such as mindfulness-based interventions, acceptance and commitment therapy, and interpersonal psychotherapy, were excluded. Nonpsychological interventions, including exercise or physical activities, music, and art therapy, were excluded. Gratitude-based therapies and journaling were also excluded.

Comparison Groups

No restriction was imposed on the type or use of control or comparison groups.

Outcomes

Primary outcomes were symptoms of depression or anxiety measured using standardized, validated, and reliable instruments or scales, suitable for adolescents. Included studies were required to report at least one measure of adherence and any of the following: descriptions, statements, or instructions for appropriate use or recommendations of use; descriptions of intervention features supporting appropriate use, including adherence-promoting features; or the association between adherence and depression outcomes and adherence and anxiety outcomes.

Studies

Studies were included if they were written in English, published in peer-reviewed journals, and published after January 1, 1991. Studies were not excluded based on study type or quality; both controlled and uncontrolled studies (eg, pre-post studies without a control group) were included. Case studies were excluded.

Search Strategy

Overview

The electronic databases, Embase, MEDLINE, PsycINFO, and Cochrane Library, were searched for articles published from January 1, 1991, to April 7, 2021. The start date was selected to coincide with the year in which the World Wide Web was introduced. The following search terms were used in the title, abstract, and keywords: (adolescent* OR youth OR child* OR teen* OR young adult) AND (online OR digital OR internet OR app OR mHealth OR eHealth OR web OR web-based OR smartphone OR smart phone OR computer*) AND (anx* OR depress* OR affect OR mood) AND (cognitive therapy OR cognitive behavioural therapy OR cognitive behavior therapy OR cognitive behavioral therapy OR CBT). Additional sources were included through a hand search that comprised examining reference lists of key articles and systematic reviews and authors’ knowledge of manuscripts related to digital CBT for youth. Furthermore, the details from included studies were used to conduct a search to identify relevant manuscripts reporting on secondary or adherence outcomes from the included studies’ data sets.

Data Extraction and Synthesis

Articles were identified from the search strategy, and after duplicates were removed, titles and then abstracts were reviewed for relevance by the first author (SHL). A second reviewer (AWS) independently screened 10% of the identified studies to ensure the reliability of the eligibility criteria. Full texts of the remaining studies (n=174) were appraised by 2 reviewers (SHL and MRA) to determine eligibility. Reasons for exclusion were discussed, and consensus for eligibility was confirmed for 6 articles. In cases where a consensus between the 2 reviewers could not be reached, a third reviewer (MSK) assessed for eligibility and a consensus was reached. The 2 reviewers (SHL and MRA) independently performed the data extraction using a data extraction template (Multimedia Appendix 1) designed to identify information, including the following: study details, sample details, intervention details, and outcomes of interest. It is worth noting that, for delivery mode, interventions were categorized as sequential where modular content was presented sequentially or explicitly delivered in a sequential order with content only becoming available when preceding modules were completed, or nonsequential where content was unrestricted allowing complete self-navigation.

Study outcomes of interest included the following: descriptions, statements, or instructions of appropriate use for young people; measures of adherence; and associations between adherence and symptom outcomes. We relied only on the information provided in the study manuscripts. Using the components of manualized CBT, we identified information on appropriate use pertaining to the following: intended recipient of the intervention, intended target condition, intended number of modules to be completed, intended duration and frequency of use, instructions regarding real-world enactment of skills (homework), adherence-promoting features (embedded within the intervention), core components, and symptom assessment and monitoring. We also extracted data on when and how users were recommended to access in-person support, excluding risk management procedures that constituted the study protocol. In lieu of specific descriptions of appropriate use, we extracted any information on intervention design features or other study details that implied appropriate use. This included the following: inclusion and exclusion criteria to indicate intended recipients, duration of intervention access to indicate intended duration of use, and tunneling to indicate intended sequence of content use. The sources of information used to derive aspects of appropriate use were recorded (trial protocol, feature description, or a statement in the study manuscript). Post hoc determination of the number of modules needed to achieve benefits was not considered as constituting any aspect of appropriate use.

Adherence-promoting features included those explicitly used by the authors to improve adherence or those that fit within the following categories: supported use defined as use that involves support, encouragement or guidance from a person, reminders, rewards, gamification, social or peer support, tailoring defined as the capacity to tailor content to meet an individual’s requirements, personalized feedback including system-generated personalized feedback, customization of visual features such as avatars and color schemes, and interactive content defined as content requiring active inputs from the user. These categories were derived from past literature that endorsed supported use as an effective adherence promoter [44] and systematic reviews of user preferences [24,45]. Supported use was further categorized into autonomous, supported, or intervention-led blended, using the definitions provided by Fairburn and Patel [46] (Table 1). Interventions delivered in schools were categorized as supported, unless otherwise specified, and studies that did not specify support in the use of the intervention were recorded as autonomous. Interactive content was further coded as follows: activities, quizzes, homework activities, and multimedia content. The same intervention delivered under different conditions (eg, with or without additional in-person therapy sessions) was treated as 2 separate interventions and is presented in different rows in the tables. It is worth noting that, we elected to categorize personal feedback from a clinician or support person as supported use, not tailoring, as supported use is likely to have an element of personalization that does not necessarily involve tailored delivery of the intervention content. In addition, the included studies were screened for references to registered or published protocols. Where protocols were identified, they underwent the same data extraction and synthesis procedure as the study manuscripts to supplement data on descriptions, statements, or instructions for appropriate use. Finally, the number of aspects of digital self-administration described in each study was tallied to show the degree to which aspects of self-administration were disclosed.

Methodological Quality Assessment

The methodological quality of the included studies was assessed using the Cochrane Risk of Bias (ROB) Tool for randomized controlled trials (RCTs) [77] and the Integrated Quality Criteria for the Review of Multiple Study Designs [78] for other study designs. Quality ratings were determined independently by 2 reviewers (SHL and JRB) and then compared to resolve differences. Studies were considered methodologically sound if they met the Integrated Quality Criteria for the Review of Multiple Study Designs criteria for inclusion [78] or were rated as low in the ROB overall assessment of bias.

Results

Overview of Included Studies

Overview

The search resulted in 4641 studies, of which 174 full-text articles were identified. A total of 32 studies met all inclusion criteria and were included in the systematic review. The screening process and reasons for exclusion are presented in Figure 1. Table 1 summarizes the study, sample, and intervention characteristics of the 32 included manuscripts. On 3 occasions, >1 manuscript was produced from a single study; as such, these manuscripts were reported together, leaving a total of 28 independent studies.

‎View this figure

Study Design and Country of Origin (Setting)

Of the 28 studies, most were RCTs (n=22, 78%), and the others were open trials (n=3, 10%), feasibility trials (n=11, 39%), 1 (3%) non-RCT, and 1 (3%) factorial design. The majority were conducted in Australia and New Zealand (11/28, 39%), Europe and the United Kingdom (9/28, 32%), and the United States (5/28, 17%), with others in Canada (2/28, 7%) and Hong Kong (1/28, 3%).

Sample Characteristics

Across the 28 studies, 16,578 participants received the intervention. Sample sizes varied considerably among studies, ranging from 10 to 9079 (mean 592.07, SD 1857.44; median 61). Overall, 42% (12/28) of studies had sample sizes <50, and 57% (16/28) had sample sizes >50. Regarding gender representation, 75% (21/28) of studies had >55% of their sample as girls or women, 10% (3/28) of studies had between 45% and 55% of the sample girls or women, 7% (2/28) of studies had <45% of the sample as girls or women, and 7% (2/28) of studies did not disclose gender representation of their samples. The girl or woman representation within the samples ranged from 41% to 100%. Across all studies, the mean age was 15.74 years (range 12.95-22.60), with a total age range of 7 to 24 years. A total of 10% (3/28) of studies did not report the mean age of the participants, and 14% (4/28) of studies did not report the age range.

Intervention Characteristics

Intervention characteristics are presented in Table 2. A total of 20 interventions from 28 studies were identified and included in the review, with the same intervention used in several studies: SPARX (Smart, Positive, Active, Realistic, X-factor thoughts; University of Auckland; n=4), MoodGYM (Australian National University; n=3), Brave Online (University of Queensland; n=2), CATCH-IT (University of Illinois; Competent Adulthood Transition with Cognitive Behavioral Humanistic and Interpersonal Training; n=2, not including the modified version), and Breathe (University of Alberta; Being Real, Easing Anxiety: Tools Helping Electronically; n=2). A total of 28% (8/28) of interventions targeted depression, 25% (7/28) targeted depression and anxiety, 10% (3/28) targeted unspecified anxiety, and 7% (2/28) targeted a specific anxiety disorder (obsessive compulsive disorder and social anxiety disorder). The digital delivery formats included web-based (n=11), computer or CD-ROM (n=6), multiplatform (n=1), weblinks viewed on a smartphone (n=1), and 1 smartphone app (n=1). Several interventions included elements of other therapeutic approaches in addition to CBT, such as interpersonal therapy (2/28, 7%), family systems therapy (1/28, 3%), and positive psychology (1/28, 3%). The total number of modules ranged from 4 to 15 (mean 8.15, SD 2.74). A total of 10% (3/28) studies did not specify the number of modules contained in the intervention. Most interventions (14/20, 70%) were sequential, 10% (2/20) were nonsequential (WeClick and MoodHwb), and 20% (4/20) were undetermined.

Methodological Quality

All 6 non-RCT studies met the methodological quality criteria for inclusion. Of the 22 RCTs included in the review, 9 (41%) were assessed as having a low ROB, 9 (41%) as having some concerns, and 4 (18%) as having a high ROB. All these studies were included in the analysis.

Digital Self-administration Guidelines as Disclosed in Study Manuscripts

Overview

The aspects of digital self-administration extracted from the included studies are contained in Table 3 and Multimedia Appendix 2 [26,27,47-76]. Only 1 study (the study by O'Dea et al [62]) reported how instructions for appropriate use were relayed to young people. In this study, an animation was used to inform young people that they could use the intervention as they wished.

Intended Recipients and Target Condition

A total of 7% (2/28) of study manuscripts contained explicit statements of the intended intervention recipients: adolescents with mild to moderate depression [65] and adolescents with clinically significant risk of depression [72]. For all other manuscripts, intended recipients were derived from the participant inclusion and exclusion criteria. Of these, intended recipients were required to meet the diagnostic criteria for an anxiety disorder in 17% (5/28) of studies, were symptomatic (anxiety, depression, or both) in 32% (9/28) of studies, were at risk of developing depression in 17% (5/28) of studies (ie, symptomatic and/or history of depression), and were asymptomatic in 3% (1/28) of studies. Symptom level was not relevant to intended users in 21% (6/28) of studies (universal sample) and symptom details were not provided in 7% (2/28) of studies. A total of 39% (11/28) of study manuscripts described the intended target condition. For the remaining 43% (17/28) of studies, the intended target condition was derived from the study protocol (primary outcome measure). Depression was the target condition in 50% (14/28) of studies, anxiety in 32% (9/28) and both anxiety and depression in 17% (5/28). Finally, 7% (2/28) of studies provided statements on the intervention’s therapeutic goals; both of the goals were to reduce unhelpful cognitions and behaviors, consistent with a CBT approach.

Intended Degree of Completion, Duration and Frequency of Use, and Sequence of Use

A total of 17% (5/28) of studies stated that all intervention modules should be completed, and 3% (1/28) of studies stated that half the modules should be completed. The remaining 82% (23/28) of studies did not provide this information. Although not explicitly stated as intended use by the study authors, 10% (3/28) of studies examined noncompleters and completers. These concepts appeared to be derived post hoc for analytical purposes. Definitions of the latter included those who completed ≥4 modules out of 8 [64], ≥3 modules out of 5 [63], and more than one-third of all activities [50].

None of the studies explicitly stated the intended duration of intervention use. This information was derived from the duration of intervention access in 82% (23/28) of studies (range 4 weeks to 12 months); however, 17% (5/28) of studies did not provide any information on this (eg, duration between pre- and postintervention assessments). Intended frequency of use was stated in 60% (17/28) of studies and was derived from an intervention feature in 3% (1/28) of studies. Of these 18 studies, 9 (50%) recommended that users complete 1 module per week, 2 (11%) recommended up to 2 modules per week, 1 (5%) recommended 1 module per fortnight, and 1 (5%) recommended that users read 2 messages per day. A total of 21% (6/28) of studies stated that the frequency of intervention use was unrestricted. There were 36% (10/28) of studies that did not provide any information regarding the intended frequency of use. The intended sequence of content was derived from intervention features, including numbered modules, locked content, or module display, in 64% (18/28) of studies, from the study protocols of 11% (3/28) of studies (prescribed delivery of content), and was stated in the manuscript in 3% (1/28) of studies. Of these 22 studies, interventions were intended to be completed sequentially in 20 (90%) studies, with 2 (10%) allowing users unrestricted access to alternate between modules. A total of 27% (6/28) of studies provided no information on the intended sequence of the intervention content.

Intended Skill Enactment

A total of 50% (14/28) of studies described intervention features that prompted homework activities. The remaining 50% (14/28) provided no information regarding recommendations for skill enactment.

Adherence-Promoting Features

Regarding supported use, 17% (5/28) of interventions were autonomous, 42% (12/28) were supported, 21% (6/28) were both supported and intervention-led blended, and 10% (3/28) were intervention-led blended only. A total of 28% (8/28) of interventions used reminders (eg, email or SMS text message), 3% (1/28) implemented rewards (a snack upon completion of a module), and 21% (6/28) contained gamified concepts (eg, leveling up). Regarding interactive content, 82% (23/28) of studies described interventions that used interactive features, including 20 with interactive activities, 8 with quizzes, 7 with homework activities, and 18 with multimedia content. A total of 10% (3/28) of studies used tailoring, 10% (3/28) used personalized feedback, and 14% (4/28) used customization of visual features. None of the interventions were reported to contain peer-support features.

Only 10% (3/28) of studies have examined the influence of an adherence-promoting strategy on adherence to recommended use. Van Voorhees et al [71] examined supported use as an adherence promoter by combining the digital intervention with 3 in-person motivational interviewing sessions. These were found to significantly increase site visits, time in intervention, proportion of exercises completed, and number of characters typed into interactive exercises. Berg et al [47] evaluated 2 strategies: supported use via chat sessions with a therapist and learning support, which included short summaries, pedagogical pictures, videos, and quizzes. These strategies, alone or combined, did not lead to greater adherence when compared with the control condition. Finally, Lillevoll et al [57] evaluated the influence of personalized email reminders on adherence to MoodGYM compared with standard email reminders and no reminders. Neither form of reminder affected adherence to the intervention.

Core Therapeutic Components, Symptom Monitoring, and Accessing Crisis Support

A total of 14% (4/28) of studies reported core therapeutic components, including exposure therapy, exposure and response prevention therapy, behavioral therapy, and cognitive restructuring. The remaining 85% (24/28) of studies did not identify any. A total of 42% (12/28) of studies described intervention features for monitoring mood and assessing safety. Of these, 50% (6/12) of studies described symptom monitoring that occurred every few days or weekly. The remainder (6/12, 50%) did not provide any information regarding the frequency of mood monitoring. Furthermore, 28% (8/28) of study manuscripts described intervention features that assessed suicide risk and responded with helping-seeking information. The remaining 85% (24/28) of studies did not provide any information on when users were recommended to seek crisis support.

Reporting on Aspects of Intended Use

Of the 10 aspects of appropriate use identified to inform instructions for use (presented in Figure 1), no study manuscript reported on all 10 aspects. The average number of aspects reported was 6.29 (SD 1.33) with a range of 3 to 8. Descriptions of intervention features and study protocol details were relied upon heavily to infer aspects of appropriate use (Table 3 and Multimedia Appendix 2).

Measures of Adherence

The measures of adherence used in the included studies are presented in Table 4. To support a clearer interpretation of the results, adherence measures were coded as follows: module completion, intervention visits or log-ins, time spent in intervention, activity or homework completion and practice, and total content completion. Study attrition was not considered to be an appropriate measure of intervention adherence, as it was not possible to differentiate the noncompletion of study assessments from adherence to the intervention. Almost all the studies (24/28, 85%) measured module completion, which was reported as either mean (and SD) module completion or as percentage of the sample completing a defined number of modules (eg, 14/28, 50% of study manuscripts reported the percentage of the sample completing all modules). Moreover, 21% (6/28) of studies reported intervention visits or log-ins, either as the mean or as the percentage of the sample visiting on a defined number of occasions. Furthermore, 25% (7/28) of studies reported time spent in the intervention, either as the mean total intervention time or as the mean time spent per event (eg, visit or module). A total of 32% (9/28) of studies reported activity completion as either the mean, percentage of all activities completed, percentage of the sample that completed a set number of activities, or percentage who engaged in any of the activities. Just under half of all studies (13/28, 46%) reported >1 measure of adherence, indicating multidimensional operationalizations of the construct.

Association Between Adherence and Outcomes

The heterogeneity of adherence measures, the different statistical methods used to determine an association between adherence and outcomes, and incomplete reporting of results made it difficult to extract comparable data for a meta-analysis. Associations are, instead, reported descriptively. A total of 19 (67%) studies did not report on the association between adherence and symptom reduction. Of the remaining 8 studies, 6 (21%) examined the relationship between adherence to the intervention and treatment outcomes for depression. A total of 4 studies, 2 (7%) of which compared outcomes for completers and noncompleters (as defined by the authors), found no association between adherence and depression outcomes. Furthermore, 7% (2/28) of studies found an association between depression and adherence, but in opposing directions, such that depression decreased as time in intervention increased as per Ip et al [53], whereas Clarke et al [51] found that depression reduced when the total time spent in intervention decreased. Moreover, 14% (4/28) of studies examined the association between adherence and anxiety outcomes: 10% (3/28) found no association, and 3% (1/28) found a negative association between symptom reduction and total time spent in intervention, whereby more time in the intervention was associated with lower anxiety symptoms [53]. None of the studies that used multiple measures of adherence examined the independent associations of these measures with treatment outcomes.

Discussion

Principal Findings

This systematic review aimed to synthesize how appropriate use has been defined in digital CBT interventions for youth, how instructions for use are relayed to young people and reported in studies, and how adherence to appropriate use has been operationalized and measured. This review found that none of the included studies systematically described definitions of appropriate use, which were instead derived from the study protocol or intervention features. Despite all interventions being based on CBT, the derived definitions of appropriate use varied widely among studies, particularly in terms of intended recipients, frequency and duration of use, and use of adherence-promoting features. The only component of use with some consistency among studies was the sequence of use, with 71% (n=20) of the 28 studies using interventions intended to be used sequentially. Nearly all included studies neglected to disclose how the instructions or recommendations for appropriate intervention use were relayed to young people, with only 3% (1/28) of studies providing such information [62]. The results showed that most studies (24/28, 85%) operationalized adherence as the degree of intervention completion, with significant heterogeneity in how this was measured, regardless of how appropriate use was defined. The most consistently reported measure of adherence was the percentage of the sample that completed the entire intervention (14/28, 50%). There was little evidence of an association between degree of use and improved outcomes.

Appropriate self-administration of digital CBT is critical for achieving the therapeutic goals. Therefore, a lack of reporting on both definitions of appropriate use and how instructions for use are provided to users is somewhat surprising. Indeed, an understanding of how the intervention creators desired the intervention to be used by youth had to be derived from the study protocol and intervention features for almost all components of appropriate use, with the frequency of use the only component explicitly stated in studies. These findings suggest that inadequate consideration has been given to developing nuanced definitions of appropriate use. As early digital CBT interventions were direct translations of in-person CBT, this omission may stem from the assumption that engagement with CBT should be consistent regardless of the delivery mode (ie, full program completion). However, given the current innovative approaches to CBT delivery, particularly the movement away from sequential delivery and expectations of full program completion [38], it is imperative to investigate patterns of digital CBT use that produce the greatest improvements in defining appropriate use. There is also little evidence that young users of digital CBT for depression and anxiety are provided with adequate instructions for use. The impact of this on young people’s treatment uptake, engagement, response, and attitudes toward digital mental health care remains unknown. Poorly relayed instructions for appropriate use to young people, their guardians, and health professionals may contribute to poor intervention adherence in digital CBT. It is possible that more nuanced definitions of appropriate use and clearly delivered instructions on use could improve intervention adherence and ultimately have beneficial effects on the outcomes.

The resulting lack of guidance on how to operationalize and measure adherence is an additional consequence of poorly defined recommendations for appropriate use. Specifically, the degree to which actual use aligns with appropriate use can only be determined if the appropriate use is adequately defined. We found that adherence was most often operationalized as the degree of program completion, demonstrating a lack of correspondence among definitions of appropriate use, which varied widely, and operationalization of adherence. This indicates that definitions of appropriate use are either inadequate or not being used to operationalize adherence. This finding is also consistent with findings in the adult literature, where appropriate use is often undefined but nevertheless operationalized generically as the more use, the better [31], with significant heterogeneity in how the amount of use is measured [29,79,80]. Our conceptualization of the appropriate use of digital CBT is made up of 10 components (based on in-person CBT manuals) and is therefore multidimensional. This suggests that the measurement of adherence should be multidimensional and nuanced enough to capture adherence to each aspect of appropriate use. Taken together, adherence definitions and measures that provide a meaningful indication of the extent to which interventions are used appropriately are lacking, and require refinement in future studies.

We found little evidence to suggest an association between amount of use and symptom improvement. Most studies did not examine or report a dose-response effect for the intervention being evaluated. However, of those that did, most found no association between use and either depression or anxiety outcomes, regardless of the type of measure used. This finding demonstrates that the pervasive notion of the more use, the better is unsupported for digital CBT. A tenuous association between use and outcomes is consistent with findings reported in a recent meta-analysis of youth digital CBT interventions, which found no association between adherence (measured as the amount of use) and outcomes [25]. It is also somewhat consistent with a review of adult literature, which found that the association between use and depression outcomes depended on the usage measure used [29]. The lack of association between use and outcomes is, however, in contrast to in-person CBT interventions, where there are clear associations between intervention attendance and therapeutic benefits [81,82].

There are several possible explanations for the lack of association between adherence and outcomes. First, it is possible that measuring adherence as the degree of intervention completion or use does not capture the complexity to which a person may or may not adhere to various aspects of appropriate self-administration. Specifically, current one-dimensional measures of adherence are unlikely to quantify adherence to other components of appropriate use that may be more critical to intervention outcomes such as skill enactment or accessing crisis support. However, as digital CBT has been shown to be effective in well-controlled trials, outcomes are unlikely to be completely unrelated to intervention use [80,83]. Another explanation is that users simply stop using the intervention when sufficient benefits have been obtained. This point within the intervention would be expected to vary among individuals, so no association between the amount of intervention use and benefits would be evident. This notion is supported by a study that used graphical modeling to identify a subset of users who showed the greatest improvement in depressive and anxiety symptoms despite spending less time in the intervention than other groups of users [37]. Finally, some critical components of the intervention—the active ingredients— may drive the therapeutic response, and it is completion of these components, rather than completion of the whole intervention per se that is associated with outcomes. These explanations suggest that the current narrow definitions of appropriate use as the degree of intervention completion or use are too generic, and a more nuanced approach to the measurement of adherence is required.

Poorly defined recommendations for appropriate use may have several practical implications, including the reduced effectiveness of digital CBT because of inappropriate self-administration. Given the low uptake of evidence-based interventions [19,20], this presents a significant missed opportunity to improve young people’s mental health. It may also prevent the self-selection of interventions that are best suited to an individual’s circumstances and limit endorsement and dissemination by mental health professionals. Currently, there is little information in the scientific literature to guide the development of definitions of appropriate use or the disclosure of instructions for the use of digital CBT. In response to our findings and to rectify poorly defined appropriate use and generic operationalization of adherence, we developed a framework to support the development and disclosure of definitions of appropriate use and instructions for the self-administration of digital CBT. The results are presented in Table 5. It is based on the components of manualized CBT designed to guide clinicians in delivering in-person CBT and mapped onto definitions of intended use (the purpose of the product), indications for use (who will be using the product and why it will be used), and instructions for use (how to use the product) as defined in the medical arena [84]. The use of this framework to develop nuanced instructions for use could optimize the therapeutic benefits of interventions. Finally, we suggest an active evaluation of appropriate definitions and their refinement based on the outcomes of the evaluation. For example, a method to evaluate the extent to which the degree of completion, duration, and frequency of use incorporated in definitions of appropriate use produces therapeutic benefits is to conduct a survival analysis [85,86]. Comparing instructions for use related to degree of completion, duration, and frequency of use with those demonstrated to produce reliable and clinically significant reductions in symptoms, as determined by the survival analysis, will contribute to the refinement of the appropriate use definition. The proposed development, evaluation, and refinement of definitions of appropriate use and instructions for use is represented in Figure 2.

‎View this figure

Limitations

In this review, we limited our search to CBT for depression and anxiety, making it difficult to generalize our findings to other interventions or mental health conditions. Future studies could include other interventions and conditions to develop an understanding of how appropriate use and instructions for use have been defined and disclosed more broadly in digital health interventions. Moreover, our data failed to account for the possibility that appropriate use may have been defined or instructions for use provided to participants, but not documented in the study manuscripts or published protocols. However, consistently generic operationalization of adherence across studies suggests that this explanation is unlikely, and adequate and systematic documentation of these constructs is, nevertheless, required.

Conclusions

This review has contributed to our understanding of how appropriate self-administration of digital CBT has been defined, relayed to users, measured, and associated with outcomes across various digital CBT interventions targeting depression and anxiety in young people. Overall, there is a lack of systematic reporting of nuanced definitions of appropriate use, and measures of adherence rarely provide adequate information on the degree of actual use corresponding to recommended appropriate use. There is little evidence that the degree of program completion or use is associated with intervention benefits. Our findings may, in part, explain low engagement in digital CBT; however, more work is required to better understand engagement with digital CBT and find ways to enhance it. A framework to assist in the development of guidelines for the self-administration of digital CBT has been provided.