A protocol for the review was developed and registered with PROSPERO (Registration #CRD42016049884). Reporting of the review adheres to the Preferred Reporting Items of Systematic Reviews and Meta-Analyses statement checklist [30]. Funding for the review was provided by the Canadian Institutes of Health Research (201404KRS). This organization had no involvement in any aspect of the conduct, analysis, and manuscript preparation of this review. This systematic review did not require ethics approval nor does it contain any individual person’s data in any form.

A research librarian developed the search strategies for 5 databases: MEDLINE, EMBASE, PsycINFO, CINAHL, and the Cochrane Database of Systematic Reviews using date (2000-2016) restrictions. No restriction was placed on the study design or language to capture a broad range of evidence. The strategy was peer reviewed prior to implementation. The searches included literature published until December 5, 2016. Grey literature was searched using Google Scholar and ProQuest Dissertations & Theses Global. Clinical trials were searched using clinicialtrials.gov. Conference proceedings of the last 2 years (2014-2016) of the International Society for Research on Internet Interventions were searched as well. Reference lists of included studies were also searched. Multimedia Appendix 1 provides the search terms developed for the MEDLINE database.

Studies were included if they met the following criteria: (1) assessed an eMental healthcare technology for treating or preventing anxiety or depression; (2) the technology under investigation involved children or adolescents (<18 years), or their parents or healthcare providers. Studies that included both adolescents <18 and young adults were included if the mean age of the study sample was ≤19 years to ensure that the results largely reflected implementation with children and adolescents; (3) the technology needed to be an internet-, computer-, tablet-, or mobile-based program or mobile app; (4) the technology was used within the primary or secondary healthcare system (as opposed to the school system); (5) reported on an implementation outcome as a primary or secondary measure. The 8 outcomes of interest were drawn from Proctor and colleagues’ implementation framework [31]. These constructs were defined as follows: acceptability (ie, a measure of satisfaction with a technology including attitudes, functionality, preferences, and user experience); adoption (ie, the intention, initial decision, or action to take up or utilize a technology); appropriateness (ie, the perceived fit, relevance, usefulness/helpfulness, or compatibility of a technology for a given practice setting or problem); cost, (ie, the financial impact of an implementation effort); feasibility, (ie, the extent to which a technology had utility and compatibility within the practice setting); fidelity, (ie, the degree to which a technology was implemented as it was intended); penetration, (ie, the spread and reach of a technology within a service setting and its subsystems); and sustainability, (ie, the extent to which a technology was maintained within standard operations) [31]. We excluded protocols, editorials, and studies assessing telehealth interventions, including telepsychiatry and videoconferencing.

References were organized and screened using EndNote X7.2.1. Three reviewers (AS, NDG, and MO) independently screened the titles and abstracts in the EndNote library and calculated the interrater agreement with the kappa statistic for every 100 articles screened [32]. Once a sufficiently high kappa was reached (≥0.80), the remaining references in the library were divided into 3 equally sized groups. Each reviewer was given 2 of the 3 groups, allowing each article to be assessed by 2 reviewers, and each reviewer screened the studies using the title and abstract. Three reviewers (AS, NDG, MO) independently reviewed the full-text of studies that were identified as potentially eligible using the review’s inclusion and exclusion criteria. Any discrepancies were discussed among the reviewers and taken to a third party (ASN) if no agreement could be reached.

Data were extracted by one reviewer (AS, NDG, or MO), and reviewed for accuracy and completeness by another. After verifying all of the extracted data, discrepancies were resolved by discussion or adjudication by another party (ASN). Extracted data included information on study characteristics (eg, authors, date of publication, country, and design) and implementation objectives, characteristics of the technology, study population, study setting, and implementation results. We coded statistically significant favorable ratings on measurement scales as “positive results” (eg, healthcare providers rating an intervention as highly acceptable) and statistically significant unfavorable ratings on measurement scales as “negative results” (eg, parents did not think the activities in the program were acceptable for their child’s age). Those studies that reported both positive and negative findings were coded as having “mixed results” (eg, child and parents did not show the same level of satisfaction with the intervention).

Methodological quality was assessed independently by 2 of the 3 assessors (AS, NDG, and MO). Disagreements were resolved through discussion. ASN participated when consensus could not be reached. The quality of studies was assessed using the Mixed Methods Appraisal Tool (MMAT) [33]. The scoring scale ranges from 0 (low quality) to 100 (high quality) and was pilot tested for reliability [34]. The MMAT consists of 2 screening questions applicable to all types of study designs and 3-4 questions applicable to specific study designs (eg, The questions relevant to each study design were scored with the number of ‘yes’ answers summed, divided by the total number of questions, and multiplied by 100 to give a final percentage score.) Qualitative studies were appraised for the relevance of data sources, processes used for data analyses, consideration of study context, and the researchers’ potential influences. Randomized controlled trials (RCTs) were appraised for sequence generation, allocation concealment, the completeness of outcome data, and study attrition. All other quantitative studies were appraised for recruitment strategies and sample representativeness, outcome measurements, the completeness of outcome data and study response rates, and the comparability of comparison groups (when applicable). Mixed methods studies were assessed for the relevance of the design, integration of methods, and limitations to integration. We did not exclude any studies on the basis of low-quality assessment scores.

A codebook approach [35] was used to organize data extraction according to the 8 implementation outcome categories [31]. When no implementation data were available for a particular outcome in the included paper, the category remained empty. Four team members (NDG, MO, AS, and ASN) reviewed the assignments of the study outcome data to the implementation categories, and assignments were finalized after all team members were confident that the data were categorized accurately. Descriptive statistics (counts, frequencies) were used to summarize patterns across studies.

The search strategy identified 6269 citations after removal of duplicates. Of these, 727 studies were considered potentially relevant based on their title and abstract (Figure 1). After full-text review, 46 studies (plus one erratum) articles met the inclusion criteria.

Table 1 outlines the format and delivery characteristics of the technologies assessed in the included studies. The implementation of eMental healthcare technologies for anxiety and depressive disorders in childhood or adolescence was assessed in 23 and 18 studies, respectively. Five studies assessed a technology that targeted both anxiety and depression. The location of studies was restricted to economically developed countries with the United States (20 studies) and Australia (13 studies) being the most common locations. A total of 32 studies examined internet-based technologies, 11 examined computer-based technologies, and 3 examined smartphone-based (app/short message service, SMS, text message) technologies as part of treatment.

Details on the quality of the studies are provided in Multimedia Appendix 2. In total, 11 studies on eMental healthcare technologies for anxiety were of excellent quality with a score of 100 [37,46-50,52,55,78-80], 6 were of very good quality with a score of 75 [36,38,39,44,51,53], 4 were of moderate quality with a score of 50 [42,43,45,54], and 2 were of poor quality with a MMAT score of 25 [40,41]. Studies on technologies for depression also varied in quality: 10 studies were of excellent quality [56,57,59,63-67,71,81], 2 were of very good quality [60,68], 4 were of moderate quality [58,61,69,72], and 2 were of low quality and received a score of 25 [62] and 0 [70]. Studies evaluating technologies applicable to both anxiety and depression were of excellent [75,77], very good [74], and moderate [73,76] quality. The most common factors impacting the quality scores for quantitative studies were the lack of description on how randomization sequences were generated and if/how allocation was concealed (ie, see MMAT items 2.1 and 2.2 in Multimedia Appendix 2). The common factor impacting quality scores for mixed-method studies was the lack of consideration of data triangulation (ie, see MMAT item 5.3 in Multimedia Appendix 2).

Figure 2 displays the frequency by which implementation outcomes were studied for eMental healthcare technologies. Studies on eMental healthcare technologies for anxiety most commonly evaluated acceptability (78%), adoption (43%), and feasibility (43%) of the technologies, while studies on technologies for depression evaluated appropriateness (56%) and acceptability (50%). Studies testing technologies relevant to both anxiety and depression tended to evaluate acceptability (100%), adoption (40%), and appropriateness (40%). Across all studies, cost, fidelity, and penetration were the least measured implementation outcomes, and none of the studies evaluated technology sustainability in the healthcare service/system in which the technology was employed. While positive findings were reported 60% of the time or more in relation to measures of acceptability and costs across all included studies (Figure 3), mixed findings were reported more than 50% of the time in studies that measured adoption, feasibility, and fidelity outcomes.

Table 2 outlines the implementation findings among eMental healthcare technologies for anxiety. Both positive (61%) [36,38,39,41,43,45,50,54,55,78,80] and mixed (39%) [37,40,42,46,48,49,79] findings were reported across 18 studies on technology acceptability. Positive results included high satisfaction and positive technology recommendations, with acceptability reported by parents [36,39,41,43,50], children [38,39,41,43,45,50,54,55,78,80], and healthcare providers [55]. Technology adoption was examined by 10 studies with studies reporting positive (60%) [42-44,47,50,53] and mixed (40%) [45,46,55,79] findings for technology compliance and adherence. Of the 6 studies that examined appropriateness, 4 described positive results (67%) [39,50,51,78] such as positive attitudes and perceived helpfulness of the technology among healthcare providers [39,51], while 2 studies [53,55] reported mixed results (33%) including moderate usefulness and helpfulness of the program for the youth [53]. Of the 23 studies on anxiety-directed technologies, only one examined cost, including initial implementation challenges such as startup costs, designated computers and clinic space, and technical assistance requirement [39]. Studies that examined the feasibility of anxiety technologies described more mixed (70%) [38-40,44,52,53,55] than positive (30%) [36,45,80] results, including barriers to participation such as finding time to complete tasks and ease of use. Only one study investigated technology penetration, reporting positive penetration with technology purchased by 56 child psychiatric institutions or practitioners within 1 year [51]. Studies examining eMental healthcare technologies for anxiety did not investigate or report on fidelity or sustainability.

Table 3 displays the implementation findings among eMental healthcare technologies for depression. Most studies reported the technologies as acceptable (67%) with high satisfaction [56,61,63,72], recommendations for use [71], acceptability, and ease of use among children, parents, and healthcare providers [58]. The remainder (33%) reported mixed acceptability [60,70,81]. Of the 6 studies that examined adoption, one study (17%) described high usage [59], while the remaining studies (83%) described moderate or mixed adherence [56,63-65] and usage [60]. Appropriateness was the most commonly measured outcome among eMental healthcare technologies for depression, although results varied. Four studies (40%) reported high helpfulness [57,60-62], while 6 studies (60%) reported mixed outcomes [58,63,68-70,72]. Two studies examined cost outcomes [66,67] and described intervention implementation as economically viable. Of the 6 studies that investigated feasibility, 3 (50%) reported positive or high outcomes [60,62,69], while the other 3 (50%) described mixed ease of use [70,81] and attitudes [57]. Four studies examining fidelity reported mixed results [62-65], particularly healthcare provider adherence to the program. The CATCH-IT program was the only intervention that was examined for penetration [66]. Although penetration was successful, implementing the technology successfully in 12 practices, several barriers to implementation were described, such as low levels of interest from healthcare providers and lack of established procedures and guidelines [66]. Studies examining eMental healthcare technologies for depression did not investigate or report on sustainability.

Table 4 shows the implementation findings among eMental healthcare technologies for both anxiety and depression. All 5 studies examined the acceptability of technologies aimed at treating anxiety and depression. Of these, 3 (60%) reported high satisfaction [73-75], with children and parents describing that they would not change any aspects of the program, and 2 studies (40%) reported moderate satisfaction [76,77]. Two studies examined adoption and reported low adherence to program sessions [75] and high website usage rates [77]. Of the 2 studies that examined appropriateness, both found positive attitudes and perceived helpfulness of the intervention from children and parents [73] and healthcare providers [74]. One study examined penetration of technology, reporting successful integration of the program into a practice of 2000 healthcare providers [74]. None of the 5 studies examined cost, feasibility, fidelity, or sustainability.

Complimentary to recent reviews [12,82], this systematic review reports on how the implementation of eMental healthcare technologies for children and adolescents with anxiety or depression has been studied and reported. The majority of studies included in the review were RCTs, and the methodological quality of studies was scored as moderate to high in all but a few cases. Broadly synthesized using Proctor’s [31] 8 dimensions of implementation, our review suggests that measures of acceptability, adoption, and appropriateness are more frequently reported than indicators of cost, fidelity, and sustainability. Further, the review highlights the lack of measurement precision around implementation constructs and the need to elucidate the relationship between implementation and effectiveness. Below, we highlight 5 key implications of our findings for advancing this emerging literature. Results derived from new lines of research can have significant practical value for decision-makers and administrators by providing the design of training, helping promote provider engagement, assisting in troubleshooting the obstacles that adolescents and parents encounter, and guiding projects that scale-up interventions in new contexts.

The vast majority of studies included in the review examined some dimension of acceptability, signifying that this construct is important as an indicator of effective implementation, but its measurement varied. Satisfaction, a frequent acceptability metric, was reported as high among participants (generally >70%), but was largely derived from self-reports of parents and adolescents taken at a single time-point (typically posttreatment). This means we still know little about satisfaction/dissatisfaction among those who fail to complete the treatment, or how early perceptions of satisfaction might impact effort and adherence during the later stages of treatment. More than half of the studies used nonvalidated measures of acceptability, which are problematic for assessing reliability and psychometric sensitivity. Given that almost all validated psychiatric patient satisfaction measures are validated for adults (not children and adolescents) and that developmental age affects perceptions of satisfaction with healthcare [83], our findings raise the possibility of overestimated satisfaction ratings within this literature. Low actual adherence rates reported in many studies, particularly those treating depression [84], suggest that we need to know more about the relationship among satisfaction, adherence, and clinical improvement. Most importantly, differences between those who do and do not respond to inquiries about service satisfaction (ie, bias in nonresponse [85]) and the impact of novelty (ie, bias resulting from perceived “new” or “innovative” technology [86]) imply that satisfaction is a potentially tendentious implementation metric. Without psychometrically strong and developmentally appropriate measures of satisfaction and acceptability for eMental health, stakeholders run the risk of focusing on the wrong “pragmatic” attributes when determining if a given adolescent-focused intervention is worth long-term investment. As a metric frequently used to inform decision-making around service delivery, a more systematic approach to instrumentation around the acceptability construct is vital. Future research can use pragmatic trial designs and hybrid effectiveness-implementation designs that aim to elucidate mechanisms of action between acceptability and effectiveness.

Reframing Adoption as Process not Product

While reporting on adoption (namely adherence) was fairly common in studies we reviewed, authors reported mixed findings. Moreover, none of the studies in our review formatively examined adolescent, parent, or clinician adoption in terms of readiness for eMental health, intent-to-use, or ongoing decision-making. All of these factors play a central role in behavior change associated with effective mental health treatments [87,88,89]. As adoption continues to be conceptualized in the literature primarily as a posthoc measure of “adherence,” our review suggests process-related measures of adoption could be a valuable new line of research. Most of the studies in our review reported on interventions involving multiple sessions (ie, anxiety interventions had a minimum of 8 sessions), requiring the youth to sustain and repeat interactions over time. Research from other fields, like adolescent online learning and gaming, could provide important insights here. For example, research has shown that young people’s internet self-efficacy, self-regulatory skills, and perceived quality of online learner-instructor interaction are important predictors of online engagement [90]. Rather than viewing adoption as a relatively stable end-product of individual effort, emphasis should be placed on understanding the situated, mutually constitutive relationship of a young person and the eMental healthcare environment. For example, use of modeling and path analysis techniques to identify the direct and indirect effects of provider (eg, therapeutic alliance, communication style) and technical (eg, persuasive system design components) or therapeutic (eg, comorbidity, treatment credibility) factors on adoption may provide valuable and practical insights. Improved knowledge of these processes could help administrators design training, promote provider engagement, and pre-emptively address obstacles for youth and their families.

A little less than half of the studies testing depression interventions and a third of those focused on anxiety measured some dimension of “appropriateness,” with many reporting overall mixed results. Perceptions about the suitability of a given healthcare service in a particular setting, for a particular purpose, with a particular provider and clientele can be a function of organizational culture and climate, as well as a public opinion. In practice, eMental healthcare is still considered outside standard practice by most youth mental health service providers [91]; yet, investments in eMental healthcare are rarely withdrawn because of purported safety risks or over concerns about the quality of care. This suggests that administrators, providers, and the general public feel that eMental healthcare is an appropriate treatment modality, but still continue to prioritize its use in some contexts over others. It may because the treatment ideologies (ie, beliefs about the etiology of illness, the roles of the provider/patient, and the efficacy of various treatments) [92] held by clinicians, parents, and children/youth lead them to greater skepticism about whether eMental healthcare can deliver the same quality of care [93] as face-to-face services for children and adolescents. In particular, public opinion and clinician beliefs about depression-associated risks (eg, suicide, self-harm [94]), privacy [95], and the changes in provider-patient interaction via eHealthcare delivery [96]) could impact perceived appropriateness. This could be one explanation for the higher acceptability rates of anxiety-focused interventions than of depression-focused ones. Research on appropriateness would benefit from an exploration of how eMental healthcare treatment ideologies develop for different clinical contexts (ie, diagnosis, severity) and technological modalities (ie, teleconsultation, mobile apps, SMS text messaging) and assess their subsequent influence on other implementation factors. These lines of research could eventually assist providers in selecting eMental healthcare technologies to match the intensity of treatment with the complexity of the condition (ie, stepped care).

Findings from this review also make an important contribution to expanding our understanding of feasibility. The feasibility results observed in our review were most frequently related to provider-level concerns (eg, issues of training, need for technical support). This suggests that the workflow impacts of eMental health services are a vital area for future implementation research. Given that most of the eMental healthcare technologies in our review included some form of healthcare provider interaction before or during the treatment, their role cannot be underappreciated. Many of the studies described atypical interactions for providers trained in traditional psychotherapy, including use of SMS text messages, frequent short emails, and bidirectional electronic exchanges, technical support, etc. Our review echoes recent calls to move beyond the simplistic analyses of barriers and facilitators to models of feasibility that allow researchers to test how eHealth modalities disrupt established professional roles, responsibilities, and working styles [97]. We recommend increased emphasis on underdeveloped implementation outcomes like feasibility, where few comprehensive and validated instruments exist [98]. Knowledge generated from this research could inform strategic targeting of resources and the tailoring of implementation strategies at an early stage to maximize opportunities for normalization of new eMental health workflows. Studies in our review were limited by small sample sizes and were mostly focused on measuring clinician attitudes with a lesser focus on quantifying actual clinician behaviors. Policy-focused research involving clinical practice models for eMental health [99], effective training practices for eMental health, and guidelines for selecting safe and effective eMental health tools are needed to shape behaviors that will make eMental health feasible in routine care settings.

Finally, this review highlights persistent gaps in the measurement of fidelity, penetration, and sustainability constructs. These implementation facets are important macro-level determinants of policies and strategies for technology integration [100,101]. However, because these factors often require longer-term follow-up to adequately assess, they pose unique methodological challenges for researchers. For example, sustainability and penetration constructs typically require very large sample sizes that are hard to obtain [102] and there is concern that the current methodological approaches for eMental healthcare technology have a long lag-time from initiation, to publication of outcomes or implementation. While the promise of scalable, more cost-effective treatments is widely argued in eMental health planning, there are knowledge gaps pertaining to how these services are costed, billed, and supported in the long term. As implementation research matures in this area, it will be critical to apply research methodologies that optimize the ecological validity of constructs and address these practical, real-world implications [103]. The use of structured, theory-driven implementation methodologies would provide flexibility to allow interventions to be adapted for use in routine care settings [99,104].

Although this review was rigorous, carefully executed, and employed a robust methodological approach, it is not without limitations. Technologies being deployed in healthcare systems that have not been scientifically investigated and without reported implementation data were not available for our review. We did not search databases such as the NIH Reporter, which may have yielded additional eMental healthcare technology studies. While some of the studies in the NIH Reporter may have been additionally registered in the clinicaltrials.gov registry after our search, some may not have been. Given that eMental healthcare technologies are constantly appearing and disappearing from the behavioral health service landscape, published accounts of the state of this field will likely always be slightly outdated. This is true for all eHealth-related research syntheses, and it only underscores the need to promote an “evergreen” mentality for research that acknowledges that the evidence base is always evolving. The inclusion of multiple study designs created a challenge for summarizing study features and generalizing study findings. Nonetheless, this approach allowed for the comparison of different kinds of evidences that shape real-world policy and service delivery. By not limiting our search based on study design, but rather reporting on quality via a validated appraisal tool, we established a starting point for broad critical appraisal. Finally, the inconsistent use of eHealth terminology [105-107] across the literature required us to make judgment calls regarding how to group implementation data across the 8 outcome categories. This could have resulted in the misclassification of some factors within the wrong outcome category [31].

Acceptability of eMental healthcare technology appears to be high among users, and it is the most commonly investigated implementation outcome. Perceptions of the appropriateness of eMental healthcare technology for use in healthcare varied, as did the adoption of technologies in healthcare practice. These findings suggest that the implementation science of eMental health for adolescent anxiety and depression needs to mature. Validated implementation measures as well as research designs and analytic techniques that model complex interactions and implementation contexts should be pursued in earnest. Future studies should help bridge gaps in knowledge about the fidelity of eMental health interventions over time and how eMental health technologies spread through the healthcare system, direct and indirect costs, and sustainability models. Closing these knowledge gaps has the potential to make treatments more accessible and reduce the burden of anxiety and depression on affected children and adolescents.