The Karma system is currently undergoing maintenance (Monday, January 29, 2018).
The maintenance period has been extended to 8PM EST.

Karma Credits will not be available for redeeming during maintenance.
Advertisement

Citing this Article

Right click to copy or hit: ctrl+c (cmd+c on mac)

Published on 26.06.18 in Vol 5, No 2 (2018): Apr-Jun

Preprints (earlier versions) of this paper are available at http://preprints.jmir.org/preprint/9655, first published Dec 14, 2017.

This paper is in the following e-collection/theme issue:

    Review

    eMental Healthcare Technologies for Anxiety and Depression in Childhood and Adolescence: Systematic Review of Studies Reporting Implementation Outcomes

    1Izaak Walton Killam Centre, Centre for Research in Family Health, Halifax, NS, Canada

    2Department of Psychiatry, Dalhousie University, Halifax, NS, Canada

    3Department of Pediatrics, Faculty of Medicine & Dentistry, University of Alberta, Edmonton, AB, Canada

    4Offord Centre for Child Studies, Department of Health Research Methods, Evidence and Impact (formerly Clinical Epidemiology and Biostatistics), McMaster University, Hamilton, ON, Canada

    5Department of Community Health and Epidemiology, Dalhousie University, Halifax, NS, Canada

    *these authors contributed equally

    Corresponding Author:

    Amanda S Newton, PhD

    Department of Pediatrics

    Faculty of Medicine & Dentistry

    University of Alberta

    3-526 Edmonton Clinic Health Academy

    11405 - 87 Avenue

    Edmonton, AB, T6G 1C9

    Canada

    Phone: 1 780 248 5581

    Fax:1 8883531548

    Email:


    ABSTRACT

    Background: Anxiety disorders and depression are frequent conditions in childhood and adolescence. eMental healthcare technologies may improve access to services, but their uptake within health systems is limited.

    Objective: The objective of this review was to examine and describe how the implementation of eMental healthcare technologies for anxiety disorders and depression in children and adolescents has been studied.

    Methods: We conducted a search of 5 electronic databases and gray literature. Eligible studies were those that assessed an eMental healthcare technology for treating or preventing anxiety or depression, included children or adolescents (<18 years), or their parents or healthcare providers and reported findings on technology implementation. The methodological quality of studies was evaluated using the Mixed Methods Appraisal Tool. Outcomes of interest were based on 8 implementation outcomes: acceptability (satisfaction with a technology), adoption (technology uptake and utilization), appropriateness (“fitness for purpose”), cost (financial impact of technology implementation), feasibility (extent to which a technology was successfully used), fidelity (implementation as intended), penetration (“spread” or “reach” of the technology), and sustainability (maintenance or integration of a technology within a healthcare service). For extracted implementation outcome data, we coded favorable ratings on measurement scales as “positive results” and unfavorable ratings on measurement scales as “negative results.” Those studies that reported both positive and negative findings were coded as having “mixed results.”

    Results: A total of 46 studies met the inclusion criteria, the majority of which were rated as very good to excellent in methodological quality. These studies investigated eMental healthcare technologies for anxiety (n=23), depression (n=18), or both anxiety and depression (n=5). Studies of technologies for anxiety evaluated the following: (1) acceptability (78%) reported high levels of satisfaction, (2) adoption (43%) commonly reported positive results, and (3) feasibility (43%) reported mixed results. Studies of technologies for depression evaluated the following: (1) appropriateness (56%) reported moderate helpfulness and (2) acceptability (50%) described a mix of both positive and negative findings. Studies of technologies designed to aid anxiety and depression commonly reported mixed experiences with acceptability and adoption and positive findings for appropriateness of the technologies for treatment. Across all studies, cost, fidelity, and penetration and sustainability were the least measured implementation outcomes.

    Conclusions: Acceptability of eMental healthcare technology is high among users and is the most commonly investigated implementation outcome. Perceptions of the appropriateness and adoption of eMental healthcare technology were varied. Implementation research that identifies, evaluates, and reports on costs, sustainability, and fidelity to clinical guidelines is crucial for making high-quality eMental healthcare available to children and adolescents.

    JMIR Ment Health 2018;5(2):e48

    doi:10.2196/mental.9655

    KEYWORDS



    Introduction

    Worldwide, at least 6.5% and 2.6% of children and adolescents meet the criteria for anxiety and depressive disorders, respectively [1]. The burden associated with these disorders rises sharply in childhood and peaks in adolescence and young adulthood (ages, 15-24 years) [2]. The long-term impact of anxiety and depression on children and adolescents includes significant interference with relationships, academic performance, school attendance, and daily functioning, making early intervention vital [3-8].

    Underdiagnosis and undertreatment of childhood and adolescent depression and anxiety are well-documented concerns [9,10]. The current distribution, demand, structure, and costs that underpin services for these young people make them relatively unavailable to many who need them [11]. Electronic mental (eMental) healthcare technologies, which include internet-, mobile-, and, computer-based programs as well as mobile phone apps, supposedly improve mental healthcare access and availability [12-17]. In the past 5 years, a number of literature reviews have highlighted the increase in research and development activities for eMental healthcare technologies for children and adolescents [18-22]. While conclusions regarding the efficacy and effectiveness of technologies vary depending on the review and employed methodology, reviews are unified in their assessment that eMental healthcare technologies have potential utility in healthcare systems. However, despite increased emphasis on the potential value for improving health outcomes for children and adolescents, eMental health technologies are not widely adopted within health systems [23-26].

    Distinguishing implementation effectiveness from the viewpoint of treatment effectiveness is critical for integrating eMental healthcare technologies. When uptake efforts fail, it is important to know if the failure occurred because the intervention was ineffective in the new setting (eg, lacked cultural relevance), or if an effective intervention was deployed ineffectively (eg, clinicians failed to send reminder emails as the protocol indicated). Current research on eMental healthcare technologies lack implementation frameworks [27], and the implementation literature has traditionally focused on the broadly defined eHealth [28,29], lacking a specific focus on mental healthcare. Conceptualizing and assessing implementation outcomes (ie, how implementation of a program works in specifics contexts) can advance the understanding of implementation processes (eg, cost, required in-service training, required infrastructure), enable studies of the comparative effectiveness of implementation strategies, and enhance efficiency in translating research into practice. The aim of this systematic review was to examine how the implementation of eMental healthcare technologies for children and adolescents with anxiety or depression has been studied (ie, the research questions asked, populations studied, and the rigor of the methodology used) and to describe implementation findings with respect to implementation processes and outcomes.


    Methods

    Design

    A protocol for the review was developed and registered with PROSPERO (Registration #CRD42016049884). Reporting of the review adheres to the Preferred Reporting Items of Systematic Reviews and Meta-Analyses statement checklist [30]. Funding for the review was provided by the Canadian Institutes of Health Research (201404KRS). This organization had no involvement in any aspect of the conduct, analysis, and manuscript preparation of this review. This systematic review did not require ethics approval nor does it contain any individual person’s data in any form.

    Search Strategy

    A research librarian developed the search strategies for 5 databases: MEDLINE, EMBASE, PsycINFO, CINAHL, and the Cochrane Database of Systematic Reviews using date (2000-2016) restrictions. No restriction was placed on the study design or language to capture a broad range of evidence. The strategy was peer reviewed prior to implementation. The searches included literature published until December 5, 2016. Grey literature was searched using Google Scholar and ProQuest Dissertations & Theses Global. Clinical trials were searched using clinicialtrials.gov. Conference proceedings of the last 2 years (2014-2016) of the International Society for Research on Internet Interventions were searched as well. Reference lists of included studies were also searched. Multimedia Appendix 1 provides the search terms developed for the MEDLINE database.

    Criteria for Considering Studies in the Review

    Studies were included if they met the following criteria: (1) assessed an eMental healthcare technology for treating or preventing anxiety or depression; (2) the technology under investigation involved children or adolescents (<18 years), or their parents or healthcare providers. Studies that included both adolescents <18 and young adults were included if the mean age of the study sample was ≤19 years to ensure that the results largely reflected implementation with children and adolescents; (3) the technology needed to be an internet-, computer-, tablet-, or mobile-based program or mobile app; (4) the technology was used within the primary or secondary healthcare system (as opposed to the school system); (5) reported on an implementation outcome as a primary or secondary measure. The 8 outcomes of interest were drawn from Proctor and colleagues’ implementation framework [31]. These constructs were defined as follows: acceptability (ie, a measure of satisfaction with a technology including attitudes, functionality, preferences, and user experience); adoption (ie, the intention, initial decision, or action to take up or utilize a technology); appropriateness (ie, the perceived fit, relevance, usefulness/helpfulness, or compatibility of a technology for a given practice setting or problem); cost, (ie, the financial impact of an implementation effort); feasibility, (ie, the extent to which a technology had utility and compatibility within the practice setting); fidelity, (ie, the degree to which a technology was implemented as it was intended); penetration, (ie, the spread and reach of a technology within a service setting and its subsystems); and sustainability, (ie, the extent to which a technology was maintained within standard operations) [31]. We excluded protocols, editorials, and studies assessing telehealth interventions, including telepsychiatry and videoconferencing.

    Screening for Eligibility

    References were organized and screened using EndNote X7.2.1. Three reviewers (AS, NDG, and MO) independently screened the titles and abstracts in the EndNote library and calculated the interrater agreement with the kappa statistic for every 100 articles screened [32]. Once a sufficiently high kappa was reached (≥0.80), the remaining references in the library were divided into 3 equally sized groups. Each reviewer was given 2 of the 3 groups, allowing each article to be assessed by 2 reviewers, and each reviewer screened the studies using the title and abstract. Three reviewers (AS, NDG, MO) independently reviewed the full-text of studies that were identified as potentially eligible using the review’s inclusion and exclusion criteria. Any discrepancies were discussed among the reviewers and taken to a third party (ASN) if no agreement could be reached.

    Data Extraction

    Data were extracted by one reviewer (AS, NDG, or MO), and reviewed for accuracy and completeness by another. After verifying all of the extracted data, discrepancies were resolved by discussion or adjudication by another party (ASN). Extracted data included information on study characteristics (eg, authors, date of publication, country, and design) and implementation objectives, characteristics of the technology, study population, study setting, and implementation results. We coded statistically significant favorable ratings on measurement scales as “positive results” (eg, healthcare providers rating an intervention as highly acceptable) and statistically significant unfavorable ratings on measurement scales as “negative results” (eg, parents did not think the activities in the program were acceptable for their child’s age). Those studies that reported both positive and negative findings were coded as having “mixed results” (eg, child and parents did not show the same level of satisfaction with the intervention).

    Quality Assessment

    Methodological quality was assessed independently by 2 of the 3 assessors (AS, NDG, and MO). Disagreements were resolved through discussion. ASN participated when consensus could not be reached. The quality of studies was assessed using the Mixed Methods Appraisal Tool (MMAT) [33]. The scoring scale ranges from 0 (low quality) to 100 (high quality) and was pilot tested for reliability [34]. The MMAT consists of 2 screening questions applicable to all types of study designs and 3-4 questions applicable to specific study designs (eg, The questions relevant to each study design were scored with the number of ‘yes’ answers summed, divided by the total number of questions, and multiplied by 100 to give a final percentage score.) Qualitative studies were appraised for the relevance of data sources, processes used for data analyses, consideration of study context, and the researchers’ potential influences. Randomized controlled trials (RCTs) were appraised for sequence generation, allocation concealment, the completeness of outcome data, and study attrition. All other quantitative studies were appraised for recruitment strategies and sample representativeness, outcome measurements, the completeness of outcome data and study response rates, and the comparability of comparison groups (when applicable). Mixed methods studies were assessed for the relevance of the design, integration of methods, and limitations to integration. We did not exclude any studies on the basis of low-quality assessment scores.

    Data Analysis

    A codebook approach [35] was used to organize data extraction according to the 8 implementation outcome categories [31]. When no implementation data were available for a particular outcome in the included paper, the category remained empty. Four team members (NDG, MO, AS, and ASN) reviewed the assignments of the study outcome data to the implementation categories, and assignments were finalized after all team members were confident that the data were categorized accurately. Descriptive statistics (counts, frequencies) were used to summarize patterns across studies.


    Results

    Literature Search and Selection

    The search strategy identified 6269 citations after removal of duplicates. Of these, 727 studies were considered potentially relevant based on their title and abstract (Figure 1). After full-text review, 46 studies (plus one erratum) articles met the inclusion criteria.

    Description of Included Studies

    Table 1 outlines the format and delivery characteristics of the technologies assessed in the included studies. The implementation of eMental healthcare technologies for anxiety and depressive disorders in childhood or adolescence was assessed in 23 and 18 studies, respectively. Five studies assessed a technology that targeted both anxiety and depression. The location of studies was restricted to economically developed countries with the United States (20 studies) and Australia (13 studies) being the most common locations. A total of 32 studies examined internet-based technologies, 11 examined computer-based technologies, and 3 examined smartphone-based (app/short message service, SMS, text message) technologies as part of treatment.

    Study Quality

    Details on the quality of the studies are provided in Multimedia Appendix 2. In total, 11 studies on eMental healthcare technologies for anxiety were of excellent quality with a score of 100 [37,46-50,52,55,78-80], 6 were of very good quality with a score of 75 [36,38,39,44,51,53], 4 were of moderate quality with a score of 50 [42,43,45,54], and 2 were of poor quality with a MMAT score of 25 [40,41]. Studies on technologies for depression also varied in quality: 10 studies were of excellent quality [56,57,59,63-67,71,81], 2 were of very good quality [60,68], 4 were of moderate quality [58,61,69,72], and 2 were of low quality and received a score of 25 [62] and 0 [70]. Studies evaluating technologies applicable to both anxiety and depression were of excellent [75,77], very good [74], and moderate [73,76] quality. The most common factors impacting the quality scores for quantitative studies were the lack of description on how randomization sequences were generated and if/how allocation was concealed (ie, see MMAT items 2.1 and 2.2 in Multimedia Appendix 2). The common factor impacting quality scores for mixed-method studies was the lack of consideration of data triangulation (ie, see MMAT item 5.3 in Multimedia Appendix 2).

    Figure 1. Literature search flow diagram.
    View this figure
    Table 1. Reported format and delivery characteristics of eMental Health technologies for adolescents with anxiety and depression.
    View this table

    Trends in the Study of Implementation Among eMental Healthcare Technologies

    Figure 2 displays the frequency by which implementation outcomes were studied for eMental healthcare technologies. Studies on eMental healthcare technologies for anxiety most commonly evaluated acceptability (78%), adoption (43%), and feasibility (43%) of the technologies, while studies on technologies for depression evaluated appropriateness (56%) and acceptability (50%). Studies testing technologies relevant to both anxiety and depression tended to evaluate acceptability (100%), adoption (40%), and appropriateness (40%). Across all studies, cost, fidelity, and penetration were the least measured implementation outcomes, and none of the studies evaluated technology sustainability in the healthcare service/system in which the technology was employed. While positive findings were reported 60% of the time or more in relation to measures of acceptability and costs across all included studies (Figure 3), mixed findings were reported more than 50% of the time in studies that measured adoption, feasibility, and fidelity outcomes.

    Implementation Findings for eMental Healthcare Technologies for Anxiety

    Table 2 outlines the implementation findings among eMental healthcare technologies for anxiety. Both positive (61%) [36,38,39,41,43,45,50,54,55,78,80] and mixed (39%) [37,40,42,46,48,49,79] findings were reported across 18 studies on technology acceptability. Positive results included high satisfaction and positive technology recommendations, with acceptability reported by parents [36,39,41,43,50], children [38,39,41,43,45,50,54,55,78,80], and healthcare providers [55]. Technology adoption was examined by 10 studies with studies reporting positive (60%) [42-44,47,50,53] and mixed (40%) [45,46,55,79] findings for technology compliance and adherence. Of the 6 studies that examined appropriateness, 4 described positive results (67%) [39,50,51,78] such as positive attitudes and perceived helpfulness of the technology among healthcare providers [39,51], while 2 studies [53,55] reported mixed results (33%) including moderate usefulness and helpfulness of the program for the youth [53]. Of the 23 studies on anxiety-directed technologies, only one examined cost, including initial implementation challenges such as startup costs, designated computers and clinic space, and technical assistance requirement [39]. Studies that examined the feasibility of anxiety technologies described more mixed (70%) [38-40,44,52,53,55] than positive (30%) [36,45,80] results, including barriers to participation such as finding time to complete tasks and ease of use. Only one study investigated technology penetration, reporting positive penetration with technology purchased by 56 child psychiatric institutions or practitioners within 1 year [51]. Studies examining eMental healthcare technologies for anxiety did not investigate or report on fidelity or sustainability.

    Figure 2. Implementation outcomes measured according to the mental health condition targeted.
    View this figure
    Figure 3. Conclusions reported by the authors for implementation outcomes.
    View this figure
    Table 2. Implementation findings among eMental healthcare technologies for anxiety.
    View this table
    Table 3. Implementation findings among eMental healthcare technologies for depression.
    View this table

    Implementation Findings for eMental Healthcare Technologies for Depression

    Table 3 displays the implementation findings among eMental healthcare technologies for depression. Most studies reported the technologies as acceptable (67%) with high satisfaction [56,61,63,72], recommendations for use [71], acceptability, and ease of use among children, parents, and healthcare providers [58]. The remainder (33%) reported mixed acceptability [60,70,81]. Of the 6 studies that examined adoption, one study (17%) described high usage [59], while the remaining studies (83%) described moderate or mixed adherence [56,63-65] and usage [60]. Appropriateness was the most commonly measured outcome among eMental healthcare technologies for depression, although results varied. Four studies (40%) reported high helpfulness [57,60-62], while 6 studies (60%) reported mixed outcomes [58,63,68-70,72]. Two studies examined cost outcomes [66,67] and described intervention implementation as economically viable. Of the 6 studies that investigated feasibility, 3 (50%) reported positive or high outcomes [60,62,69], while the other 3 (50%) described mixed ease of use [70,81] and attitudes [57]. Four studies examining fidelity reported mixed results [62-65], particularly healthcare provider adherence to the program. The CATCH-IT program was the only intervention that was examined for penetration [66]. Although penetration was successful, implementing the technology successfully in 12 practices, several barriers to implementation were described, such as low levels of interest from healthcare providers and lack of established procedures and guidelines [66]. Studies examining eMental healthcare technologies for depression did not investigate or report on sustainability.

    Implementation Findings for eMental Healthcare Technologies for Anxiety and Depression

    Table 4 shows the implementation findings among eMental healthcare technologies for both anxiety and depression. All 5 studies examined the acceptability of technologies aimed at treating anxiety and depression. Of these, 3 (60%) reported high satisfaction [73-75], with children and parents describing that they would not change any aspects of the program, and 2 studies (40%) reported moderate satisfaction [76,77]. Two studies examined adoption and reported low adherence to program sessions [75] and high website usage rates [77]. Of the 2 studies that examined appropriateness, both found positive attitudes and perceived helpfulness of the intervention from children and parents [73] and healthcare providers [74]. One study examined penetration of technology, reporting successful integration of the program into a practice of 2000 healthcare providers [74]. None of the 5 studies examined cost, feasibility, fidelity, or sustainability.

    Table 4. Implementation findings among eMental healthcare technologies for both anxiety and depression.
    View this table

    Discussion

    Principal Findings

    Complimentary to recent reviews [12,82], this systematic review reports on how the implementation of eMental healthcare technologies for children and adolescents with anxiety or depression has been studied and reported. The majority of studies included in the review were RCTs, and the methodological quality of studies was scored as moderate to high in all but a few cases. Broadly synthesized using Proctor’s [31] 8 dimensions of implementation, our review suggests that measures of acceptability, adoption, and appropriateness are more frequently reported than indicators of cost, fidelity, and sustainability. Further, the review highlights the lack of measurement precision around implementation constructs and the need to elucidate the relationship between implementation and effectiveness. Below, we highlight 5 key implications of our findings for advancing this emerging literature. Results derived from new lines of research can have significant practical value for decision-makers and administrators by providing the design of training, helping promote provider engagement, assisting in troubleshooting the obstacles that adolescents and parents encounter, and guiding projects that scale-up interventions in new contexts.

    Improving the Validity of Acceptability Measures

    The vast majority of studies included in the review examined some dimension of acceptability, signifying that this construct is important as an indicator of effective implementation, but its measurement varied. Satisfaction, a frequent acceptability metric, was reported as high among participants (generally >70%), but was largely derived from self-reports of parents and adolescents taken at a single time-point (typically posttreatment). This means we still know little about satisfaction/dissatisfaction among those who fail to complete the treatment, or how early perceptions of satisfaction might impact effort and adherence during the later stages of treatment. More than half of the studies used nonvalidated measures of acceptability, which are problematic for assessing reliability and psychometric sensitivity. Given that almost all validated psychiatric patient satisfaction measures are validated for adults (not children and adolescents) and that developmental age affects perceptions of satisfaction with healthcare [83], our findings raise the possibility of overestimated satisfaction ratings within this literature. Low actual adherence rates reported in many studies, particularly those treating depression [84], suggest that we need to know more about the relationship among satisfaction, adherence, and clinical improvement. Most importantly, differences between those who do and do not respond to inquiries about service satisfaction (ie, bias in nonresponse [85]) and the impact of novelty (ie, bias resulting from perceived “new” or “innovative” technology [86]) imply that satisfaction is a potentially tendentious implementation metric. Without psychometrically strong and developmentally appropriate measures of satisfaction and acceptability for eMental health, stakeholders run the risk of focusing on the wrong “pragmatic” attributes when determining if a given adolescent-focused intervention is worth long-term investment. As a metric frequently used to inform decision-making around service delivery, a more systematic approach to instrumentation around the acceptability construct is vital. Future research can use pragmatic trial designs and hybrid effectiveness-implementation designs that aim to elucidate mechanisms of action between acceptability and effectiveness.

    Reframing Adoption as Process not Product

    While reporting on adoption (namely adherence) was fairly common in studies we reviewed, authors reported mixed findings. Moreover, none of the studies in our review formatively examined adolescent, parent, or clinician adoption in terms of readiness for eMental health, intent-to-use, or ongoing decision-making. All of these factors play a central role in behavior change associated with effective mental health treatments [87,88,89]. As adoption continues to be conceptualized in the literature primarily as a posthoc measure of “adherence,” our review suggests process-related measures of adoption could be a valuable new line of research. Most of the studies in our review reported on interventions involving multiple sessions (ie, anxiety interventions had a minimum of 8 sessions), requiring the youth to sustain and repeat interactions over time. Research from other fields, like adolescent online learning and gaming, could provide important insights here. For example, research has shown that young people’s internet self-efficacy, self-regulatory skills, and perceived quality of online learner-instructor interaction are important predictors of online engagement [90]. Rather than viewing adoption as a relatively stable end-product of individual effort, emphasis should be placed on understanding the situated, mutually constitutive relationship of a young person and the eMental healthcare environment. For example, use of modeling and path analysis techniques to identify the direct and indirect effects of provider (eg, therapeutic alliance, communication style) and technical (eg, persuasive system design components) or therapeutic (eg, comorbidity, treatment credibility) factors on adoption may provide valuable and practical insights. Improved knowledge of these processes could help administrators design training, promote provider engagement, and pre-emptively address obstacles for youth and their families.

    Perceived Suitability of eMental Healthcare for Adolescents With Anxiety and Depression

    A little less than half of the studies testing depression interventions and a third of those focused on anxiety measured some dimension of “appropriateness,” with many reporting overall mixed results. Perceptions about the suitability of a given healthcare service in a particular setting, for a particular purpose, with a particular provider and clientele can be a function of organizational culture and climate, as well as a public opinion. In practice, eMental healthcare is still considered outside standard practice by most youth mental health service providers [91]; yet, investments in eMental healthcare are rarely withdrawn because of purported safety risks or over concerns about the quality of care. This suggests that administrators, providers, and the general public feel that eMental healthcare is an appropriate treatment modality, but still continue to prioritize its use in some contexts over others. It may because the treatment ideologies (ie, beliefs about the etiology of illness, the roles of the provider/patient, and the efficacy of various treatments) [92] held by clinicians, parents, and children/youth lead them to greater skepticism about whether eMental healthcare can deliver the same quality of care [93] as face-to-face services for children and adolescents. In particular, public opinion and clinician beliefs about depression-associated risks (eg, suicide, self-harm [94]), privacy [95], and the changes in provider-patient interaction via eHealthcare delivery [96]) could impact perceived appropriateness. This could be one explanation for the higher acceptability rates of anxiety-focused interventions than of depression-focused ones. Research on appropriateness would benefit from an exploration of how eMental healthcare treatment ideologies develop for different clinical contexts (ie, diagnosis, severity) and technological modalities (ie, teleconsultation, mobile apps, SMS text messaging) and assess their subsequent influence on other implementation factors. These lines of research could eventually assist providers in selecting eMental healthcare technologies to match the intensity of treatment with the complexity of the condition (ie, stepped care).

    Disruption of Established Professional Roles, Responsibilities, and Working Styles

    Findings from this review also make an important contribution to expanding our understanding of feasibility. The feasibility results observed in our review were most frequently related to provider-level concerns (eg, issues of training, need for technical support). This suggests that the workflow impacts of eMental health services are a vital area for future implementation research. Given that most of the eMental healthcare technologies in our review included some form of healthcare provider interaction before or during the treatment, their role cannot be underappreciated. Many of the studies described atypical interactions for providers trained in traditional psychotherapy, including use of SMS text messages, frequent short emails, and bidirectional electronic exchanges, technical support, etc. Our review echoes recent calls to move beyond the simplistic analyses of barriers and facilitators to models of feasibility that allow researchers to test how eHealth modalities disrupt established professional roles, responsibilities, and working styles [97]. We recommend increased emphasis on underdeveloped implementation outcomes like feasibility, where few comprehensive and validated instruments exist [98]. Knowledge generated from this research could inform strategic targeting of resources and the tailoring of implementation strategies at an early stage to maximize opportunities for normalization of new eMental health workflows. Studies in our review were limited by small sample sizes and were mostly focused on measuring clinician attitudes with a lesser focus on quantifying actual clinician behaviors. Policy-focused research involving clinical practice models for eMental health [99], effective training practices for eMental health, and guidelines for selecting safe and effective eMental health tools are needed to shape behaviors that will make eMental health feasible in routine care settings.

    Toward Sustainable, Cost-Effective, Scalable eMental Health for Anxiety and Depression

    Finally, this review highlights persistent gaps in the measurement of fidelity, penetration, and sustainability constructs. These implementation facets are important macro-level determinants of policies and strategies for technology integration [100,101]. However, because these factors often require longer-term follow-up to adequately assess, they pose unique methodological challenges for researchers. For example, sustainability and penetration constructs typically require very large sample sizes that are hard to obtain [102] and there is concern that the current methodological approaches for eMental healthcare technology have a long lag-time from initiation, to publication of outcomes or implementation. While the promise of scalable, more cost-effective treatments is widely argued in eMental health planning, there are knowledge gaps pertaining to how these services are costed, billed, and supported in the long term. As implementation research matures in this area, it will be critical to apply research methodologies that optimize the ecological validity of constructs and address these practical, real-world implications [103]. The use of structured, theory-driven implementation methodologies would provide flexibility to allow interventions to be adapted for use in routine care settings [99,104].

    Limitations

    Although this review was rigorous, carefully executed, and employed a robust methodological approach, it is not without limitations. Technologies being deployed in healthcare systems that have not been scientifically investigated and without reported implementation data were not available for our review. We did not search databases such as the NIH Reporter, which may have yielded additional eMental healthcare technology studies. While some of the studies in the NIH Reporter may have been additionally registered in the clinicaltrials.gov registry after our search, some may not have been. Given that eMental healthcare technologies are constantly appearing and disappearing from the behavioral health service landscape, published accounts of the state of this field will likely always be slightly outdated. This is true for all eHealth-related research syntheses, and it only underscores the need to promote an “evergreen” mentality for research that acknowledges that the evidence base is always evolving. The inclusion of multiple study designs created a challenge for summarizing study features and generalizing study findings. Nonetheless, this approach allowed for the comparison of different kinds of evidences that shape real-world policy and service delivery. By not limiting our search based on study design, but rather reporting on quality via a validated appraisal tool, we established a starting point for broad critical appraisal. Finally, the inconsistent use of eHealth terminology [105-107] across the literature required us to make judgment calls regarding how to group implementation data across the 8 outcome categories. This could have resulted in the misclassification of some factors within the wrong outcome category [31].

    Conclusions

    Acceptability of eMental healthcare technology appears to be high among users, and it is the most commonly investigated implementation outcome. Perceptions of the appropriateness of eMental healthcare technology for use in healthcare varied, as did the adoption of technologies in healthcare practice. These findings suggest that the implementation science of eMental health for adolescent anxiety and depression needs to mature. Validated implementation measures as well as research designs and analytic techniques that model complex interactions and implementation contexts should be pursued in earnest. Future studies should help bridge gaps in knowledge about the fidelity of eMental health interventions over time and how eMental health technologies spread through the healthcare system, direct and indirect costs, and sustainability models. Closing these knowledge gaps has the potential to make treatments more accessible and reduce the burden of anxiety and depression on affected children and adolescents.

    Acknowledgments

    The authors would like to thank Robin Featherstone, University of Alberta, who developed our search strategy; Tara Landry, Montreal General Hospital, who peer reviewed the search strategy; and Marcus O’Neill, University of Alberta, who contributed to the screening and data extraction. The authors would also like to thank the corresponding authors of included studies who responded to inquiries requesting further information about their research.

    Funding for this project was provided by the Canadian Institutes of Health Research (CIHR 201404KRS). Drs. Newton and Hartling held CIHR New Investigator Awards. Dr. McGrath held a Tier I Canada Research Chair. The CIHR had no role in 1) review design; 2) the collection, analysis, and interpretation of data; 3) the writing of the review; and 4) the decision to submit the manuscript for publication.

    Authors' Contributions

    PJM conceptualized the project idea and all authors were involved in the design of this work. NDG, AS, and ASN were responsible for the screening, data extraction, and analysis. LW, NDG, and ASN were responsible for the initial drafting of the manuscript and PJM, KB, AH, LH, MPD, and AS provided comments and edits on it at all stages. All authors have read and approved the final version of the manuscript.

    Conflicts of Interest

    None declared.

    Multimedia Appendix 1

    Search strategy for Epub ahead of print, in-process, & other non-indexed citations, Ovid MEDLINE(R) Daily and Ovid MEDLINE(R).

    PDF File (Adobe PDF File), 13KB

    Multimedia Appendix 2

    Study Quality according to Mixed Methods Appraisal Tool (MMAT).

    PDF File (Adobe PDF File), 178KB

    References

    1. Polanczyk GV, Salum GA, Sugaya LS, Caye A, Rohde LA. Annual research review: A meta-analysis of the worldwide prevalence of mental disorders in children and adolescents. J Child Psychol Psychiatry 2015 Mar;56(3):345-365. [CrossRef] [Medline]
    2. Whiteford HA, Degenhardt L, Rehm J, Baxter AJ, Ferrari AJ, Erskine HE, et al. Global burden of disease attributable to mental and substance use disorders: findings from the Global Burden of Disease Study 2010. Lancet 2013 Nov 9;382(9904):1575-1586. [CrossRef] [Medline]
    3. Essau CA, Lewinsohn PM, Olaya B, Seeley JR. Anxiety disorders in adolescents and psychosocial outcomes at age 30. J Affect Disord 2014 Jul;163:125-132 [FREE Full text] [CrossRef] [Medline]
    4. Fergusson DM, Woodward LJ. Mental health, educational, and social role outcomes of adolescents with depression. Arch Gen Psychiatry 2002 Mar;59(3):225-231. [Medline]
    5. Mazzone L, Ducci F, Scoto MC, Passaniti E, D'Arrigo VG, Vitiello B. The role of anxiety symptoms in school performance in a community sample of children and adolescents. BMC Public Health 2007;7:347 [FREE Full text] [CrossRef] [Medline]
    6. Naicker K, Galambos NL, Zeng Y, Senthilselvan A, Colman I. Social, demographic, and health outcomes in the 10 years following adolescent depression. J Adolesc Health 2013 May;52(5):533-538. [CrossRef] [Medline]
    7. Verboom CE, Sijtsema JJ, Verhulst FC, Penninx BWJH, Ormel J. Longitudinal associations between depressive problems, academic performance, and social functioning in adolescent boys and girls. Dev Psychol 2014 Jan;50(1):247-257. [CrossRef] [Medline]
    8. Woodward LJ, Fergusson DM. Life course outcomes of young people with anxiety disorders in adolescence. J Am Acad Child Adolesc Psychiatry 2001 Sep;40(9):1086-1093. [CrossRef] [Medline]
    9. Merikangas KR, He J, Brody D, Fisher PW, Bourdon K, Koretz DS. Prevalence and treatment of mental disorders among US children in the 2001-2004 NHANES. Pediatrics 2010 Jan;125(1):75-81 [FREE Full text] [CrossRef] [Medline]
    10. Pelletier L, O'Donnell S, Dykxhoorn J, McRae L, Patten SB. Under-diagnosis of mood disorders in Canada. Epidemiol Psychiatr Sci 2017 Aug;26(4):414-423. [CrossRef] [Medline]
    11. Hickie IB, McGorry PD. Increased access to evidence-based primary mental health care: will the implementation match the rhetoric? Med J Aust 2007 Jul 16;187(2):100-103. [Medline]
    12. Boydell KM, Hodgins M, Pignatiello A, Teshima J, Edwards H, Willis D. Using technology to deliver mental health services to children and youth: a scoping review. J Can Acad Child Adolesc Psychiatry 2014 May;23(2):87-99 [FREE Full text] [Medline]
    13. Christensen H, Hickie IB. E-mental health: a new era in delivery of mental health services. Med J Aust 2010 Jun 07;192(11 Suppl):S2-S3. [Medline]
    14. Christensen H, Hickie IB. Using e-health applications to deliver new mental health services. Med J Aust 2010 Jun 7;192(11 Suppl):S53-S56. [Medline]
    15. Hollis C, Morriss R, Martin J, Amani S, Cotton R, Denis M, et al. Technological innovations in mental healthcare: harnessing the digital revolution. Br J Psychiatry 2015 Apr;206(4):263-265. [CrossRef] [Medline]
    16. Lal S, Adair CE. E-mental health: a rapid review of the literature. Psychiatr Serv 2014 Jan 1;65(1):24-32. [CrossRef] [Medline]
    17. Riper H, Andersson G, Christensen H, Cuijpers P, Lange A, Eysenbach G. Theme issue on e-mental health: a growing field in internet research. J Med Internet Res 2010;12(5):e74 [FREE Full text] [CrossRef] [Medline]
    18. Ebert DD, Zarski A, Christensen H, Stikkelbroek Y, Cuijpers P, Berking M, et al. Internet and computer-based cognitive behavioral therapy for anxiety and depression in youth: a meta-analysis of randomized controlled outcome trials. PLoS One 2015;10(3):e0119895 [FREE Full text] [CrossRef] [Medline]
    19. Farrer L, Gulliver A, Chan JKY, Batterham PJ, Reynolds J, Calear A, et al. Technology-based interventions for mental health in tertiary students: systematic review. J Med Internet Res 2013;15(5):e101 [FREE Full text] [CrossRef] [Medline]
    20. Huguet A, Rao S, McGrath PJ, Wozney L, Wheaton M, Conrod J, et al. A Systematic Review of Cognitive Behavioral Therapy and Behavioral Activation Apps for Depression. PLoS One 2016;11(5):e0154248 [FREE Full text] [CrossRef] [Medline]
    21. Pennant ME, Loucas CE, Whittington C, Creswell C, Fonagy P, Fuggle P, et al. Computerised therapies for anxiety and depression in children and young people: a systematic review and meta-analysis. Behav Res Ther 2015 Apr;67:1-18. [CrossRef] [Medline]
    22. Reyes-Portillo JA, Mufson L, Greenhill LL, Gould MS, Fisher PW, Tarlow N, et al. Web-based interventions for youth internalizing problems: a systematic review. J Am Acad Child Adolesc Psychiatry 2014 Dec;53(12):1254-1270.e5. [CrossRef] [Medline]
    23. Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O. Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q 2004;82(4):581-629 [FREE Full text] [CrossRef] [Medline]
    24. Kazdin AE, Blase SL. Rebooting Psychotherapy Research and Practice to Reduce the Burden of Mental Illness. Perspect Psychol Sci 2011 Jan;6(1):21-37. [CrossRef] [Medline]
    25. Li J, Talaei-Khoei A, Seale H, Ray P, Macintyre CR. Health Care Provider Adoption of eHealth: Systematic Literature Review. Interact J Med Res 2013;2(1):e7 [FREE Full text] [CrossRef] [Medline]
    26. World Health Organization. Global diffusion of eHealth: Making universal health coverage achievable   URL: http://apps.who.int/iris/bitstream/10665/252529/1/9789241511780-eng.pdf?ua=1 [WebCite Cache]
    27. Lyon AR, Wasse JK, Ludwig K, Zachry M, Bruns EJ, Unützer J, et al. The Contextualized Technology Adaptation Process (CTAP): Optimizing Health Information Technology to Improve Mental Health Systems. Adm Policy Ment Health 2016 May;43(3):394-409. [CrossRef] [Medline]
    28. Mair FS, May C, O'Donnell C, Finch T, Sullivan F, Murray E. Factors that promote or inhibit the implementation of e-health systems: an explanatory systematic review. Bull World Health Organ 2012 May 1;90(5):357-364 [FREE Full text] [CrossRef] [Medline]
    29. Ross J, Stevenson F, Lau R, Murray E. Factors that influence the implementation of e-health: a systematic review of systematic reviews (an update). Implement Sci 2016 Oct 26;11(1):146 [FREE Full text] [CrossRef] [Medline]
    30. Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Int J Surg 2010;8(5):336-341 [FREE Full text] [CrossRef] [Medline]
    31. Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health 2011 Mar;38(2):65-76 [FREE Full text] [CrossRef] [Medline]
    32. Altman D. Practical statistics for medical research. In: Practical statistics for medical research. London: Chapman & Hall; 1991.
    33. Pluye P, Robert E, Cargo M, Bartlett G, O'Cathain A, Griffiths F. McGill University. 2011. Proposal: a mixed methods appraisal tool for systematic mixed studies reviews   URL: http:/​/mixedmethodsappraisaltoolpublic.​pbworks.com/​w/​file/​fetch/​84371689/​MMAT%202011%20criteria%20and%20tutorial%202011-06-29updated2014.​08.​21.​pdf [accessed 2018-06-05] [WebCite Cache]
    34. Pace R, Pluye P, Bartlett G, Macaulay AC, Salsberg J, Jagosh J, et al. Testing the reliability and efficiency of the pilot Mixed Methods Appraisal Tool (MMAT) for systematic mixed studies review. Int J Nurs Stud 2012 Jan;49(1):47-53. [CrossRef] [Medline]
    35. Crabtree B, Miller W. A template approach to text analysis: developing and using codebooks. In: Research methods for primary care. Thousand Oaks, CA: Sage Publications; 1992:93-109.
    36. Morgan AJ, Rapee RM, Bayer JK. Prevention and early intervention of anxiety problems in young children: A pilot evaluation of Cool Little Kids Online. Internet Interventions 2016 May;4:105-112. [CrossRef]
    37. Crawford EA, Salloum A, Lewin AB, Andel R, Murphy TK, Storch EA. A Pilot Study of Computer-Assisted Cognitive Behavioral Therapy for Childhood Anxiety in Community Mental Health Centers. J Cogn Psychother 2013 Aug 01;27(3):221-234. [CrossRef]
    38. Khanna MS, Kendall PC. Computer-assisted cognitive behavioral therapy for child anxiety: results of a randomized clinical trial. J Consult Clin Psychol 2010 Oct;78(5):737-745. [CrossRef] [Medline]
    39. Salloum A, Crawford EA, Lewin AB, Storch EA. Consumers' and providers' perceptions of utilizing a computer-assisted cognitive behavioral therapy for childhood anxiety. Behav Cogn Psychother 2015 Jan;43(1):31-41. [CrossRef] [Medline]
    40. Salloum A, Johnco C, Lewin AB, McBride NM, Storch EA. Barriers to access and participation in community mental health treatment for anxious children. J Affect Disord 2016 May 15;196:54-61. [CrossRef] [Medline]
    41. Storch EA, Salloum A, King MA, Crawford EA, Andel R, McBride NM, et al. A Randomized Controlled Trial In Community Mental Health Centers Of Computer-Assisted Cognitive Behavioral Therapy Versus Treatment As Usual For Children With Anxiety. Depress Anxiety 2015 Nov;32(11):843-852. [CrossRef] [Medline]
    42. Vigerland S, Ljótsson B, Thulin U, Öst L, Andersson G, Serlachius E. Internet-delivered cognitive behavioural therapy for children with anxiety disorders: A randomised controlled trial. Behav Res Ther 2016 Jan;76:47-56 [FREE Full text] [CrossRef] [Medline]
    43. Reuland MM, Teachman BA. Interpretation bias modification for youth and their parents: a novel treatment for early adolescent social anxiety. J Anxiety Disord 2014 Dec;28(8):851-864 [FREE Full text] [CrossRef] [Medline]
    44. Cunningham M, Wuthrich V. Examination of Barriers to Treatment and User Preferences With Computer-based Therapy Using The Cool Teens CD for Adolescent Anxiety. EJAP 2008 Dec 23;4(2):12-17. [CrossRef]
    45. Pramana G, Parmanto B, Kendall PC, Silk JS. The SmartCAT: an m-health platform for ecological momentary intervention in child anxiety treatment. Telemed J E Health 2014 May;20(5):419-427 [FREE Full text] [CrossRef] [Medline]
    46. Donovan CL, March S. Online CBT for preschool anxiety disorders: a randomised control trial. Behav Res Ther 2014 Jul;58:24-35. [CrossRef] [Medline]
    47. Anderson REE, Spence SH, Donovan CL, March S, Prosser S, Kenardy J. Working alliance in online cognitive behavior therapy for anxiety disorders in youth: comparison with clinic delivery and its role in predicting outcome. J Med Internet Res 2012;14(3):e88 [FREE Full text] [CrossRef] [Medline]
    48. Spence SH, Donovan CL, March S, Gamble A, Anderson RE, Prosser S, et al. A randomized controlled trial of online versus clinic-based CBT for adolescent anxiety. J Consult Clin Psychol 2011 Oct;79(5):629-642. [CrossRef] [Medline]
    49. March S, Spence SH, Donovan CL. The efficacy of an internet-based cognitive-behavioral therapy intervention for child anxiety disorders. J Pediatr Psychol 2009 Jun;34(5):474-487 [FREE Full text] [CrossRef] [Medline]
    50. Spence SH, Holmes JM, March S, Lipp OV. The feasibility and outcome of clinic plus internet delivery of cognitive-behavior therapy for childhood anxiety. J Consult Clin Psychol 2006 Jun;74(3):614-621. [CrossRef] [Medline]
    51. Brezinka V. Ricky and the spider - a video game to support cognitive behavioural treatment of children with obsessive-compulsive disorder. Clinical Neuropsychiatry 2013;3(1):6-12. [CrossRef]
    52. Cunningham M, Rapee R, Lyneham H. Feedback to a prototype self-help computer program for anxiety disorders in adolescents. Aust E J Adv Ment Health 2006;5(3):1-9.
    53. Wuthrich VM, Rapee RM, Cunningham MJ, Lyneham HJ, Hudson JL, Schniering CA. A randomized controlled trial of the Cool Teens CD-ROM computerized program for adolescent anxiety. J Am Acad Child Adolesc Psychiatry 2012 Mar;51(3):261-270. [CrossRef] [Medline]
    54. Tillfors M, Andersson G, Ekselius L, Furmark T, Lewenhaupt S, Karlsson A, et al. A randomized trial of Internet-delivered treatment for social anxiety disorder in high school students. Cogn Behav Ther 2011;40(2):147-157. [CrossRef] [Medline]
    55. Sarver NW, Beidel DC, Spitalnick JS. The feasibility and acceptability of virtual environments in the treatment of childhood social anxiety disorder. J Clin Child Adolesc Psychol 2014;43(1):63-73 [FREE Full text] [CrossRef] [Medline]
    56. Simmons MB, Elmes A, McKenzie JE, Trevena L, Hetrick SE. Right choice, right time: Evaluation of an online decision aid for youth depression. Health Expect 2017 Aug;20(4):714-723 [FREE Full text] [CrossRef] [Medline]
    57. Hetrick SE, Dellosa MK, Simmons MB, Phillips L. Development and pilot testing of an online monitoring tool of depression symptoms and side effects for young people being treated for depression. Early Interv Psychiatry 2015 Feb;9(1):66-69. [CrossRef] [Medline]
    58. Hetrick SE, Goodall J, Yuen HP, Davey CG, Parker AG, Robinson J, et al. Comprehensive Online Self-Monitoring to Support Clinicians Manage Risk of Suicide in Youth Depression. Crisis 2017 May;38(3):147-157. [CrossRef]
    59. Rice S, Gleeson J, Davey C, Hetrick S, Parker A, Lederman R, et al. Moderated online social therapy for depression relapse prevention in young people: pilot study of a 'next generation' online intervention. Early Interv Psychiatry 2016 Jun 17. [CrossRef] [Medline]
    60. Ranney ML, Freeman JR, Connell G, Spirito A, Boyer E, Walton M, et al. A Depression Prevention Intervention for Adolescents in the Emergency Department. J Adolesc Health 2016 Oct;59(4):401-410. [CrossRef] [Medline]
    61. Kobak KA, Mundt JC, Kennard B. Integrating technology into cognitive behavior therapy for adolescent depression: a pilot study. Ann Gen Psychiatry 2015;14:37 [FREE Full text] [CrossRef] [Medline]
    62. Davidson TM, Yuen EK, Felton JW, McCauley J, Gros KS, Ruggiero KJ. Feasibility assessment of a brief, web-based behavioral activation intervention for adolescents with depressed mood. Int J Psychiatry Med 2014;48(1):69-82. [CrossRef] [Medline]
    63. Van Voorhees BW, Fogel J, Pomper BE, Marko M, Reid N, Watson N, et al. Adolescent Dose and Ratings of an Internet-Based Depression Prevention Program: A Randomized Trial of Primary Care Physician Brief Advice versus a Motivational Interview. J Cogn Behav Psychother 2009;9(1):1-19 [FREE Full text] [Medline]
    64. Van Voorhees BW, Fogel J, Reinecke MA, Gladstone T, Stuart S, Gollan J, et al. Randomized clinical trial of an Internet-based depression prevention program for adolescents (Project CATCH-IT) in primary care: 12-week outcomes. J Dev Behav Pediatr 2009 Feb;30(1):23-37. [CrossRef] [Medline]
    65. Van Voorhees BW, Vanderplough-Booth K, Fogel J, Gladstone T, Bell C, Stuart S, et al. Integrative internet-based depression prevention for adolescents: a randomized clinical trial in primary care for vulnerability and protective factors. J Can Acad Child Adolesc Psychiatry 2008 Nov;17(4):184-196 [FREE Full text] [Medline]
    66. Van Voorhees BW, Watson N, Bridges JFP, Fogel J, Galas J, Kramer C, et al. Development and pilot study of a marketing strategy for primary care/internet-based depression prevention intervention for adolescents (the CATCH-IT intervention). Prim Care Companion J Clin Psychiatry 2010;12(3) [FREE Full text] [CrossRef] [Medline]
    67. Ruby A, Marko-Holguin M, Fogel J, Van Voorhees BW. Economic analysis of an internet-based depression prevention intervention. J Ment Health Policy Econ 2013 Sep;16(3):121-130. [Medline]
    68. Iloabachie C, Wells C, Goodwin B, Baldwin M, Vanderplough-Booth K, Gladstone T, et al. Adolescent and parent experiences with a primary care/Internet-based depression prevention intervention (CATCH-IT). Gen Hosp Psychiatry 2011;33(6):543-555. [CrossRef] [Medline]
    69. Gladstone T, Marko-Holguin M, Henry J, Fogel J, Diehl A, Van Voorhees BW. Understanding adolescent response to a technology-based depression prevention program. J Clin Child Adolesc Psychol 2014;43(1):102-114. [CrossRef] [Medline]
    70. Eisen JC, Marko-Holguin M, Fogel J, Cardenas A, Bahn M, Bradford N, et al. Pilot Study of Implementation of an Internet-Based Depression Prevention Intervention (CATCH-IT) for Adolescents in 12 US Primary Care Practices: Clinical and Management/Organizational Behavioral Perspectives. Prim Care Companion CNS Disord 2013;15(6) [FREE Full text] [CrossRef] [Medline]
    71. Merry SN, Stasiak K, Shepherd M, Frampton C, Fleming T, Lucassen MFG. The effectiveness of SPARX, a computerised self help intervention for adolescents seeking help for depression: randomised controlled non-inferiority trial. BMJ 2012;344:e2598 [FREE Full text] [Medline]
    72. Demaso DR, Marcus NE, Kinnamon C, Gonzalez-Heydrich J. Depression experience journal: a computer-based intervention for families facing childhood depression. J Am Acad Child Adolesc Psychiatry 2006 Feb;45(2):158-165. [CrossRef] [Medline]
    73. Sapru I, Khalid-Khan S, Choi E, Alavi N, Patel A, Sutton C, et al. Effectiveness of online versus live multi-family psychoeducation group therapy for children and adolescents with mood or anxiety disorders: a pilot study. Int J Adolesc Med Health 2016 Oct 14. [CrossRef] [Medline]
    74. Brezinka V. Computer games supporting cognitive behaviour therapy in children. Clin Child Psychol Psychiatry 2014 Jan;19(1):100-110. [CrossRef] [Medline]
    75. Bobier C, Stasiak K, Mountford H, Merry S, Moor S. When ‘e’ therapy enters the hospital: Examination of the feasibility and acceptability of SPARX (a cCBT programme) in an adolescent inpatient unit. Advances in Mental Health 2014 Dec 17;11(3):286-292. [CrossRef]
    76. Hoek W, Schuurmans J, Koot HM, Cuijpers P. Effects of Internet-based guided self-help problem-solving therapy for adolescents with depression and anxiety: a randomized controlled trial. PLoS One 2012;7(8):e43485 [FREE Full text] [CrossRef] [Medline]
    77. Ercan S. Evaluation of a mental health website for teenagers. Psychiatric Bulletin 2006 May 01;30(5):175-178. [CrossRef]
    78. Lenhard F, Vigerland S, Andersson E, Rück C, Mataix-Cols D, Thulin U, et al. Internet-delivered cognitive behavior therapy for adolescents with obsessive-compulsive disorder: an open trial. PLoS One 2014;9(6):e100773 [FREE Full text] [CrossRef] [Medline]
    79. Vigerland S, Thulin U, Ljótsson B, Svirsky L, Ost L, Lindefors N, et al. Internet-delivered CBT for children with specific phobia: a pilot study. Cogn Behav Ther 2013;42(4):303-314. [CrossRef] [Medline]
    80. Lenhard F, Vigerland S, Engberg H, Hallberg A, Thermaenius H, Serlachius E. “On My Own, but Not Alone” - Adolescents' Experiences of Internet-Delivered Cognitive Behavior Therapy for Obsessive-Compulsive Disorder. PLoS One 2016;11(10):e0164311 [FREE Full text] [CrossRef] [Medline]
    81. Carrasco A. Acceptability of an adventure video game in the treatment of female adolescents with symptoms of depression. Research in Psychotherapy: Psychopathology, Process and Outcome 2016;19(1):10-18.
    82. Struthers A, Charette C, Bapuji SB, Winters S, Ye X, Metge C, et al. The Acceptability of E-mental Health Services for Children, Adolescents, and Young Adults: A Systematic Search and Review. Canadian Journal of Community Mental Health 2015 Jul;34(2):1-21. [CrossRef]
    83. Madan A, Sharp C, Newlin E, Vanwoerden S, Fowler JC. Adolescents Are Less Satisfied with Inpatient Psychiatric Care than Their Parents: Does It Matter? J Healthc Qual 2016;38(4):e19-e28. [CrossRef] [Medline]
    84. Christensen H, Griffiths K, Farrer L. Adherence in internet interventions for anxiety and depression. Journal of Medical Internet Research 2011 Apr;11(2) [FREE Full text] [CrossRef] [Medline]
    85. Cheung KL, Ten KPM, Smit C, de VH, Pieterse ME. The impact of non-response bias due to sampling in public health studies: A comparison of voluntary versus mandatory recruitment in a Dutch national survey on adolescent health. BMC Public Health 2017 Dec 23;17(1):276 [FREE Full text] [CrossRef] [Medline]
    86. Kalyanaraman S, Sundar SS. The Psychological Appeal of Personalized Content in Web Portals: Does Customization Affect Attitudes and Behavior? J Communication 2006 Mar;56(1):110-132. [CrossRef]
    87. John J. Patient satisfaction: the impact of past experience. J Health Care Mark 1992 Sep;12(3):56-64. [Medline]
    88. Michie S, Johnston M, Abraham C, Lawton R, Parker D, Walker A. Making psychological theory useful for implementing evidence based practice: a consensus approach. Qual Saf Health Care 2005 Feb;14(1):26-33 [FREE Full text] [CrossRef] [Medline]
    89. Eccles MP, Hrisos S, Francis J, Kaner EF, Dickinson HO, Beyer F, et al. Do self- reported intentions predict clinicians' behaviour: a systematic review. Implement Sci 2006 Nov 21;1:28 [FREE Full text] [CrossRef] [Medline]
    90. Kuo Y, Walker AE, Belland BR, Schroder KEE. A predictive study of student satisfaction in online education programs. Int Rev of Res in Open and Dis Learn 2013;14(1):16-39.
    91. Montague AE, Varcin KJ, Simmons MB, Parker AG. Putting Technology Into Youth Mental Health Practice: Young People's Perspectives. SAGE Open 2015 Apr 15;5(2). [CrossRef]
    92. Scheid TL. An explication of treatment ideology among mental health care providers. Sociol Health & Illness 1994 Nov;16(5):668-693. [CrossRef]
    93. Baker-Ericzén MJ, Jenkins MM, Haine-Schlagel R. Therapist, Parent, and Youth Perspectives of Treatment Barriers to Family-Focused Community Outpatient Mental Health Services. J Child Fam Stud 2013 Aug 01;22(6):854-868 [FREE Full text] [CrossRef] [Medline]
    94. Flewett T. Clinical Risk Management: An introductory text for mental health clinicians. Australia: Elsevier; 2010.
    95. Laxman K, Krishnan S, Dhillon J. Barriers to adoption of consumer health informatics applications for health self management. Health Science Journal 2015;9(5):1-7 [FREE Full text]
    96. Sarvet B, Torous J. Health information technology for child and adolescent psychiatry. Child and Adolescent Psychiatric Clinics of North America 2016;26(1).
    97. Broens THF, Huis IVRMHA, Vollenbroek-Hutten MMR, Hermens HJ, van HAT, Nieuwenhuis LJM. Determinants of successful telemedicine implementations: a literature study. J Telemed Telecare 2007;13(6):303-309. [CrossRef] [Medline]
    98. Lewis CC, Fischer S, Weiner BJ, Stanick C, Kim M, Martinez RG. Outcomes for implementation science: an enhanced systematic review of instruments using evidence-based rating criteria. Implement Sci 2015 Nov 04;10:155 [FREE Full text] [CrossRef] [Medline]
    99. Reynolds J, Griffiths KM, Cunningham JA, Bennett K, Bennett A. Clinical Practice Models for the Use of E-Mental Health Resources in Primary Health Care by Health Professionals and Peer Workers: A Conceptual Framework. JMIR Ment Health 2015;2(1):e6 [FREE Full text] [CrossRef] [Medline]
    100. Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci 2009;4:50 [FREE Full text] [CrossRef] [Medline]
    101. Durlak JA, DuPre EP. Implementation matters: a review of research on the influence of implementation on program outcomes and the factors affecting implementation. Am J Community Psychol 2008 Jun;41(3-4):327-350. [CrossRef] [Medline]
    102. Proctor EK, Landsverk J, Aarons G, Chambers D, Glisson C, Mittman B. Implementation research in mental health services: an emerging science with conceptual, methodological, and training challenges. Adm Policy Ment Health 2009 Jan;36(1):24-34 [FREE Full text] [CrossRef] [Medline]
    103. Curran GM, Bauer M, Mittman B, Pyne JM, Stetler C. Effectiveness-implementation hybrid designs: combining elements of clinical effectiveness and implementation research to enhance public health impact. Med Care 2012 Mar;50(3):217-226 [FREE Full text] [CrossRef] [Medline]
    104. Cohen DJ, Crabtree BF, Etz RS, Balasubramanian BA, Donahue KE, Leviton LC, et al. Fidelity versus flexibility: translating evidence-based research into practice. Am J Prev Med 2008 Nov;35(5 Suppl):S381-S389. [CrossRef] [Medline]
    105. Dixon B, Zafar A, McGowan J. Development of a taxonomy for health information technology. Studies in Health Technology and Informatics 2007;129:616-620.
    106. Oh H, Rizo C, Enkin M, Jadad A. What is eHealth?: a systematic review of published definitions. World Hosp Health Serv 2005;41(1):32-40. [Medline]
    107. Pagliari C, Sloan D, Gregor P, Sullivan F, Detmer D, Kahan JP, et al. What is eHealth (4): a scoping exercise to map the field. J Med Internet Res 2005;7(1):e9 [FREE Full text] [CrossRef] [Medline]


    Abbreviations

    MEDLINE: Medical Literature Analysis and Retrieval System Online, or MEDLARS Online
    EMBASE: Excerpta Medica dataBASE
    CINAHL: Cumulative Index to Nursing and Allied Health Literature
    MMAT: Mixed Methods Appraisal Tool
    RCT: randomized controlled trials


    Edited by J Prescott; submitted 14.12.17; peer-reviewed by D Gratzer, J Stapleton, S March, A Radovic-Stakic, G Pramana; comments to author 29.01.18; revised version received 15.04.18; accepted 19.05.18; published 26.06.18

    ©Lori Wozney, Patrick J McGrath, Nicole D Gehring, Kathryn Bennett, Anna Huguet, Lisa Hartling, Michele P Dyson, Amir Soleimani, Amanda S Newton. Originally published in JMIR Mental Health (http://mental.jmir.org), 26.06.2018.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Mental Health, is properly cited. The complete bibliographic information, a link to the original publication on http://mental.jmir.org/, as well as this copyright and license information must be included.