Maintenance Notice

Due to necessary scheduled maintenance, the JMIR Publications website will be unavailable from Wednesday, July 01, 2020 at 8:00 PM to 10:00 PM EST. We apologize in advance for any inconvenience this may cause you.

Who will be affected?

Advertisement

Citing this Article

Right click to copy or hit: ctrl+c (cmd+c on mac)

Published on 24.07.20 in Vol 7, No 7 (2020): July

Preprints (earlier versions) of this paper are available at http://preprints.jmir.org/preprint/13356, first published Jan 10, 2019.

This paper is in the following e-collection/theme issue:

    Original Paper

    An Internet-Based Cognitive Behavioral Program for Adolescents With Anxiety: Pilot Randomized Controlled Trial

    1Glenrose Rehabilitation Hospital, Edmonton, AB, Canada

    2IWK Health Centre, Halifax, NS, Canada

    3Dalhousie University, Halifax, NS, Canada

    4Department of Pediatrics, Faculty of Medicine & Dentistry, University of Alberta, Edmonton, AB, Canada

    5Department of Neurosurgery, University of Ottawa, Ottawa, ON, Canada

    6University of Calgary, Calgary, AB, Canada

    7School of Public Health, University of Alberta, Edmonton, AB, Canada

    8University of Alberta, Edmonton, AB, Canada

    Corresponding Author:

    Amanda Newton, PhD

    Department of Pediatrics

    Faculty of Medicine & Dentistry

    University of Alberta

    3-526 Edmonton Clinic Health Academy

    11405 - 87 Avenue

    Edmonton, AB, T6G 1C9

    Canada

    Phone: 1 7802485581

    Email: mandi.newton@ualberta.ca


    ABSTRACT

    Background: Internet-based cognitive behavioral therapy (ICBT) is a treatment approach recently developed and studied to provide frontline treatment to adolescents with anxiety disorders.

    Objective: This study aimed to pilot procedures and obtain data on methodological processes and intervention satisfaction to determine the feasibility of a definitive randomized controlled trial (RCT) to test the effectiveness of a self-managed ICBT program, Breathe (Being Real, Easing Anxiety: Tools Helping Electronically), for adolescents with anxiety concerns.

    Methods: This study employed a two-arm, multisite, pilot RCT. Adolescents aged 13 to 17 years with a self-identified anxiety concern were recruited online from health care settings and school-based mental health care services across Canada between April 2014 and May 2016. We compared 8 weeks of ICBT with ad hoc telephone and email support (Breathe experimental group) to access to a static webpage listing anxiety resources (control group). The primary outcome was the change in self-reported anxiety from baseline to 8 weeks (posttreatment), which was used to determine the sample size for a definitive RCT. Secondary outcomes were recruitment and retention rates, a minimal clinically important difference (MCID) for the primary outcome, intervention acceptability and satisfaction, use of cointerventions, and health care resource use, including a cost-consequence analysis.

    Results: Of the 588 adolescents screened, 94 were eligible and enrolled in the study (49 adolescents were allocated to Breathe and 45 were allocated to the control group). Analysis was based on 74% (70/94) of adolescents who completed baseline measures and progressed through the study. Enrolled adolescents were, on average, 15.3 years old (SD 1.2) and female (63/70, 90%). Retention rates at 8 weeks were 28% (13/46; Breathe group) and 58% (24/43; control group). Overall, 39% (14/36) of adolescents provided feedback on completion of the Breathe program. Adolescents’ scores on a satisfaction survey indicated a moderate level of satisfaction. All but one adolescent indicated that Breathe was easy to use and they understood all the material presented. The most frequent barrier identified for program completion was difficulty in completing exposure activities. The power analysis indicated that 177 adolescents per group would be needed to detect a medium effect size (d=0.3) between groups in a definitive trial. Data for calculating an MCID or conducting a cost-consequence analysis were insufficient due to a low response rate at 8 weeks.

    Conclusions: Adolescents were moderately satisfied with Breathe. However, program adjustments will be needed to address attrition and reduce perceived barriers to completing key aspects of the program. A definitive RCT to evaluate the effectiveness of the program is feasible if protocol adjustments are made to improve recruitment and retention to ensure timely study completion and increase the completeness of the data at each outcome measurement time point.

    Trial Registration: ClinicalTrials.gov NCT02059226; http://clinicaltrials.gov/ct2/show/NCT02059226.

    JMIR Ment Health 2020;7(7):e13356

    doi:10.2196/13356

    KEYWORDS



    Introduction

    Background: Anxiety, Cognitive Behavioral Therapy, and Internet-Based Approaches

    Anxiety disorders are the most prevalent of mental illnesses to be diagnosed before the age of 18 years. Approximately 1 in every 3 adolescents will meet criteria for an anxiety disorder in their lifetime [1]. The global burden of such disorders in adolescence is significant. In 2010, anxiety disorders accounted for 14.6% of disability-adjusted life years (DALYs), with the highest proportion of total DALYs seen in young people aged 10 to 29 years [2].

    The current classification system for diagnosing anxiety disorders identifies several types: separation anxiety disorder, selective mutism, generalized anxiety disorder (GAD), social phobia, specific phobia, panic disorder, and agoraphobia [3]. In general, these disorders are characterized by excessive, persistent fear or worry that interferes with day-to-day functioning; such impairments can be pervasive, affecting activities of daily living, school performance, and interpersonal relationships [4]. General worries are common among adolescents, and those who experience impairments that do not meet the threshold for any particular diagnosis may be considered as having a subthreshold disorder. Research exploring subthreshold anxiety among adolescents is limited; however, a 2014 study by Burstein et al [5] found that the prevalence of subthreshold GAD among adolescents in the United States was 2-fold compared with adolescents diagnosed with GAD. Eventually, subthreshold disorders may lead to a need for treatment [6].

    Cognitive behavioral therapy (CBT) is a well-established first-line treatment for anxiety disorders in adolescents [4,7,8] and can reduce the risk of chronic anxiety if delivered early and effectively [9]. CBT conceptualizes anxiety as arising from maladaptive patterns of cognition and behavior, with treatment focusing on addressing the factors that maintain an adolescent’s symptoms rather than understanding the etiology of the disorder [7]. Accordingly, therapeutic content focuses on teaching skills for replacing anxious thoughts with a more realistic and adaptive approach, developing skills to cope with and reduce anxiety symptoms, and exposure to feared situations to address anxiety-driven behavior and avoidance.

    Although trained mental health professionals have traditionally delivered CBT, the structured, skill-based, and sequential nature of CBT translates well to computer-based delivery. A computer-based approach to CBT delivery can involve accessing a program via the internet (internet-based CBT; ICBT), which typically involves therapeutic content being presented in web-based, structured modules in a progressive format. Technology-based features such as multimedia (eg, videos and audio files) and interactive user formats (eg, drop-down response menus) may be used to deliver therapeutic content. In some ICBT programs, therapist support may also be included in the form of messages, phone calls, or in-person contact [10].

    A total of four randomized controlled trials (RCTs) provide evidence of the treatment effects of ICBT for adolescents with anxiety disorders [11-14]; two other studies with adolescent populations have been recently published, but their focus was on establishing feasibility [15,16]. The 9-session program developed by Tillfors et al [13], which represents the earliest published RCT of ICBT, targeted social fears (namely public speaking) among high school students who met diagnostic criteria for social anxiety disorder. Posttreatment, significant improvements were reported in favor of ICBT (compared with wait-list control) on measures of social anxiety (Social Phobia Screening Questionnaire for Children [17], between-group Cohen d effect size=1.28), general anxiety (Beck Anxiety Inventory [18], d=1.47), and depression (Montgomery-Åsberg Depression Rating Scale [19], d=1.39) [13]. In another study, Spence et al [12] reported comparable, significant changes in clinician ratings of anxiety severity among adolescents with a markedly impairing anxiety disorder (predominantly GAD and social phobia) who completed 10 ICBT sessions (P<.001) and clinic-based, face-to-face delivery of CBT (P<.001); in contrast, those in a wait-list control condition displayed no significant change in severity [12]. In a recent single-group open trial published by Silfvernagel et al [11], a large within-group treatment effect emerged for adolescents with mild-to-moderate anxiety who completed 6 to 9 ICBT treatment modules (d=2.51). Most recently, Stjerneklar et al [14] reported that, compared with wait-list control, more adolescents who received ICBT were classified as being free of their primary anxiety disorder as well as any other anxiety disorder posttreatment (P<.05), with the odds of being free of their primary disorder appearing 3.60 times greater for recipients of ICBT. Participation in ICBT was also associated with greater improvements in clinician-rated diagnostic severity (P<.05) and adolescent- and mother-rated improvement in anxiety symptoms (P=.001). Many positive effects of ICBT were maintained at the 3-month follow-up, including freedom from anxiety diagnoses and subjective improvement in symptomatology. At this time, broad recommendations for future research within the ICBT field include conducting power calculations to ensure adequate sample sizes, defining primary outcomes before conducting the study, presenting results from intention-to-treat analyses, and measuring and reporting treatment adherence [20].

    Objectives

    We conducted a pilot RCT to inform the planning of a definitive RCT to test the effectiveness of the ICBT program, Breathe (Being Real, Easing Anxiety: Tools Helping Electronically) compared with a static webpage listing anxiety resources (considered a form of usual self-led care during internet use). In the pilot RCT, we set out to (1) determine a sample size for the definitive RCT; (2) define a minimal clinically important difference (MCID), as defined by adolescents, for the primary outcome measure; (3) estimate recruitment and retention rates to determine the number of study sites needed and the timeline for recruitment; (4) measure intervention acceptability to inform critical intervention changes; (5) determine the use of cointerventions; and (6) conduct a cost-consequence analysis to inform a cost-effectiveness analysis for the definitive RCT.


    Methods

    The study design was a two-arm pilot RCT (Breathe vs a static webpage) conducted with adolescents aged 13 to 17 years across Canada. We received approval from the Health Research Ethics Boards at the University of Alberta (Edmonton, Alberta), Izaak Walton Killam Health Centre (Halifax, Nova Scotia), and the Children’s Hospital of Eastern Ontario (Ottawa, Ontario) to conduct the study. The study protocol was registered with ClinicalTrials.gov (NCT02059226) and published [21].

    Recruitment

    We recruited adolescents between April 2014 and May 2016 over the course of 3 recruitment cycles. Near the end of each cycle, we reviewed the effect of the different recruitment strategies that we employed. Cycle 1 recruitment spanned April 2014 to August 2015 and involved a soft launch with health care professionals providing study pamphlets to prospective participants seeking mental health care from emergency departments, mobile or school-based crisis teams, and primary care clinics in Edmonton, Alberta; Halifax, Nova Scotia; and Ottawa, Ontario. Cycle 2 spanned 12 months (September 2014 to September 2015) and involved the implementation of a communication strategy with health care providers and study site contacts. The communication strategy involved the distribution of study updates through email (using MailChimp) [22] on a monthly basis and fostering relationships between Breathe research staff and recruitment partners through teleconferences and site visits, as requested. In cycle 3 (October 2015 to May 2016), we introduced a social media recruitment strategy (Facebook, Twitter, and Instagram) with posts that appeared when adolescents searched or posted about anxiety or stress. These posts directed adolescents to the study website, which provided details about the study, instructions for eligibility screening and potential enrolment, information on anxiety disorders, and contact information for the research team.

    Screening for Study Eligibility

    Youth eligible for participation were Canadian adolescents aged 13 to 17 years who (1) could read and write English, (2) had regular access to a telephone and a computer system with a high-speed internet service, (3) were able to use a computer to interact with web-based material, and (4) reported the presence of anxiety symptoms (Multimedia Appendix 1).

    The exclusion criterion was adolescent self-report of suicidal thoughts in the past week. In cycle 1 recruitment, we initially had a second exclusion criterion, receipt of face-to-face CBT; however, we removed this criterion midway in the first recruitment cycle (cycle 1) due to emails from adolescents who found it confusing (eg, unaware of what CBT is, difficulty distinguishing CBT from other health care services such as support from a guidance counselor). Upon reviewing the questions we asked adolescents about their participation in other services that would result in ineligibility, we inferred that adolescents were seeking the Breathe program as an adjunct to other counseling and school-based services (not necessarily CBT-based) and that this would likely reflect how the program would be used in a real-world setting. Although we originally implemented this criterion as a way of reducing the potential for cointervention during the definitive trial, removing it increased the extent to which the planned definitive trial would evaluate real-world treatment effectiveness. At the time of this protocol change, 150 adolescents had been deemed ineligible for study enrolment due to this criterion.

    We screened adolescents for study eligibility using a 2-stage process:

    • Stage 1: During stage 1, we screened adolescents on inclusion criteria 1 to 3 and the second exclusion criterion until it was removed. Adolescents used a secure web-based process to answer questions to determine eligibility [23]. Telephone-based and email support during this stage were available from a research team member.
    • Stage 2: Adolescents who met the first set of criteria proceeded to stage 2 screening. Stage 2 screening was conducted via the secure, internet-based platform, Intelligent Research Intervention Software (IRIS) [24]. During this stage, we assessed adolescents on inclusion criteria 4 and 5, and exclusion criterion 1.

    We screened adolescents for anxiety symptoms using the Screen for Child Anxiety-Related Emotional Disorders (SCARED) [25].

    To be eligible for study participation at this stage, SCARED scores needed to indicate the presence of anxiety symptoms. Adolescents were not excluded from study participation based on their SCARED scores. Although we originally thought that the Breathe program could be used by adolescents with mild-to-moderate anxiety symptoms [21], we did not exclude adolescents whose SCARED scores indicated severe anxiety symptoms. This approach in our pilot trial allowed us to determine who was accessing the program and identify the target population for the definitive trial.

    We assessed the risk of deliberate self-harm using the 4-item Ask Suicide-Screening Questions (ASQ) [26]. Adolescents who responded yes to any of the questions received a safety telephone call from a research team member who evaluated intent/severity/immediacy of risk before deciding on the adolescent’s safety and ability to participate. Adolescents who indicated an immediate risk of self-harm by responding yes to question 3 on the ASQ (“In the past week, have you been having thoughts about killing yourself?”) were excluded from the trial and received brief telephone-based support from the research team member who encouraged the adolescent to seek mental health care appropriate to their level of need.

    Informed Consent/Assent

    Adolescents aged 15 to 17 years were asked to consent to the study on their own behalf; adolescents aged 13 and 14 years were asked to assent to study participation. We also required parental consent for all adolescents aged 13 and 14 years, even if they were assessed as being able to consent. The intent was to have parents involved so that they could support their child with the enrolment process.

    Consent/assent from eligible adolescents was indicated electronically via the secure myStudies website [23]. The first webpage confirmed that the individual understood that he/she could ask questions about the study at any time during the study or in the future. Each webpage included a Contact Us button that provided a pop-up email box with toll-free phone and email contact information for a member of the research team. The Contact Us button triggered a message to the participant that a research team member would contact them to answer any questions they may have before proceeding with consent/assent. Individuals were also given the option of saving or printing a blank copy of the informed consent form to read and review on their own instead of proceeding immediately to consent at that time. During the consent/assent process, individuals were guided through a series of sections describing what it meant to participate (eg, reiterating the youth’s right to withdraw from the study at any time) and asking them to confirm (through true/false and yes/no questions) that they understood the study, its risks and benefits, and/or had any questions. True/False questions were added before the final consent/assent webpage to ensure that the individual understood the study information. If an individual answered incorrectly, a pop-up box with the correct answer and explanation appeared. The script was designed to give individuals ample time to make an informed decision about participation. Once consent/assent was indicated, the date and time of consent/assent were recorded by the myStudies website [23].

    Randomization and Blinding

    Randomization took place after informed consent/assent was obtained. Adolescents were randomly assigned using a computer-generated allocation sequence with a 1:1 ratio to 1 of 2 groups. A graduate student trainee affiliated with the project generated this sequence and an email was sent to each participant with information on their assigned intervention and log-in/website information to begin participation. A permuted block randomization procedure [27] with random block sizes of 4 to 6 was used. Given the methodological objectives of the pilot study, no blinding took place.

    Experimental Group: The Breathe Program

    Program Details

    Consistent with published treatment recommendations [7], the Breathe program is a newly developed 8-module CBT program that involves: (1) multimedia-based education about anxiety problems and approaches to overcoming anxiety (eg, reviewing why exposure exercises are important); (2) self-assessment activities to determine level of treatment and safety needs; (3) activities that teach users about anxiety sensitivity, how to identify anxious thoughts, and how to develop realistic thinking about anxiety-producing situations; (4) activities for practicing coping and relaxation skills with self-assessment of performance and rewards; (5) development of a hierarchy of feared situations and steps for gradual and repeated exposure to feared situations (using imagery and in vivo activities); (6) contingency management (examining the function of anxiety from a reinforcement perspective) and modeling (viewing videos of others confronting feared situations); and (7) skills for maintenance and relapse prevention. An overview of module content is provided in Table 1.

    Animations, embedded video, audio playback, graphic novel style vignettes, image maps, timed prompts, and on-screen pop-ups were used in each module to provide an interactive and multimodal experience. Each module included 4 components: Check-in, which asked the youth to assess and rate their social-emotional functioning over the past week (Figure 1); Discover, which introduced the module’s key topics; Check-out, which asked the youth to reflect on their responses to module content; and Try Out, which outlined activities for the adolescent to choose to practice the module’s key concepts and skills. Check-in/Check-out ratings that indicated thoughts of self-harm triggered a safety video and pop-up box, encouraging the adolescent to notify a parent/guardian of their thoughts and to seek immediate help. For each module, adolescents were given a choice as to whether they wanted parents to receive an email that included educational materials about the nature of adolescent anxiety and highlights of key topics that they worked on for that module. During program use, adolescents were also provided with the email contact of a trained research team member who could answer questions about the program and/or treatment (including discussion of distressing issues that may be activated during treatment). The program underwent an evaluation for usability with adolescents and clinicians before the start of the trial to improve the intervention’s technical interface, therapeutic messaging, and user experience (eg, esthetics, presentation of rating scales) [28].

    Table 1. Overview of Being Real, Easing Anxiety: Tools Helping Electronically (Breathe) program content.
    View this table
    Figure 1. Being Real, Easing Anxiety: Tools Helping Electronically check-in activity.
    View this figure
    Persuasive Design Mechanisms

    Breathe was delivered via IRIS, the same platform used for eligibility screening. The platform supports the integration of persuasive design [29] and enables a personalized program experience for adolescents via 3 primary mechanisms: (1) tailoring content, which involved adolescents providing information for the program to use during the module (Figure 2); (2) self-monitoring that enabled the adolescent to track their own behavior toward intended outcomes; and (3) automated reminders—for example, adolescents were instructed at the beginning of the trial to use the program weekly and those who did not log in for 1 week received an email encouraging them to complete their weekly module.

    Figure 2. Example of adolescent-provided personal information for module tailoring.
    View this figure

    Control Group: Static Webpage

    Adolescents assigned to the control group received minimal intervention—access to a secure, password-protected static study webpage housed in IRIS. The website offered suggested anxiety-related trade publications, print-based workbooks for adolescents, and the names of national and local organizations and websites where the adolescent might find support. There was no interactivity or personalization included in the webpage. Adolescents assigned to the control group were provided with the option to access the Breathe program for clinical use at the end of their 8-week control group participation.

    Safety Monitoring

    Among those adolescents allocated to the Breathe program, a graduate student trainee monitored adolescent well-being under the supervision of a child and adolescent psychiatrist and the primary investigator. Adolescent well-being was monitored via adolescents’ answers in the check-in and check-out components of the program. Automated indicators built into the IRIS program flagged safety issues (eg, decompensation in anxiety symptoms, thoughts of self-harm), and an email notification was sent to the trainee. The trainee would then (1) develop a plan of action and discuss it with the psychiatrist and principal investigator and (2) subsequently contact the adolescent and their parent(s) by phone follow-up within 36 hours. Serious adverse events were to be reported to the institutional ethics board. Among adolescents randomized to the control group, the website provided contact information for local emergency resources (crisis lines, emergency department, and/or other crisis mental health resources).

    Study Procedures

    Study participation did not begin until the adolescent logged into their assigned intervention via IRIS. Once logged in, participants could access either the Breathe website or the static webpage and could complete the study outcome measures. Outcome measures were available to complete at baseline (available immediately upon first log-in; preintervention), after the completion of module 8 (postintervention; experimental group) or 8 weeks of website access (postintervention; control group), and at 3 months postintervention (follow-up).

    Data Collection

    During Eligibility Screening

    We used the SCARED [25] to screen interested adolescents for anxiety symptoms. The SCARED is a 41-item self-report screen for symptoms of panic disorder, social anxiety disorder, and GAD in clinical and community adolescent samples as based on diagnostic criteria [30-32]. Adolescents responded to the 41 items of anxiety symptoms/experiences as not/hardly ever true, somewhat/sometimes true, or very/often true.

    During Study Participation
    Study Recruitment and Retention

    A study log was used by a research coordinator to track the number of adolescents who were screened as eligible and were enrolled or not enrolled in the study as well as those who completed outcome measures at postintervention and 3-month follow-up.

    Demographic Characteristics

    Self-reported age, gender, and province of residence, as well as adolescent and parent contact information (ie, telephone number and email), were collected from enrolled adolescents via IRIS before starting the intervention.

    Anxiety Symptoms

    We used the total score from the Multidimensional Anxiety Scale for Children–Second Edition (MASC2; 50 items) to measure anxiety symptoms pre- and postintervention. The MASC2 is one of the most widely used self-report measures in clinical trials in adolescents with anxiety disorders. It assesses physical symptoms, social anxiety, harm avoidance, separation/panic, and total anxiety, and has excellent 3-month test-retest reliability [33] and validity [34,35]. MASC2 software scores adolescent responses, produces a total raw score, and converts raw scores to T scores (a standardized score that allows for individual scores on a dimension to be compared with those from a broader population in which the dimension is normally distributed). On the MASC2, converting raw scores into T scores allows for anxiety scores to be differentiated as average/typical (scores between 45 and 55), slightly above average (scores between 56 and 60), above average (scores between 61 and 65), much above average (scores between 66 and 70), and clinical diagnosis (score>70) [33].

    We administered the MASC2 to adolescents in the experimental group at baseline and at each posttreatment time point. We used the data collected at the 3-month time point to estimate data completion rates for the full-scale trial. Adolescents in the control group completed the MASC2 at baseline and 8 weeks after study enrolment.

    Minimal Clinically Important Difference

    The MCID was defined using data collected from adolescents allocated to the experimental group following module 8 completion (posttreatment). Adolescents were asked to indicate the minimum change in anxiety for which they would consider it worthwhile to participate in the Breathe program. We used adolescents’ self-reported global ratings of change on a 10-point Likert scale (−5=a very great deal worse to +5=a very great deal better), a commonly used anchor [36,37]. We did not collect data from adolescents who completed the Breathe program after their 8-week period in the control group.

    Program Acceptability

    Adolescents allocated to the Breathe program answered 16 questions on program acceptability after completing module 8. An instrument was designed specifically for this study to assess ease of program use, sense of privacy, and delivery format and content. The research team reviewed the instrument for face and content validity. For 10 questions, a 4-point Likert scale (strongly disagree to strongly agree) was used. We originally intended to use a 5-point scale but removed a neutral option on the scale to improve the interpretability of adolescents’ ratings. Scores ranged from 10 to 40, with higher scores indicating higher acceptability. Of the remaining 6 questions, one asked adolescents to identify topics they would like to see in future Breathe programs by selecting from a list of options (eg, bullying, specific phobias). Two questions allowed adolescents to identify the top 3 most motivating program features and 3 most helpful modules from provided lists. Three questions were open-ended, allowing adolescents to note challenges or barriers that they faced in taking part in the trial, the extent to which they used the skills learned, and any technical issues encountered while completing the program.

    We assessed treatment adherence to further evaluate program acceptability. Adherence was measured by documenting the number of modules completed by adolescents allocated to the Breathe program. We also recorded whether adolescents allocated to the control intervention accessed the website during the 8-week assignment period. Adherence data were recorded by and stored in IRIS.

    Health Care Resource Use

    We asked adolescents allocated to Breathe to report on the nature and frequency of health care use (cointerventions, emergency department visits, other treatments, and medication) during completion of the program. This information was collected following module 8 completion. We also detailed software development and maintenance costs (for Breathe program maintenance and delivery) and any training and personnel costs associated with the Breathe program.

    Outcomes

    The primary outcome was self-reported change in anxiety symptoms from baseline to 8 weeks posttreatment. Given the purpose of this pilot trial, primary outcome data were not used to estimate treatment effects. Rather, the outcome data were described and used to inform the sample size needed for a definitive trial. Secondary outcomes were study recruitment and retention rates, MCID, program acceptability, and health care resource use during the trial.

    Sample Size

    The sample size calculation for the pilot RCT was based on obtaining data to determine the sample size necessary for a definitive RCT [38,39]. We set out to enroll 80 adolescents (40 assigned to each group) and expected to retain 40 adolescents (20 per group) at the 8-week time point. We estimated that we would need 20 adolescents per group at the 8-week time point to estimate SDs and provide 80% CIs for SDs and 95% CIs for recruitment and retention proportions with sufficient levels of precision in our calculations.

    Analysis

    Primary Outcome

    For each group, we calculated the mean difference in raw MASC2 scores (and SDs) from baseline to 8 weeks (postintervention). Per protocol, we used data from adolescents who had completed the MASC2 both pre- and postintervention for this analysis. A two-sided two-sample t test power analysis was conducted for the change in score from baseline to postintervention to calculate the sample required per group in a definitive RCT [40]. Given the various effect sizes based on different comparators and the heterogeneity between previous studies [20], we were conservative and decided that we wanted to be able to detect a medium treatment effect (d=0.3) of the intervention on our primary outcome for the experimental group in the definitive RCT. We used the software R for the sample calculation (type I error=0.05; power=0.80).

    Secondary Outcomes

    We used descriptive statistics (eg, mean and frequency) to summarize demographic characteristics, recruitment and retention rates, health care utilization, and program acceptability. Participant data were considered unknown if no answer was recorded. We used SPSS version 24 for all secondary analyses [41].

    We defined the recruitment rate as the number of adolescents enrolled during the study period divided by the number of adolescents eligible to participate during the study period. The recruitment rate was iteratively calculated throughout the trial to assess the adequacy of the recruitment strategy and formally determined at 26 months (the conclusion of the study recruitment/enrolment period) to determine an overall timeline for the definitive trial. Retention rates were defined as the number of adolescents who completed outcome measures at the 8-week (posttreatment retention) and 3-month (follow-up) time points divided by the number of adolescents enrolled. We used the 8-week retention rate to adjust the sample size for the definitive trial (eg, to adjust for anticipated study attrition).

    We intended to use adolescent global ratings of change (within the ranges of +2 to +3 or −3 to −2 for reported change using a 10-point Likert scale) reported at the end of Breathe program completion to estimate the MCID value [42]. However, only 13 of the 36 adolescents who received the Breathe program reported data for this outcome, and we considered the data set inadequate to support the analysis. Instead, we report the global ratings of change for these participants with no calculation of the MCID. We will carry over the MCID objective to the definitive trial with a larger sample size.

    Analysis of program acceptability data included summarizing the number of modules completed, and examining whether there were differences (eg, by gender) between program completers (>75% of modules) and noncompleters to identify potential confounders that would need to be adjusted for in the definitive RCT. We summarized responses to the satisfaction instrument using mean and frequencies and collated answers to open-ended questions. Completers and noncompleters were compared using two-sided two-sample t tests for continuous data and chi-square tests of independence for categorical data. A P value less than .05 was considered to be statistically significant.

    We intended for health care resource use data to inform a preliminary cost-consequence analysis [43,44]. However, data for this outcome were largely missing, and the data set was not adequate to support this analysis. Instead, we report the type and frequency of health care use and the crude costs associated with the Breathe program. We will investigate the cost-consequence objective in the definitive trial with a larger sample size.


    Results

    Recruitment and Retention Rates

    Recruitment commenced in April 2014 and continued until the end of May 2016 for a 26-month recruitment period. During this time, 588 adolescents were screened for study eligibility. A total of 94 adolescents were confirmed eligible for study participation (94/588, 15.9% of those screened; 95% CI 13.2% to 19.3%); all consented and were enrolled in the study. The success rates of the 3 recruitment cycles are presented in Table 2. The most dramatic increase in recruitment was observed following the introduction of social media to the recruitment strategy with an approximate increase of 300% in enrolled adolescents when comparing cycles 1 and 2 (29 months combined) with cycle 3 (8 months).

    The flow of participants through the trial is shown in Figure 3. We could not confirm the eligibility of 146 adolescents at stage 2 screening as they did not complete screening measures: 111 did not complete the SCARED and the ASQ, and 35 completed the SCARED but not the ASQ. Thus, we enrolled 94 of 240 potentially eligible adolescents (39.2% recruitment rate). A total of 49 adolescents were allocated to the Breathe program, and of these adolescents, 36 accessed the program; 45 adolescents were allocated to the control intervention, and of these adolescents, 34 accessed this intervention. The study’s retention rate at 8 weeks was 28% (13/46; 95% CI 17%-44%) for the Breathe intervention group and 58% (25/43; 95% CI 42%-73%) for the control group. The overall 8-week retention rate (both groups combined) was 43% (38/89; 95% CI 32%-54%). The retention rate at 3 months among adolescents allocated to the Breathe intervention group was 24% (11/46). The analysis was based on the 70 adolescents who completed baseline measures and did not withdraw from the study. No serious adverse events were detected during the study.

    Table 2. Number of youth recruited in each recruitment cycle.
    View this table
    Figure 3. Consolidated Standards of Reporting Trials diagram describing flow of participants through the study. ASQ: Ask Suicide-Screening Questions; CBT: cognitive behavioral therapy; SCARED: Screen for Child Anxiety-Related Emotional Disorders.
    View this figure

    Description of the Study Sample

    Participant characteristics are shown in Table 3. The average age was 15.3 years (SD 1.2; range 13 to 17 years), and the majority of participants were female (63/70, 90%). On the basis of baseline MASC2 T scores, 7/70 (10%) adolescents reported average levels of anxiety, 3/70 (4%) reported slightly above-average levels, 5/70 (7%) reported above-average levels, 7/70 (10%) reported much above-average levels, and 46/70 (66%) were at a level consistent with a clinical diagnosis of anxiety.

    Table 3. Baseline characteristics of participants by study group.
    View this table

    Anxiety Change Scores and Sample Size for a Definitive Randomized Control Trial

    A total of 38 participants completed the MASC2 measure at both the baseline and the 8-week posttreatment time points (13/36 in the Breathe intervention group and 25/34 in the control group). Among adolescents in the experimental group, the mean change in raw MASC2 scores from 8-weeks posttreatment to baseline was −7.9 (SD 15.7). The 80% CI for the SD generated for the 8-weeks posttreatment to baseline change score was 12.6 to 21.7. For the control group, the mean change in MASC2 scores from 8-weeks posttreatment to baseline was −9.0 (SD 15.4). The 80% CI for the SD generated for the 8-weeks posttreatment to baseline change score was 13.1 to 19.1. Assuming 80% power and 5% type I error rate, 177 adolescents per group (354 total) will be able to detect an effect size of 0.3 using a two-sided two-sample t test for means. The pilot data suggest that the pooled SD could be 15.7, translating the effect size of 0.3 to a detectable difference of 4.7 in change scores between the Breathe group and the control group.

    Global Ratings of Change

    Of the 13 adolescents who rated their change in anxiety symptoms after completing the Breathe program, 6 reported a somewhat better change and 7 reported a much better change.

    Program Acceptability

    Of the 36 adolescents who received the Breathe program, 13 (36%) completed all 8 modules (Figure 4) and 2 (6%) did not complete any modules. Program completers and noncompleters did not differ significantly in their responses to any of the 4 ASQ screening questions (P=.32, .93, .49, and .49), the manner in which they learned about the study (social media/on the web, health care provider/guidance counselor, friend, or not specified; P=.17), age (P=.85), or baseline MASC2 T scores (P=.44). Completers and noncompleters could not be compared on self-identified gender due to the limited number of males enrolled in the study.

    Figure 4. Percentage of adolescents who completed each Being Real, Easing Anxiety: Tools Helping Electronically module.
    View this figure

    In total, 39% (14/36) of adolescents provided feedback on the Breathe program (Table 4). The mean satisfaction score among these adolescents was 28.5/40 (SD 4.0), indicating modest satisfaction. All but 1 adolescent indicated that the Breathe program was easy to use and that they understood all the material presented within the program; 36% (5/14) of participants noted that it was difficult to complete the Try Out (homework) pages each week. All participants liked that the program was completed on the web, with 79% (11/14) indicating no concerns with privacy. Responses were divided as to whether the program should include a social media component (5/14 in agreement), be more personalized to the participant (7/14 in agreement), and include a module for parents (8/14 in agreement). The most common barriers to program completion were difficulty completing exposure activities and remembering/finding time to complete modules, among other life commitments. Additional feedback provided by Breathe users is provided in the Multimedia Appendices 2-Multimedia Appendices 4.

    Table 4. Adolescent feedback on the Being Real, Easing Anxiety: Tools Helping Electronically (Breathe) program.
    View this table

    Health Care Resource Use

    In terms of costs to develop the Breathe program, software development totaled Can $51,405 (US $36,462), whereas personnel costs associated with program development and maintenance during the study (eg, technician and programmer costs) totaled Can $73,172 (US $51,911).

    With regard to health care resource use outside Breathe, 39% (14/36) of adolescents in the study answered questions about health care resource use during the Breathe program. We summarize the findings from these adolescents, but acknowledge that the results may not reflect all adolescents allocated to the Breathe program. Counsellors (of unspecified professional background, designation, or theoretical orientation) were the most commonly used resources; 50% (7/14) respondents reported having seen a counselor for anxiety at least once during their participation in Breathe, with 5 having attended 5 or fewer visits and 2 having attended 9 or more. A total of 43% (6/14) respondents had visited family physicians for anxiety-related concerns, although these visits occurred only once for all but one of this group. Other resources were accessed relatively infrequently, with the exception of 1 adolescent who accessed social work 7 times over the course of Breathe (Table 5). During the trial, no adolescents were identified as requiring additional support based on their responses to survey questions completed before and after each module.

    Table 5. Health care resource use reported by adolescents after completion of module 8 of the Being Real, Easing Anxiety: Tools Helping Electronically program.
    View this table

    Discussion

    Principal Findings

    This RCT piloted procedures and obtained data on acceptability to determine feasibility for a definitive RCT that would test the effectiveness of an ICBT program for adolescent anxiety. The 3 key lessons learned from conducting the pilot study and being applied to plan for the definitive trial are as follows: (1) adolescents did not use all the Breathe resources provided, and adjustments to the program were necessary to increase program completion in the definitive trial; (2) recruitment by social media was the most successful modality for recruiting adolescents into the study and should be the primary recruitment strategy in the definitive trial to ensure timely study completion; and (3) protocol adjustments are necessary to increase study retention at each measurement time point to improve outcome data collection.

    Several adjustments were made to the Breathe program in an effort to support adolescents’ abilities to navigate and complete the program. First, we streamlined content and reduced the number of modules from 8 to 6. This decision was based on our observation of a leveling off of completion around sessions 5 to 6. To achieve this reduction, the flow of Breathe content was streamlined and focused, and content considered unessential or potentially overwhelming was removed (eg, numerous exercises highlighting the same concept). We also increased the use of video content in the 6 modules as this mode was described as inspiring by adolescents and created new first-person narrative videos to reinforce concepts and support adolescents in relating topics to their own lives. Second, we noted that participant feedback pointed to Breathe’s exposure component as a significant barrier to successful program completion. Although exposure is widely viewed as one of the most important components of therapy in terms of producing lasting change in anxiety symptoms, it does produce discomfort and may have contributed to the lower retention rate in the Breathe group as compared with the control intervention. We made 2 changes to exposure activities in the Breathe program. First, we adjusted the flow of our content so that exposures were first introduced in module 2 (not module 6, as originally designed); with this change, the concept could be introduced gradually and promote adolescents’ sense of positive change as a result of participating in the program. Second, we added telephone-based support from a coach to module 2. The coach will help the adolescent build an exposure activity plan tailored to their specific needs, clarify any confusion about how to set up an exposure activity, and address perceptions of self-efficacy in completing exposure activities. A recent systematic review suggests that this type of ICBT support (ie, human components) may boost ICBT adherence and engagement, particularly when delivered at critical points in a program that participants may find difficult or taxing [45].

    In this pilot study, we enrolled 94 adolescents over a 26-month period (approximately 4 adolescents enrolled per month). This recruitment rate is not feasible for enrolling 354 adolescents (before attrition) in the definitive RCT. In testing different recruitment strategies, however, we learned that recruitment was most successful via social media (39/94, 42% enrolled). We were able to recruit approximately 4 times the number of adolescents in cycle 3 (n=75) once we launched our social media strategy, compared with cycles 1 and 2 during which we relied on health care providers. For the definitive trial, we plan to hire a communications specialist to assist with social media recruitment efforts. The communication specialist’s role will be governed by the following key objectives: (1) to create study awareness and inform the target audiences (parents, adolescents, and health care providers) about the Breathe program and study; (2) to increase traffic to the study website; and (3) to increase recruitment of study participants, including specifically targeting males. More broadly, our social media strategy will involve consideration of the functions of each communication strategy and any associated costs (eg, advertising), development of study-branded, tailored content, including that geared toward males (eg, sport performance stress) to increase their recruitment; an established approach for social media use (eg, frequency of posts, refreshing content), and a strategy to enhance user privacy (eg, use of marketing headlines so that personal disclosure is limited when social media content is viewed) and safety (eg, monitoring of web-based posts).

    To improve study retention in the definitive trial, we will budget for financial tokens of appreciation in an effort to increase the response rate to study questionnaires (US $25 for completing posttreatment questionnaires; US $25 for completing follow-up questionnaires). We will also streamline the initial screening steps for consent and eligibility to reduce early dropout and will include those identifying as receiving CBT and other forms of treatment for anxiety. As other similar trials did not report such low retention rates nor offered incentives [8-11], we do not know what to expect in terms of the impact of this strategy on study retention, but hope that it will increase our retention to 75% at postintervention and 50% at the 3-month follow-up. Accounting for a 50% attrition rate at the 3-month follow-up, 708 adolescents would need to be enrolled in the definitive trial to achieve our desired sample size.

    Another important aspect of this pilot trial was our intent to define an MCID for adolescent anxiety. However, challenges with retention did not permit us to calculate an MCID as planned. A critical aspect of all ICBT programs is the degree of improvement adolescent users experience as a result of their use. The use of MCID estimates could help adolescents, parents, and health care providers select among ICBT programs with different effects and anticipate the meaningfulness of the expected differences in their effects (ie, their clinical significance). Moving forward, within the broader literature, there is no minimum sample size necessary for calculating an MCID for a patient-reported outcome for adolescent anxiety. However, studies exploring the use of MCID scores in intervention-based research noted that this value has been calculated and used meaningfully in studies with sample sizes of a minimum of approximately 60 participants [46]. Should our efforts to improve study retention in the definitive trial be successful, we will have sufficient data to calculate an MCID. This MCID will be used to support the interpretation of results from this trial as opposed to defining a sufficient sample size (to calculate statistical significance) as originally planned.

    Limitations

    The most significant limitation of this pilot RCT was the lack of data at posttreatment and 3-month follow-up. Unlike in the pilot, where posttreatment and 3-month follow-up MASCs were only administered if/when participants completed all program modules, administration of follow-up questionnaires in the definitive RCT will occur independently of program progress—ideally increasing the data available at both follow-up points and limiting the extent to which it is subject to selection bias. Another important limitation was our reliance on adolescents’ own recall when providing information about their utilization of other health care services. Addressing this limitation is challenging, given the privacy considerations and logistics that would be associated with verifying self-report data with information from other sources. However, as discussed previously, support from research staff will be made available to participants throughout their involvement in the full-scale trial, and they will be encouraged to contact staff with any questions that may arise as they provide this information.

    A final limitation was the exclusion of 150 adolescents early on in the trial due to their report of CBT participation. Although we do not know how many of these adolescents would have consented/assented to participate in Breathe, their exclusion introduces the potential for further selection bias in the study. In this study, initial recruitment in cycles 1 and 2 was also very slow until a social media component was included in cycle 3. That these changes occurred during the pilot study and not the definitive trial is important, and we do not anticipate needing to adjust this inclusion/exclusion criterion or the recruitment modality utilizing social media further. In the definitive trial, efforts will also be focused on increasing the number of male participants, which was a limitation to our pilot trial’s study population.

    Conclusions

    This study aimed to determine the feasibility of a definitive RCT exploring the effectiveness of Breathe, an ICBT program for adolescents reporting anxiety, by piloting procedures and assessing intervention acceptability. Adolescents enrolled in Breathe reported modest satisfaction with the program, with most indicating that it had been easy to use and was readily understood. Still, adjustments to the program are required to reduce attrition and address the barriers that adolescents encounter when attempting to complete key program elements. These adjustments, including streamlining program access and content and providing phone coaching support for adolescents around challenging elements of the program, will ideally increase recruitment and retention within the study, thereby promoting timely completion of the Breathe program and supporting the completeness of our dataset at each outcome time point. Ultimately, with these adjustments, the definitive RCT will allow for a more in-depth exploration of the impact of the Breathe program on adolescents with anxiety and inform ICBT utilization within mental health care systems.

    Acknowledgments

    This study was conducted using the Pediatric Emergency Research Canada network. This research was funded by the generosity of the Stollery Children’s Hospital Foundation and supporters of the Lois Hole Hospital for Women through the Women and Children’s Health Research Institute. The authors would like to thank the clinicians from the Children, Youth, and Families portfolio of Addiction and Mental Health Service, Edmonton Zone, and Alberta Health Services. Funding for this project was provided by a grant from the Canadian Institutes of Health Research (CIHR; Funding Reference Number 119531) and the Royal Bank of Canada Foundation.

    Authors' Contributions

    KO analyzed and interpreted the data, and drafted the initial manuscript. AB contributed to the design of the study, obtained funding for the study, supervised the study, and analyzed and interpreted the data. PM contributed to the design of the study, obtained funding for the study, supervised the study, and helped interpret the data. LW supervised the study and analyzed and interpreted the data. AR acquired the data and analyzed and interpreted the data. SC contributed to the design of the study, supervised the study, and interpreted the data. MJ contributed to the design of the study, supervised the study, and interpreted the data. EF supervised the study and interpreted the data. DJ contributed to the design of the study and interpreted the data. RR contributed to the design of the study and oversaw data analysis. AO contributed to the design of the study and interpreted the data. AJ contributed to the design of the study and interpreted the data. AN conceptualized and designed the study, obtained funding for the study, supervised the study, acquired the data, analyzed and interpreted the data, and helped draft the initial manuscript. All authors critically revised the manuscript for important intellectual content and approved the final manuscript as submitted.

    Conflicts of Interest

    During this work, AN held a CIHR New Investigator Award and RR held a Health Scholar award from Alberta Innovates—Health Solutions. PM holds a Tier I Canada Research Chair. The other authors have no conflicts to declare.

    Multimedia Appendix 1

    Trial inclusion and exclusion criteria.

    PNG File , 92 KB

    Multimedia Appendix 2

    Adolescents’ ratings of most inspiring Being Real, Easing Anxiety: Tools Helping Electronically features.

    PNG File , 303 KB

    Multimedia Appendix 3

    Adolescents’ ratings of most helpful Being Real, Easing Anxiety: Tools Helping Electronically modules.

    PNG File , 160 KB

    Multimedia Appendix 4

    Open-ended responses from adolescents on the extent to which they used the skills learned and topics to include in the Being Real, Easing Anxiety: Tools Helping Electronically program.

    PNG File , 107 KB

    References

    1. Merikangas KR, He J, Burstein M, Swanson SA, Avenevoli S, Cui L, et al. Lifetime prevalence of mental disorders in US adolescents: results from the national comorbidity survey replication-adolescent supplement (NCS-A). J Am Acad Child Adolesc Psychiatry 2010 Oct;49(10):980-989 [FREE Full text] [CrossRef] [Medline]
    2. Whiteford HA, Degenhardt L, Rehm J, Baxter AJ, Ferrari AJ, Erskine HE, et al. Global burden of disease attributable to mental and substance use disorders: findings from the global burden of disease study 2010. Lancet 2013 Nov 9;382(9904):1575-1586. [CrossRef] [Medline]
    3. American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders: DSM-5. Fifth Edition. Washington, DC: American Psychiatric Publishing; 2013.
    4. Connolly SD, Bernstein GA, Work Group on Quality Issues. Practice parameter for the assessment and treatment of children and adolescents with anxiety disorders. J Am Acad Child Adolesc Psychiatry 2007 Feb;46(2):267-283. [CrossRef] [Medline]
    5. Burstein M, Beesdo-Baum K, He J, Merikangas KR. Threshold and subthreshold generalized anxiety disorder among US adolescents: prevalence, sociodemographic, and clinical characteristics. Psychol Med 2014 Aug;44(11):2351-2362 [FREE Full text] [CrossRef] [Medline]
    6. Cartwright-Hatton S, McNicol K, Doubleday E. Anxiety in a neglected population: prevalence of anxiety disorders in pre-adolescent children. Clin Psychol Rev 2006 Nov;26(7):817-833. [CrossRef] [Medline]
    7. Seligman LD, Ollendick TH. Cognitive-behavioral therapy for anxiety disorders in youth. Child Adolesc Psychiatr Clin N Am 2011 Apr;20(2):217-238 [FREE Full text] [CrossRef] [Medline]
    8. James AC, James G, Cowdrey FA, Soler A, Choke A. Cognitive behavioural therapy for anxiety disorders in children and adolescents. Cochrane Database Syst Rev 2013 Jun 3(6):CD004690. [CrossRef] [Medline]
    9. Ginsburg G, Becker-Haimes E, Keeton C, Kendall P, Iyengar S, Sakolsky D, et al. Results from the Child/Adolescent Anxiety Multimodal Extended Long-Term Study (CAMELS): Primary Anxiety Outcomes. J Am Acad Child Adolesc Psychiatry 2018 Jul;57(7):471-480. [CrossRef] [Medline]
    10. Andersson G, Carlbring P, Berger T, Almlöv J, Cuijpers P. What makes Internet therapy work? Cogn Behav Ther 2009;38(Suppl 1):55-60. [CrossRef] [Medline]
    11. Silfvernagel K, Gren-Landell M, Emanuelsson M, Carlbring P, Andersson G. Individually tailored internet-based cognitive behavior therapy for adolescents with anxiety disorders: a pilot effectiveness study. Internet Interv 2015 Sep 26;2(3):297-302 [FREE Full text] [CrossRef]
    12. Spence SH, Donovan CL, March S, Gamble A, Anderson RE, Prosser S, et al. A randomized controlled trial of online versus clinic-based CBT for adolescent anxiety. J Consult Clin Psychol 2011 Oct;79(5):629-642. [CrossRef] [Medline]
    13. Tillfors M, Andersson G, Ekselius L, Furmark T, Lewenhaupt S, Karlsson A, et al. A randomized trial of Internet-delivered treatment for social anxiety disorder in high school students. Cogn Behav Ther 2011;40(2):147-157. [CrossRef] [Medline]
    14. Stjerneklar S, Hougaard E, McLellan LF, Thastum M. A randomized controlled trial examining the efficacy of an internet-based cognitive behavioral therapy program for adolescents with anxiety disorders. PLoS One 2019;14(9):e0222485 [FREE Full text] [CrossRef] [Medline]
    15. March S, Spence SH, Donovan CL, Kenardy JA. Large-scale dissemination of internet-based cognitive behavioral therapy for youth anxiety: feasibility and acceptability study. J Med Internet Res 2018 Jul 4;20(7):e234 [FREE Full text] [CrossRef] [Medline]
    16. Stjerneklar S, Hougaard E, Nielsen AD, Gaardsvig MM, Thastum M. Internet-based cognitive behavioral therapy for adolescents with anxiety disorders: a feasibility study. Internet Interv 2018 Mar;11:30-40 [FREE Full text] [CrossRef] [Medline]
    17. Gren-Landell M, Björklind A, Tillfors M, Furmark T, Svedin CG, Andersson G. Evaluation of the psychometric properties of a modified version of the social phobia screening questionnaire for use in adolescents. Child Adolesc Psychiatry Ment Health 2009 Nov 11;3(1):36 [FREE Full text] [CrossRef] [Medline]
    18. Beck AT, Epstein N, Brown G, Steer RA. An inventory for measuring clinical anxiety: psychometric properties. J Consult Clin Psychol 1988 Dec;56(6):893-897. [CrossRef] [Medline]
    19. Svanborg P, Asberg M. A new self-rating scale for depression and anxiety states based on the Comprehensive Psychopathological Rating Scale. Acta Psychiatr Scand 1994 Jan;89(1):21-28. [CrossRef] [Medline]
    20. Vigerland S, Lenhard F, Bonnert M, Lalouni M, Hedman E, Ahlen J, et al. Internet-delivered cognitive behavior therapy for children and adolescents: a systematic review and meta-analysis. Clin Psychol Rev 2016 Dec;50:1-10 [FREE Full text] [CrossRef] [Medline]
    21. Newton AS, Wozney L, Bagnell A, Fitzpatrick E, Curtis S, Jabbour M, et al. Increasing access to mental health care with Breathe, an internet-based program for anxious adolescents: study protocol for a pilot randomized controlled trial. JMIR Res Protoc 2016 Jan 29;5(1):e18 [FREE Full text] [CrossRef] [Medline]
    22. Rocket Science Group. Mailchimp   URL: https://mailchimp.com [accessed 2020-05-13]
    23. myStudies. myStudies: A Connec Service   URL: https://mystudies.ca [accessed 2020-05-13]
    24. Wozney L, McGrath PJ, Newton A, Huguet A, Franklin M, Perri K, et al. Usability, learnability and performance evaluation of Intelligent Research and Intervention Software: a delivery platform for ehealth interventions. Health Informatics J 2016 Sep;22(3):730-743. [CrossRef] [Medline]
    25. Birmaher B, Brent DA, Chiappetta L, Bridge J, Monga S, Baugher M. Psychometric properties of the Screen for Child Anxiety Related Emotional Disorders (SCARED): a replication study. J Am Acad Child Adolesc Psychiatry 1999 Oct;38(10):1230-1236. [CrossRef] [Medline]
    26. Horowitz LM, Bridge JA, Teach SJ, Ballard E, Klima J, Rosenstein DL, et al. Ask suicide-screening questions (ASQ): a brief instrument for the pediatric emergency department. Arch Pediatr Adolesc Med 2012 Dec;166(12):1170-1176 [FREE Full text] [CrossRef] [Medline]
    27. Zelen M. The randomization and stratification of patients to clinical trials. J Chronic Dis 1974 Sep;27(7-8):365-375. [CrossRef] [Medline]
    28. Wozney L, Baxter P, Newton AS. Usability evaluation with mental health professionals and young people to develop an internet-based cognitive-behaviour therapy program for adolescents with anxiety disorders. BMC Pediatr 2015 Dec 16;15:213 [FREE Full text] [CrossRef] [Medline]
    29. Kelders SM, Kok RN, Ossebaard HC, van Gemert-Pijnen JE. Persuasive system design does matter: a systematic review of adherence to web-based interventions. J Med Internet Res 2012 Nov 14;14(6):e152 [FREE Full text] [CrossRef] [Medline]
    30. Hale III WW, Raaijmakers QA, van Hoof A, Meeus WH. Improving screening cut-off scores for DSM-5 adolescent anxiety disorder symptom dimensions with the Screen for Child Anxiety Related Emotional Disorders. Psychiatry J 2014;2014:517527 [FREE Full text] [CrossRef] [Medline]
    31. Desousa DA, Salum GA, Isolan LR, Manfro GG. Sensitivity and specificity of the Screen for Child Anxiety Related Emotional Disorders (SCARED): a community-based study. Child Psychiatry Hum Dev 2013 Jun;44(3):391-399. [CrossRef] [Medline]
    32. Crocetti E, Hale WW, Fermani A, Raaijmakers Q, Meeus W. Psychometric properties of the screen for child anxiety related emotional disorders (SCARED) in the general Italian adolescent population: a validation and a comparison between Italy and the Netherlands. J Anxiety Disord 2009 Aug;23(6):824-829. [CrossRef] [Medline]
    33. March JS, Parker JD, Sullivan K, Stallings P, Conners CK. The Multidimensional Anxiety Scale for Children (MASC): factor structure, reliability, and validity. J Am Acad Child Adolesc Psychiatry 1997 Apr;36(4):554-565. [CrossRef] [Medline]
    34. March J, Parker JD. The Multidimensional Anxiety Scale for Children (MASC). In: Maruish ME, editor. The Use of Psychological Testing for Treatment Planning and Outcomes Assessment: Volume 1: General Considerations. New York, USA: Routledge; 2004:39-62.
    35. Dierker LC, Albano AM, Clarke GN, Heimberg RG, Kendall PC, Merikangas KR, et al. Screening for anxiety and depression in early adolescence. J Am Acad Child Adolesc Psychiatry 2001 Aug;40(8):929-936. [CrossRef] [Medline]
    36. Jaeschke R, Singer J, Guyatt GH. Measurement of health status. Ascertaining the minimal clinically important difference. Control Clin Trials 1989 Dec;10(4):407-415. [CrossRef] [Medline]
    37. Crosby RD, Kolotkin RL, Williams GR. Defining clinically meaningful change in health-related quality of life. J Clin Epidemiol 2003 May;56(5):395-407. [CrossRef] [Medline]
    38. Arain M, Campbell MJ, Cooper CL, Lancaster GA. What is a pilot or feasibility study? A review of current practice and editorial policy. BMC Med Res Methodol 2010 Jul 16;10:67 [FREE Full text] [CrossRef] [Medline]
    39. Lancaster GA, Dodd S, Williamson PR. Design and analysis of pilot studies: recommendations for good practice. J Eval Clin Pract 2004 May;10(2):307-312. [CrossRef] [Medline]
    40. Browne RH. On the use of a pilot sample for sample size determination. Stat Med 1995 Sep 15;14(17):1933-1940. [CrossRef] [Medline]
    41. IBM SPSS Statistics for Windows, Version 24.0.   URL: https://www.ibm.com/ [accessed 2020-05-18]
    42. Juniper EF, Guyatt GH, Willan A, Griffith LE. Determining a minimal important change in a disease-specific quality of life questionnaire. J Clin Epidemiol 1994 Jan;47(1):81-87. [CrossRef] [Medline]
    43. Drummond MF, Sculpher MJ, Torrance GW, O?Brien BJ, Stoddart GL. Methods for the Economic Evaluation of Health Care Programmes. Third Edition. Oxford, UK: Oxford University Press; 2005.
    44. Marshall DA, Hux M. Design and analysis issues for economic analysis alongside clinical trials. Med Care 2009 Jul;47(7 Suppl 1):S14-S20. [CrossRef] [Medline]
    45. Wozney L, Huguet A, Bennett K, Radomski AD, Hartling L, Dyson M, et al. How do ehealth programs for adolescents with depression work? A realist review of persuasive system design components in internet-based psychological therapies. J Med Internet Res 2017 Aug 9;19(8):e266 [FREE Full text] [CrossRef] [Medline]
    46. Wright A, Hannon J, Hegedus EJ, Kavchak AE. Clinimetrics corner: a closer look at the minimal clinically important difference (MCID). J Man Manip Ther 2012 Aug;20(3):160-166 [FREE Full text] [CrossRef] [Medline]


    Abbreviations

    ASQ: Ask Suicide-Screening Questions
    Breathe: Being Real, Easing Anxiety: Tools Helping Electronically
    CIHR: Canadian Institutes of Health Research
    DALY: disability-adjusted life year
    GAD: Generalized Anxiety Disorder
    ICBT: internet-based cognitive behavioral therapy
    IRIS: Intelligent Research Intervention Software
    MASC2: Multidimensional Anxiety Scale for Children–Second Edition
    MCID: minimal clinically important difference
    RCT: randomized controlled trial
    SCARED: Screen for Child Anxiety-Related Emotional Disorders


    Edited by G Eysenbach; submitted 10.01.19; peer-reviewed by M Subotic-Kerry, F Lenhard, MS Aslam; comments to author 27.04.19; revised version received 20.09.19; accepted 21.03.20; published 24.07.20

    ©Kathleen O'Connor, Alexa Bagnell, Patrick McGrath, Lori Wozney, Ashley Radomski, Rhonda J Rosychuk, Sarah Curtis, Mona Jabbour, Eleanor Fitzpatrick, David W Johnson, Arto Ohinmaa, Anthony Joyce, Amanda Newton. Originally published in JMIR Mental Health (http://mental.jmir.org), 24.07.2020.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Mental Health, is properly cited. The complete bibliographic information, a link to the original publication on http://mental.jmir.org/, as well as this copyright and license information must be included.