Tips and Traps: Lessons From Codesigning a Clinician E-Monitoring Tool for Computerized Cognitive Behavioral Therapy

Introduction

Depression in adolescence is a major cause of disability that is often underreported, unrecognized [1,2], and not addressed by timely and effective treatments [3-5]. Psychological therapies are recommended as a first-line treatment for mild-to-moderate forms of depression [6,7] but are difficult to access. Computer-delivered therapies, especially those based on cognitive behavioral therapy (CBT), have been developed to help address this [8]. Computerized cognitive behavioral therapy (cCBT) can be offered as pure self-help or guided by a clinician [9,10]. The advantages of pure self-help tools include improved cost-effectiveness and nonreliance on clinicians to guide or support adolescents, and ultimately it may be some adolescents’ preference to utilize programs independently. However, there is some evidence that clinician-guided interventions or blended therapies (where face-to-face therapy and computerized approaches are used side by side) have better completion rates and are more effective than unsupported self-help therapy [11-15]. An example of a supported program is the virtual MindSpot Clinic in Australia that allows remote screening assessments (phone- or Web-based) and clinician-guided treatment for anxiety and depression [16]. On the contrary, “Beating the Blues” is an unguided Web-based depression treatment program completed by a user at home that enables the prescribing general practitioner (GP) to receive risk alerts and progress updates [17,18].

To date most cCBT programs have been designed for adults, although more recently a number of programs have been created for depressed or anxious children and adolescents. They too range from self-help tools to supported programs. For example, “Think, Feel, Do” [19] relies on a facilitator (who is not required to be a clinician) to be available during the entire delivery of the program to discuss the content covered. BRAVE-ONLINE is a clinician supported anxiety program in which young people receive weekly emails from a clinician designed to give them feedback and encourage their program progression [20]. However, BRAVE-ONLINE does not include the ability to monitor symptoms. SPARX (Smart, Positive, Active, Realistic, X-factor thoughts) is a gamified cCBT program for adolescents, which was shown to be effective in 12- to 19-year-olds presenting to primary care with mild-to-moderate symptoms of depression [21]. SPARX was designed as a pure self-help program in response to young people wanting to maintain privacy and access the program independently. A freely available self-help resource such as SPARX, which is accessible to all young people in New Zealand, can help bring treatment to the estimated three quarters of young people with depression who do not seek help [22]. In the original trial of SPARX, the referring clinicians provided a minimal degree of oversight with only brief contact after a month to ensure adequate progress. Our experience from the trial suggested that while most clinicians endorsed the use of a self-help program, some wanted to have a more formal means of monitoring adolescent users’ progress while using SPARX (Merry, personal communication).

Therefore, our study aimed to develop and implement an electronic monitoring (e-monitoring) tool relevant to adolescents with depression who attend youth-oriented primary care settings. The monitoring tool was envisaged to work in settings such as school health and counseling services because this is where many adolescents are likely to seek help [23]. To achieve this, we (1) assessed clinicians’ and adolescents’ views on using an e-monitoring tool and used this information to help shape development of the tool and (2) assessed clinician experiences with a fully developed version of the tool that was implemented in their clinical service. Of note, the tool was not expected to replace face-to-face contact but rather to offer another means of clinician oversight.

Methods

Study Design and Setting

A qualitative study was undertaken in New Zealand using focus groups with adolescents (aged 12-19 years) who were in contact with mental health services and also clinicians in primary care settings from the services where they worked (Table 1). We approached local secondary schools, a youth health center, and a nongovernmental organization using a snowball approach to recruit English-speaking participants. The clinicians approached were known to the researchers, and had an interest in adolescent health and in using the tool. The clinicians were GPs and allied health staff (school guidance counselors, clinical psychologists, youth workers, and nurses).

We set out to embed within SPARX an e-monitoring tool that would allow a clinician to “prescribe” SPARX and remotely oversee users’ progress and mood scores. As a minimum, the tool would have the following features:

1. A registration system for clinicians so that they could use the SPARX e-monitoring tool

2. Electronic linking of SPARX user data to the prescribing clinician

3. A dashboard accessible on the Internet to allow clinicians to track all users they had referred to SPARX

4. An algorithm to automatically generate alerts at specific and concerning levels of depression, or self-harm, and a system to encourage users to access more help

5. A system to deliver alerts or “flags” via email to the clinician

In order to track mood in SPARX, we aimed to use the Patient Health Questionnaire for Adolescents (PHQ-A) [24,25] at 3 time points: module 1 (baseline), module 4 (mid-point), and module 7 (end of intervention). We also consulted a software developer to investigate potential solutions, and schematic blueprints (wireframes) were then developed to visually guide participants through the flow of a given system and illustrate what an e-monitoring system or dashboard might look like. These were then presented at focus groups with the clinicians and adolescents, who provided feedback during the various phases of development of the tool.

Ethics Approval and Consent

Ethics approval was obtained from the Health and Disability Ethics Committee, New Zealand (Reference: 12/CEN/62), and written informed consent was gained from each clinician, adolescent, and parents of any adolescent aged less than 16 years.

Focus Groups and Data Collection

Focus groups (Table 1) were conducted during 3 phases. Phase 1 was the consultation stage and was carried out prior to the creation of the e-monitor to canvass ideas that would help the design of the e-monitoring system, based on early wireframe designs (December 2012 to March 2013). Phase 2 was the development stage and was conducted after preliminary wireframes had been revised, and it included a broader discussion of how e-monitoring could be used clinically (June to July 2013). Phase 3 was the postimplementation stage and was conducted after clinicians trialed the e-monitoring tool in day-to-day practice (August 2013 to February 2014). A semistructured schedule was used for all the focus groups (Multimedia Appendix 1), and these were digitally recorded and professionally transcribed. Each group was run by 2 of 6 experienced facilitators (FS, KS, ML, TF, MS, and SM) and the duration, while ultimately determined by the participants, was usually 60-90 minutes. FS checked fidelity of the Phases 1 and 2 clinician transcripts, and AG reviewed the transcripts of the adolescent groups and Phase 3 clinician groups.

Data Analysis

A theoretical thematic analysis framework was used for examining the data from focus groups [26]. Transcripts were imported into NVivo before thematic analysis [27], and more specifically, the general inductive approach [28] was used to organize the dataset into multiple coded blocks. A higher level of data interpretation was then performed, and responses were assessed for what might be implied or inferred. FS checked and coded the transcripts of the Phases 1 and 2 clinician focus groups, and AG similarly reviewed and coded the transcripts of the adolescent focus groups and Phase 3 clinician focus groups. Each dataset was then independently coded (SH, Phases 1 and 2 clinician groups; MS, all adolescent groups; KS, Phase 3 clinician groups). The independent coding by the 2 researchers was then discussed until consensus was reached to determine the final themes. Quotations from focus groups, where relevant, are provided verbatim.

Results

Common themes were identified from both the clinician and adolescent focus groups, as well as from issues reported upon as unique to either group. During Phases 1 and 2, themes were organized into 3 broad categories: clinical progress; confidentiality and privacy; and technical issues. Clinicians who were involved in clinical testing (Phase 3) reflected on some concepts (for instance, the initial themes) and drew on their “hands-on” experiences of using the actual tool postimplementation. Refinements to improve the tool were also suggested (Textboxes 1 and 2).

Development of the E-Monitor (Phases 1 and 2)

Clinical Progress

Engagement, Adherence, and Offering Help

Most participants were positive and supportive of the idea of adding e-monitoring to enhance the effectiveness of cCBT. Some adolescents noted that it might be hard to initiate contact with a clinician if they received a message from SPARX saying “you should seek help,” and that the e-monitoring system would help clinicians to make contact with adolescents.

I think the monitoring is definitely quite key for like if you’re you know making sure you get like consistently doing it and stuff. It could be quite easier if no one was checking up on you and stuff to be like just oh that’s enough that kind of thing.
[Adolescent]

Clinical Burden and Effects on Therapeutic Relationship

E-monitoring was seen as a positive step by clinicians and adolescents, but there was also concern that it could potentially contribute to increased clinician burden. Some clinicians felt that the clinician-user therapeutic relationship may be impacted upon by a tool used in place of an ongoing face-to-face relationship, while others were concerned at the possible risk of rapid change in depression severity in adolescents. Similarly, some adolescents were concerned that clinicians might become over-burdened with the monitoring, with potential negative impacts on the clinical relationship and rapport. Clinicians and adolescents suggested the need for a discussion around e-monitoring, so that the adolescent was aware that a clinician had oversight of their progress, and could maintain the therapeutic relationship while simultaneously enhancing the adolescent’s autonomy and sense of control.

I personally think this is great, so you can give the child a bit more power if you like, to take control of how they think and what they’re going to do, because it seems often that it’s one of the parents driving what’s going to happen next and how quickly they are able to get better.
[Clinician]

Responsibility of Monitoring Risk, Parallel Monitoring, and Backup Systems

Clinicians were concerned about how they could discharge the clinical responsibility of monitoring an adolescent who was using a self-help therapy and might not be seeing the clinician regularly. In particular, there was concern about managing self-harm or suicide if the clinician was unable to monitor progress regularly due to after-hours cover, being part-time, or on annual leave. A potential solution proposed was for other people in the service to check alerts. Having a robust system in place with cross-cover arrangements would allow monitoring to continue in their absence. Similarly, having e-monitoring alerts sent to more than one clinician would allow more staff to better assess and manage risk.

I mean if I was working here and I saw it come through and I glance at it out of interest because I’m seeing the patient, I would be very happy and comfortable that the nurses had full ability to follow-up on that and someone else was doing it. It would be great. It would take too much time so that would be good.
[Clinician]

Adolescents realized that mood monitoring was based on honest responses by the user of the program, which may not always happen. They suggested that there should be “check-ins” alongside SPARX to ensure that the adolescent user was truly getting benefits from the treatment.

Confidentiality and Privacy

In keeping with the literature [29], confidentiality was an important consideration for both adolescents and clinicians. There were concerns about privacy, information use, and who had access to the information collected during the process of e-monitoring. Clinicians also had the impression that adolescents would prefer if their parents were not involved and that adolescents preferred to retain anonymity.

There was some discussion about social media, and this was in the context of adolescents possibly using their Facebook login credentials to access the cCBT program, an idea posed as a possible means of reducing registration overheads. Another idea raised was whether mental health programs should have a social media presence, given the potential stigma associated with mental illness. Adolescents were worried that others could see that they had signed up for SPARX through a Facebook announcement, and consequently asked that the option to register using a social media account be removed.

It’s like when you are depressed or something, you don’t like want to tell the whole world about it really.
[Adolescent]

Some clinicians believed there would be reduced adolescent uptake should their details be identifiable via social media. However, many clinicians did not have privacy concerns especially if details were provided on the e-monitoring arrangements to the adolescent beforehand. Both adolescents and clinicians generally agreed that SPARX users should be given the option of selecting what information they wanted to share and how to be contacted.

Technical Considerations

Both adolescents and clinicians agreed that the registration process for e-monitoring should not be time-consuming. Ideally, clinicians would like to do it with the individual still present in their office, time permitting. Adolescents may change their contact details frequently, and therefore discussing the right communication medium is necessary. Technical aids such as information booklets with pictorial guides could be provided. Some adolescents suggested that the registration process should be “short and sweet” and there should be shortcuts, for example, a “Play Now” button that takes you straight to the program. Others said it would be easier for their clinician to complete the registration on their behalf and then send them the details.

Some clinicians wondered who should have access to the tool and whether it should be restricted to those with appropriate clinical skills and experience. They suggested that before a clinician was given access to e-monitoring (or the ability to “prescribe” SPARX), their credentials should be verified with the relevant regulatory agencies that maintain the professional registration of clinicians. A minority of clinicians thought that the positive and supportive material within the cCBT program may cause some adolescents to avoid being e-monitored. This is because mood may improve while undergoing cCBT, and consequently the adolescent may view engagement with the clinician as unnecessary. Despite concerns about the security of emails and the way alerts would be sent, overall clinicians were supportive of e-monitoring.

Many clinicians also suggested integrating the e-monitor and alerts with existing patient management systems (including electronic mailboxes that receive laboratory results). Clinicians supported the use of the Web-based tools built into the e-monitoring system for assessing mood, for example, PHQ-A, as these would potentially allow the “teasing out” of mood symptoms in greater detail so that energy, sleep, concentration, and other dimensions could be clarified. When adolescents describe low-mood symptoms, e-monitoring would potentially allow the assessment of the pattern of change over time. Clinicians felt that the acquisition of such finer detail is currently not possible with many face-to-face meetings. However, internet connectivity is key to e-monitoring, and adolescents experiencing sociodemographic deprivation may be particularly disadvantaged in this regard.

Adolescents had a number of suggestions concerning the technical aspects of e-monitoring, but perhaps unsurprisingly, they were different from those expressed by the clinicians. Adolescents wanted to select a communication method (eg, email, texting) and agree this with the clinician. Adolescents were cognizant that the severity of symptoms varies between individuals and wanted alerts to the clinician set at a threshold high enough to warrant attention. Some also wanted to include a function that would allow the users to determine the urgency of the situation. Adolescents suggested that the following information be included when informing potential SPARX users about the ways to get help: “A recommended doctor,” “What to do if you’re depressed and you got no one,” “And if you don’t trust anyone to talk to.” Information where and how to access help (including after-hours help) should be prominent, clear, and available to all users (including those who may not need it immediately). Adolescents did not wish to receive many “unnecessary” alerts (via email or text). Therefore, the alerts going back to users should be sent at milestones (eg, completion of levels) and should not be sent too frequently. Personalized messages were favored over automated ones as they would “make you want to read it more.”

Clinician Feedback on the E-Monitoring Tool Postimplementation (Phase 3)

Following the use of the e-monitor in clinical practice, clinicians provided feedback that was often consistent with their prior responses during the consultation and development phases of the e-monitoring tool.

Clinical Progress, Confidentiality, and Privacy

Clinicians noted that e-monitoring was “another thing to remember,” requiring additional work, and following up with adolescents took more time than they allocated. For many, although the e-monitor was available, it was not used. During use of the tool, one primary contact was allocated to receive all the alerts in a clinic and then notify clinicians as appropriate. This process worked well but required dedicated time. The use of e-monitoring in this way was different from how it was originally conceptualized by the clinicians.

Overall, clinicians explained that e-monitoring was helpful in checking whether adolescents had completed SPARX. They also described that e-monitoring provided another form of communication between them and the adolescent. Clinicians who used the tool believed that the quality of their relationship with adolescents was unaffected by the use of e-monitoring. Some clinicians, contrary to their earlier expectations, reported that it strengthened the therapeutic relationship as the adolescent felt more supported. Clinicians reported that e-monitoring was not on adolescents’ “radar” when they were doing it. So, it did not impact the therapeutic relationship, and clinicians continued to engage and talk with adolescents as they normally would. Overall, clinicians thought that the process worked well and shared similar views in terms of privacy considerations described earlier. They clarified with adolescents the purpose of e-monitoring and what information they could access from the e-monitor.

Technical Considerations

Despite attempts to keep the registration process simple, the log-on process was a barrier and “could be simpler.” Clinicians suggested an e-monitoring icon that could be integrated onto their current desktop to simplify access and serve as a reminder or prompt to check adolescents’ progress. Email alerts were widely discussed, and clinicians thought that these could serve a variety of purposes, including alerting of at-risk adolescents, reminding them to log in, and providing regular summaries in terms of adolescents’ engagement and progress with SPARX.

Final Version of the E-Monitor

In order to minimize burden on users and increase engagement with the program, mood was tracked in SPARX at the 3 time points originally proposed. Other suggestions from adolescents and clinicians are found in Textboxes 1 and 2, alongside considerations for an e-monitoring system to work effectively. Following the development and postimplementation phases, the e-monitor created (Figures 1 and 2) had the following features in (Textbox 3)

‎

‎

Discussion

Principal Findings

Our study focused on the development and use of an e-monitoring tool in clinical practice. It is therefore a timely contribution, as the area of electronically supported psychotherapy is very much in its infancy [30,31]. Using a codesign process, we developed an e-monitoring tool that would enable clinicians to electronically supervise adolescents with depression completing Web-based cCBT. There are various options for providing supported therapy, and the proposed e-monitoring tool for SPARX not only provides a mechanism for clinicians to monitor an adolescent’s progress, but also affords them an opportunity to gain regular depression assessment data via the Internet in the absence of face-to-face therapy.

In line with recent treatment preferences, clinicians were in favor of a system of monitoring adolescents with mild-to-moderate depression that embraced technology, various communication modalities, and a clinician-adolescent shared decision-making approach [32,33]. E-monitoring in tandem with face-to-face check-ins could potentially address clinician and adolescent concerns of no one providing oversight of a cCBT program such as SPARX, and this is a solution that has been previously identified in the literature [34]. While computer-based clinician support systems are increasingly gaining traction, it is also important that these do not place additional demands on the clinician’s workload [35]. Providing e-monitoring as well as face-to-face supervision may result in duplication of assessments and an increased clinician workload, but the need for such additional face-to-face assessment may reflect the reluctance of clinicians in depending on systems other than their own clinical judgment gathered from direct clinical contact.

Clinicians in our study also recognized the potential difficulties associated with after-hours monitoring and responding to concerns. Therefore, monitoring arrangements should be preagreed with adolescents in relation to checking alerts and also with regard to their preferred communication methods (phone, text alerts, emails). As has been recommended by others, relevant contact numbers can be provided by a GP [36] should a user or client experience a crisis, or these details could be embedded in the cCBT program, or alternatively, integrated between hotlines or emergency services and the cCBT program. Whereas the latter would be a novel and useful contribution (because most programs do not have such integration built-in), there may be additional challenges to implement this, as it involves a wider group of stakeholders. Additionally, clinicians would like e-monitoring integrated with the existing patient data management systems they are familiar with for receiving and monitoring alerts. Logistically this is a challenging prospect, as clinicians in primary care utilize different systems, and there may be little motivation on the part of the developers of patient data management systems to include e-monitors (like the one we have developed) into their systems.

There have been e-monitoring studies on various health conditions such as asthma and human immunodeficiency virus infection focusing on treatment adherence [37,38]. However, there are only a limited number of studies that have directly involved clinicians in a codesign process. For example, a Swiss qualitative study conducted with pharmacists on the implementation of an e-monitoring system for supporting treatment adherence similarly highlighted some of the difficulties we encountered, specifically the challenges of integrating an e-monitor into existing systems, but preexisting collaboration with physicians on codesigning the system helped [39]. Understandably clinicians would like to use an e-monitoring system that is simple to access, but also ensures privacy and confidentiality. Future developments should strive to fully integrate e-monitoring into existing data management systems, as challenging as this may be, so that e-monitoring is maximally useful to clinicians and acceptable to users [40].

During the development phase of the e-monitor in our study, clinicians noted several concerns. However, when they were followed up after using the tool postimplementation, they found that some of the initially identified concerns did not eventuate. For example, clinicians expected a negative impact on the therapeutic relationship but this did not arise. Also, during the development phase, alerts were envisaged to be sent to clinicians to highlight adolescents who might be critical in terms of mood severity or risks. However, they used the tool somewhat differently from what had been proposed during the postimplementation phase, because clinicians flagged that it required additional time for e-monitoring to occur. So, a dedicated clinician at the primary care service reviewed the dashboard and alerts regularly, and informed the other monitoring clinicians about the progress of their adolescent users. This was not how the e-monitor was originally designed to work, but it demonstrated a practical adaptation at this particular service. As clinicians are also concerned about the practicalities of e-monitoring, when a GP becomes the main crisis-intervention provider, there have been suggestions of a “zero-risk” approach for GPs and sharing of clinical responsibility with crisis teams [36,41].

Adolescents valued their privacy and preferred not to use their social networking login details. They also preferred a quick registration process, but one that either they or clinicians could undertake on their behalf. Moreover, adolescents wanted to discuss and customize e-monitoring arrangements and the timing and frequency of feedback they received via the e-monitor. While adolescents preferred their parents not to be involved in the e-monitoring process, they welcomed parallel monitoring with face-to-face check-ins with their clinician, despite reported difficulties with adolescents seeking help or expressing their concerns directly via face-to-face conversations [42]. Adolescents also wanted information on how to access support, especially after-hours when their clinician might be unavailable. As most adolescents have access to the Internet [43], this makes a Web-based e-monitoring system feasible in practice. With regard to SPARX (with or without e-monitoring), it has the capacity to advise adolescents to seek immediate help if they achieve a high score on mood symptoms or risk, but in practice, it may be difficult for the adolescent to ask for help. Information sheets could be provided prior to e-monitoring or during primary care consults, as depressed adolescents often do not recall details of consultations due to impaired concentration [36].

Limitations

This qualitative study involved adolescents and clinicians in New Zealand who, respectively, had experienced mental health difficulties and those who had managed adolescents with depression in primary care. The main limitation was that we were unable to take our interpretations of the data back to the participants for verification, which limited the study’s trustworthiness. Additionally, we were not able to undertake a systematic evaluation of the e-monitoring tool at multiple primary care sites or involve adolescents in the postimplementation phase. This was due to time and resource constraints. However, the multidisciplinary research team consisted of experienced clinicians who were directly involved in adolescent mental health and therefore very familiar with the issues.

Conclusions

Depression often commences during a critical period of early development, so addressing the needs of adolescents yields not only significant improvements in their health care outcomes, but also longer-term benefits over their life span [44,45]. Clinician and adolescent views were acquired on e-monitoring for cCBT, an area with little previous empirical research and even less so in primary care where adolescents are usually seen for depression. Subsequently, an e-monitoring tool was designed in line with clinicians’ wishes for oversight of adolescents using SPARX. The e-monitor was also designed with the “needs and wants” of adolescents and included a system that was easy to use with information and alerts. We are currently exploring ways in which the e-monitor for SPARX can be integrated into patient data management systems. Overall, clinical progress, confidentiality or privacy, and technical issues need to be considered alongside discussion between the clinician and the adolescent prior to commencement of e-monitoring arrangements. The findings from this study are potentially applicable beyond the adolescent population and will be of interest to developers of the various cCBT packages who may be considering a monitoring or feedback mechanism.