Clinical Use of Mental Health Digital Therapeutics in a Large Health Care Delivery System: Retrospective Patient Cohort Study and Provider Survey

Introduction

The digital therapeutics (DTx) market has experienced rapid growth in recent years, with the global DTx market valued around US $5.2 billion in 2022 and market research analysis projecting double-digit compounded annual growth from 2023 through 2030 [1]. In the first half of 2021 alone, the DTx business sector raised US $1.6 billion in venture capital [2]. Evidence suggests that DTx resources may be acceptable and scalable and therefore are a logical adjunct for mental health treatment, particularly given the growing need for mental health care [3].

However, despite DTx proliferation and potential for application in the mental health care space, patient engagement has been minimal [4]. Reasons for this poor engagement may include a lack of personal support in DTx (ie, integration of the tool into treatment rather than passive deployment of digital resources), which has been shown as important for the acceptability and usability of mental health DTx [5]. Further, there is a notable evidence gap for DTx’s mental health treatment effectiveness, with most evidence-based DTx having minimal market share [6], which may impact provider recommendations of DTx. Small, randomized controlled trials (RCTs) of DTx have shown small to moderate effect sizes for smartphone interventions in reducing depression and anxiety symptoms compared to controls [7,8]. A recent meta-analysis of 22 RCTs of mindfulness DTx showed small to moderate positive effects on depression and stress symptoms but not on anxiety [9]. An RCT of 88 college students using the Calm app (CALM.com, Inc) showed an improvement in perceived stress, mindfulness, and self-compassion in the users compared to controls [10]. Finally, an RCT of 146 employees with depressive symptoms using the myStrength app (Livongo Health, Inc) showed a more rapid reduction in depressive symptoms compared to controls [11]. The literature is not yet robust on which clinical populations are most appropriate for DTx, but those with mild to moderate symptoms of depression and anxiety, rather than severe, appear to benefit most [12,13]. DTx that are based on cognitive behavioral therapy may also be more effective, but much depends on the patient population as well as DTx design [13].

A mental health provider recommendation could be an influential factor for patient engagement with DTx tools. A 2016 survey of physicians found broad interest in DTx, but physicians also reported concerns over limited experience with such tools as well as lack of DTx data availability in the electronic medical record as barriers to DTx implementation [14]. Provider lack of knowledge regarding DTx, lack of investment and infrastructure, technical issues, workflow, and workload concerns have been identified as barriers to mobile health app adoption and implementation by providers [15-17]. In mental health, DTx tool adoption may vary by provider type and scope of practice, with psychiatrists reporting less bandwidth to focus on therapeutic DTx tools and therapists reporting that caseload, familiarity with DTx content and evidence, and limits on access to DTx data impact the ability to engage with patients around these tools [18]. Despite these descriptions of barriers to DTx tool adoption, there are little data on who providers recommend DTx tools to, the characteristics of providers recommending DTx tools, or their behaviors in recommending DTx tools. Understanding the characteristics of providers and their behaviors in recommending DTx tools can inform how these tools are integrated into mental health care, which is of particular importance given the increasing patient need for mental health resources and the dominant role mental health is assuming in the DTx space.

This study sought to characterize the clinical use of DTx resources within a large, integrated health care delivery system in two ways: (1) using electronic health record (EHR) data to identify the clinical and demographic characteristics of patients receiving recommendations from mental health providers to use DTx tools and (2) using an anonymous provider survey to determine characteristics and DTX recommending behaviors of mental health providers in clinical practice. Findings can inform the integration of DTx into mental health clinical care.

Methods

Setting

Kaiser Permanente Northern California (KPNC) is a large, diverse, integrated health care delivery system with approximately 4.6 million members. The membership is largely representative of the region, with less representation at the extreme ends of the income distribution [19]. Mental health services are primarily provided internally, both individually and in group format. Services are also available by referral to external community providers as needed.

DTx Resources

KPNC offers multiple DTx tools to its members at no cost. During the study time frame, the portfolio included Calm, myStrength, Headspace (Headspace Health, Inc), SilverCloud (Amwell), Thrive (Waypoint), and Whil (RethinkFirst) [20]. Providers can recommend DTx tools to patients based on their clinical judgment using smart text elements in the EHR, which are inserted into electronic secure messages to the patients with an associated link to the DTx. This electronic secure message contact is documented in the after-visit summary if providers use these smart text elements.

Study Design and Data Sources

Overview

This study had 2 components. The first used a retrospective cohort design with KPNC EHR data to identify adult patients who were seen in the mental health department, diagnosed with a mental health disorder, and received a provider recommendation to a DTx, as well as a matched cohort of patients who were also seen in the mental health department with a diagnosed mental health disorder but did not receive a recommendation to a DTx. The second component consisted of an anonymous web-based survey of health system mental health providers. This study followed the STROBE (Strengthening the Reporting of Observational Studies in Epidemiology) guidelines for observational studies [21] and the CHERRIES (Checklist for Reporting Results of Internet E-Surveys) for web-based surveys (Checklists 1 and 2) [22].

EHR Cohort

Adult patients aged 18 years or older with a mental health department visit and mental health diagnosis between May 1, 2020, and December 31, 2021, were included (Multimedia Appendix 1). Qualifying mental health disorders based on the International Classification of Diseases, 10th Revision (ICD-10) diagnosis codes included generalized anxiety disorder, bipolar disorder, major depressive disorder, adjustment disorders, posttraumatic stress disorder, psychosis, and sleep disorders (Multimedia Appendix 2). The index date was the first mental health visit; subsequent visits were excluded. Patients were excluded if they had less than 1-year health plan membership prior to their index mental health visit, in order to ensure the ability to capture baseline diagnoses assigned at patient encounters.

EHR Measures

The primary outcomes were the number and proportion of patients who received a DTx recommendation within 30 days of their index date. Within this time frame, we assumed that the primary diagnosis of their index mental health visit was the reason for a DTx recommendation. The recommendation was determined through text string searches (Multimedia Appendix 3) using relevant phrases within provider progress notes, smart data element capturing questionnaires, and secure messages from the patient’s provider.

Additional EHR-based measures included patient demographic variables (sex, race, ethnicity, and age group), type of psychiatric provider seen (nurse, therapist, and physician), neighborhood deprivation index (as a proxy measure for socioeconomic status, categorized into quintiles), insurance type (commercial, Medicare, or Medicaid), Charlson Comorbidity Index [23], and KPNC medical service area. The Patient Health Questionnaire 9-Item (PHQ-9) [24] and Generalized Anxiety Disorder 7-Item (GAD-7) [25] questionnaires, which identify symptoms for depression and anxiety, respectively, were extracted from the EHR if available within 30 days of the index visit. Each item in the PHQ-9 is scored 0 to 3, providing a 0 to 27 severity total score. The total score is used to determine cut points for depression: 0‐4 is normal, 5‐9 is mild, 10‐14 is moderate, 15‐19 is moderate severe, and 20 or more is severe.  The GAD-7 is a 7-item generalized anxiety disorder scale. Each item is scored 0 to 3, providing a 0 to 21 severity total score. The total score is used to determine cut points for generalized anxiety disorder: 0‐4 is normal, 5‐9 is mild, 10‐14 is moderate, and 15 or more is severe. 

Provider Survey

We conducted an anonymous web-based survey of a convenience sample of mental health providers throughout the KPNC region to understand the characteristics of providers recommending DTx tools and their recommendation behaviors. This survey was distributed 3 times in December 2022 to department managers and chiefs of psychiatry via an email that contained a Microsoft Forms link and a request to distribute it to clinical staff of approximately 2130 therapists and 375 psychiatrists. Participation was voluntary.

The survey included 26 multiple-choice questions and a free-text comment section (Multimedia Appendix 4). Specifically, we collected data on providers’ attitudes toward DTx adoption, DTx utility and effectiveness, preference for the type of DTx, awareness of different DTx modules, patient criteria for DTx recommendation, patterns of recommendation to DTx (self-assessment regarding manner of recommendation and monthly volume), barriers to recommendation, and demographics (professional role, practice experience, service area, work hours, sex [female or male self-identified hereafter referred to as “female” or “male”], age, race or ethnicity, and work hours). The primary outcome was the percentage of KPNC providers who reported recommending DTx.

Ethical Considerations

This study received approval from the KPNC Institutional Review Board (1899252) and was determined exempt, with a waiver of consent for the EHR-based analyses and a waiver of written consent for the anonymized provider survey, which did not gather protected health information. No compensation was provided to the providers per health system policy.

Statistical Analysis

EHR-Based Characteristics of Patients Receiving Versus Not Receiving DTx Recommendations

Demographic variables, clinical characteristics, and depression and anxiety symptom burden scores (PHQ-9 and GAD-7) were compared between patients who were and were not recommended to DTx with a series of bivariate analyses. Among those who received a DTx recommendation, we also examined the relationships of mental health symptoms by sex. To examine depression and anxiety diagnoses and symptom burden associated with DTx recommendations while accounting for potential confounding factors, we performed a frequency match to create a 1:1 matched cohort of patients without a recommendation to those recommended to DTx. Patients were matched based on a 5-year age group, sex, and geographical service area within KPNC. We then conducted bivariate analyses comparing the PHQ-9 and GAD-7 using the cases and matched patients.

Provider Survey

We performed descriptive statistics (eg, frequencies and bivariate tests) to characterize provider characteristics and DTx recommendations based on multiple-choice responses. Comparisons were done based on whether respondents were “prescribers” (eg, psychiatrists and nurse practitioners and able to prescribe medications to patients) or “nonprescribers” (eg, therapists such as psychologists and marriage and family therapists) to examine potential differences by provider role.

For both the EHR-based provider recommendation and survey analyses, the Pearson or Mantel-Haenszel chi-square test or Fisher exact test was used to calculate P values for categorical variables, the 2-tailed t test was used to calculate P values for continuous variables when the assumptions for a parametric test assumption were met, and the Mann-Whitney U test was applied to calculate ordinal data when the assumptions for a nonparametric test assumption was met. All item responses were used. All analyses were performed using SAS (version 9.4) for Windows and SAS Studio (version 3.81; SAS Institute Inc).

Results

Characteristics of Patients Recommended Versus Not Recommended DTx

Of the 335,250 adults eligible during the study time period, 16% (53,546/335,250) were recommended a DTx by a provider (Table 1). Patients recommended versus not recommended to a DTx were more likely to be younger (age 18‐44 years: 37,744/53,546, 70.5% vs 164,456/281,704, 58.4%; P<.001), Asian or Hispanic (20,360/53,546, 38% vs 88,827/281,704, 31.6%; P<.001), and female (38,338/53,546, 71.6% vs 193,063/281,704, 68.5%; P<.001). Additionally, there was a higher proportion of patients in the lowest (“0“) category of the Charlson Comorbidity Index (40,261/53,546, 75.2% vs 188,414,/281,704, 66.9%; P<.001) and a higher proportion of commercially insured patients (44,960/53,363, 84.3% vs 205,425/280,546, 73.2%; P<.001) among those who received recommendations versus those without a recommendation. Most DTx recommendations originated from therapy providers (48,916/53,546, 91.4%).

Matched Analyses—Mental Health Diagnoses and Symptoms of Patients Recommended Versus Not Recommended DTx

Given the significant demographic differences between patients recommended a DTx versus those not recommended, we performed a matched analysis to compare mental health diagnoses and symptoms between these 2 groups. Patients recommended to DTx versus not recommended had higher proportions of anxiety (22,247/53,545, 41.5% vs 19,039/53,545, 35.6%) and adjustment disorders (11,024/53,545, 20.6% vs 7592/53,545, 14.2%) and lower proportions of depression (13,151/53,545, 24.6% vs 16,517/53,545, 30.8%), bipolar (588/53,545, 1.1% vs 2685/53,545, 5%), or psychotic diagnoses (173/53,545, 0.3% vs 1239/53,545, 2.3%; all P<.001; Table 2). Additionally, more patients recommended to DTx had PHQ-9 depressive symptom scores in the mild (5-9: 10,643/37,282, 28.5% vs 6926/25,749, 26.9%), moderate (10-14: 10,597/37,282, 28.4% vs 7104/25,749, 27.6%), and moderate severe (15-19: 7513/37,282, 20.2% vs 5115/25,749, 19.9%) ranges compared to patients not recommended to DTx (P<.001 for all). The distribution of GAD-7 anxiety scores was similar between patients recommended and not recommended to DTx, albeit with the distribution of recommended patients having a slightly higher proportion of mild (5-9: 4852/23,251, 20.9% vs 2579/13,080, 19.7%) and lower proportion of severe (≥15: 9283/23,251, 39.9% vs 5391/13,080, 41.2%) GAD-7 scores (P=.003 for all).

Mental Health Symptom Burden by Sex in DTx-Recommended Patients

Since approximately two-thirds of patients recommended to DTx were female, we conducted ad hoc analyses examining symptom burden stratified by sex among those who received a recommendation to DTx. Female patients recommended to DTx had higher proportions of PHQ-9 depressive symptom scores in the moderate (7575/26,416, 28.7% vs 3022/10,867, 27.8%), moderate severe (5414/26,416, 20.5% vs 2099/10,867, 19.3%), and severe (3246/26,416, 12.3% vs 1220/10,867, 11.2%) range compared to male patients; and male patients had higher proportions of PHQ-9 scores in the normal (2804/26,416, 10.6% vs 1260/10,867, 11.6%) and mild (7377/26,416, 27.9% vs 3266/10,867, 30.1%) ranges (P=.003; Table 3). Female patients recommended to DTx had higher proportions of GAD-7 scores in the severe (6839/16,526, 41.4% vs 2444/6725, 36.3%) range compared to male patients; and male patients had more representation in the normal (516/16,526, 3.1% vs 341/6725, 5.1%) and mild (3296/16,526, 19.9% vs 1556/6725, 23.1%) ranges for GAD-7 (P<.001).

Characteristics of Providers Recommending DTx

A total of 211 individuals responded to the provider survey on DTx prescribing patterns for a participation rate of 8.4% (211/2505). Of those, 19.6% (40/204) identified as prescribers, 68.6% (140/204) as nonprescribers, and the remainder as having an administrative or managerial role in the clinic. Of respondents, 91.9% (192/209) were between the ages of 30 and 59 years, 54.1% (113/209) identified as White, 17.2% (36/209) as Asian, 12% (25/209) as Hispanic or Latino, and 6.2% (13/209) as Black. Nearly all respondents reported working at least half-time, and two-thirds of respondents reported working a full-time schedule.

Provider DTx Recommendation Behaviors

Overall, a higher proportion of nonprescribers (127/164, 77.4%) indicated they recommend DTx to patients compared to prescribers (22/39, 56.4%; P=.008). Of providers who recommend DTx, 83.6% (122/146) do so via an electronic secure message to patients, and only 12.3% (18/146) do so during a clinic appointment. Calm was the most commonly recommended DTx, followed by myStrength, Headspace, SilverCloud, Thrive, and Whil. Only 29.4% (45/153) of providers indicated using the suggested internal EHR–based smart text to recommend DTx. Prescribers were more likely than nonprescribers to report that they did not know how to make patient recommendations using EHR-based protocols (P=.01). Of providers recommending DTx to patients, 64.1% (98/153) report that they follow up with patients to inquire whether the resource was helpful. Only 38.4% (58/151) of providers recommend specific modules of DTx resources, and of those 58.6% (34/58) report following up with patients on these specific recommendations. On a scale of 0‐10 regarding the perceived effectiveness of DTx resources to impact symptoms (0=none and 10=extreme), both prescribers and nonprescribers rated cognitive behavioral therapy and mindfulness content between 6 and 7 or moderate to highly effective.

Discussion

Principal Findings

These are some of the first pragmatic results regarding provider recommendation patterns, characteristics, and attitudes toward mental health DTx. In this large, integrated health system with mental health DTx tools included as a benefit, a very modest percentage of patients with mental health disorders received recommendations to DTx tools. Those who did were more likely to be young, female, Asian or Hispanic, with less medical comorbidity, and to have commercial insurance compared to patients not recommended to DTx. Recommendations for DTx were more often made for anxiety or adjustment diagnoses and for patients with less severe mental health symptoms, although female patients recommended to DTx tools had a higher proportion of anxiety symptoms compared to male patients. A provider survey showed that DTx are recommended more frequently by nonprescribers and via electronic secure messages rather than during clinical appointments. Of providers recommending DTx, a modest majority follow up with the patient regarding DTx effectiveness, and few knew how to follow the recommended EHR protocols to do so.

Previous studies reporting demographic characteristics of patients using DTx have been from pilot trials or RCTs [7-9]. Using real-world data collected in the course of clinical care, we found that patient characteristics were similar to those of DTx trials. Overall, patients who received DTx recommendations were younger with less severe mental health symptoms, suggesting that providers are likely tailoring their recommendations to those they feel are most suitable. This may also reflect the initial emphasis of the health system DTx implementation focusing on patients with mild to moderate symptoms of depression or anxiety [20]. These results highlight a gap in the evidence base regarding the effectiveness of DTx for patients with mild to severe mental health symptoms or multiple comorbidities.

Asian or Hispanic patients were more likely to receive DTx recommendations. There are many potential drivers of this finding, including provider beliefs about patient DTx uptake, patient ability to interface with DTx, if providers felt the symptom severity reported by these patients was more appropriate for DTx, or other unrecognized factors. This finding merits further investigation. Of note, the significant association for insurance type with a DTx recommendation suggests slightly more patients with commercial insurance receiving recommendations. This may reflect provider-perceived barriers to care (eg, differences in disease severity or mobile phone access) for patients with other types of insurance coverage.

The finding that a higher proportion of female patients with severe mental health symptoms were referred to DTx compared to male patients could reflect a bias to implement more tools to treat symptoms in female patients, a reporting bias of symptoms by male or female patients, refusal of the DTx recommendation by male patients, or other sex-associated biases in care [26]. These findings warrant further research.

Provider survey results indicate that a higher proportion of nonprescribers recommend DTx and reported familiarity with EHR-based protocols to recommend DTx compared to prescribers. These results could reflect uneven training by provider type during the initial rollout or a lack of awareness or interest in the DTx tools by prescribers. Nonprescribers may appreciate having another treatment option to recommend beyond their typical strategies, given the therapeutic role DTx can fill and the silos of mental health care (ie, “therapy” vs “medication management” tasks). Interestingly, a minority of providers reported using the EHR-based protocols to recommend DTx to patients, with very few recommending specific modules or follow-up on module recommendations. Providers may view DTx tools as a passive, adjunctive resource rather than part of an active mental health treatment plan. Further, only a modest majority of providers reported asking patients if they found the DTx effective. A similar follow-up rate for a recommended or prescribed medical device, medication, or therapeutic program such as intensive outpatient or partial hospitalization would likely be regarded as very low. It is unclear why providers perceive DTx offerings in this way; it is possible that providers do not see DTx as having robust effectiveness data, which may impact their rate of follow-up. Similarly, providers may not view DTx resources as part of clinical care or within their scope of practice to monitor. This may reflect the early stage of growth and implementation of DTx in mental health and may evolve over time. If DTx are to be integrated into clinical mental health care, it may be more effective as a formalized part of the treatment plan and followed in the same way as other treatment aspects.

Limitations

This study was conducted in an integrated health care system, which limits generalizability to providers and patients in different systems of care. The survey participation rate was low, given constraints related to direct dissemination and reapproaching for response, though consistent with reported response rates in similar investigations [15]. While our integrated health system made multiple DTx available at no cost to patients during the study time period, such resources may not be as easily available to patients in other health care systems. Recommendations that were in person only or sent in messages without using the EHR protocols and smart text were not captured, nor were recommendations related to later visits, although our intent was to identify options presented earlier in the treatment course. The large sample size of EHR-based recommendation data makes it likely to detect associations that may have statistical significance but may not have meaningful clinical implications. The data we explored are observational and exploratory; future work warrants further examination into these initial findings along with an examination of patient-level outcomes associated with DTx.

Conclusions

The use of DTx is growing at a rapid pace, and health systems and patients see these as potentially valuable resources. However, the extent of provider recommendations to DTx is modest, as is provider knowledge about DTx and their follow-up with patients about DTx use, which suggests a largely passive DTx uptake by providers. Given the likely continued interest by patients, health care systems, and industry, future research on how to effectively implement these tools is critical.