Evaluation of Digital Mental Health Technologies in the United States: Systematic Literature Review and Framework Synthesis

Introduction

Background

Digital health comprises a broad range of technologies, including mobile health, health information technology, wearable devices, and personalized medicine, which serve as tools to enhance health care delivery. Recently, several digital mental health (MH) therapeutics, a category of digital MH technologies (DMHTs), have received US Food and Drug Administration (FDA) approval to prevent, manage, or treat a medical disorder or disease based on evidence from superiority trials and compliance with technical guidelines [1,2]. However, most DMHTs, particularly apps, fall outside FDA jurisdiction because they are not intended to diagnose, treat, or prevent disease and because they are “low risk” in that they would not cause harm in the event of malfunction [3]. Due to this lack of regulatory framework, few DMHTs are supported by published efficacy studies. One study found that only 16% of MH apps recommended by college counseling centers were supported by efficacy studies published in peer-reviewed journals [4].

Nonetheless, many health care providers (HCPs) use MH apps in clinical practice. Up to 83% of behavioral health providers in a small study covering the Greater Boston area reported using apps as part of their clinical care, particularly mindfulness apps for patient anxiety management [5]. As many DMHTs are currently widely used in clinical practice without undergoing any formal assessment for quality or relevance, understanding how DMHTs should be assessed based on factors impacting their value from the perspective of key stakeholders, such as patients, caregivers, providers, payers, and employers, could improve the selection of DMHTs for use by patients, thereby increasing care quality and outcomes for those seeking MH support.

Objective

To address identified gaps, a systematic literature review (SLR) was conducted using a published framework to synthesize emerging themes from mixed methods evidence in order to understand how digital health solutions, encompassing both digital therapeutics and direct-to-consumer digital health technologies, are valued, with a focus on MH disorders, Alzheimer disease, epilepsy, autism spectrum disorder (ASD), and attention-deficit/hyperactivity disorder (ADHD) in the United States.

Methods

Overview

The SLR was performed in accordance with a prespecified protocol and reported in line with the Enhancing Transparency in Reporting the Synthesis of Qualitative Research and PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines [6,7]. The protocol was not registered.

Search Strategy

Electronic databases, encompassing MEDLINE (including MEDLINE In-Process, MEDLINE Daily, and MEDLINE Epub Ahead of Print); Embase; the Cochrane Library (including Cochrane Database of Systematic Reviews and Cochrane Central Register of Controlled Trials); PsycINFO; and the Health Technology Assessment Database, were selected in alignment with this SLR’s target indications and were searched on June 17, 2022. The search terms included combinations of free-text and Medical Subject Heading or Emtree terms related to indications of interest, DMHTs, and relevant outcomes or assessment types (eg, technology assessments and cost; Tables S1-S5 in Multimedia Appendix 1). Searches were limited to studies performed in the United States and to those published from 2017 onward.

Manual hand searches of gray literature, namely, the bibliographies of relevant SLRs identified from the electronic database searches and key conference proceedings (2019-2022), were performed to identify additional studies of relevance (Table S6 in Multimedia Appendix 1). The FDA website was also searched to identify factors involved in the FDA’s appraisal of relevant MH apps, which could supplement the factors identified in this SLR (Table S7 in Multimedia Appendix 1).

Study Selection

Studies were included in the SLR if they met prespecified criteria defined using the SPIDER (Sample, Phenomenon of Interest, Design, Evaluation, Research type) framework, which is appropriate for mixed methods research questions. Eligible studies were published in the English language, were set in the United States, and reported quantitative or qualitative outcomes relating to the factors considered in the evaluation of DMHTs. Only studies published in 2017 or later were included because of the rapidly evolving research area. Eligible studies reported on MH, Alzheimer disease, epilepsy, ASD, or ADHD from user, payer, or employer perspectives (Table S8 in Multimedia Appendix 1). While the primary focus of the SLR was MH, neurological conditions were also of interest because their pathologies, symptoms, and treatment strategies can overlap with those of mental illnesses. Alzheimer disease, epilepsy, ASD, and ADHD were selected because they are highly researched and represent diverse types of neurological conditions.

The titles and abstracts of records were assessed for inclusion against these eligibility criteria by 2 independent reviewers, and discrepancies were resolved by consensus, with arbitration by a third reviewer if necessary. Full texts of potentially relevant articles were acquired and screened using the same methodology.

Study Prioritization

Due to the large volume of the evidence identified, additional eligibility criteria were applied to prioritize primary research on theoretical DMHT valuation factors. In line with the thematic framework synthesis objective, theoretical valuation factors were defined as user or DMHT attributes that impact interaction with or perception of DMHTs; therefore, studies that reported only efficacy outcomes, such as mental illness symptom improvement, were deprioritized for full-text review. Secondary research was also deprioritized for full-text review. Studies that reviewed a select app against a framework and studies that reported only the outcomes specific to a select app were deprioritized for data extraction. For example, a study reporting the usability of a specific app’s features would have been deprioritized, while a study reporting what types of features increase MH app usability in general would not.

Data Extraction

All relevant data were extracted into a prespecified Microsoft Excel grid, and a quality assessment was performed for each study. Studies that reported only qualitative data were assessed with the Critical Appraisal Skills Program Qualitative Studies Checklist. Studies that reported only quantitative data were evaluated with the Critical Appraisal Skills Program Cohort Study Checklist, and studies reporting both qualitative and quantitative data were evaluated with both checklists [8]. Data extractions and quality assessments were performed by a single individual for each study, with the information verified by a second independent individual. Discrepancies were resolved by consensus, with arbitration by a third individual if necessary.

Framework Synthesis

A framework synthesis approach was undertaken to synthesize qualitative and quantitative data identified from the SLR. In line with the “best fit” framework synthesis approach, data were indexed deductively against an existing framework where possible, and novel themes were added inductively as needed [9,10]. The American Psychiatric Association (APA) Mental Health App Evaluation framework was considered the most appropriate framework to address the research objectives of this SLR because its key valuation themes were developed using psychiatrist and patient input, are broadly shared by other evaluation frameworks, are widely acknowledged in the literature, and have been described as durable and adaptable [11-13].

The APA model follows a hierarchical and chronological order whereby the evaluator moves through the framework using prompting questions (eg, “Does the app work offline?”). For this SLR, these questions were either thematically grouped into subthemes or left as prompting questions, as appropriate. The framework was therefore ultimately adapted into 3 levels: themes, subthemes, and more granular valuation criteria. It should be emphasized that this SLR did not aim to formally develop an updated framework to be used in practice by HCPs and their patients but rather was used to form a theoretical basis for understanding DMHT valuation factors, for which novel themes were expected to emerge.

A data-based convergent approach was used to synthesize quantitative and qualitative data [14]. Data were initially indexed deductively against the prespecified themes within the data collection instrument and then further synthesized within Docear [15], a mind-map software used to organize and connect data and concepts. Indexing was performed by 1 reviewer and checked by a second independent reviewer. New themes and subthemes that emerged from the literature through inductive coding were added post hoc to the thematic framework, with all extracted data then considered against both the prespecified and novel themes. The evidence identified for each theme was synthesized narratively, taking into consideration the context and design of each study.

Results

Included Studies

A total of 4974 records were retrieved from the electronic databases. Of the 3374 (67.83%) unique records identified following deduplication across the databases, 2891 (85.68%) were excluded based on the eligibility criteria, and an additional 456 (13.52%) were deprioritized because they were not directly related to the topic of interest for this SLR. Excluded and deprioritized full texts are listed in Tables S9 and S10 in Multimedia Appendix 1, respectively. Therefore, 27 (0.54%) articles were included from the electronic database searches. In addition, 1 article reporting on the same study as an already-included conference abstract was identified during supporting targeted searches and included as a supplementary record, resulting in a total of 28 articles reporting on 26 unique studies (Figure S1 in Multimedia Appendix 1). No relevant FDA appraisals were identified in the supplementary search.

Of the 26 included studies, 8 (31%) were quantitative, 12 (46%) were qualitative, and 6 (23%) used a mixed methods approach. While 5 (19%) studies assessed prospective cohorts, 22 (85%) used a cross-sectional approach, including 1 (4%) study that contained both a prospective cohort and a cross-sectional cohort (Table 1). All studies (26/26, 100%) investigated a user perspective, with none specifically investigating payer or employer perspectives. Only 1 (4%) study, which examined ingestible sensor pills and smart pill dispensers to track adherence, investigated a DMHT that was not an app [16].

Most frequently, studies focused on indications for mood, anxiety, or psychotic disorders (15/26, 58%), with other indications of focus including ADHD (2/26, 8%), ASD (1/26, 4%), and epilepsy (1/26, 4%). No relevant studies focused on Alzheimer disease were identified.

A total of 8 (31%) studies assessed the perspectives toward DMHTs of general population participants who were not necessarily diagnosed with relevant conditions [19,28,29,33-37]. Of these populations, several were identified as having an increased risk of MH conditions, such as patients with cancer [28], college students [29,34], deaf or hard-of-hearing individuals [19], and people who were unemployed or furloughed during the COVID-19 pandemic [33]. In addition, 1 (4%) study included a mix of patients who were above and below the referral criteria for psychotherapy for depression [37].

Thematic Analysis

Overview

Evidence was identified for all 5 themes included in the APA framework: engagement style (23/26, 88%), background and accessibility (16/26, 62%), privacy and security (13/26, 50%), therapeutic goal (12/26, 46%), and clinical foundation (8/26, 31%; Table 2). Five novel criteria were identified and added to the framework post hoc, 1 each under engagement style (forgetting or feeling unmotivated to use DMHTs) and privacy and security (personal image and stigma) and 3 under background and accessibility (willingness to pay, insurance restrictions, and cost savings compared with professional care).

Theme 1: Engagement Style

Engagement style was the most reported theme, with evidence identified from 23 (88%) of the 26 studies. Engagement style encompasses how and why users do or do not interact with DMHTs. The long-term usability subtheme was reported by 96% (22/23) of studies, short-term usability by 12 (52%) studies, and customizability by 7 (30%) studies. Findings from short- and long-term usability subthemes were highly interconnected.

A total of 4 studies reported that ease of use promoted short-term DMHT engagement. In the study by Schueller et al [35], 89.6% of a general population of smartphone users reported ease of use for MH apps as “important” or “very important,” and users qualitatively reported dislike of “overwhelming,” difficult-to-navigate apps. In addition, users valued apps that were “simplistic” [34], fit into their daily schedules, and were available when needed (eg, during acute symptom experiences) [5,25]. Select supporting qualitative data are presented in Table 3.

Users valued DMHT features that aligned with their needs and priorities, as reflected by findings within the long-term usability subtheme. Across 9 studies, quantitative and qualitative findings demonstrated high interest in anxiety management features such as relaxation tools, breathing exercises, and mindfulness or meditation activities, and 10 studies identified interest in mood, symptom, or sleep tracking (Tables 3 and 4). While most studies (24/26, 92%) focused on MH, patients with epilepsy also reported high interest in features to record seizure dates and types [17]. Importantly, users in 2 studies emphasized the need for developers to tailor DMHTs to the needs and priorities of the target population (Table 3) [28,31]. Relatedly, mixed attitudes were reported toward positive affirmations and words of encouragement, with many users expressing interest but others emphasizing the value of a human component to DMHTs or cautioning against blanket encouragement and automated messages that could feel insincere [19,25,31].

Both patients and caregivers expressed interest in psychoeducational content that aligned with their needs and priorities. When surveyed, >60% of veterans with anxiety or major depressive disorder (MDD), patients with epilepsy, young adults with psychosis, and essential and furloughed workers during the COVID-19 pandemic expressed interest in relevant psychoeducational content [17,22,32,33]. In contrast, only 4% of college students in another study reported using an MH app for information about MH, although an MH diagnosis was not required for study participation [29].

Caregivers of young adults with psychosis, caregivers of children with ADHD, and spouses and partners of people with bipolar disorder (BD) were all interested in information related to caring for the individual with the given disorder, such as information on psychological and pharmacological treatments, symptoms and symptom changes, and the MH system [21,24,26]. Comparatively smaller, but still notable, proportions of caregivers of patients with psychosis were interested in caregiver-focused information; for instance, 62% to 69% were interested in relaxation exercises, stress and mood management, and community events for caregivers, while 85% to 90% were interested in the aforementioned patient-focused information [21].

Information delivery–style preference was captured under the short-term usability subtheme. One study in young adults with psychosis and another study with their caregivers revealed that delivering information in a variety of formats was important; when presented with nonmutually exclusive options, >50% of both populations were interested in text content, video content, audio content, and discussion boards [21,22].

Social interaction promoted long-term engagement. Qualitatively, 3 studies found that users valued learning about similar experiences from others via social media–like features, which normalized their experiences and could provide new symptom management strategies (Table 3) [28,31,36]. Similarly, 67% of both young adults with psychosis and deaf or hard-of-hearing survey participants (N=9) reported interest in peer support via chat features [19,22]. However, a comparatively smaller proportion of veterans with anxiety or MDD (48.3% of the full cohort and 51.3% of the smartphone user subgroup) were interested in peer support [32].

Overall, users endorsed social features to support their MH. In the study by Casarez et al [24], spouses and partners of people with BD likewise desired features to communicate with other caregivers and also emphasized that DMHTs could facilitate conversation and understanding with patients, a sentiment echoed by peer support specialists by Storm et al [38] (Table 3). However, one oncology HCP cautioned that similar to support groups, “very strict guidelines of what is said” should be implemented to manage potential risks from shared social media–like content, although little additional context was provided [28].

Spouses and partners of people with BD also suggested both in-app information on accessing professional resources and direct counseling for the patient at times when other support might be inaccessible [24]. More than half of all workers, employed or unemployed during the COVID-19 pandemic, likewise endorsed links to resources, counseling, and crisis support as DMHT features, and 81.6% of young adults with psychosis endorsed a feature to communicate with professional experts [22,33]. Importantly, compared with patients attending public clinics, patients attending private psychiatric clinics expressed a higher comfort level for in-app communication with HCPs, suggesting demographic differences in the valuation of access to professional support through DMHTs [39].

A total of 9 studies reported an interest in DMHT reminders and notifications. Across 3 studies, >70% of patients or caregivers were interested in appointment reminders [17,21,22]. In addition, 73% and 68% of patients with epilepsy reported interest in reminders for medication refills and adherence, respectively [17]. Beyond apps, caregivers of patients with MDD, BD, and schizophrenia preferred an ingestible pill sensor that tracked medication adherence, physical activity, mood, and rest 9.79 (95% CI 4.81-19.9), 7.47 (95% CI 3.81-14.65), and 6.71 (95% CI 3.29-13.69) times more than a nondigital pill organizer, respectively [16]. Qualitatively, patients and caregivers also appreciated reminders, especially if reasonably timed or delivered via text messages [27,31].

Short-term DMHT engagement was also supported by games and graphics, which could communicate information in an accessible way [24], provide tools for stress management [17,33], and be used therapeutically with children [20,30]. However, some HCPs and caregivers expressed concerns that graphics and games may be distracting for certain children (Table 3) [20].

In a novel finding, 3 studies reported forgetfulness or lack of motivation as an influence on DMHT engagement. In some cases, disuse was related to stress, other MH symptoms, or poor technical literacy (Table 3) [5,25,31]. In contrast, “forgetting to use” DMHTs and “lack of motivation” were perceived as relatively small barriers to use in the study by Stiles-Shields et al [37].

The third subtheme under engagement style was customizability, which was generally valued by users; 70.9% of a general population of smartphone users noted customization was an important factor [35]. Similarly, 9.4% of all surveyed veterans and 10.9% of those with smartphones reported disliking a prior DMHT due to a lack of personalization [32]. Users specifically wanted to be able to opt out of irrelevant features, customize audiovisual and design elements, add personal notes to tracked mood data, and provide ongoing feedback to facilitate personalization [20,24,28,31,34].

Theme 2: Background and Accessibility

A total of 16 (62%) studies reported findings related to DMHT background and accessibility, which considers the developer of the DMHT, as well as functionality and accessibility. Of these, 12 (75%) studies reported on the technical considerations subtheme, 9 (56%) on costs, and 2 (13%) on stability.

Under technical considerations, 9 studies assessed diverse accessibility concerns. Broadly, Storm et al [38] emphasized that DMHTs should be developed in consideration of patients’ social, cognitive, and environmental needs to avoid overwhelming users. Specifically, 2 studies reported language as a barrier. Deaf or hard-of-hearing participants recommended visual content presentation, such as videos and icons, alongside text and American Sign Language translations where possible [19]. Similarly, when discussing English-only apps, 1 provider stated as follows: “language is a barrier for some [patients]” [5]. Mobility issues related to MH symptoms or other conditions and technical literacy, such as difficulties remembering passwords and navigating smartphones or apps, created accessibility barriers as well (Table 3) [5,25,27,28]. Additional concerns included apps that restricted use based on geographic location [19], user difficulty in finding relevant, useful apps [32], and limited mobile device memory for downloading apps [5,19].

Offline functionality, reported by 6 studies, was also captured under the technical considerations subtheme. A majority (5/9, 56%) of participants included in the study by Borghouts et al [19] expressed concern about their mobile data plans when using their devices. Correspondingly, “availability of Wi-Fi” was noted as a top barrier to the use of apps for depression by Stiles-Shields et al [37], and several veterans in another study reported that home Wi-Fi connectivity facilitated app use by eliminating cellular data fees [25,37]. Quotes from patients and HCPs echoed the concern about apps without offline functionality (Table 3) [23,30].

Data fees were also captured under the costs subtheme, with hidden or additional costs described as a barrier to app use by 2 studies [26,37]. Parents of children with ADHD reported that difficulty paying phone bills could result in their phones being shut off, limiting DMHT use; one MH clinic administrator stated as follows: “We often encounter parents’ phones being shut off because they haven’t paid their bill...If the app were free or low cost, I imagine it could be very helpful” [26]. In addition to hidden costs, this quote identifies up-front app costs as a barrier. Quantitatively, more than half of a general population of surveyed college students expressed that cost was a top concern for the use of MH apps [34]. Qualitative findings from 2 additional studies likewise identified cost as a barrier to DMHT use [25,27].

Three novel cost attributes were identified by this SLR: willingness to pay, insurance restrictions, and cost savings compared with professional care. Four studies, 3 of which focused on apps, explored willingness to pay for DMHTs from a user perspective. Willingness to pay varied based on user preference; some surveyed college students and smartphone users among general populations valued free apps due to financial restrictions or uncertainty around app effectiveness, although 1 student commented that the quality of free trials might be inferior [34,35]. Some smartphone users also voiced a limit on how much they would be willing to spend for an app subscription (Table 3) [35]. Forma et al [16] found that caregivers were willing to pay US $255.04 (95% CI US $123.21-US $386.86) more per month for a pill with an ingestible sensor that tracked medication adherence, physical activity, and rest and could connect to an app that also collected self-reported mood data. Moreover, the caregivers were willing to pay US $124.50 (95% CI US $48.18-US $200.81) more per month for an app-connected pill organizer alone than for a nondigital pill organizer [16]. In contrast, some veterans expressed total disinterest in paid apps, with 1 user citing poor technical literacy (“don’t have the knowledge”) in addition to cost as affecting willingness to pay [25].

In another novel finding, a speech-language pathologist working with children with ASD preferred a single app including multiple features over separate apps for particular features due to insurance restrictions: “I agree that teaching Apps should be an in-App feature versus their own app because sometimes insurance doesn’t allow us to open the iPads purchased through insurance” [20]. Although no further detail was provided for this finding, it suggests that there may be restrictions on the use of other apps on devices purchased under insurance, which may have implications for DMHT use in formal care settings due to the lack of financial support.

In a third novel cost-related finding, a small number of participants from a general population of students (3.6%) in one study preferred using an MH app to seeing an MH professional due to cost savings [29].

A total of 13% (2/16) of studies reported on the subtheme of app stability and technical difficulties, with crashes and poor display quality decreasing DMHT value [35,37]. Participants in the study by Schueller et al [35] reported that technical difficulties were often an issue for apps developed by medical institutions, which might be effective and safe but less usable than apps from other developers.

Theme 3: Privacy and Security

A total of 13 (50%) out of 26 studies reported findings related to the privacy and security theme, which covered the use and protection of user data by DMHTs. Subthemes were reported relatively equally: data collection and storage (5/13, 38%), personal health information (PHI; 5/13, 38%), privacy policies (4/13, 31%), general privacy (3/13, 23%), and security measures (3/13, 23%).

Quantitative and qualitative findings on general privacy (ie, evidence not categorized under any specific subtheme), the data collection and storage subtheme, and the privacy policies subtheme revealed heterogeneous concerns (Table 3). A total of 74% of a general population of college students reported privacy as a top concern for MH apps, although further details on the specific area of concern were unclear [34]. In the study by Stiles-Shields et al [37], participants were highly concerned with data access but less so with general privacy. Echoing the concerns about data collection and storage, 59.1% of veterans with anxiety or MDD in 1 study were concerned about in-app PHI protection [32]; however, a qualitative study in veterans with posttraumatic stress disorder, alcohol use disorder, or MDD reported that a relatively small number of participants expressed privacy concerns. In the latter study, reasons for the concerns included distrust in Veterans Affairs, belief that digital data are inherently not confidential, and fear of phone hacking [25]. From an HCP perspective, none of the surveyed behavioral health HCPs agreed with the statement “My patients are concerned about data security,” despite multiple patients within the same study reporting privacy concerns [5].

Still, privacy policies were important overall, with 70.5% of smartphone MH app users rating having a privacy policy as “very important” or “important” [35]. Melcher et al [34] found that although users valued data protection, some reported a lack of awareness about data privacy, and others were concerned about obscure privacy policies and PHI use. As noted in the data collection and storage subtheme, veteran concerns about government use of PHI were heterogeneous [25].

A novel valuation factor not included in the APA framework related to user concern with PHI privacy and security regarding MH diagnoses and MH app use is a desire to upkeep their personal image or avoid stigma (Table 3) [5,25,29,40]. For instance, 21.1% of a general college student population preferred MH app use to seeing an MH professional due to anonymity or reduced stigma [29]. One participant in a study of Veterans Affairs health service users described access to professional care via MH apps as convenient because they could avoid disclosing their use of MH services to explain leaving work early for an appointment [25].

In line with the overarching concern about PHI privacy and security, users valued app security measures. Schueller et al [35] reported that 74.2% of users rated data encryption as “important” or “very important.” Users in another study perceived the level of privacy protection as the highest for apps using a combination of a generic app name (ie, not reflecting the indicated MH disorder); easily hidden modules; and secure, user-authenticated web portals for making module changes [40]. Behavioral health clinic staff echoed the importance of discreet MH app names (Table 3) [30].

Theme 4: Therapeutic Goal

There were 12 (46%) studies that reported on the factors relating to the integration of DMHTs with users’ therapeutic goals. The clinical actionability and therapeutic alliance subthemes were reported by 83% (10/12) and 58% (7/12) of studies, respectively.

A total of 9 studies reported the value of clinically actionable insights from apps where the users could acquire and practice new skills to make positive changes in their lives (Table 3). For instance, patient and caregiver app users reported interests in “daily tips,” “new ideas,” and “solutions or recommendations” for symptom management [26,27,36]. Furthermore, an app that could serve as a resource for multiple management strategies was preferable [26,28,31]. Quantitatively, 4% of patients receiving acute treatment in a partial hospitalization program for MH conditions, including mood and psychotic disorders, reported that the primary purpose of their DMHT use was therapy skills practice [18]. HCPs similarly appreciated that DMHTs could facilitate patients practicing skills outside of formal treatment sessions [5]. In particular, clinicians from a youth behavioral health clinic noted that DMHTs might be especially beneficial for young users because they could be conveniently and discreetly incorporated into their daily lives [30].

Users valued easy data sharing with clinicians, particularly for mood- or symptom-tracking features, which could improve communication and the accuracy of symptom reporting during clinical visits [5,25-27,34,36]. For instance, 53% of a general college student population believed that the potential to share information with their clinician was “one of the top benefits” of using DMHTs [34]. In addition, many HCPs reported active use or interest in the use of DMHTs in clinical practice to facilitate asynchronous communication and increase patient engagement with treatments outside of formal appointments; however, some preferred traditional care strategies for their personalization and flexibility (Table 3) [5,26,30].

Theme 5: Clinical Foundation

A total of 8 (31%) studies reported findings related to the clinical foundation of DMHTs, that is, their utility and appropriateness for patients. Clinical validity was the most reported subtheme, with evidence identified from 6 (75%) studies; 2 (25%) studies reported on the user feedback subtheme and 2 (25%) on the impressions of use subtheme, which captured users’ perceptions of app content as accurate and relevant.

Across subthemes, users valued evidence of DMHT benefit or efficacy from various sources. A total of 71.8% of surveyed veterans said that they would use a DMHT if they “saw proof that it worked” for their MH conditions [32]. Similarly, among the 811 general population participants surveyed, 69.5% ranked direct research evidence as “important” or “very important” for DMHT, and 66.8% ranked indirect research evidence the same [35]. Qualitative data identified recommendations from HCPs or academic institutions, as well as evidence of DMHT benefit from publications or research studies, as specific sources for clinically valid evidence of benefits (Table 3) [27,34,35].

In addition to academic and professional support, the user feedback subtheme captured user interest in whether DMHTs were beneficial for peers or recommended by other trusted individuals. Patients with depression reported that other users’ experiences influenced their app use, with one user wanting to know “...if other people had success using it” [27]. Quantitatively, user ratings and user reviews were ranked as “important” or “very important” factors in DMHT use by 59.4% and 58.7% of the general population participants, respectively [35].

Quality Assessment

The risk of bias was overall moderate. Of the 14 studies including quantitative components, only 1 (7%) used relevant validated outcome measurement instruments [33]; all others used custom questionnaires. Of the 18 studies with qualitative components, 4 (22%) were at risk of selection bias due to participants being exclusively recruited using web-based postings and research registries [33-35,37], and only 1 (6%) considered the relationship between researcher and participant when interpreting the results [36]. Full quality assessments for qualitative and quantitative study components can be found in Tables S11 and S12 in Multimedia Appendix 1, respectively.

Discussion

Principal Findings

This SLR aimed to identify and synthesize qualitative and quantitative evidence on how DMHTs are valued by users, payers, and employers in the United States. Evidence from users with or without diagnosed relevant disorders, caregivers, and HCPs was captured across a wide range of demographics. No study reported evaluating an app from a payer or employer perspective. Furthermore, all but one included study focused on mobile apps.

No relevant appraisals of DMHTs were identified from the FDA website searches; however, 8 relevant FDA approval labels or notifications for MH apps or guidance documents for industry and FDA staff were identified. The content of these materials overlapped with some valuation factors identified in this SLR, including evidence of clinical efficacy and safety, app maintenance, and privacy and security.

Engagement style, although not covered by the FDA materials, was the most reported theme by the studies included in this SLR and was found to overlap heavily with other themes. Engagement may be a key consideration for app developers, as app user retention can be low: 1 study showed that >90% of users had abandoned free MH apps within 30 days of installation [41]. Engagement is also a key clinical concern in terms of DMHT efficacy; one meta-analysis of 25 studies showed that increased use of DMHT modules was significantly associated with positive outcomes regardless of the target MH condition [42]. The findings of this SLR may therefore be informative to both DMHT designers and HCPs who integrate DMHTs into clinical care by providing insight on DMHT valuation and thus how use and benefit can be improved. For instance, users valued DMHTs that were easy to use and aligned with their needs and priorities, particularly through features that supported their therapeutic goals. In addition, content presented through multiple delivery modes, such as both text and visuals, promoted engagement as well as accessibility.

However, engagement and feature preference varied across populations. For instance, DMHT valuation was affected by technical literacy, which may relate to user demographics; in this SLR, veterans repeatedly emphasized technical literacy as a barrier to DMHT use [25]. Similarly, offline functionality may be more important for some users. Although 85% of the total United States population owns smartphones, only 59% of Medicare beneficiaries have access to a smartphone with a wireless plan. Moreover, beneficiaries who are older, less educated, disabled, or Black or Hispanic have even lower digital access [43,44]. These findings emphasize the importance of customizability and suggest that app development and selection in the clinical setting should consider the demographics of the target population, particularly in relation to ease of use and offline functionality.

Background and accessibility findings also identified up-front and hidden costs as barriers to DMHT use, with the willingness to pay varying among individuals. This has important implications for app development, considering that many MH apps currently on the market are direct-to-consumer sales and require out-of-pocket payment. App developers often take this approach as it does not require the accumulation of formal evidence of clinical benefit for FDA approval [45], but it may present a financial barrier to use for consumers.

Privacy and security, reported by 13 (50%) out of 26 studies, was a prevalent theme, with users primarily concerned with data and PHI security within apps. This finding reflects wider research; a 2019 review of 116 depression-related apps retrieved from iTunes and Google Play stores in 2017 found that only 4% of the identified apps had acceptable transparency in privacy and security, with many completely lacking a privacy policy [46]. Similarly, 39% of MH apps recommended by college counseling centers had no privacy policy, and of those with a policy, 88% collected user data, and 49% shared that data with third parties [4]. Most evidence identified in this SLR under this theme, as well as findings previously published in the wider literature, focuses on these remote privacy risks. However, local privacy concerns are also important to users. In particular, inconspicuous naming and the ability to hide sensitive modules within MH apps were rated as highly important by both patients and HCPs to maintain user privacy. Users emphasized a desire to avoid the stigma associated with mental illness, which was also reflected by the findings in the engagement style theme: more young adults with psychosis were more interested in in-app messaging with other patients in psychosis recovery (67.1%) than a provider and family member together (47.3%) or their personal support network (59.8%) [22]. Similarly, youths were interested in apps that could be used discreetly in school or other public settings to avoid potential MH stigma. This is a key, novel finding of this SLR, considering that many app or DMHT components on the market are named after their target disorder.

The use of DMHTs to achieve therapeutic goals was discussed from patient, caregiver, and HCP perspectives, all of which valued DMHTs that had evidence of efficacy, presented clinically actionable information, and facilitated patient-clinician relationships. Of the 5 studies that explored how HCPs value DMHTs in clinical practice, 2 (40%) were restricted to the oncology or ASD settings and were not readily generalizable to wider MH settings [20,28]. In other studies, providers reported interest in using DMHTs to facilitate asynchronous communication with patients and their caregivers, promote patient skill practice, and improve care for children through the use of games and visuals [26,30]. However, while HCPs overall believed that DMHTs improved care, some believed that their clinical training allowed for care personalization beyond what DMHTs could provide. Feature customizability and receipt of input from HCPs and users during app development and testing may be a way to mitigate these concerns, as well as concerns about safety and efficacy, as many available apps do not appropriately address user health concerns [47].

Findings additionally suggested that training and resources on DMHTs would be beneficial to ensure that HCPs were equipped to integrate DMHTs into their practices [5]. Collaboration between DMHT specialists and HCPs, along with a shift from randomized controlled trials to effectiveness-implementation hybrid trials, may be a way to streamline the integration of DMHTs into clinical care and provide more training and resources for HCPs [30,48].

Strengths

This review followed a prespecified protocol and used systematic methods in line with the York Centre for Reviews and Dissemination guidelines [49] to conduct an exhaustive search of the literature, identifying evidence relevant to the review objectives from multiple databases and supplementary sources. The framework synthesis approach allowed for the inclusion and analysis of both qualitative and quantitative data, providing a detailed picture of not only what DMHT features users value but why they value them, especially in areas where valuation varies across patient demographics. In addition, the APA framework is a robust model created with patient and HCP input that incorporates key valuation themes broadly shared by other frameworks and widely acknowledged in the literature [11-13].

Limitations

Methodological limitations should be considered when interpreting the findings of this SLR. Only publications in English and in United States populations were included. As perceptions of value are influenced by factors including cultures, laws, and health care settings, the findings of this SLR should not be generalized to other countries. For instance, trust in HCPs and rates of longstanding relationships between patients and primary care providers are lower in the United States than in many European nations [50,51], which could impact the type of support users want from DMHTs (ie, engagement style) or interest in DMHT integration with therapeutic goals.

In addition to the prespecified eligibility criteria, deprioritization strategies were implemented due to the large volume of the identified evidence, and this may have resulted in missing relevant articles. In particular, the deprioritization of secondary research and opinion pieces likely led to the exclusion of relevant discussion around payer perspectives and reimbursement, for which no evidence was included in this SLR. Furthermore, although unlikely, there may have been reporting biases in the included studies due to missing results, which this SLR was not able to assess.

This SLR identified no evidence for 3 subthemes included in the APA framework: business model (background and accessibility), which covers DMHT funding sources and potential sources of conflict, medical claims (background and accessibility), which examines whether DMHTs claim to be medical and the trustworthiness of their creators, and data ownership, access, and export (therapeutic goal), which includes sharing data with eHealth records or wellness devices (eg, Apple HealthKit [Apple Inc], Fitbit [Google LLC]). The valuation of these subthemes should be evaluated in future research.

Conclusions

In summary, app usability, cost, accessibility and other technical considerations, and alignment with therapeutic goals were the most reported valuation factors identified by this SLR. Many studies also reported user preference for apps that incorporated privacy and security features that provided protection from stigma. However, individual DMHTs and their features are valued differently across individuals based on demographics and personal preferences. MH apps should be developed and selected with these specific user needs in mind. Feature customizability and input from users and HCPs during development may improve app usability and clinical benefit.