Review
Abstract
Background: Digital mental health technologies (DMHTs) have the potential to enhance mental health care delivery. However, there is little information on how DMHTs are evaluated and what factors influence their use.
Objective: A systematic literature review was conducted to understand how DMHTs are valued in the United States from user, payer, and employer perspectives.
Methods: Articles published after 2017 were identified from MEDLINE, Embase, PsycINFO, Cochrane Library, the Health Technology Assessment Database, and digital and mental health congresses. Each article was evaluated by 2 independent reviewers to identify US studies reporting on factors considered in the evaluation of DMHTs targeting mental health, Alzheimer disease, epilepsy, autism spectrum disorder, or attention-deficit/hyperactivity disorder. Study quality was assessed using the Critical Appraisal Skills Program Qualitative and Cohort Studies Checklists. Studies were coded and indexed using the American Psychiatric Association’s Mental Health App Evaluation Framework to extract and synthesize relevant information, and novel themes were added iteratively as identified.
Results: Of the 4353 articles screened, data from 26 unique studies from patient, caregiver, and health care provider perspectives were included. Engagement style was the most reported theme (23/26, 88%), with users valuing DMHT usability, particularly alignment with therapeutic goals through features including anxiety management tools. Key barriers to DMHT use included limited internet access, poor technical literacy, and privacy concerns. Novel findings included the discreetness of DMHTs to avoid stigma.
Conclusions: Usability, cost, accessibility, technical considerations, and alignment with therapeutic goals are important to users, although DMHT valuation varies across individuals. DMHT apps should be developed and selected with specific user needs in mind.
doi:10.2196/57401
Keywords
Introduction
Background
Digital health comprises a broad range of technologies, including mobile health, health information technology, wearable devices, and personalized medicine, which serve as tools to enhance health care delivery. Recently, several digital mental health (MH) therapeutics, a category of digital MH technologies (DMHTs), have received US Food and Drug Administration (FDA) approval to prevent, manage, or treat a medical disorder or disease based on evidence from superiority trials and compliance with technical guidelines [,]. However, most DMHTs, particularly apps, fall outside FDA jurisdiction because they are not intended to diagnose, treat, or prevent disease and because they are “low risk” in that they would not cause harm in the event of malfunction []. Due to this lack of regulatory framework, few DMHTs are supported by published efficacy studies. One study found that only 16% of MH apps recommended by college counseling centers were supported by efficacy studies published in peer-reviewed journals [].
Nonetheless, many health care providers (HCPs) use MH apps in clinical practice. Up to 83% of behavioral health providers in a small study covering the Greater Boston area reported using apps as part of their clinical care, particularly mindfulness apps for patient anxiety management []. As many DMHTs are currently widely used in clinical practice without undergoing any formal assessment for quality or relevance, understanding how DMHTs should be assessed based on factors impacting their value from the perspective of key stakeholders, such as patients, caregivers, providers, payers, and employers, could improve the selection of DMHTs for use by patients, thereby increasing care quality and outcomes for those seeking MH support.
Objective
To address identified gaps, a systematic literature review (SLR) was conducted using a published framework to synthesize emerging themes from mixed methods evidence in order to understand how digital health solutions, encompassing both digital therapeutics and direct-to-consumer digital health technologies, are valued, with a focus on MH disorders, Alzheimer disease, epilepsy, autism spectrum disorder (ASD), and attention-deficit/hyperactivity disorder (ADHD) in the United States.
Methods
Overview
The SLR was performed in accordance with a prespecified protocol and reported in line with the Enhancing Transparency in Reporting the Synthesis of Qualitative Research and PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines [,]. The protocol was not registered.
Search Strategy
Electronic databases, encompassing MEDLINE (including MEDLINE In-Process, MEDLINE Daily, and MEDLINE Epub Ahead of Print); Embase; the Cochrane Library (including Cochrane Database of Systematic Reviews and Cochrane Central Register of Controlled Trials); PsycINFO; and the Health Technology Assessment Database, were selected in alignment with this SLR’s target indications and were searched on June 17, 2022. The search terms included combinations of free-text and Medical Subject Heading or Emtree terms related to indications of interest, DMHTs, and relevant outcomes or assessment types (eg, technology assessments and cost; Tables S1-S5 in ). Searches were limited to studies performed in the United States and to those published from 2017 onward.
Manual hand searches of gray literature, namely, the bibliographies of relevant SLRs identified from the electronic database searches and key conference proceedings (2019-2022), were performed to identify additional studies of relevance (Table S6 in ). The FDA website was also searched to identify factors involved in the FDA’s appraisal of relevant MH apps, which could supplement the factors identified in this SLR (Table S7 in ).
Study Selection
Studies were included in the SLR if they met prespecified criteria defined using the SPIDER (Sample, Phenomenon of Interest, Design, Evaluation, Research type) framework, which is appropriate for mixed methods research questions. Eligible studies were published in the English language, were set in the United States, and reported quantitative or qualitative outcomes relating to the factors considered in the evaluation of DMHTs. Only studies published in 2017 or later were included because of the rapidly evolving research area. Eligible studies reported on MH, Alzheimer disease, epilepsy, ASD, or ADHD from user, payer, or employer perspectives (Table S8 in ). While the primary focus of the SLR was MH, neurological conditions were also of interest because their pathologies, symptoms, and treatment strategies can overlap with those of mental illnesses. Alzheimer disease, epilepsy, ASD, and ADHD were selected because they are highly researched and represent diverse types of neurological conditions.
The titles and abstracts of records were assessed for inclusion against these eligibility criteria by 2 independent reviewers, and discrepancies were resolved by consensus, with arbitration by a third reviewer if necessary. Full texts of potentially relevant articles were acquired and screened using the same methodology.
Study Prioritization
Due to the large volume of the evidence identified, additional eligibility criteria were applied to prioritize primary research on theoretical DMHT valuation factors. In line with the thematic framework synthesis objective, theoretical valuation factors were defined as user or DMHT attributes that impact interaction with or perception of DMHTs; therefore, studies that reported only efficacy outcomes, such as mental illness symptom improvement, were deprioritized for full-text review. Secondary research was also deprioritized for full-text review. Studies that reviewed a select app against a framework and studies that reported only the outcomes specific to a select app were deprioritized for data extraction. For example, a study reporting the usability of a specific app’s features would have been deprioritized, while a study reporting what types of features increase MH app usability in general would not.
Data Extraction
All relevant data were extracted into a prespecified Microsoft Excel grid, and a quality assessment was performed for each study. Studies that reported only qualitative data were assessed with the Critical Appraisal Skills Program Qualitative Studies Checklist. Studies that reported only quantitative data were evaluated with the Critical Appraisal Skills Program Cohort Study Checklist, and studies reporting both qualitative and quantitative data were evaluated with both checklists []. Data extractions and quality assessments were performed by a single individual for each study, with the information verified by a second independent individual. Discrepancies were resolved by consensus, with arbitration by a third individual if necessary.
Framework Synthesis
A framework synthesis approach was undertaken to synthesize qualitative and quantitative data identified from the SLR. In line with the “best fit” framework synthesis approach, data were indexed deductively against an existing framework where possible, and novel themes were added inductively as needed [,]. The American Psychiatric Association (APA) Mental Health App Evaluation framework was considered the most appropriate framework to address the research objectives of this SLR because its key valuation themes were developed using psychiatrist and patient input, are broadly shared by other evaluation frameworks, are widely acknowledged in the literature, and have been described as durable and adaptable [-].
The APA model follows a hierarchical and chronological order whereby the evaluator moves through the framework using prompting questions (eg, “Does the app work offline?”). For this SLR, these questions were either thematically grouped into subthemes or left as prompting questions, as appropriate. The framework was therefore ultimately adapted into 3 levels: themes, subthemes, and more granular valuation criteria. It should be emphasized that this SLR did not aim to formally develop an updated framework to be used in practice by HCPs and their patients but rather was used to form a theoretical basis for understanding DMHT valuation factors, for which novel themes were expected to emerge.
A data-based convergent approach was used to synthesize quantitative and qualitative data []. Data were initially indexed deductively against the prespecified themes within the data collection instrument and then further synthesized within Docear [], a mind-map software used to organize and connect data and concepts. Indexing was performed by 1 reviewer and checked by a second independent reviewer. New themes and subthemes that emerged from the literature through inductive coding were added post hoc to the thematic framework, with all extracted data then considered against both the prespecified and novel themes. The evidence identified for each theme was synthesized narratively, taking into consideration the context and design of each study.
Results
Included Studies
A total of 4974 records were retrieved from the electronic databases. Of the 3374 (67.83%) unique records identified following deduplication across the databases, 2891 (85.68%) were excluded based on the eligibility criteria, and an additional 456 (13.52%) were deprioritized because they were not directly related to the topic of interest for this SLR. Excluded and deprioritized full texts are listed in Tables S9 and S10 in , respectively. Therefore, 27 (0.54%) articles were included from the electronic database searches. In addition, 1 article reporting on the same study as an already-included conference abstract was identified during supporting targeted searches and included as a supplementary record, resulting in a total of 28 articles reporting on 26 unique studies (Figure S1 in ). No relevant FDA appraisals were identified in the supplementary search.
Of the 26 included studies, 8 (31%) were quantitative, 12 (46%) were qualitative, and 6 (23%) used a mixed methods approach. While 5 (19%) studies assessed prospective cohorts, 22 (85%) used a cross-sectional approach, including 1 (4%) study that contained both a prospective cohort and a cross-sectional cohort (
Table 1). All studies (26/26, 100%) investigated a user perspective, with none specifically investigating payer or employer perspectives. Only 1 (4%) study, which examined ingestible sensor pills and smart pill dispensers to track adherence, investigated a DMHT that was not an app [].
| Study (author, year) | Designa | Perspective and populationa | Objectives | Data collection methodsa |
| Afra et al [], 2018 | Cross-sectional, quantitative |
| To develop a drug-device combination product using an app in combination with antiseizure medications as an epilepsy treatment | Custom survey |
| Beard et al [], 2019 | Cross-sectional, quantitative |
| To characterize general smartphone app and social media use in an acute transdiagnostic psychiatric sample with high smartphone ownership, characterize current engagement and interest in the use of smartphone apps to support MHe, and test demographic and clinical predictors of smartphone use | Custom survey |
| Borghouts et al [], 2022 | Cross-sectional, mixed methods |
| To investigate the MH needs of the deaf or hard-of-hearing community and how MH apps might support these needs | Custom survey; focus group |
| Boster and McCarthy [], 2018 | Cross-sectional, qualitative |
| To gain insight from speech-language pathologists and parents of children with ASD regarding appealing features of augmentative and alternative communication apps | Focus groups; poll questions |
| Buck et al [], 2021a | Cross-sectional, quantitative |
| To assess caregivers’ interest in an array of specific potential mHealthi functions to guide the development of mHealth for caregivers of young adults with early psychosis | Custom survey |
| Buck et al [], 2021b | Cross-sectional, quantitative |
| To understand the needs, interests, and preferences of young adults with early psychosis regarding mHealth by surveying interest in mHealth features and delivery modalities and by collecting information about their digital and web-based behaviors | Custom survey |
| Carpenter-Song et al [], 2018 | Prospective cohort, qualitative |
| To examine current practices and orientations toward technology among consumers in 3 mental health settings in the United States | Semistructured interviews |
| Casarez et al [], 2019 | Cross-sectional, qualitative |
| To explore how the well-being of spouses and partners of patients with BD can be improved through mHealth technology | Focus groups; minimally structured, open-ended individual interviews |
| Connolly et al [], 2018 | Cross-sectional, qualitative |
| To examine veterans’ attitudes toward smartphone apps and to assess whether openness toward this technology varies by age or rurality | Semistructured interviews informed by the State of the Art Access Model |
| Cummings et al [], 2019 | Cross-sectional, qualitative |
| To examine stakeholder perspectives regarding whether mHealth tools can improve MH treatment for low-income youth with ADHD in safety-net settings and what functions would improve treatment | Focus groups (caregivers) and interviews (HCPs and staff), both semistructured and including open-ended questions and targeted probes |
| Dinkel et al [], 2021 | Cross-sectional, qualitative |
| To explore patient and clinic-level perceptions of the use of depression self-management apps within an integrated primary care setting | Semistructured focus groups; semistructured interviews |
| Forma et al [], 2022 | Cross-sectional, quantitative |
| To assess caregivers’ preferences and willingness to pay for digital (ingestible sensor pill, medication containers with electronic monitoring, mobile apps, and smart pill dispensers) and nondigital (medication diary and simple pill organizer) tools | Custom discrete choice experiment survey |
| Hoffman et al [], 2019 | Prospective interventional, mixed methods |
| To test the feasibility of using mHealth apps to augment integrated primary care services, solicit feedback from patients and providers to guide implementation, and develop an MH app toolkit for system-wide dissemination | Custom survey |
| Huberty et al [], 2022 | Cross-sectional (current Calm (Calm.com, Inc) users) and prospective interventional (nonusers of Calm, HCPs), qualitative |
| To develop a mobile meditation app prototype specifically designed for patients with cancer and survivors of cancer | Custom surveys; focus groups |
| Kern et al [], 2018 | Cross-sectional, quantitative |
| To investigate the potential usefulness of MH apps and attitudes toward using them | Custom survey |
| Knapp et al [], 2021 | Prospective cohort, qualitative |
| To learn about considerations and perspectives of community behavioral HCPs on incorporating digital tools into their clinical care for children and adolescents | Focus groups |
| Kornfield et al [], 2022 | Prospective cohort, qualitative |
| To investigate how digital technologies can engage young adults in self-managing their MH outside the formal care system | Web-based asynchronous discussion; synchronous web-based design workshop |
| Lipschitz et al [], 2019 | Cross-sectional, quantitative |
| To assess patients’ interest in mHealth interventions for MH, to identify whether provider endorsement would impact interest, to determine reasons for nonuse of mHealth interventions for MH, and to identify what mHealth content or features are of most interest to patients | Custom survey |
| Mata-Greve et al [], 2021 | Cross-sectional, mixed methods |
| To document psychological stress, to explore DMHTo use in response to COVID-19–related stress, to explore the usability and user burden of DMHTs, and to explore which aspects and features of DMHTs were seen as necessary for managing stress during a pandemic by having participants design their own ideal DMHTs | Survey combining custom and validated measures (System Usability Scale, Use Burden Scale) |
| Melcher et al [], 2022 and Melcher and Torous [], 2020 | Cross-sectional, mixed methods |
| To examine why college students show poor engagement with MH apps and how apps may be adapted to suit this population | Custom survey; interviews |
| Schueller et al [], 2018 | Cross-sectional, mixed methods |
| To understand where users search for MH apps, what aspects of MH apps they find appealing, and what factors influence their decisions to use MH apps | Custom survey; focus group interviews |
| Schueller et al [], 2021 | Cross-sectional, qualitative |
| To understand motivations for and experiences in using mood-tracking apps from people who used them in real-world contexts | Semistructured interviews |
| Stiles-Shields et al [], 2017 | Cross-sectional, qualitative |
| To identify the barriers to the use of a mobile app to deliver treatment for depression and to provide design implications on the basis of identified barriers | Card sorting task |
| Storm et al [], 2021 | Cross-sectional, qualitative |
| To identify stakeholders’ perspectives on partnering to inform the software development life cycle of a smartphone health app intervention for people with serious mental illness | Semistructured interviews |
| Torous et al [], 2018 | Cross-sectional, quantitative |
| To understand how individuals with mental illness use their mobile phones by exploring their access to mobile phones and their use of MH apps | Custom survey |
| Zhou and Parmanto [], 2020 | Cross-sectional, mixed methods |
| To determine user preferences among the several privacy protection methods used in current mHealth apps and the reasons behind those preferences | Custom survey; interview |
aOnly information relevant to this systematic literature review is reported in this table.
bMDD: major depressive disorder.
cBD: bipolar disorder.
dOCD: obsessive-compulsive disorder.
eMH: mental health.
fGeneral users are participants who were not necessarily diagnosed with indications of interest.
gASD: autism spectrum disorder.
hHCP: Health care provider.
imHealth: mobile health.
jPTSD: posttraumatic stress disorder.
kVA: Veterans Affairs.
lADHD: attention-deficit/hyperactivity disorder.
mPHQ-9: Personal Health Questionnaire-9.
nGAD-7: Generalized Anxiety Disorder-7.
oDMHT: digital mental health technology.
Most frequently, studies focused on indications for mood, anxiety, or psychotic disorders (15/26, 58%), with other indications of focus including ADHD (2/26, 8%), ASD (1/26, 4%), and epilepsy (1/26, 4%). No relevant studies focused on Alzheimer disease were identified.
A total of 8 (31%) studies assessed the perspectives toward DMHTs of general population participants who were not necessarily diagnosed with relevant conditions [,,,-]. Of these populations, several were identified as having an increased risk of MH conditions, such as patients with cancer [], college students [,], deaf or hard-of-hearing individuals [], and people who were unemployed or furloughed during the COVID-19 pandemic []. In addition, 1 (4%) study included a mix of patients who were above and below the referral criteria for psychotherapy for depression [].
Thematic Analysis
Overview
Evidence was identified for all 5 themes included in the APA framework: engagement style (23/26, 88%), background and accessibility (16/26, 62%), privacy and security (13/26, 50%), therapeutic goal (12/26, 46%), and clinical foundation (8/26, 31%; Table 2). Five novel criteria were identified and added to the framework post hoc, 1 each under engagement style (forgetting or feeling unmotivated to use DMHTs) and privacy and security (personal image and stigma) and 3 under background and accessibility (willingness to pay, insurance restrictions, and cost savings compared with professional care).
| Subtheme | Criteria (study reference) | ||
| Engagement style | |||
| Short-term usability |
| ||
| Long-term usability |
| ||
| Customizability |
| ||
| Background and accessibility | |||
| Technical |
| ||
| Business modelb |
| ||
| Costs |
| ||
| Medical claimsb |
| ||
| Stability |
| ||
| Privacy and security | |||
| No specific subtheme |
| ||
| Data collection and storage |
| ||
| Privacy policy |
| ||
| Personal health information |
| ||
| Security measures |
| ||
| Clinical foundation | |||
| Impressions of use |
| ||
| User feedback |
| ||
| Clinical validity |
| ||
| Therapeutic goal | |||
| Clinically actionable |
| ||
| Therapeutic alliance |
| ||
| Data ownership, access, and exportb |
| ||
aNovel findings that emerged from this systematic literature review.
bThese subthemes and criteria were included in the American Psychiatric Association’s framework but were not reported on by studies included in this systematic literature review.
cHCP: health care provider.
Theme 1: Engagement Style
Engagement style was the most reported theme, with evidence identified from 23 (88%) of the 26 studies. Engagement style encompasses how and why users do or do not interact with DMHTs. The long-term usability subtheme was reported by 96% (22/23) of studies, short-term usability by 12 (52%) studies, and customizability by 7 (30%) studies. Findings from short- and long-term usability subthemes were highly interconnected.
A total of 4 studies reported that ease of use promoted short-term DMHT engagement. In the study by Schueller et al [], 89.6% of a general population of smartphone users reported ease of use for MH apps as “important” or “very important,” and users qualitatively reported dislike of “overwhelming,” difficult-to-navigate apps. In addition, users valued apps that were “simplistic” [], fit into their daily schedules, and were available when needed (eg, during acute symptom experiences) [,]. Select supporting qualitative data are presented in Table 3.
| Subtheme and criteria: findings | Key quotes | |||
| Engagement style | ||||
| Short-term usability | ||||
| Ease of use |
| |||
| Available engagement styles: use of animation and visuals |
| |||
| Long-term usability | ||||
| Alignment of app with needs and priorities: gamification |
| |||
| Alignment of app with needs and priorities: anxiety management |
| |||
| Alignment of app with needs and priorities: tracking mood, symptoms, or sleep |
| |||
| Alignment of app with needs and priorities: social media–like features |
| |||
| Alignment of app with needs and priorities: peer support and chat functions |
| |||
| Forgot or unmotivated to usee |
| |||
| Background and accessibility | ||||
| Technical considerations | ||||
| Accessibility: mobility barriers |
| |||
| Accessibility: technical literacy |
| |||
| Offline functionality: internet and mobile data access as a barrier to use |
| |||
| Costs | ||||
| Willingness to paye |
| |||
| Privacy and security | ||||
| Data collection and storage | ||||
| Security associated with collection, use, and transmission of sensitive data (including personal health information) |
| |||
| Privacy policy | ||||
| Transparency and accessibility of privacy policy |
| |||
| Personal health information | ||||
| Personal image and stigmae |
| |||
| Security measures | ||||
| Security systems used |
| |||
| Therapeutic goal | ||||
| Clinically actionable | ||||
| Positive change or skill acquisition: apps that impart skills and encourage positive change, in an easy way |
| |||
| Ease of sharing and interpretation of data: increase of engagement and symptom reporting |
| |||
| Therapeutic alliance | ||||
| Therapeutic alliance between patient and HCP |
| |||
| Clinical foundation | ||||
| Clinical validity | ||||
| Evidence of specific benefit: HCP recommendations |
| |||
| Evidence of specific benefit: increased usage if supported by research, academic institution, or reputable professional society |
| |||
aASD: autism spectrum disorder.
bMH: mental health.
cADHD: attention-deficit/hyperactivity disorder
dBD: bipolar disorder.
eNovel criteria identified by this systematic literature review.
fCHA: Cambridge Health Alliance.
gEMR: electronic medical record.
hHCP: health care provider.
Users valued DMHT features that aligned with their needs and priorities, as reflected by findings within the long-term usability subtheme. Across 9 studies, quantitative and qualitative findings demonstrated high interest in anxiety management features such as relaxation tools, breathing exercises, and mindfulness or meditation activities, and 10 studies identified interest in mood, symptom, or sleep tracking (Tables 3 and 4). While most studies (24/26, 92%) focused on MH, patients with epilepsy also reported high interest in features to record seizure dates and types []. Importantly, users in 2 studies emphasized the need for developers to tailor DMHTs to the needs and priorities of the target population (Table 3) [,]. Relatedly, mixed attitudes were reported toward positive affirmations and words of encouragement, with many users expressing interest but others emphasizing the value of a human component to DMHTs or cautioning against blanket encouragement and automated messages that could feel insincere [,,].
| Features, study, perspective, and finding | Patients, n (%) | Likert score, mean (SD) | |||||
| Anxiety management | |||||||
| Buck et al[], 2021b | |||||||
| Young adults with early psychosis | |||||||
| Interest in skill practices for managing stress and improving mood | 64 (84.2) | 3.30 (0.98)a | |||||
| Interest in skill practices for relaxation | 57 (76) | 3.09 (1.12)a | |||||
| Interest in information about relaxation exercises | 59 (77.6) | 3.00 (1.16)a | |||||
| Interest in information about healthy sleep practices | 56 (73.7) | 2.93 (1.15)a | |||||
| Interest in mindfulness or meditation practices | 44 (59.4) | 2.61 (1.34)a | |||||
| Afra et al [], 2018 | |||||||
| Patients with epilepsy | |||||||
| Interest in music to help seizure control | —b (75) | — | |||||
| Interest in relaxing music that may help alleviate stress | — (68) | — | |||||
| Interest in relaxing imagery that may help alleviate stress | — (40) | — | |||||
| Interest in drawing or writing while listening to music | — (35) | — | |||||
| Interest in practicing mindfulness | — (63) | — | |||||
| Torous et al [], 2018 | |||||||
| Outpatients attending a private psychiatric clinic | |||||||
| Comfort level for mindfulness and therapy | — | 3.75c | |||||
| Outpatients attending a state psychiatric clinic | |||||||
| Comfort level for mindfulness and therapy | — | 3.17c | |||||
| Beard et al [], 2019 | |||||||
| Patients in a partial hospitalization program in a psychiatric hospital | |||||||
| Current use of an MHd app with the primary purpose being mindfulness or meditation | — (71) | — | |||||
| Mata-Greve et al [], 2021 | |||||||
| Workers furloughed during COVID-19 | |||||||
| Most frequently endorsed mindfulness tools as a feature when provided options to build their own app | 687 (67.8) | — | |||||
| Essential workers employed during COVID-19 | |||||||
| Most frequently endorsed mindfulness tools as a feature when provided options to build their own app | 584 (60) | — | |||||
| Nondistressed essential workers employed or workers furloughed during COVID-19 | |||||||
| Most frequently endorsed mindfulness tools as a feature when provided options to build their own app | 305 (61.4) | — | |||||
| Distressed essential workers employed or workers furloughed during COVID-19 | |||||||
| Most frequently endorsed mindfulness tools as a feature when provided options to build their own app | 966 (65.3) | — | |||||
| Hoffman et al [], 2019 | |||||||
| Staff in a routine primary care behavioral health setting | |||||||
| The ability to manage mood, anxiety, or substance use through the use of DMHTse was seen as a benefit of incorporating DMHTs into clinical care | 13 (57) | — | |||||
| Symptom, mood, or sleep trackers | |||||||
| Kern et al [], 2018 | |||||||
| General population of college students | |||||||
| Willingness to use an MH app to track mood or anxiety | 41 (10.3) | — | |||||
| Afra et al [], 2018 | |||||||
| Patients with epilepsy | |||||||
| Interest in a diary to record the date of seizures | — (85) | — | |||||
| Interest in a digital diary to record the type of seizure | — (73) | — | |||||
| Interest in digital diary to log the missed dosages of their medications | — (78) | — | |||||
| Lipschitz et al [], 2019 | |||||||
| Veterans with anxiety, MDDf, or PTSDg | |||||||
| Interested in progress monitoring (track mood, stress, anxiety, or PTSD symptoms) | 95 (63.8) | — | |||||
| Subgroup of smartphone owners | |||||||
| Interested in progress monitoring (track mood, stress, anxiety, or PTSD symptoms) | 80 (67.2) | — | |||||
| Bucket al[], 2021b | |||||||
| Young adults with early psychosis | |||||||
| Interest in a feature to set and track goals | 60 (78) | 3.10 (1.05)a | |||||
| Interest in a feature to track symptoms over time | 70 (90.9) | 3.44 (0.90)a | |||||
| Interest in a feature to track changes in progress toward goals | 66 (86.9) | 3.37 (0.86)a | |||||
| Interest in a feature to track wellness behaviors (eg, steps or activity) | 48 (64.9) | 2.86 (1.22)a | |||||
| Beard et al [], 2019 | |||||||
| Patients in a partial hospitalization program in a psychiatric hospital | |||||||
| Current use of an MH app with the primary purpose being mood tracking | — (10) | — | |||||
| Willingness to use an MH app daily to monitor condition | 262 (81) | — | |||||
| Subgroup with higher education | |||||||
| Willingness to use an MH app daily to monitor condition | — (85) | — | |||||
| Subgroup with lower education | |||||||
| Willingness to use an MH app daily to monitor condition | — (77) | — | |||||
| Mata-Greve et al [], 2021 | |||||||
| Workers furloughed during COVID-19 | |||||||
| Most frequently endorsed symptom tracking (tracking sleep or mood) as a feature when provided options to build their app | 605 (59.7) | — | |||||
| Essential workers employed during COVID-19 | |||||||
| Most frequently endorsed symptom tracking (tracking sleep or mood) as a feature when provided options to build their app | 555 (57) | — | |||||
| Nondistressed essential workers employed or workers furloughed during COVID-19 | |||||||
| Most frequently endorsed symptom tracking (tracking sleep or mood) as a feature when provided options to build their app | 270 (54.3) | — | |||||
| Distressed essential workers employed or workers furloughed during COVID-19 | |||||||
| Most frequently endorsed symptom tracking (tracking sleep or mood) as a feature when provided options to build their own app | 890 (60.2) | — | |||||
| Torous et al [], 2018 | |||||||
| Outpatients attending a private psychiatric clinic | |||||||
| Comfort level for in-app symptom surveys | — | 3.50c | |||||
| Outpatients attending a state psychiatric clinic | |||||||
| Comfort level for in-app symptom surveys | — | 3.11c | |||||
| Outpatients attending a private psychiatric clinic | |||||||
| Comfort level for passive call or text monitoring | — | 2.32c | |||||
| Outpatients attending a state psychiatric clinic | |||||||
| Comfort level for passive call or text monitoring | — | 2.39c | |||||
| Outpatients attending a private psychiatric clinic | |||||||
| Comfort level for passive GPS monitoring | — | 2.31c | |||||
| Outpatients attending a state psychiatric clinic | |||||||
| Comfort level for passive GPS monitoring | — | 2.78c | |||||
aA 5-point Likert scale (0-4) was used.
bNot available.
cA 5-point Likert scale (1-5) was used.
dMH: mental health.
eDMHT: digital mental health technology.
fMDD: major depressive disorder.
gPTSD: posttraumatic stress disorder.
Both patients and caregivers expressed interest in psychoeducational content that aligned with their needs and priorities. When surveyed, >60% of veterans with anxiety or major depressive disorder (MDD), patients with epilepsy, young adults with psychosis, and essential and furloughed workers during the COVID-19 pandemic expressed interest in relevant psychoeducational content [,,,]. In contrast, only 4% of college students in another study reported using an MH app for information about MH, although an MH diagnosis was not required for study participation [].
Caregivers of young adults with psychosis, caregivers of children with ADHD, and spouses and partners of people with bipolar disorder (BD) were all interested in information related to caring for the individual with the given disorder, such as information on psychological and pharmacological treatments, symptoms and symptom changes, and the MH system [,,]. Comparatively smaller, but still notable, proportions of caregivers of patients with psychosis were interested in caregiver-focused information; for instance, 62% to 69% were interested in relaxation exercises, stress and mood management, and community events for caregivers, while 85% to 90% were interested in the aforementioned patient-focused information [].
Information delivery–style preference was captured under the short-term usability subtheme. One study in young adults with psychosis and another study with their caregivers revealed that delivering information in a variety of formats was important; when presented with nonmutually exclusive options, >50% of both populations were interested in text content, video content, audio content, and discussion boards [,].
Social interaction promoted long-term engagement. Qualitatively, 3 studies found that users valued learning about similar experiences from others via social media–like features, which normalized their experiences and could provide new symptom management strategies (Table 3) [,,]. Similarly, 67% of both young adults with psychosis and deaf or hard-of-hearing survey participants (N=9) reported interest in peer support via chat features [,]. However, a comparatively smaller proportion of veterans with anxiety or MDD (48.3% of the full cohort and 51.3% of the smartphone user subgroup) were interested in peer support [].
Overall, users endorsed social features to support their MH. In the study by Casarez et al [], spouses and partners of people with BD likewise desired features to communicate with other caregivers and also emphasized that DMHTs could facilitate conversation and understanding with patients, a sentiment echoed by peer support specialists by Storm et al [] (Table 3). However, one oncology HCP cautioned that similar to support groups, “very strict guidelines of what is said” should be implemented to manage potential risks from shared social media–like content, although little additional context was provided [].
Spouses and partners of people with BD also suggested both in-app information on accessing professional resources and direct counseling for the patient at times when other support might be inaccessible []. More than half of all workers, employed or unemployed during the COVID-19 pandemic, likewise endorsed links to resources, counseling, and crisis support as DMHT features, and 81.6% of young adults with psychosis endorsed a feature to communicate with professional experts [,]. Importantly, compared with patients attending public clinics, patients attending private psychiatric clinics expressed a higher comfort level for in-app communication with HCPs, suggesting demographic differences in the valuation of access to professional support through DMHTs [].
A total of 9 studies reported an interest in DMHT reminders and notifications. Across 3 studies, >70% of patients or caregivers were interested in appointment reminders [,,]. In addition, 73% and 68% of patients with epilepsy reported interest in reminders for medication refills and adherence, respectively []. Beyond apps, caregivers of patients with MDD, BD, and schizophrenia preferred an ingestible pill sensor that tracked medication adherence, physical activity, mood, and rest 9.79 (95% CI 4.81-19.9), 7.47 (95% CI 3.81-14.65), and 6.71 (95% CI 3.29-13.69) times more than a nondigital pill organizer, respectively []. Qualitatively, patients and caregivers also appreciated reminders, especially if reasonably timed or delivered via text messages [,].
Short-term DMHT engagement was also supported by games and graphics, which could communicate information in an accessible way [], provide tools for stress management [,], and be used therapeutically with children [,]. However, some HCPs and caregivers expressed concerns that graphics and games may be distracting for certain children (Table 3) [].
In a novel finding, 3 studies reported forgetfulness or lack of motivation as an influence on DMHT engagement. In some cases, disuse was related to stress, other MH symptoms, or poor technical literacy (Table 3) [,,]. In contrast, “forgetting to use” DMHTs and “lack of motivation” were perceived as relatively small barriers to use in the study by Stiles-Shields et al [].
The third subtheme under engagement style was customizability, which was generally valued by users; 70.9% of a general population of smartphone users noted customization was an important factor []. Similarly, 9.4% of all surveyed veterans and 10.9% of those with smartphones reported disliking a prior DMHT due to a lack of personalization []. Users specifically wanted to be able to opt out of irrelevant features, customize audiovisual and design elements, add personal notes to tracked mood data, and provide ongoing feedback to facilitate personalization [,,,,].
Theme 2: Background and Accessibility
A total of 16 (62%) studies reported findings related to DMHT background and accessibility, which considers the developer of the DMHT, as well as functionality and accessibility. Of these, 12 (75%) studies reported on the technical considerations subtheme, 9 (56%) on costs, and 2 (13%) on stability.
Under technical considerations, 9 studies assessed diverse accessibility concerns. Broadly, Storm et al [] emphasized that DMHTs should be developed in consideration of patients’ social, cognitive, and environmental needs to avoid overwhelming users. Specifically, 2 studies reported language as a barrier. Deaf or hard-of-hearing participants recommended visual content presentation, such as videos and icons, alongside text and American Sign Language translations where possible []. Similarly, when discussing English-only apps, 1 provider stated as follows: “language is a barrier for some [patients]” []. Mobility issues related to MH symptoms or other conditions and technical literacy, such as difficulties remembering passwords and navigating smartphones or apps, created accessibility barriers as well (Table 3) [,,,]. Additional concerns included apps that restricted use based on geographic location [], user difficulty in finding relevant, useful apps [], and limited mobile device memory for downloading apps [,].
Offline functionality, reported by 6 studies, was also captured under the technical considerations subtheme. A majority (5/9, 56%) of participants included in the study by Borghouts et al [] expressed concern about their mobile data plans when using their devices. Correspondingly, “availability of Wi-Fi” was noted as a top barrier to the use of apps for depression by Stiles-Shields et al [], and several veterans in another study reported that home Wi-Fi connectivity facilitated app use by eliminating cellular data fees [,]. Quotes from patients and HCPs echoed the concern about apps without offline functionality (Table 3) [,].
Data fees were also captured under the costs subtheme, with hidden or additional costs described as a barrier to app use by 2 studies [,]. Parents of children with ADHD reported that difficulty paying phone bills could result in their phones being shut off, limiting DMHT use; one MH clinic administrator stated as follows: “We often encounter parents’ phones being shut off because they haven’t paid their bill...If the app were free or low cost, I imagine it could be very helpful” []. In addition to hidden costs, this quote identifies up-front app costs as a barrier. Quantitatively, more than half of a general population of surveyed college students expressed that cost was a top concern for the use of MH apps []. Qualitative findings from 2 additional studies likewise identified cost as a barrier to DMHT use [,].
Three novel cost attributes were identified by this SLR: willingness to pay, insurance restrictions, and cost savings compared with professional care. Four studies, 3 of which focused on apps, explored willingness to pay for DMHTs from a user perspective. Willingness to pay varied based on user preference; some surveyed college students and smartphone users among general populations valued free apps due to financial restrictions or uncertainty around app effectiveness, although 1 student commented that the quality of free trials might be inferior [,]. Some smartphone users also voiced a limit on how much they would be willing to spend for an app subscription (Table 3) []. Forma et al [] found that caregivers were willing to pay US $255.04 (95% CI US $123.21-US $386.86) more per month for a pill with an ingestible sensor that tracked medication adherence, physical activity, and rest and could connect to an app that also collected self-reported mood data. Moreover, the caregivers were willing to pay US $124.50 (95% CI US $48.18-US $200.81) more per month for an app-connected pill organizer alone than for a nondigital pill organizer []. In contrast, some veterans expressed total disinterest in paid apps, with 1 user citing poor technical literacy (“don’t have the knowledge”) in addition to cost as affecting willingness to pay [].
In another novel finding, a speech-language pathologist working with children with ASD preferred a single app including multiple features over separate apps for particular features due to insurance restrictions: “I agree that teaching Apps should be an in-App feature versus their own app because sometimes insurance doesn’t allow us to open the iPads purchased through insurance” []. Although no further detail was provided for this finding, it suggests that there may be restrictions on the use of other apps on devices purchased under insurance, which may have implications for DMHT use in formal care settings due to the lack of financial support.
In a third novel cost-related finding, a small number of participants from a general population of students (3.6%) in one study preferred using an MH app to seeing an MH professional due to cost savings [].
A total of 13% (2/16) of studies reported on the subtheme of app stability and technical difficulties, with crashes and poor display quality decreasing DMHT value [,]. Participants in the study by Schueller et al [] reported that technical difficulties were often an issue for apps developed by medical institutions, which might be effective and safe but less usable than apps from other developers.
Theme 3: Privacy and Security
A total of 13 (50%) out of 26 studies reported findings related to the privacy and security theme, which covered the use and protection of user data by DMHTs. Subthemes were reported relatively equally: data collection and storage (5/13, 38%), personal health information (PHI; 5/13, 38%), privacy policies (4/13, 31%), general privacy (3/13, 23%), and security measures (3/13, 23%).
Quantitative and qualitative findings on general privacy (ie, evidence not categorized under any specific subtheme), the data collection and storage subtheme, and the privacy policies subtheme revealed heterogeneous concerns (Table 3). A total of 74% of a general population of college students reported privacy as a top concern for MH apps, although further details on the specific area of concern were unclear []. In the study by Stiles-Shields et al [], participants were highly concerned with data access but less so with general privacy. Echoing the concerns about data collection and storage, 59.1% of veterans with anxiety or MDD in 1 study were concerned about in-app PHI protection []; however, a qualitative study in veterans with posttraumatic stress disorder, alcohol use disorder, or MDD reported that a relatively small number of participants expressed privacy concerns. In the latter study, reasons for the concerns included distrust in Veterans Affairs, belief that digital data are inherently not confidential, and fear of phone hacking []. From an HCP perspective, none of the surveyed behavioral health HCPs agreed with the statement “My patients are concerned about data security,” despite multiple patients within the same study reporting privacy concerns [].
Still, privacy policies were important overall, with 70.5% of smartphone MH app users rating having a privacy policy as “very important” or “important” []. Melcher et al [] found that although users valued data protection, some reported a lack of awareness about data privacy, and others were concerned about obscure privacy policies and PHI use. As noted in the data collection and storage subtheme, veteran concerns about government use of PHI were heterogeneous [].
A novel valuation factor not included in the APA framework related to user concern with PHI privacy and security regarding MH diagnoses and MH app use is a desire to upkeep their personal image or avoid stigma (Table 3) [,,,]. For instance, 21.1% of a general college student population preferred MH app use to seeing an MH professional due to anonymity or reduced stigma []. One participant in a study of Veterans Affairs health service users described access to professional care via MH apps as convenient because they could avoid disclosing their use of MH services to explain leaving work early for an appointment [].
In line with the overarching concern about PHI privacy and security, users valued app security measures. Schueller et al [] reported that 74.2% of users rated data encryption as “important” or “very important.” Users in another study perceived the level of privacy protection as the highest for apps using a combination of a generic app name (ie, not reflecting the indicated MH disorder); easily hidden modules; and secure, user-authenticated web portals for making module changes []. Behavioral health clinic staff echoed the importance of discreet MH app names (Table 3) [].
Theme 4: Therapeutic Goal
There were 12 (46%) studies that reported on the factors relating to the integration of DMHTs with users’ therapeutic goals. The clinical actionability and therapeutic alliance subthemes were reported by 83% (10/12) and 58% (7/12) of studies, respectively.
A total of 9 studies reported the value of clinically actionable insights from apps where the users could acquire and practice new skills to make positive changes in their lives (Table 3). For instance, patient and caregiver app users reported interests in “daily tips,” “new ideas,” and “solutions or recommendations” for symptom management [,,]. Furthermore, an app that could serve as a resource for multiple management strategies was preferable [,,]. Quantitatively, 4% of patients receiving acute treatment in a partial hospitalization program for MH conditions, including mood and psychotic disorders, reported that the primary purpose of their DMHT use was therapy skills practice []. HCPs similarly appreciated that DMHTs could facilitate patients practicing skills outside of formal treatment sessions []. In particular, clinicians from a youth behavioral health clinic noted that DMHTs might be especially beneficial for young users because they could be conveniently and discreetly incorporated into their daily lives [].
Users valued easy data sharing with clinicians, particularly for mood- or symptom-tracking features, which could improve communication and the accuracy of symptom reporting during clinical visits [,-,,]. For instance, 53% of a general college student population believed that the potential to share information with their clinician was “one of the top benefits” of using DMHTs []. In addition, many HCPs reported active use or interest in the use of DMHTs in clinical practice to facilitate asynchronous communication and increase patient engagement with treatments outside of formal appointments; however, some preferred traditional care strategies for their personalization and flexibility (Table 3) [,,].
Theme 5: Clinical Foundation
A total of 8 (31%) studies reported findings related to the clinical foundation of DMHTs, that is, their utility and appropriateness for patients. Clinical validity was the most reported subtheme, with evidence identified from 6 (75%) studies; 2 (25%) studies reported on the user feedback subtheme and 2 (25%) on the impressions of use subtheme, which captured users’ perceptions of app content as accurate and relevant.
Across subthemes, users valued evidence of DMHT benefit or efficacy from various sources. A total of 71.8% of surveyed veterans said that they would use a DMHT if they “saw proof that it worked” for their MH conditions []. Similarly, among the 811 general population participants surveyed, 69.5% ranked direct research evidence as “important” or “very important” for DMHT, and 66.8% ranked indirect research evidence the same []. Qualitative data identified recommendations from HCPs or academic institutions, as well as evidence of DMHT benefit from publications or research studies, as specific sources for clinically valid evidence of benefits (Table 3) [,,].
In addition to academic and professional support, the user feedback subtheme captured user interest in whether DMHTs were beneficial for peers or recommended by other trusted individuals. Patients with depression reported that other users’ experiences influenced their app use, with one user wanting to know “...if other people had success using it” []. Quantitatively, user ratings and user reviews were ranked as “important” or “very important” factors in DMHT use by 59.4% and 58.7% of the general population participants, respectively [].
Quality Assessment
The risk of bias was overall moderate. Of the 14 studies including quantitative components, only 1 (7%) used relevant validated outcome measurement instruments []; all others used custom questionnaires. Of the 18 studies with qualitative components, 4 (22%) were at risk of selection bias due to participants being exclusively recruited using web-based postings and research registries [-,], and only 1 (6%) considered the relationship between researcher and participant when interpreting the results []. Full quality assessments for qualitative and quantitative study components can be found in Tables S11 and S12 in , respectively.
Discussion
Principal Findings
This SLR aimed to identify and synthesize qualitative and quantitative evidence on how DMHTs are valued by users, payers, and employers in the United States. Evidence from users with or without diagnosed relevant disorders, caregivers, and HCPs was captured across a wide range of demographics. No study reported evaluating an app from a payer or employer perspective. Furthermore, all but one included study focused on mobile apps.
No relevant appraisals of DMHTs were identified from the FDA website searches; however, 8 relevant FDA approval labels or notifications for MH apps or guidance documents for industry and FDA staff were identified. The content of these materials overlapped with some valuation factors identified in this SLR, including evidence of clinical efficacy and safety, app maintenance, and privacy and security.
Engagement style, although not covered by the FDA materials, was the most reported theme by the studies included in this SLR and was found to overlap heavily with other themes. Engagement may be a key consideration for app developers, as app user retention can be low: 1 study showed that >90% of users had abandoned free MH apps within 30 days of installation []. Engagement is also a key clinical concern in terms of DMHT efficacy; one meta-analysis of 25 studies showed that increased use of DMHT modules was significantly associated with positive outcomes regardless of the target MH condition []. The findings of this SLR may therefore be informative to both DMHT designers and HCPs who integrate DMHTs into clinical care by providing insight on DMHT valuation and thus how use and benefit can be improved. For instance, users valued DMHTs that were easy to use and aligned with their needs and priorities, particularly through features that supported their therapeutic goals. In addition, content presented through multiple delivery modes, such as both text and visuals, promoted engagement as well as accessibility.
However, engagement and feature preference varied across populations. For instance, DMHT valuation was affected by technical literacy, which may relate to user demographics; in this SLR, veterans repeatedly emphasized technical literacy as a barrier to DMHT use []. Similarly, offline functionality may be more important for some users. Although 85% of the total United States population owns smartphones, only 59% of Medicare beneficiaries have access to a smartphone with a wireless plan. Moreover, beneficiaries who are older, less educated, disabled, or Black or Hispanic have even lower digital access [,]. These findings emphasize the importance of customizability and suggest that app development and selection in the clinical setting should consider the demographics of the target population, particularly in relation to ease of use and offline functionality.
Background and accessibility findings also identified up-front and hidden costs as barriers to DMHT use, with the willingness to pay varying among individuals. This has important implications for app development, considering that many MH apps currently on the market are direct-to-consumer sales and require out-of-pocket payment. App developers often take this approach as it does not require the accumulation of formal evidence of clinical benefit for FDA approval [], but it may present a financial barrier to use for consumers.
Privacy and security, reported by 13 (50%) out of 26 studies, was a prevalent theme, with users primarily concerned with data and PHI security within apps. This finding reflects wider research; a 2019 review of 116 depression-related apps retrieved from iTunes and Google Play stores in 2017 found that only 4% of the identified apps had acceptable transparency in privacy and security, with many completely lacking a privacy policy []. Similarly, 39% of MH apps recommended by college counseling centers had no privacy policy, and of those with a policy, 88% collected user data, and 49% shared that data with third parties []. Most evidence identified in this SLR under this theme, as well as findings previously published in the wider literature, focuses on these remote privacy risks. However, local privacy concerns are also important to users. In particular, inconspicuous naming and the ability to hide sensitive modules within MH apps were rated as highly important by both patients and HCPs to maintain user privacy. Users emphasized a desire to avoid the stigma associated with mental illness, which was also reflected by the findings in the engagement style theme: more young adults with psychosis were more interested in in-app messaging with other patients in psychosis recovery (67.1%) than a provider and family member together (47.3%) or their personal support network (59.8%) []. Similarly, youths were interested in apps that could be used discreetly in school or other public settings to avoid potential MH stigma. This is a key, novel finding of this SLR, considering that many app or DMHT components on the market are named after their target disorder.
The use of DMHTs to achieve therapeutic goals was discussed from patient, caregiver, and HCP perspectives, all of which valued DMHTs that had evidence of efficacy, presented clinically actionable information, and facilitated patient-clinician relationships. Of the 5 studies that explored how HCPs value DMHTs in clinical practice, 2 (40%) were restricted to the oncology or ASD settings and were not readily generalizable to wider MH settings [,]. In other studies, providers reported interest in using DMHTs to facilitate asynchronous communication with patients and their caregivers, promote patient skill practice, and improve care for children through the use of games and visuals [,]. However, while HCPs overall believed that DMHTs improved care, some believed that their clinical training allowed for care personalization beyond what DMHTs could provide. Feature customizability and receipt of input from HCPs and users during app development and testing may be a way to mitigate these concerns, as well as concerns about safety and efficacy, as many available apps do not appropriately address user health concerns [].
Findings additionally suggested that training and resources on DMHTs would be beneficial to ensure that HCPs were equipped to integrate DMHTs into their practices []. Collaboration between DMHT specialists and HCPs, along with a shift from randomized controlled trials to effectiveness-implementation hybrid trials, may be a way to streamline the integration of DMHTs into clinical care and provide more training and resources for HCPs [,].
Strengths
This review followed a prespecified protocol and used systematic methods in line with the York Centre for Reviews and Dissemination guidelines [] to conduct an exhaustive search of the literature, identifying evidence relevant to the review objectives from multiple databases and supplementary sources. The framework synthesis approach allowed for the inclusion and analysis of both qualitative and quantitative data, providing a detailed picture of not only what DMHT features users value but why they value them, especially in areas where valuation varies across patient demographics. In addition, the APA framework is a robust model created with patient and HCP input that incorporates key valuation themes broadly shared by other frameworks and widely acknowledged in the literature [-].
Limitations
Methodological limitations should be considered when interpreting the findings of this SLR. Only publications in English and in United States populations were included. As perceptions of value are influenced by factors including cultures, laws, and health care settings, the findings of this SLR should not be generalized to other countries. For instance, trust in HCPs and rates of longstanding relationships between patients and primary care providers are lower in the United States than in many European nations [,], which could impact the type of support users want from DMHTs (ie, engagement style) or interest in DMHT integration with therapeutic goals.
In addition to the prespecified eligibility criteria, deprioritization strategies were implemented due to the large volume of the identified evidence, and this may have resulted in missing relevant articles. In particular, the deprioritization of secondary research and opinion pieces likely led to the exclusion of relevant discussion around payer perspectives and reimbursement, for which no evidence was included in this SLR. Furthermore, although unlikely, there may have been reporting biases in the included studies due to missing results, which this SLR was not able to assess.
This SLR identified no evidence for 3 subthemes included in the APA framework: business model (background and accessibility), which covers DMHT funding sources and potential sources of conflict, medical claims (background and accessibility), which examines whether DMHTs claim to be medical and the trustworthiness of their creators, and data ownership, access, and export (therapeutic goal), which includes sharing data with eHealth records or wellness devices (eg, Apple HealthKit [Apple Inc], Fitbit [Google LLC]). The valuation of these subthemes should be evaluated in future research.
Conclusions
In summary, app usability, cost, accessibility and other technical considerations, and alignment with therapeutic goals were the most reported valuation factors identified by this SLR. Many studies also reported user preference for apps that incorporated privacy and security features that provided protection from stigma. However, individual DMHTs and their features are valued differently across individuals based on demographics and personal preferences. MH apps should be developed and selected with these specific user needs in mind. Feature customizability and input from users and HCPs during development may improve app usability and clinical benefit.
Acknowledgments
The authors thank Max Lee, Costello Medical, US, for medical writing and editorial assistance based on the authors’ input and direction.
Conflicts of Interest
DCM is a consultant for Otsuka Pharmaceutical Development & Commercialization (OPDC) Inc for this project and has received consulting funds from Pear Therapeutics, Sanofi, Avidity, Sarepta, Novartis, and BioMarin. ML, HG, and HCW are employees of OPDC. JC, SB, RSK, and EW are employees of Costello Medical. This research was funded by OPDC.
Multimedia Appendix 1
Electronic database and supplementary search terms, systematic literature review eligibility criteria, publications excluded or deprioritized at full-text review, quality assessments of included studies, and the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flow diagram of the identified publications.
DOCX File , 189 KBReferences
- Content of premarket submissions for device software functions: guidance for industry and Food and Drug Administration staff. U.S. Food and Drug Administration. Jun 14, 2023. URL: https://www.fda.gov/media/153781/download [accessed 2024-07-19]
- Patel NA, Butte AJ. Characteristics and challenges of the clinical pipeline of digital therapeutics. NPJ Digit Med. Dec 11, 2020;3(1):159. [FREE Full text] [CrossRef] [Medline]
- Policy for device software functions and mobile medical applications: guidance for industry and Food and Drug Administration staff. U.S. Food and Drug Administration. URL: https://www.fda.gov/media/80958/download [accessed 2023-01-06]
- Melcher J, Torous J. Smartphone apps for college mental health: a concern for privacy and quality of current offerings. Psychiatr Serv. Nov 01, 2020;71(11):1114-1119. [CrossRef] [Medline]
- Hoffman L, Benedetto E, Huang H, Grossman E, Kaluma D, Mann Z, et al. Augmenting mental health in primary care: a 1-year study of deploying smartphone apps in a multi-site primary care/behavioral health integration program. Front Psychiatry. 2019;10:94. [FREE Full text] [CrossRef] [Medline]
- Tong A, Flemming K, McInnes E, Oliver S, Craig J. Enhancing transparency in reporting the synthesis of qualitative research: ENTREQ. BMC Med Res Methodol. Nov 27, 2012;12(1):181. [FREE Full text] [CrossRef] [Medline]
- Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ. Jul 21, 2009;339(jul21 1):b2535. [FREE Full text] [CrossRef] [Medline]
- CASP critical appraisal checklists. Critical Appraisal Skills Programme. URL: https://casp-uk.net/casp-tools-checklists/ [accessed 2022-01-06]
- Carroll C, Booth A, Cooper K. A worked example of "best fit" framework synthesis: a systematic review of views concerning the taking of some potential chemopreventive agents. BMC Med Res Methodol. Mar 16, 2011;11(1):29. [FREE Full text] [CrossRef] [Medline]
- Carroll C, Booth A, Leaviss J, Rick J. "Best fit" framework synthesis: refining the method. BMC Med Res Methodol. Mar 13, 2013;13(1):37. [FREE Full text] [CrossRef] [Medline]
- Kolasa K, Kozinski G. How to value digital health interventions? a systematic literature review. Int J Environ Res Public Health. Mar 23, 2020;17(6):2119. [FREE Full text] [CrossRef] [Medline]
- Lagan S, Aquino P, Emerson MR, Fortuna K, Walker R, Torous J. Actionable health app evaluation: translating expert frameworks into objective metrics. NPJ Digit Med. 2020;3:100. [FREE Full text] [CrossRef] [Medline]
- Lagan S, Emerson MR, King D, Matwin S, Chan SR, Proctor S, et al. Mental health app evaluation: updating the American Psychiatric Association's framework through a stakeholder-engaged workshop. Psychiatr Serv. Sep 01, 2021;72(9):1095-1098. [CrossRef] [Medline]
- Hong QN, Pluye P, Bujold M, Wassef M. Convergent and sequential synthesis designs: implications for conducting and reporting systematic reviews of qualitative and quantitative evidence. Syst Rev. Mar 23, 2017;6(1):61. [FREE Full text] [CrossRef] [Medline]
- Beel J, Gipp B, Langer S, Genzmehr M. Docear: an academic literature suite for searching, organizing and creating academic literature. In: Proceedings of the 11th annual international ACM/IEEE joint conference on Digital libraries. 2011. Presented at: JCDL '11; June 13-17, 2011:565-566; Ottawa, ON. URL: https://dl.acm.org/doi/10.1145/1998076.1998188 [CrossRef]
- Forma F, Chiu K, Shafrin J, Boskovic DH, Veeranki SP. Are caregivers ready for digital? caregiver preferences for health technology tools to monitor medication adherence among patients with serious mental illness. Digit Health. 2022;8:20552076221084472. [FREE Full text] [CrossRef] [Medline]
- Afra P, Bruggers CS, Sweney M, Fagatele L, Alavi F, Greenwald M, et al. Mobile software as a medical device (SaMD) for the treatment of epilepsy: development of digital therapeutics comprising behavioral and music-based interventions for neurological disorders. Front Hum Neurosci. 2018;12:171. [FREE Full text] [CrossRef] [Medline]
- Beard C, Silverman AL, Forgeard M, Wilmer MT, Torous J, Björgvinsson T. Smartphone, social media, and mental health app use in an acute transdiagnostic psychiatric sample. JMIR Mhealth Uhealth. Jun 07, 2019;7(6):e13364. [FREE Full text] [CrossRef] [Medline]
- Borghouts J, Neary M, Palomares K, de Leon C, Schueller SM, Schneider M, et al. Understanding the potential of mental health apps to address mental health needs of the deaf and hard of hearing community: mixed methods study. JMIR Hum Factors. Apr 11, 2022;9(2):e35641. [FREE Full text] [CrossRef] [Medline]
- Boster JB, McCarthy JW. Designing augmentative and alternative communication applications: the results of focus groups with speech-language pathologists and parents of children with autism spectrum disorder. Disabil Rehabil Assist Technol. May 10, 2018;13(4):353-365. [CrossRef] [Medline]
- Buck B, Chander A, Monroe-DeVita M, Cheng SC, Stiles B, Ben-Zeev D. Mobile health for caregivers of young adults with early psychosis: a survey study examining user preferences. Psychiatr Serv. Aug 01, 2021;72(8):955-959. [FREE Full text] [CrossRef] [Medline]
- Buck B, Chander A, Tauscher J, Nguyen T, Monroe-DeVita M, Ben-Zeev D. mHealth for young adults with early psychosis: user preferences and their relationship to attitudes about treatment-seeking. J Technol Behav Sci. 2021;6(4):667-676. [CrossRef] [Medline]
- Carpenter-Song E, Noel VA, Acquilano SC, Drake RE. Real-world technology use among people with mental illnesses: qualitative study. JMIR Ment Health. Nov 23, 2018;5(4):e10652. [FREE Full text] [CrossRef] [Medline]
- Casarez RL, Barlow E, Iyengar SM, Soares JC, Meyer TD. Understanding the role of m-health to improve well-being in spouses of patients with bipolar disorder. J Affect Disord. May 01, 2019;250:391-396. [CrossRef] [Medline]
- Connolly SL, Miller CJ, Koenig CJ, Zamora KA, Wright PB, Stanley RL, et al. Veterans' attitudes toward smartphone app use for mental health care: qualitative study of rurality and age differences. JMIR Mhealth Uhealth. Aug 22, 2018;6(8):e10748. [FREE Full text] [CrossRef] [Medline]
- Cummings JR, Gaydos LM, Mensa-Kwao A, Song M, Blake SC. Perspectives on caregiver-focused mHealth technologies to improve mental health treatment for low-income youth with ADHD. J Technol Behav Sci. Mar 9, 2019;4(1):6-16. [FREE Full text] [CrossRef] [Medline]
- Dinkel D, Harsh Caspari J, Fok L, Notice M, Johnson DJ, Watanabe-Galloway S, et al. A qualitative exploration of the feasibility of incorporating depression apps into integrated primary care clinics. Transl Behav Med. Sep 15, 2021;11(9):1708-1716. [CrossRef] [Medline]
- Huberty J, Bhuiyan N, Neher T, Joeman L, Mesa R, Larkey L. Leveraging a consumer-based product to develop a cancer-specific mobile meditation app: prototype development study. JMIR Form Res. Jan 14, 2022;6(1):e32458. [FREE Full text] [CrossRef] [Medline]
- Kern A, Hong V, Song J, Lipson SK, Eisenberg D. Mental health apps in a college setting: openness, usage, and attitudes. Mhealth. Jun 2018;4:20. [FREE Full text] [CrossRef] [Medline]
- Knapp AA, Cohen K, Nicholas J, Mohr DC, Carlo AD, Skerl JJ, et al. Integration of digital tools into community mental health care settings that serve young people: focus group study. JMIR Ment Health. Aug 19, 2021;8(8):e27379. [FREE Full text] [CrossRef] [Medline]
- Kornfield R, Meyerhoff J, Studd H, Bhattacharjee A, Williams JJ, Reddy MC, et al. Meeting users where they are: user-centered design of an automated text messaging tool to support the mental health of young adults. In: Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems. 2022. Presented at: CHI '22; April 29-May 5 2022:1-6; New Orleans, LA. URL: https://dl.acm.org/doi/abs/10.1145/3491102.3502046
- Lipschitz J, Miller CJ, Hogan TP, Burdick KE, Lippin-Foster R, Simon SR, et al. Adoption of mobile apps for depression and anxiety: cross-sectional survey study on patient interest and barriers to engagement. JMIR Ment Health. Jan 25, 2019;6(1):e11334. [FREE Full text] [CrossRef] [Medline]
- Mata-Greve F, Johnson M, Pullmann MD, Friedman EC, Griffith Fillipo I, Comtois KA, et al. Mental health and the perceived usability of digital mental health tools among essential workers and people unemployed due to COVID-19: cross-sectional survey study. JMIR Ment Health. Aug 05, 2021;8(8):e28360. [FREE Full text] [CrossRef] [Medline]
- Melcher J, Camacho E, Lagan S, Torous J. College student engagement with mental health apps: analysis of barriers to sustained use. J Am Coll Health. Oct 13, 2022;70(6):1819-1825. [CrossRef] [Medline]
- Schueller SM, Neary M, O'Loughlin K, Adkins EC. Discovery of and interest in health apps among those with mental health needs: survey and focus group study. J Med Internet Res. Jun 11, 2018;20(6):e10141. [FREE Full text] [CrossRef] [Medline]
- Schueller SM, Neary M, Lai J, Epstein DA. Understanding people's use of and perspectives on mood-tracking apps: interview study. JMIR Ment Health. Aug 11, 2021;8(8):e29368. [FREE Full text] [CrossRef] [Medline]
- Stiles-Shields C, Montague E, Lattie EG, Kwasny MJ, Mohr DC. What might get in the way: barriers to the use of apps for depression. Digit Health. Jun 08, 2017;3:2055207617713827. [FREE Full text] [CrossRef] [Medline]
- Storm M, Venegas M, Gocinski A, Myers A, Brooks J, Fortuna KL. Stakeholders' perspectives on partnering to inform the software development lifecycle of smartphone applications for people with serious mental illness: enhancing the software development lifecycle through stakeholder engagement. In: Proceedings of the 2021 IEEE Global Humanitarian Technology Conference. 2021. Presented at: GHTC '21; October 19-23, 2021:195-199; Seattle, WA. URL: https://ieeexplore.ieee.org/document/9612444 [CrossRef]
- Torous J, Wisniewski H, Liu G, Keshavan M. Mental health mobile phone app usage, concerns, and benefits among psychiatric outpatients: comparative survey study. JMIR Ment Health. Nov 16, 2018;5(4):e11715. [FREE Full text] [CrossRef] [Medline]
- Zhou L, Parmanto B. User preferences for privacy protection methods in mobile health apps: a mixed-methods study. Int J Telerehabil. Dec 08, 2020;12(2):13-26. [FREE Full text] [CrossRef] [Medline]
- Baumel A, Muench F, Edan S, Kane JM. Objective user engagement with mental health apps: systematic search and panel-based usage analysis. J Med Internet Res. Sep 25, 2019;21(9):e14567. [FREE Full text] [CrossRef] [Medline]
- Gan DZ, McGillivray L, Han J, Christensen H, Torok M. Effect of engagement with digital interventions on mental health outcomes: a systematic review and meta-analysis. Front Digit Health. 2021;3:764079. [FREE Full text] [CrossRef] [Medline]
- Mobile fact sheet. Pew Research Center. 2021. URL: https://www.pewresearch.org/internet/fact-sheet/mobile/ [accessed 2024-04-29]
- Roberts ET, Mehrotra A. Assessment of disparities in digital access among Medicare beneficiaries and implications for telemedicine. JAMA Intern Med. Oct 01, 2020;180(10):1386-1389. [FREE Full text] [CrossRef] [Medline]
- Powell AC, Torous JB, Firth J, Kaufman KR. Generating value with mental health apps. BJPsych Open. Feb 05, 2020;6(2):e16. [FREE Full text] [CrossRef] [Medline]
- O'Loughlin K, Neary M, Adkins EC, Schueller SM. Reviewing the data security and privacy policies of mobile apps for depression. Internet Interv. Mar 2019;15:110-115. [FREE Full text] [CrossRef] [Medline]
- Akbar S, Coiera E, Magrabi F. Safety concerns with consumer-facing mobile health applications and their consequences: a scoping review. J Am Med Inform Assoc. Feb 01, 2020;27(2):330-340. [FREE Full text] [CrossRef] [Medline]
- Curran GM, Bauer M, Mittman B, Pyne JM, Stetler C. Effectiveness-implementation hybrid designs: combining elements of clinical effectiveness and implementation research to enhance public health impact. Med Care. Mar 2012;50(3):217-226. [FREE Full text] [CrossRef] [Medline]
- Systematic reviews: CRD's guidance for undertaking reviews in health care. Centre for Reviews and Dissemination. 2008. URL: https://www.york.ac.uk/media/crd/Systematic_Reviews.pdf [accessed 2024-04-29]
- Gumas ED, Lewis C, Horstman C, Gunja MZ. Finger on the pulse: the state of primary care in the U.S. and nine other countries. The Commonwealth Fund. URL: https://www.commonwealthfund.org/publications/issue-briefs/2024/mar/finger-on-pulse-primary-care-us-nine-countries [accessed 2024-04-29]
- Huang EC, Pu C, Chou YJ, Huang N. Public trust in physicians-health care commodification as a possible deteriorating factor: cross-sectional analysis of 23 countries. Inquiry. Mar 05, 2018;55:46958018759174. [FREE Full text] [CrossRef] [Medline]
Abbreviations
| ADHD: attention-deficit/hyperactivity disorder |
| APA: American Psychiatric Association |
| ASD: autism spectrum disorder |
| BD: bipolar disorder |
| DMHT: digital mental health technology |
| FDA: Food and Drug Administration |
| HCP: health care provider |
| MDD: major depressive disorder |
| MH: mental health |
| PHI: personal health information |
| PRISMA: Preferred Reporting Items for Systematic Reviews and Meta-Analyses |
| SLR: systematic literature review |
| SPIDER: Sample, Phenomenon of Interest, Design, Evaluation, Research type |
Edited by J Torous; submitted 15.02.24; peer-reviewed by A Mathieu-Fritz, K Stawarz; comments to author 05.05.24; revised version received 20.06.24; accepted 21.06.24; published 30.08.24.
Copyright©Julianna Catania, Steph Beaver, Rakshitha S Kamath, Emma Worthington, Minyi Lu, Hema Gandhi, Heidi C Waters, Daniel C Malone. Originally published in JMIR Mental Health (https://mental.jmir.org), 30.08.2024.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Mental Health, is properly cited. The complete bibliographic information, a link to the original publication on https://mental.jmir.org/, as well as this copyright and license information must be included.