Review
Abstract
Background: Digital mental health interventions (DMHIs) to monitor and improve the health of people with psychosis or bipolar disorder show promise; however, user engagement is variable, and integrated clinical use is low.
Objective: This prospectively registered systematic review examined barriers and facilitators of clinician and patient engagement with DMHIs, to inform implementation within real-world settings.
Methods: A systematic search of 7 databases identified empirical studies reporting qualitative or quantitative data about factors affecting staff or patient engagement with DMHIs aiming to monitor or improve the mental or physical health of people with psychosis or bipolar disorder. The Consolidated Framework for Implementation Research was used to synthesize data on barriers and facilitators, following a best-fit framework synthesis approach.
Results: The review included 175 papers (150 studies; 11,446 participants) describing randomized controlled trials; surveys; qualitative interviews; and usability, cohort, and case studies. Samples included people with schizophrenia spectrum psychosis (98/150, 65.3% of studies), bipolar disorder (62/150, 41.3% of studies), and clinicians (26/150, 17.3% of studies). Key facilitators were a strong recognition of DMHIs’ relative advantages, a clear link between intervention focus and specific patient needs, a simple, low-effort digital interface, human-supported delivery, and device provision where needed. Although staff thought patients would lose, damage, or sell devices, reviewed studies found only 11% device loss. Barriers included intervention complexity, perceived risks, user motivation, discomfort with self-reflection, digital poverty, symptoms of psychosis, poor compatibility with existing clinical workflows, staff and patient fears that DMHIs would replace traditional face-to-face care, infrastructure limitations, and limited financial support for delivery.
Conclusions: Identified barriers and facilitators highlight key considerations for DMHI development and implementation. As to broader implications, sustainable business models are needed to ensure that evidence-based DMHIs are maintained and deployed.
Trial Registration: PROSPERO CRD42021282871; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=282871
doi:10.2196/65246
Keywords
Introduction
Background
The costs and burdens of schizophrenia spectrum psychosis and bipolar disorder are huge and often neglected in research [,]. Digital mental health interventions (DMHIs) can help people with psychosis or bipolar disorder monitor, manage, and improve their symptoms and health [-]. Although there is promising evidence of feasibility, acceptability, and efficacy [-], and widespread expectations that DMHIs will become core to health care [], user uptake and engagement are highly variable [], and integrated clinical use is low []. The challenges of implementing DMHIs in health care are well established; attempts to implement well-evidenced DMHIs for common mental health problems in the United States and United Kingdom health services have frequently failed due to lack of patient and provider engagement and difficulty integrating DMHIs into clinical care []. For example, while randomized controlled trials (RCTs) of computerized cognitive behavior therapy (cCBT) for depression have reported relatively high engagement (47% of participants completing all cCBT sessions []), pragmatic trials delivering cCBT in real-world settings report much lower engagement (16%-18% of participants completing all sessions; most participants completing 1 session only) []. Implementing DMHIs for people with psychosis or bipolar disorder, within secondary care mental health services (eg, community mental health teams), presents additional challenges (eg, high staff caseloads and reactive care in response to increased risk or crisis), which have been examined in this review.
Implementation science acknowledges that intervention effectiveness cannot guarantee uptake in services; uptake depends largely on contextual barriers and facilitators []. Examining such contextual factors is therefore crucial to improving uptake in real-world settings, allowing tailoring of interventions to maximize engagement, and informing development and testing of implementation strategies. Barriers and facilitators of engagement with DMHIs for common mental health problems (eg, depression or anxiety) were examined relatively recently [], but the last systematic review of factors affecting DMHI implementation in samples with psychosis or bipolar disorder was 5 years ago []. This review provides a timely update on this rapidly developing literature by examining barriers and facilitators of patient and staff engagement with DMHIs for this group.
Using a best-fit framework synthesis approach [,], the Consolidated Framework for Implementation Research (CFIR) [] guided this analysis, enabling consideration of user engagement barriers and facilitators in the context of the complex, multilevel systems within and surrounding health care []. In supplementary analyses, we also examined safety reporting within reviewed studies to explore whether perceived harms may be additional barriers to DMHI uptake and to further explore recent findings on safety reporting quality [,].
Objective
In summary, this review examines evidence from qualitative and quantitative studies regarding the barriers and facilitators of patient and staff engagement with DMHIs for psychosis or bipolar disorder. By synthesizing this information using CFIR, this review presents important findings about how to maximize engagement with such DMHIs, and ideal conditions for DMHI implementation in secondary mental health services.
Methods
Design
This systematic review follows PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses; see for PRISMA checklist) guidance [], with eligibility criteria developed according to the PICOS (participants, interventions, comparators, outcomes, study design) framework. The review was registered with PROSPERO (main review: CRD42021282871; safety review: CRD42022306123).
Search Strategy and Selection Criteria
We conducted a PRISMA-compliant search of 7 databases using search terms relating to psychosis and bipolar disorder, DMHIs, and barriers, facilitators or implementation (see search terms and database list in ). Searches were restricted to peer-reviewed articles reporting on studies with human participants, published in English between January 2010 and October 2021. By restricting the search in this way, we were able to capture the period between the first studies of digital mental health tools being conducted (consistent with previous reviews [,]) and the COVID-19 pandemic. The search end date (1.5 years after the worldwide COVID-19 restrictions peaked []), allows for the lag from data collection to publication. The search retrieved very few papers reporting data gathered during or after the pandemic, suggesting this was a suitable watershed date. A future systematic review will examine studies from October 2021 onward.
Author EE combined, deduplicated, and screened the titles and abstracts of the search results; author AP independently screened a random selection (10%). Full-text of the remaining articles were independently screened against PICOS inclusion and exclusion criteria by authors EE and AP. Studies included and excluded by each researcher were compared, disagreements were resolved by consensus, and final reasons for all exclusions were recorded. Full inclusion and exclusion criteria are provided in . In summary, the review included all published studies reporting qualitative or quantitative data on hypothetical or actual barriers or facilitators of patient or staff engagement with a DMHI aiming to monitor or improve the mental or physical health of adults with a bipolar or schizophrenia spectrum diagnosis using a digital method (eg, smartphone app, SMS text messaging, website, or wearable).
Data Extraction
A data extraction form was used to extract relevant data: study metadata, design, DMHI details, and sample characteristics. Quantitative findings relating to barriers and facilitators of user engagement were extracted and imported into NVivo software (Lumivero) [], as were the full results and discussion sections of qualitative studies. A second researcher checked all the extracted data. Study quality was assessed using the Mixed Methods Appraisal Tool []. In addition, from the main paper text and supplements of studies that tested the actual use of a DMHI, we extracted data on how adverse events (AEs) were monitored, classified (relatedness or severity), analyzed, reported, and discussed, and whether reporting followed the CONSORT (Consolidated Standards of Reporting Trials) harms checklist []. Given that the reporting of formal AE data is typically poor for DMHIs [,], we also extracted information about incidents that might be classifiable as AEs but were not described as such by the authors.
Data Analysis
Study characteristics, quality checks, and AE information were tabulated and summarized descriptively. A best-fit framework synthesis approach [,] was used to synthesize qualitative and quantitative data from all included studies to address the following research question: What are the barriers and facilitators of service user or staff engagement with DMHIs for people with psychosis or bipolar disorder? Generally, within a best-fit framework approach, relevant published frameworks, models, or theories are systematically identified, and key elements of these are integrated into an a priori coding framework for use in the review []. Primary studies are then deductively coded into the a priori coding framework, with data not fitting within the framework coded inductively. In this review, we were able to use a single existing implementation framework, the CFIR, as the a priori coding framework [,], followed by inductive coding. CFIR was developed using a comprehensive systematic review of the implementation science literature [,]. It is a meta-theoretical framework incorporating constructs from a wide range of existing implementation theories into a single comprehensive framework and is used by researchers and practitioners to identify barriers and facilitators of successful implementation and tailor their strategies accordingly. Thus, it is an appropriate choice of framework to guide the systematic evaluation of potential barriers and facilitators of user engagement with DMHIs in this review. CFIR has 5 major domains as provided in 
Textbox 1.
- Innovation characteristics: features of the innovation or intervention being implemented, including its complexity, adaptability, design, and evidence base. For example, a smartphone app for monitoring symptoms of psychosis.
- Outer setting: the external context in which the organization is situated, including economic, political, legal, and social factors that may influence implementation. For example, a particular country or state.
- Inner setting: features of the organization in which the intervention is being implemented, including its culture, structure, resources, and readiness for change. For example, a secondary care mental health service.
- Individuals: characteristics of individuals involved in the implementation process, such as their knowledge, beliefs, and motivation. For example, patients with psychosis or bipolar disorder, and mental health staff.
- Implementation process: the implementation process itself, including planning, engaging, executing, and reflecting.
Results
Search Results
Our search retrieved 7617 records after deduplication (
Figure 1); 503 underwent full-text review, of which 175 were included based on inclusion and exclusion criteria. A list of the excluded studies and their reasons for exclusion are provided in Table S1 in . Studies that did not contain data on barriers or facilitators of DMHIs were excluded (psychosis samples n=36; bipolar samples n=57). However, to supplement findings regarding the potential harms of DMHIs, we examined AE reporting in the 36 studies of psychosis samples.
Description of Included Studies
We included 175 papers reporting 150 studies (11,446 participants). Tables S2-S5 in outline the study design, sample characteristics, and quality checks. Most DMHIs focused on general symptom management, relapse prevention, or similar. Some focused on a specific symptom (eg, cognitive difficulties and paranoia), and a minority on medication adherence, lifestyle, physical activity, or smoking cessation. Common technology types were mobile apps (77/150, 51.3% studies), websites (42/150, 28% studies), SMS text messaging (13/150, 8.7% studies), and wearables (15/150, 10% studies). Slightly more DMHIs were blended (ie, delivered with human support, 49/150, 32.7% studies) or stand-alone (24/150, 16% studies) than minimally blended (23/150, 15.3% studies) or included setup or technical input only (20/150, 13.3% studies). The median DMHI use was 8 (IQR 4-19) weeks. Study types included single-arm interventional studies (47/150, 31.3%), RCTs (25/150, 16.7%), qualitative studies (20/150, 13.3%), cross-sectional surveys (14/150, 9.3%), observational studies (7/150, 4.7%), intervention development studies (13/150, 8.7%), usability testing studies (11/150, 7.3%), and case studies (7/150, 4.7%). Most studies examined actual DMHI use (106/150, 70.7%), though some were hypothetical (ie, asking participants about potential DMHI use, 44/150, 29.3%), and some tested in-session usability rather than use in daily life (9/150, 6%). Samples included people with schizophrenia spectrum disorders (98/150, 65.3% studies), bipolar disorder (62/150, 41.3% studies), and clinicians (26/150, 17.3% studies). Participants were recruited from a range of settings, including inpatient services (21/150, 14% studies), community services (115/150, 76.7% studies), and online adverts (16/150, 10.7% studies). The Mixed Methods Appraisal Tool subsections (scored out of 5; Table S4 in ) indicated the moderate quality of qualitative (mean 4.1, SD 1.57) and quantitative (mean 4.0, SD 1.0) study elements, and lower mixed methods quality (mean 3.1, SD 1.9). The CONSORT harms checklist showed AE reporting to be mostly poor or missing (Table S5 in ); thus, it is difficult to draw informative conclusions regarding the harms of DMHIs.
Findings Not Fitting Within CFIR Framework
Although most relevant findings were coded into CFIR, a few could not be coded: patterns of DMHI use over time, associations between demographic variables and DMHI engagement, and safety reporting findings.
Patterns of Use
Sixteen studies reported reduced DMHI use over time in this study, 7 reported stable use and none reported an increase.
Demographics
Demographic findings are outlined in the Table S6 in . Only 28% (11/39) studies examining age found significant effects: 6 found higher engagement in older people, 4 in younger people, and 1 found mixed effects. Conversely, all staff participants in qualitative studies hypothesized higher DMHI engagement among younger people [-]. Only 28% (7/25) studies examining gender or sex reported significant effects; most (n=5) found higher DMHI engagement among women. Only 20% (4/20) studies examining education reported significant effects; all 4 found higher education associated with higher engagement. Only 20% (2/11) studies reported significant effects of race, ethnicity, or immigration: Ben-Zeev et al [] reported significantly higher engagement among White participants than African American participants (and among White participants than Hispanic participants for some, but not all, engagement indices), while Bonet et al [] reported significantly higher agreement to use a DMHI among non-native participants than native Spanish participants. Of studies examining the effects of IQ, 67% (2/3) showed significant effects, with higher IQ associated with higher DMHI engagement [,]. Reading level (0/1), employment (0/5), and marital status (0/2) showed no significant effects.
Safety Reporting
describes safety information extracted from 168 papers (139 studies) reporting actual DMHI use. Only 86 papers (76 studies) reported AE-related content; 23 reported AE frequency or percentage (median number of AEs reported per study 2, IQR 0.8-39.2). AEs included hospitalizations, psychological distress, suicidal ideation, physical health-related events, and death. Relatedness of AEs to the intervention was only reported in 11 papers, 8 of which deemed some events related.
CFIR Coding
Tables 1-5 provide detailed findings regarding the coding of reviewed studies into the 5 domains of the CFIR framework: Innovation (Table 1), Outer setting (Table 2), Inner setting (Table 3), Individual characteristics (Table 4), and Implementation process (Table 5). Findings were most frequently related to Innovation (Table 1; n=149 papers) and Individual (Table 4; n=137) domains of CFIR, with aspects of the Outer setting (Table 2; n=9), Inner setting (Table 3; n=20), and Implementation process (Table 5; n=5) domains rarely described. 
Figure 2 shows the number of papers coded into each CFIR domain per year of publication. Studies published between 2010 and 2015 were coded almost exclusively against the Innovation or Individual CFIR domains. Occasional studies were then coded against the Inner setting domain from 2016 onward, and against the Inner setting, Outer setting, and Implementation process domains from 2018 onward.
Prominent subcodes within the Innovation domain were Relative advantage (99/175, 56.6% papers), Complexity (56/175, 32% papers), and Design (128/175, 73.1% papers). As many papers commented on DMHI design, details of inductive subcodes are provided separately in Table 6. These included design features to increase DMHI engagement, customization, repetition vs variety, type of device used, user interface design, passive sensing, data security and privacy, and technical difficulties. Need (48/175, 27.4% papers), Capability (88/175, 50.3% papers), Opportunity (48/175, 27.4% papers), and Motivation (73/175, 41.7% papers) from the Individuals domain were all widely used, although almost exclusively in relation to innovation recipients rather than other roles. Details of how prominent findings from the CFIR-coded data related to the 5 crosscutting themes, are provided in the Crosscutting Themes section.
| CFIR construct | Findings in relation to this construct | Papers, n (%) | Data type | Study typea | Sample |
| Qualitative; quantitative; mixed | Hypothetical; usability; actual | Patient; staff; both | |||
| Source | Participants hypothesized that clear endorsement by a known institution (eg, health service or university) would increase engagement. One study [] noted that online information from pharmaceutical companies was less trusted than other sources. Some preferred endorsement by individual staff because “organizations have hidden agendas” []. Perceived credibility of teams developing or testing the DMHIb impacted engagement [-]. Teams that included people with lived experience of psychosis or bipolar disorder were especially credible. However, one study [] highlighted that co-design is not a panacea; despite being co-designed, the study innovation (MindFrame) “was neither applicable nor appealing to all.” | 14 (8) | 11; 0; 3 | 7; 1; 8 | 10; 0; 4 |
| Evidence base | A patient perspective piece [] suggested that health care professionals discussing the evidence base for specific DMHIs with patients would likely increase engagement. Two surveys [,] collected mixed methods data on whether information on the evidence base of bipolar apps influenced users’ app choice. Qualitatively, “participants seldom reported evidence of efficacy or scientific quality to justify app choice,” but in quantitative responses, 93% [] and 90% of participants [] rated scientific quality as important. Health care professionals [] cite a lack of knowledge regarding which apps were evidence based as the foremost barrier to discussing apps with patients with bipolar disorder. | 4 (2) | 1; 1; 2 | 4; 0; 0 | 3; 1; 0 |
| Relative advantage | Connection with peers and normalization: qualitative data highlighted that DMHIs can facilitate peer connection and normalize psychosis or bipolar disorder experiences. Peer contact, both as a formal part of the intervention (peer facilitators or video resources) or via a bespoke or mainstream social media platform, provided “the opportunity to learn from others managing the same condition, reducing self-stigma through normalization of shared experiences” []. Peer support appeared to influence participants’ actual level of engagement with DMHIs [], or participants hypothesized that it would [,,]. Participants valued encouragement and accountability from others on the platform [-] and the opportunity to socialize with peers [-]. Even DMHIs without specific peer content were normalizing as their presence implied others with similar needs [-], and they used a similar format to everyday things (“...everyone uses an app these days... It’s normal now,” [,]). However, some were anxious that others would ask what the device or app was for [,,,], or preferred apps for “the general consumer, so I do not feel stigmatized as a patient for using it” []. | 43 (25) | 31; 2; 10 | 18; 0; 29 | 31; 3; 9 |
| Relative advantage | Availability and autonomy: reviewed studies highlighted DMHIs’ availability (anytime, anywhere) as a key advantage and reported that DMHIs increased participants’ autonomy. Availability was particularly beneficial for people constrained by working hours [], childcare [], geography [,,], reversed sleep cycles [], or during recovery from psychosis [,,]. Patients contrasted DMHI availability with that of their care team [] and valued DMHIs’ ecologic validity [] and being able to visit, or revisit, content at their own pace [-]. Quantitative data [] indicated similar app engagement off- and on-hours; ie, DMHIs are used outside clinic hours when available. Choosing when and where to use DMHIs enhanced feelings of autonomy, control, or empowerment [,], encouraging individuals to play an active role in managing their mental health (“They actually feel like they have part ownership in the process,” []) and decreasing reliance on health professionals (“it gives you a bit of freedom to say ‘hold on a second, I don’t have to wait for my CPNc’” []). Accessing information without a clinician’s direct involvement was particularly empowering, with patients frequently describing DMHIs as a friend or therapist. Patients and staff noted that internet-based mental health information ranges in accuracy, so providing “curated knowledge from a bunch of professionals rather than just what Google tells you” [] was a significant advantage. Participants from 10 studies described a DMHI in human terms, including as a “CPN in your pocket” [], “therapist in your phone,” “personal smoking trainer” [], “life-coach” [], “friend” [,,,,], “buddy” [], “close companion” [] or “constant companion” []. Staff [,,] and patients [,,,,,,,,] across 12 studies highlighted a DMHI’s anonymity can be a substantial advantage. For example, some DMHIs provided an opportunity to vent without fears of a subsequent medication increase [], to “scream on the keyboard” at any time [], or to disclose sensitive or potentially disturbing information in a nonjudgmental, anonymous place instead of directly to a person [,,,,,]. A minority of participants viewed some automated DMHI functions as potentially detracting from autonomy; for example, automatic transfer of data from the app to services [,,], and medication reminders []. Availability and autonomy contributed to perceptions of DMHIs as widening the reach of services. Symptom-monitoring or cognitive behavior therapy apps helped users feel safe with a more hands-off clinical approach (“it did... make me feel safe... I knew he was keeping an eye on those graphs,” []) while allowing prompt intervention if needed. More trivially, technology provided a convenient way to keep in touch with clinicians (eg, SMS text messaging, email, and instant messaging). | 65 (37) | 42; 8; 15 | 29; 2; 38 | 43; 7 15 |
| Relative advantage | Self-awareness and memory: DMHIs’ availability helped patients notice patterns in their experiences (“to see what’s making me happy, what’s doing my head in” []), remember those experiences, and talk to others about them. Participants said DMHIs gave a useful structure for therapeutic conversations [,-], promoting shared understanding, strengthening therapeutic relationships, helping patients feel “heard” [] or “connected” [,,], and facilitating help-seeking. One study highlighted a self-monitoring app’s value in translating individuals’ subjective experiences into an objective form, enabling others to access their inner experiences and take them more seriously (“If you were to answer the questions and go to the doctor and say, ‘look... you can see clearly there’s a change, and these are my experiences,’ that would be substantial evidence for the doctor to then sit up and take note” []). In some cases, this process was formalized as part of an Ecological Momentary Intervention [,]. One study drew a specific link between self-awareness and user engagement (“EMAd self-monitoring and EMA-derived feedback were seen as helpful for improving awareness, highlighting a willingness to engage with more intense monitoring for clinical purposes” []. As well as sharing symptom data with clinicians, patients often shared it with family, facilitating communication, understanding, and support. However, 2 surveys reported that only a minority of patients [,] and staff [] thought symptom information should be shared with family, and a qualitative study cautioned that users should always retain control over who had data access []. | 61 (35) | 34; 9; 18 | 17; 0; 45 | 45; 1; 15 |
| Relative advantage | Should enhance but not replace human care: qualitative studies outlined ways that DMHIs did not present a relative advantage over human care. Neither staff [] nor patients [] liked “fake empathy” offered by a machine, considering it degrading. Many saw DMHIs as a cost-cutting exercise [,,,,], which might feel “dismissive” to patients, “as though they are not worthy of a clinician or therapist” [], leading them to feel “fobbed-off” []. Some worried that using DMHIs blended within face-to-face sessions might render clinicians less “present” in the interaction [,,] because they are “increasingly focused on a computer or on data rather than on the suffering human seated in front of them” []. Patients considered DMHIs a “poor substitute for seeing a person that knows you” [], lacking personal contact [,], opportunities for “being listened to” [], tone of voice, body language [] and emotional reassurance [], and unable to address everyday practical problems (eg, housing difficulties; []. Staff worried that DMHIs lacked nonverbal communication [], context [,] and personalization [,], left patients “alone to deal with any issues that surface” [], removed a key source of socialization and warmth, potentially reinforcing social avoidance [], and that DMHI-generated data would be difficult to interpret without staff members’ clinical experience []. | 16 (9) | 13; 0; 3 | 11; 0; 7 | 9; 4; 3 |
| Adaptability | No studies provided information on whether the ability to adapt a DMHI to a specific service affected engagement. Personalization of DMHIs for individual users is coded under Innovation design. | 0 | —e | — | — |
| Trialability | An expert consensus study [] and a patient perspective piece [] highlighted that the availability of a free trial version of a DMHI would increase the likelihood that staff and patients would use it. | 2 (1) | 1; 1; 0 | 2; 0; 0 | 1; 1; 0 |
| Complexity | Technology-related complexity, stemming from poor user experience design, increased user burden, and “mental fatigue” [], prompting disengagement [,,,]. Lengthy login processes reduced engagement [,,,,], as did difficult-to-navigate websites or apps (“I wanted to go and do it, but then I just couldn’t navigate it. And I was just like, ‘Oh, I can’t be bothered’,” []. Qualitative and quantitative data found shallow website and app structures, and pages with <200 words, most usable by patients. Unwieldy, multistep systems (eg, SMS text messaging–based symptom monitoring) were perceived as “awkward, laborious, and burdensome” [] and associated with significantly lower engagement than a more streamlined native smartphone app []. One study noted that tailoring DMHI content increased complexity []. Patients suggested that training and support may mitigate the effects of technology-related complexity []. Individual capability (eg, cognitive ability, digital literacy; [,] also moderated the extent to which complexity hindered engagement and the amount of training that was needed to mitigate this effect [,]. Some studies noted that practical complexity, such as remembering to carry, charge, or use an additional device [,-], hindered engagement. Although most issues of complexity were related to the technology, rather than the intervention more broadly, staff in several studies [,,,,] found the information they received from symptom-monitoring apps complex to interpret, making it difficult to know how to respond clinically. Patients expressed concerns that some apps and websites had overwhelming amounts of information or features [,,]. A patient perspective piece highlighting the fallacy of the “killer app,” preferring apps with a clear aim that do not try to do too much at once []. | 55 (31) | 31; 10; 14 | 16; 7; 37 | 41; 2; 12 |
| Design | As most reviewed studies commented on aspects of DMHI design, details are provided separately in Table 6. | 128 (73) | 56; 43; 29 | 38; 5; 93 | 101; 7; 20 |
| Cost | Most participants (85%) in a large international survey of people with bipolar disorder (n=919) preferred free or low-cost apps [] and qualitative survey data suggested DMHIs for bipolar should be “free or inexpensive so accessible to all or many” []. Five studies [-] listed financial costs associated with delivering a particular DMHI (eg, web hosting, licensing fees, texting fees, technical support, and staff time to deliver) but did not examine the effect on engagement. | 9 (5) | 1; 6; 2 | 4; 0; 6 | 7; 0; 2 |
aSome papers fall into more than one category, so categories may add to more than the overall total N.
bDMHI: digital mental health intervention.
cCPN: community psychiatric nurse.
dEMA: ecological momentary assessment.
eNot applicable.
| CFIR construct | Findings in relation to this construct | Papers, n (%) | Data type | Study typea | Sample |
| Qualitative; quantitative; mixed | Hypothetical; usability; actual | Patient; staff; both | |||
| Critical incidents | Patients and clinicians noted their use of technology had increased since the COVID-19 pandemic []. | 1 (1) | 1; 0; 0 | 1; 0; 0 | 0; 0; 1 |
| Local attitudes | —b | 0 | — | — | — |
| Local conditions | US clinicians [] said local technological conditions were a major barrier to DMHIc implementation. A lack of regional infrastructure (mobile phone signal) prevented app use by clinicians and patients. Many participants had government-sponsored smartphones, incapable of running the study app. Technological infrastructure is not a universal barrier though a survey with EISd clinicians across 30 US states indicated that 93% had high-speed internet available [] and a study testing an app in a Canadian city indicated 100% mobile phone coverage []. | 3 (2) | 1; 1 1 | 2; 0; 3 | 1; 2; 0 |
| Partnerships | — | 0 | — | — | — |
| Policies and laws | Clinicians from Indian health services highlighted the need to understand regulations and abilities governing app-generated data before using apps with clients []; US clinicians in the same study did not raise such concerns. Study authors attributed the difference to recently introduced telehealth policies in India. Clinicians from Australia [] and the United Kingdom [] mental health services noted the lack of guidelines and policies as a barrier to DMHI engagement. | 3 (2) | 3; 0; 0 | 3; 0; 0 | 0; 1; 2 |
| Financing | Empirical results did not directly report on financing in relation to DMHI engagement or implementation, but study authors noted that a sustainable business model for maintaining evidence-based DMHIs is often absent. One study noted that costs were offset by the billable nature of the service []. | 3 (2) | 1; 0; 2 | 2; 0; 2 | 2; 0; 1 |
| External pressure | — | 0 | — | — | — |
aSome papers fall into more than one category, so categories may add to more than the overall total N.
bNot applicable.
cDMHI: digital mental health intervention.
dEIS: early intervention service
| CFIR construct | Findings in relation to this construct | Papers, n (%) | Data type | Study typea | Sample |
| Qualitative; quantitative; mixed | Hypothetical; usability; actual | Patient; staff; both | |||
| Structural characteristics | IT: poor IT infrastructure, including aging computers [], poor bandwidth [,], and no wireless connection in hospitals [] hindered engagement. Staff in one study expressed low confidence that their organization could implement DMHIsb due to its historical failures to successfully deliver past IT projects []. | 4 (2) | 3; 0; 1 | 2; 0; 2 | 1; 2; 1 |
| Structural Characteristics | Physical infrastructure: a lack of quiet space in the clinic hindered implementation []. | 1 (1) | 0; 0; 1 | 0; 0; 1 | 0; 0; 1 |
| Structural characteristics | Work infrastructure: work infrastructure impacted staff DMHI engagement in one study []. Local mental health teams had been relocated and merged shortly before the DMHI was introduced. Study authors hypothesized that staff hesitated to fully engage with the DMHI for fear that it would increase their workload further. | 1 (1) | 1; 0; 0 | 0; 0; 1 | 0; 0; 1 |
| Relational connections | —c | 0 | — | — | — |
| Communications | — | 0 | — | — | — |
| Culture | Human equality-centeredness: none | 0 | — | — | — |
| Culture | Recipient-centeredness: staff felt was it important to offer patients a choice to use a DMHI: “It’s also nice to think, to be able to give someone an option that you’re not forcing down their throat” []. Patients in this study appreciated this attitude, although the impact on DMHI engagement was not directly investigated. | 1 (1) | 1; 0; 0 | 0; 0; 1 | 0; 0; 1 |
| Culture | Deliverer-centeredness: none | 0 | — | — | — |
| Culture | Learning-centeredness: none | 0 | — | — | — |
| Tension for change | — | 0 | — | — | — |
| Compatibility | The lack of compatibility between DMHIs and existing workflows was a key barrier to engagement and implementation. Clinicians believed DMHIs would create extra work, a major barrier in overstretched services [,,,,]. Some DMHIs duplicated existing work [,]: “reporting symptoms via the app system, instead of saving time, added another workflow to manage.” Participants often questioned who should be responsible for managing and responding to real-time symptom data [,,,,], particularly out-of-hours [,,,,], as doing so did not fit well with their current ways of working. Several studies emphasized that compatibility with existing electronic medical record systems would be important [,,]. Clinicians in one study [] strongly suggested patients should bring symptom-monitoring data to appointments, rather than uploading data in real-time. Although arguably undermining a key advantage of DMHIs (real-time data), this suggestion highlights how challenging the staff find the idea of being presented with a “constant stream of information” [], and the perceived additional responsibility that comes with that information. | 17 (10) | 11; 3; 3 | 9; 0; 9 | 2; 5; 10 |
| Relative priority | — | 0 | — | — | — |
| Incentive systems | “Reimbursement by payers for time spent training patients and family members about DMHI and for time spent using or reviewing data from the DMHI” was a strong potential facilitator of HCPd engagement with DMHIs []. Similarly, the “difficulty having the DMHI approved by insurance” was considered a substantial barrier to implementation []. | 1 (1) | 0; 1; 0 | 1; 0; 0 | 0; 1; 0 |
| Mission alignment | Although staff appeared to consider some aspects of DMHIs aligned with their mission (see Relative advantage in Table 1), 5 studies highlighted ways that implementing DMHIs was not aligned with the inner setting’s central mission. Staff felt funding was “channeled into technological advancement when perhaps it would be better channeled into staffing and training” []. Given limited time, clinicians considered other on-mission tasks more important than using DMHIs; eg, emergent crises [], the client’s own priorities [,], and simply having “more pressing things to deal with” []. Staff in one study felt “a huge burden of responsibility toward protecting their clients from harm and take this responsibility incredibly seriously” []; they perceived some aspects of DMHIs to present additional risks, directly conflicting with this perceived mission. Other staff were cautious of DMHIs disrupting the trust they had built with clients (“I’m not prepared to do anything that could damage that relationship... you work so hard to get that trust,” []); hence, they would always prescreen DMHI content before showing it to a client. | 5 (3) | 4; 0; 1 | 3; 0; 2 | 0; 3; 2 |
| Available resources | Materials and equipment: in response to digital exclusion challenges, it has been suggested that health services could provide devices to facilitate DMHI access. None of the reviewed studies directly investigated the impact of this, but staff in one study “felt the NHS should not provide the required technology because of concerns that tablets and mobile phones may get lost, sold, or damaged” []. However, across 15 reviewed studies that reported device loss rates, only 112 (11.1%) of 1011 provided devices were lost, broken, stolen, or damaged. Staff [] and patients [] also mentioned the need for devices for staff themselves, to facilitate blended DMHIs; eg, for SMS text messaging–based DMHIs [], or as a backup if the data connection is lost during video-based aspects part of a blended DMHI []. | 20 (11) | 7; 6; 7 | 5; 0; 15 | 14; 2; 4 |
| Available resources | Funding: none | 0 | — | — | — |
| Available resources | Space: none | 0 | — | — | — |
| Access to knowledge and information | Providing staff with training in basic digital skills [,], how to evaluate DMHI credibility and quality [,], and how to use specific DMHIs [,,] was hypothesized to facilitate engagement. One participant said, “There’s no point in [a patient] being a whizz on that computer and smartphone and me not having a clue cos I wouldn’t be able to support adequately, there would be... ongoing training needs,” []. Even staff who were digitally confident felt they would need more training to assist others [] and lacked knowledge about which DMHIs are credible [,]. As well as clear training or instructions on how to support patients with specific DMHIs [,], staff also valued the opportunity to try the DMHI themselves as part of this training []. | 5 (3) | 2; 3; 0 | 4; 0; 2 | 0; 4; 1 |
aSome papers fall into more than one category, so categories may add to more than the overall total N.
bDMHI: digital mental health intervention.
cNot applicable.
dHCP: health care professional.
| CFIR construct | Findings in relation to this construct | Papers, n (%) | Data type | Study typea | Sample |
| Qualitative; quantitative; mixed | Hypothetical; usability; actual | Patient; staff; both | |||
| Need | Perceived benefit: users need to perceive a DMHI’sb benefit to try it (“People do not do things just because their doctors tell them to. They need to see the benefit of what they do,” [] and continue using it (“If people think an app is helping them, they will continue using it,” []). Clinicians noted that quickly finding benefits helped sustain use: “People tend to download the app and then it decreases in use if people don’t see a quick benefit” []. | 34 (19) | 16; 10; 8 | 10; 0; 25 | 24; 4; 6 |
| Need | Stage of illness: qualitative data highlighted that needs may differ depending on patients’ stage of illness or recovery and that DMHI content (eg, psychoeducation) and functions (eg, symptom tracking) are often best suited to the needs of people with recent onset illness or ongoing symptoms. Quantitative data implied effects on engagement: people with recent onset psychosis were more likely to use an app than those with established psychosis (037); those with longer psychosis duration rated digital monitoring as less important to managing their illness than more recently diagnosed people. Several studies reported DMHI discontinuations due to improved mental health [,,]. Some adapted their DMHI use to meet their own needs []. | 27 (15) | 14; 8; 5 | 7; 0; 21 | 21; 1; 5 |
| Capability | Physical capability: 4 studies mentioned that medication side effects (eg, fine motor tremors and blurry vision), interfered with participants’ ability to use a DMHI [,,,]. | 4 (2) | 2; 0; 2 | 1; 0; 3 | 3; 0; 1 |
| Capability | Psychological capability: Table S7 in outlines data from studies quantitatively examining the effects of specific symptoms, functioning, and diagnosis on DMHI engagement. Around a third of quantitative analyses found a significant effect of symptoms on engagement, with higher baseline positive, negative, or cognitive symptoms associated with lower DMHI engagement. Results for functioning and depression were more mixed. Quantitative studies did not examine the impact of relapse on DMHI engagement, but 11 qualitative studies noted decreased DMHI use during relapse due to thought disorder (“I wouldn’t have been able to make sense of the topics,” []), using a DMHI feeling “too confrontational” [], avoiding certain symptom-related questions [], difficulty staying awake during app training [], or lack of energy or motivation to engage [] when unwell. Nevertheless, relapsing participants were typically able to continue using a DMHI once symptom exacerbation had resolved [,], and one study noted that, for some, stopping the app might be a sign of imminent relapse []. A few participants avoided DMHIs due to interactions between their symptoms and technology, such as delusions or paranoia about technology [,,,], adverse effects of technology in mania [,], or social anxiety during DMHI social media use []. Some variables relating to psychological capability were examined in very few studies. Neither baseline biological rhythm disturbance [] nor baseline medication adherence predicted DMHI engagement [,,], although those who chose to use a DMHI were less likely to be using injectable medication []. Better premorbid adjustment was associated with higher DMHI engagement []. Cannabis use predicted lower DMHI engagement []. | 76 (43) | 26; 33; 17 | 16; 0; 61 | 64; 4; 8 |
| Capability | Digital literacy: lack of digital literacy was a barrier for some, and expert consensus participants considered self-efficacy beliefs about using devices as highly likely to promote DMHI engagement []. Some clinicians reported lacking the technological capability to deliver a DMHI [,,]. Three studies quantitatively examined the effects of patients’ digital literacy on DMHI engagement. Two reported no significant effect [,]; in one study participants with prior smartphone experience engaged significantly more in a digital than a nondigital intervention, whereas no such difference was observed for people without previous smartphone experience. | 33 (19) | 14; 10; 9 | 9; 3; 22 | 24; 6; 3 |
| Opportunity | Space: participants reported a lack of a suitable environment as a barrier. Busy homes, hospitals, public libraries, or homelessness presented challenges of noise, repeated interruptions, and lack of privacy. For blended DMHIs, the in-person location was sometimes a barrier [,,]. | 12 (7) | 7; 2; 3 | 1; 0; 11 | 9; 1; 2 |
| Opportunity | Time: lack of time due to work [,], physical illness [], preoccupation with delusions [], or other commitments was a barrier to engagement for some patient participants. Conversely, being able to fit the DMHI within their daily or weekly routine was a facilitator, and inpatient participants enjoyed using a DMHI as “something to do” []. | 22 (13) | 7; 10; 5 | 1; 0; 21 | 20; 0; 2 |
| Opportunity | Technology: lack of access to technology was a barrier. This included not owning a phone, tablet, or computer [,,,], or having a device with limited functionality [], insufficient storage [], insufficient data allowance [,], poor internet connection [], or frequently changed phone numbers [,]. Staff highlighted a key difficulty with using technology for health care: “phones are... a luxury item... if people aren’t working because of health needs... they are being denied healthcare because they can’t afford a phone” []. | 30 (17) | 15; 12; 3 | 16; 0; 15 | 19; 6; 5 |
| Motivation | General: some were motivated by financial or material factors (eg, vouchers or free study devices). Others described intrinsic motivations: commitment to participating [], wanting to provide accurate data [], agreement with DMHI tasks or goals [], less “controlled motivations for treatment” [], and interest in technology []. One study noted that people who tended to be more engaged with services were also more engaged with the DMHI []. | 12 (7) | 7; 3; 2 | 4; 1; 9 | 9; 2; 1 |
| Motivation | Support: across 45 qualitative studies, the following types of human support were mentioned: training in basic digital skills, managing technical difficulties, support navigating the DMHI, ongoing coaching to sustain use, discussion of psychoeducational materials, and interpreting or responding to changes in real-time symptoms data. Four randomized controlled trials quantitatively compared unsupported vs supported DMHI engagement. Two reported significantly higher engagement in a peer-supported group than in an unsupported group (P<.05 [,]; P=.005 []). One reported a significant interaction effect such that reviewing symptoms had a significant positive effect on app adherence only for younger women (P=.001), with no significant overall effect []. Finally, a pilot randomized controlled trial found moderate effect sizes in favor of the group allocated to support from a clinical helper (not powered to detect significant differences []). Regarding social support more generally, there were mixed quantitative results. One study reported lower social support among more actively engaged participants [], with a second reporting the opposite pattern []. | 48 (27) | 24; 13; 11 | 13; 1; 39 | 36; 4; 8 |
| Motivation | Fear of relapse (barrier): anticipated or actual anxiety about DMHI use was sometimes a barrier. Staff [] and patients [,] hypothesized that using a DMHI may increase awareness of symptoms, heightening users’ anxiety about deterioration, and potentially making symptoms worse. There was some evidence to indicate that this belief may hinder engagement. Quantitatively, users reporting higher fear of relapse at baseline showed lower engagement with a symptom-monitoring app over the following 6 months []. Elsewhere, one participant reported: “Being notified of all the changes sometimes made me anxious. It made me wonder if the illness was maybe about to get out of control” [] and a minority of participants in other studies reported increased rumination [,,]. This could be mitigated to some extent by support: “It really helped to have her sit there and talk things through with me... and just to have her to bring me back down, or bring me back into reality” []. Others were hesitant to (accurately) report symptoms via a DMHI for fear that it may lead to more restrictive treatment, such as hospital admission or increased medication [,,,]. | 26 (15) | 11; 5; 10 | 8; 0; 19 | 20; 3; 3 |
| Motivation | DMHI too confronting (barrier): some participants found DMHIs to be confronting: being reminded about their diagnosis or noticing how unwell they were, was disheartening and often prompted disengagement. A striking number of quotations related to this, for example, “My best days are when I typically forget that I have any kind of illness... to be very heavily reminded about the gravity of it all can be a bit unpleasant [];” “It started me comparing how I am now to how I used to be... and its not good. That’s why I canceled it. I found it too personal in the end [];” “I wasn’t ready to accept the illness... I wasn’t willing to change my life according to the program [].” In one study [], half the patients felt negatively influenced by frequent confrontations with their symptoms. In another study [] only a minority (4%) found the DMHI confronting and others reported that it helped them recognize their progress. One study quantitatively examined if recovery style predicted DMHI engagement, finding no significant effect []. | 11 (6) | 7; 3; 1 | 2; 0; 9 | 8; 0; 3 |
aSome papers fall into more than one category, so categories may add to more than the overall total N.
bDMHI: digital mental health intervention.
| CFIR construct | Findings in relation to this construct | Papers, n (%) | Data type | Study typea | Sample |
| Qualitative; quantitative; mixed | Hypothetical; usability; actual | Patient; staff; both | |||
| Teaming | —b | 0 | — | — | — |
| Assessing needs | Innovation deliverers: none | 0 | — | — | — |
| Assessing needs | Innovation recipients: a patient perspective piece highlighted the current lack of assessment of patient needs regarding DMHIc implementation: “It is important for clinicians to explore with their patients what kinds of problems they are facing living their day-to-day lives and whether there are apps that can assist them” []. | 1 (1) | 1; 0; 0 | 1; 0; 0 | 1; 0; 0 |
| Assessing context | — | 0 | — | — | — |
| Planning | — | 0 | — | — | — |
| Tailoring strategies | — | 0 | — | — | — |
| Engaging | Innovation deliverers: none | 0 | — | — | — |
| Engaging | Innovation recipients: there were mixed findings regarding whether a recommendation from a health professional increased DMHI engagement. Three out of 4 studies reported that such a recommendation would usually prompt engagement [,,], particularly in the context of a strong relationship between a patient and staff member: “I trust the early intervention team... so I would be fairly confident that it would be secure if they said so” []. Patients in another study were asked whether they would use an app recommended by a clinician and all said “no” [], though reasons for this were not outlined. | 4 (2) | 3; 0; 1 | 3; 0; 1 | 1; 0; 3 |
| Doing | — | 0 | — | — | — |
| Reflecting and evaluating | — | 0 | — | — | — |
| Adapting | — | 0 | — | — | — |
aSome papers fall into more than one category, so categories may add to more than the overall total N.
bNot applicable.
cDMHI: digital mental health intervention.
| CFIR construct | Findings in relation to this construct | Papers, n (%) | Data type | Study typea | Sample |
| Qualitative; quantitative; mixed | Hypothetical; usability; actual | Patient; staff; both | |||
| Digital reminder | Most participants found digital reminders (eg, push notifications or email) to engage with the DMHIb, or remember medication or appointments, helpful; some found them nagging [], irritating [], “pushy” [], or repetitive []. Ideal timing and frequency of alerts were often discussed, eg, personalizing alert timing and frequency [,,], allowing users to “snooze” alerts to receive a later reminder [], considering users’ specific needs and schedules (eg, afternoon alerts for those sleeping late due to medication; [,], and allowing self-initiated use [,]. Studies testing self-initiated use report 47% [], 52% [], 63% [], and 97% [] of app engagements were self-initiated rather than in response to app alerts. Two quantitative studies examined the impact of alerts on engagement: individuals randomized to receive weekly emails had significantly higher DMHI engagement than a comparison group []; 10-times daily reminders were significantly more inconvenient than daily monitoring []. | 57 (33) | 26; 14; 17 | 16; 0; 43 | 45; 2; 10 |
| Gamification | Most participants across 12 studies thought gamification would increase engagement though one reported mismatching qualitative (positive) and quantitative (neutral or negative) survey data on this, and another that focus groups barely mentioned it []. Three studies testing DMHIs with gamification elements [,,] reported impacts on enjoyment and engagement (eg, “I really like the badges idea... that really helped me to keep going,” []). | 13 (7) | 9; 1; 3 | 8; 0; 6 | 11; 1; 1 |
| Feedback from the DMHI | Qualitative data suggested DMHI feedback would increase engagement; eg, a symptom data summary, symptom graph, summary of passive data (eg, step count), a summary of DMHI engagement (eg, minutes used) or encouraging messages. Two studies quantitatively investigated the effects of feedback on engagement; one reported significantly higher Fitbit wear rates in those receiving feedback than a nonrandomized comparison group []; another found no significant between-group difference in symptom-monitoring app use []. | 29 (17) | 13; 7; 9 | 8; 0; 22 | 23; 0; 6 |
| Tailoring, customization, personalization | “Tailoring,” “customization,” or “personalization” of DMHIs was a facilitator, as mental health experiences vary hugely (“10 people with the same diagnosis as me will have 10 different ways of experiencing the illness,” [], and DMHI users have different symptoms, treatment goals, schedules, illness durations, and learning styles. Customizing aspects of a DMHI is advantageous, eg, alert schedules [,,], symptom questions [,,], self-management strategies [,], psychoeducation content [,,,,] and delivery method []. Personalizing more superficial elements (eg, colors, []), was occasionally mentioned but much less prominent. Indeed, a survey of people with bipolar disorder [] found 92% of participants rated “flexibility/customization” (DMHI frequency or what things are tracked) as important, 84% rated “tailored content” as important but only 33% rated “personalization” (changing interface or colors) as important. | 34 (19) | 19; 4; 11 | 12; 0; 23 | 26; 2; 6 |
| Repetitiveness and variety | DMHI repetitiveness was a barrier (18 studies) and greater variety in DMHI content was a potential facilitator (12 studies). Participants “wanted more tips and strategies... to minimize repetition and boredom, particularly toward the end of the intervention period when users had selected the same domain repeatedly and started receiving the same tips and strategies” []. “Periodic content updates or staggered unlocking of content based on the user’s evolving needs” would encourage repeated use [,]. A patient testing a cognitive behavior therapy–based app suggested adding different “levels” for different needs: “Level one... at the beginning of their recovery... starting to learn about the illness. Level 2 is in the middle... they’re getting more self-awareness or understanding of the illness and how to cope with certain symptomatic episodes. And then you have the advanced level” []. There was some coding overlap with personalization [,,], as a wider variety of content would allow people to select content that is more relevant to their needs. | 27 (15) | 13; 5; 9 | 7; 0; 21 | 22; 0; 5 |
| Type of device used | Participants preferred smartphone apps more than computer-based [,,,,] or SMS text messaging–based systems [,], reporting that smartphone apps were more convenient and easier to integrate into daily life. SMS text messaging contact was preferred to emails by most (87%, 74%, and 82%, respectively [-]). A comparison of smartphone app and wearable use for >6 weeks in people with bipolar disorder [] reported a significant time-by-device interaction: participants used the wearable more at study start and the app more at the study end. Subjective reports indicated more (72%) participants would use a wearable for a year than the app (47%) and that the wearable required less effort and was slightly easier to use. Most participants (80%) in a small survey of people with bipolar disorder said they would use a wearable device but would not want stationary installations of sensors in their homes []. Five studies mentioned the importance of DMHIs being compatible across Android and iOS devices [,,,,]. | 28 (16) | 13; 9; 6 | 7; 1; 23 | 22; 2; 4 |
| User interface design | Clear language was a facilitator of patient engagement and complex language was a barrier. Clinicians recommended that DMHIs should be usable by people with low literacy [] and in languages other than English []. The “tone” of messages or information presented in DMHIs was important to users. The ideal tone was positive, relevant, validating, motivating, and encouraging without being patronizing, glib, or “mind-numbingly positive.” Given that users were people with experiences of mental health problems, some felt that an overly positive tone hindered discussion of difficult topics []. Users preferred visually appealing DMHIs (eg, attractive colors, modern-looking interface, and suitably sized fonts), which presented information in a variety of formats (video, text, audio, graphics) and were engaging and interactive. If DMHIs include personas or characters, these should be diverse in ethnicity and gender [,]. Graphs, while popular, were not always easy for patients to understand []. Connectivity with other apps was not generally important to users []. | 46 (26) | 28; 5; 15 | 19; 5; 27 | 35; 2; 9 |
| Passive sensing | Patients often considered passively gathered data to be more objective and accurate than self-reported data [,,] though some highlighted the possibility of false positives and inaccuracies and suggested that DMHIs include disclaimers acknowledging this [,,]. Two studies highlighted a tradeoff between increased data accuracy provided by passive sensing and increased agency and self-efficacy provided by active symptom monitoring [,]. Some participants found the idea of passive sensing invasive [,,], whereas others were unconcerned [], or embraced the idea: “I know that my phone knows everything about me. Why not get it to help me?” []. Data indicated that participants were most enthusiastic about passive sensing being used to track sleep [,,,,] and activity patterns [,,,]. There was some evidence patients were willing for data to be passively gathered about neurocognition [], psychophysiology [], medication adherence [], and heart rate []. Audio and video data collection was less popular []; GPS and social media usage data collection was less popular still []. Participants with bipolar were positive about an “intelligent” system that “learns” their routines and automatically detects changes [,], and staff said that passively collecting data provided an overwhelming amount of information []. | 11 (6) | 5; 2; 4 | 7; 0; 4 | 8; 0; 3 |
| Data security and privacy | Data security and privacy concerns were frequently mentioned barriers, including general concerns about security breaches and mental health-specific concerns. Some associated DMHIs with surveillance [,] or emphasized that mental health stigma made them worried about sensitive symptom data being disclosed [,,,]. Measures to protect data or mitigate the effects of data loss included the use of pseudonyms, password protection, and the use of general terms (eg, “illness”) rather than specific diagnoses. Five studies suggested, including a clear statement about what would happen to data, though, staff and patients in a study that included such a statement admitted they did not read it []. | 36 (21) | 23; 5; 8 | 20; 0; 18 | 25; 2; 9 |
| Technical difficulties | Twenty-five studies reported that participants experienced technical difficulties with the software (eg, notifications firing multiple times in error) or hardware (eg, poor battery life). These usually affected a minority; responses varied from mild irritation to total DMHI disengagement. | 25 (14) | 5; 13; 7 | 1; 0; 24 | 23; 0; 2 |
aSome papers fall into more than one category; therefore, categories may add to more than the overall total N.
bDMHI: digital mental health intervention.
Crosscutting Themes
Crosscutting Theme 1: DMHIs to Meet Specific Needs, Not Replace Human Care
Findings emphasized that DMHI functions and content should align with specific patient needs and that patients and staff must perceive a benefit if they are to engage with a DMHI (n=34 studies; Individuals, Need; Table 4). Patients preferred DMHIs with a clear aim, which did not try to do too many things (Innovation: Complexity; Table 1). Several DMHI benefits came through clearly in reviewed studies (Innovation: Relative advantage), including their value in being available anywhere and anytime (n=37 studies), promoting autonomy (n=26), facilitating self-awareness and memory (n=61), connecting people with peers (n=29), and normalizing mental health experiences (n=26; Innovation: Relative advantage). These advantages contributed to perceptions that, if successfully implemented, DMHIs could widen the reach of services. Nevertheless, while patients and staff acknowledged that DMHIs might enhance care in specific ways, they emphasized that they should not replace human care (n=16; Innovation: Relative advantage) and staff worried that the increasing emphasis on DMHIs might detract from health services’ ability to deliver care (Inner setting: Mission alignment; Table 3).
Seeing the benefit of a DMHI in meeting their needs prompted patients’ initial use and helped sustain use (Individuals: Need). Studies (n=27) highlighted that patient needs may change over time, depending on illness stage and symptom levels. Certain DMHI content (eg, psychoeducation) and functions (eg, symptom tracking) may be best suited to the needs of people with recent onset illness or ongoing symptoms. Indeed, 3 studies [,,] testing DMHI use qualitatively reported discontinuations due to improved mental health, as participants no longer felt they needed the DMHI. Tailoring or customization (Design; Table 6) allows DMHIs to be adapted for varied needs; eg, different psychoeducation “levels,” staggered unlocking of content based on users’ evolving needs, or adaptive frequency of symptom monitoring depending on symptom levels.
While staff considered some aspects of DMHIs (Innovation: Relative advantage) aligned with their core “mission”, they saw human contact as a central tenet of their mission that was potentially undermined by DMHIs (Inner setting: Mission alignment). Staff worried that “patients would see using the app system as a replacement for care, something it was not designed to do” [], and doubted DMHIs could be effective without a therapeutic relationship: “If you are going to talk evidence-based, the biggest thing in every single intervention... is therapeutic relationship, so you are just removing the most effective part” []. Such views could be a prominent barrier to DMHI implementation, particularly for stand-alone or minimally blended DMHIs. This underlines the importance of clearly communicating the specific purpose of the DMHI, and its limits, and reassuring staff that it is not intended to replace them.
Crosscutting Theme 2: Human Support for DMHIs
Forty-nine studies highlighted human support as a key facilitator of DMHI engagement (Individuals: Motivation;Table 4), underlining a need to align DMHI implementation with clinical workflows. Reviewed studies found a lack of compatibility with current job roles and workflows (n=17), with DMHIs seen as creating extra work in already stretched teams (Inner setting: Compatibility; Table 3) and as less important than other tasks (Inner setting: Mission alignment). Clinicians did not want to receive a “constant stream of information” about patients []. They thought symptom monitoring-based DMHIs would increase their responsibilities, and questioned who would respond out-of-hours. Staff lacked time to facilitate DMHI use, found data from symptom-monitoring apps complex to interpret (Innovation: Complexity; Table 1), and sometimes lacked digital skills (Inner setting: Access to knowledge and information).
Data on types of support needed suggested that not all support must be provided by someone clinically trained (Individuals: Motivation). Support needs varied, depending on the innovation’s or individual’s status on other CFIR constructs (see CFIR codes in parentheses), but included: training in basic digital skills (Individuals: Capability), recommending suitable DMHIs, onboarding, managing technical difficulties (Design; Table 6), support navigating DMHIs (Innovation Complexity), ongoing coaching (Individuals: Motivation), moderation of social media components, and interpreting and triaging real-time symptom data. Arguably, these tasks could be undertaken by a suitably trained nonclinical staff member, with referrals made to clinicians as needed (eg, when symptoms increase above a predefined threshold, or risk is indicated). Similar observations have prompted the idea of restructuring services to include new job roles for DMHI facilitation (eg, digital navigator, peer supporter). Reviewed studies did not test the implementation of such roles within clinical services; although, many DMHIs were successfully delivered by someone in an equivalent role in a research context. Of 4 RCTs quantitatively examining the effects of support, 2 demonstrated significantly higher DMHI engagement in peer-supported than unsupported trial arms [,,], 1 found a significant positive effect of brief interviewer-facilitated symptom reviews on app adherence for younger women (no significant overall effect []), and 1 found moderate effect sizes in favor of the supported group [] but was not powered to detect a significant difference (Individuals: Motivation).
Six studies provided evidence regarding whom staff or patients might prefer to deliver DMHI support (ie, clinicians or nonclinicians). Studies with staff suggested a preference for nonclinician support for initial onboarding [], technical [,,], or peer-supported [] aspects of DMHI use, with input from a clinician or team leader for more clinical aspects [,]. Patients expressed mixed views, with some preferring support from a known clinician [,], but others viewing this as more “intrusive” than nonclinician support []. Qualitative studies reported that, as well as overcoming practical barriers (eg, poor digital literacy, technical difficulties, complex DMHI layout), human connection helped address barriers like low motivation and anxiety about DMHI use, and provided accountability: “I’m going to see [coach]... within a few days, and we were going to discuss this, then there was accountability, I had to get some things done” []. Peer support was valued, whether as a formal part of the intervention (peer facilitators) or via social media (Innovation: Relative advantage), as peers “...can relate to each other and they know what they’ve been through” [], and participants did “not feel scared people will judge you because they have been through the same thing and will be supportive” [].
Crosscutting Theme 3: Practical Helps and Hindrances
Reviewed studies outlined practical factors that can help or hinder DMHI implementation. Arising across CFIR constructs, these factors tended to hinge on the principle that DMHI use should be as low-effort as possible for users, and practical barriers to access should be addressed.
To maximize engagement, DMHIs should ideally be free or inexpensive to patients (Innovation: Cost;Table 1), use similar formats to everyday technology (eg, smartphone app), and allow on-demand, out-of-hours use (Innovation: Relative advantage). DMHIs should use straightforward language, be visually appealing (Design; Table 6), minimize complexity, avoid lengthy login processes, use an easy-to-navigate app or website structure (shallow structure, <200 words/page), and be compatible with users’ own devices (Innovation: Complexity). Patients found smartphone apps more convenient than computer or SMS text messaging systems, though apps must be thoroughly tested to minimize technical difficulties (Design). Other design elements to increase engagement included push-notification reminders; gamification features (eg, badges for completed tasks); automated feedback about symptoms or DMHI use (eg, symptom graph, step count, and minutes DMHI used); and tailored or varied content to minimize repetition.
These factors interact with each other. For example, tailoring DMHIs may increase engagement, but can also increase complexity, an engagement barrier. In turn, other factors, such as training and support, and individual capability (eg, cognitive ability and digital literacy) may moderate the negative effects of complexity on engagement. Nevertheless, even with training and support, and high individual capability, more complex interventions require more time and effort from users, making disengagement more likely. DMHI design teams must weigh the relative advantages and disadvantages of such interacting factors.
Some engagement barriers could be mitigated given sufficient resources. Lack of access to technology (Individuals: Opportunity; Table 4) and lack of digital literacy (Individuals: Capability) could be addressed, in theory, by providing devices and training in how to use them. Although staff thought patients would lose, damage, or sell devices, reviewed studies found only an 11% loss of study-provided devices. Data security and privacy concerns were frequently mentioned barriers; patients were concerned about sensitive symptom data being disclosed, due to the potential stigma associated with it. Measures to protect data or mitigate the effects of data loss included the use of pseudonyms, password protection, and the use of general terms (eg, “illness”) rather than specific diagnoses.
Some broader practical barriers may be difficult for implementation teams to address directly. These included the lack of mobile phone signal, publicly-funded smartphones being too basic, lack of national guidelines and policies on DMHI use in health services (Outer setting; Table 2), aging computers, poor bandwidth, no wireless connection in hospitals, poor organizational support for IT developments, and lack of quiet space (Inner setting; Table 3), patient time, or suitable environment (Individuals: Opportunity).
Crosscutting Theme 4: Interactions Between Symptoms and DMHI Engagement
Approximately one-third of quantitative analyses (18/52, 35%) reported a significant effect of symptoms on DMHI engagement (Individuals: Capability; Table 4; Table S7 in ). Among these, higher baseline positive, negative, or cognitive symptoms were associated with lower DMHI engagement. Qualitative data suggested that staff and patients anticipated such a pattern (and so may recommend or agree to DMHI use accordingly), and some patients avoided DMHIs due to experiencing difficult interactions between symptoms and technology (eg, technology-related delusions or paranoia). Findings for functioning and depression were more mixed, with higher baseline functioning or depression sometimes predicting higher and sometimes lower DMHI engagement. Qualitative studies suggested that while symptoms of depression (eg, poor concentration and motivation) may interfere with DMHI engagement, low mood may prompt higher engagement for some “because of a desire to find meanings and solutions for their depressive symptoms” []. No quantitative studies examined the effects of relapse on engagement; qualitative studies noted decreased DMHI use during relapse; however, participants usually restarted DMHIs on remission. Some were hesitant to (accurately) report symptoms via DMHIs for fear it may lead to more restrictive treatment (hospital admission and increased medication).
Negative interactions between DMHIs and symptoms, or perceived risk of such interactions, emerged as possible motivational barriers. Patients sometimes experienced DMHIs as “deflating, confronting, or triggering,” reminding them how unwell they were, or about their diagnosis, which could be disheartening, “self-stigmatizing,” and often prompted disengagement (Individuals: Motivation). There were numerous quotations about this, across 10 studies; however, it was difficult to tell how many participants were typically affected. While 1 study reported that half the sample felt negatively influenced by logging their moods and symptoms 5 times per day [], in other studies with less intensive report schedules, very few (1/27, 4%) participants found the DMHI to be confronting []. Relatedly, fear of relapse was a barrier to DMHI use. Staff and patients anticipated that, by increasing awareness of symptoms, DMHIs would heighten users’ anxiety about deterioration, which may worsen symptoms. In studies testing actual DMHI use, there was qualitative [,,,] and quantitative [] evidence that fear of relapse and increased rumination about symptoms prevented DMHI engagement but the proportion of participants affected was unclear. Reviewed papers did not typically report such negative reactions to DMHI use (ie, finding the DMHI “confronting,” or experiencing fear of relapse) as adverse reactions in the context of formal AE monitoring (). Arguably, doing so would allow a quantitative examination of how many participants were affected across studies.
Crosscutting Theme 5: Financial Provision for Implementing DMHIs
There was scant empirical information across CFIR constructs about the effects of financial provision on DMHI implementation, probably because most studies examined DMHI use in research studies rather than services. However, findings implied that financial limitations concerning software development and maintenance may indirectly reduce engagement. One study noted that some participant-suggested technical improvements were not possible due to financial limitations []. Another noted that the look and feel of the study app were considered “old school” but insufficient funding precluded improvements [] (Innovation: Cost; Table 1); study authors highlighted the challenge of competing with commercial apps: “It is difficult to obtain... funding at a level that can compete with the funding that goes into app design for functions that people routinely use... leaving [study app] looking ‘old school’” []. Others noted that, while patients prefer free-to-use DMHIs (Outer setting: Financing;Table 2), sustainable business models for maintaining evidence-based DMHIs are often absent: “...apps that have been empirically evaluated and developed in accordance with clinical guidelines are rarely made available via the app marketplace... Thoughtful plans for commercialization and fiscal sustainability are important, given respondents with BD endorsed free/low-cost apps” []. This statement was supported by a striking absence of information from other reviewed studies about their intended model for DMHI dissemination and implementation, except a US-based study which mentioned costs were offset by the service’s billable nature [] and a Canadian study that anticipated care provider networks would purchase DMHI licenses [].
Some studies (5/150, 3.3%) listed costs associated with delivering DMHIs in the research context, including web hosting, software licensing fees, texting fees, technical support, and staff time for DMHI delivery. Financial provision for these and other DMHI-associated costs will be key to scaling up DMHI use in services but was not directly discussed in reviewed studies. Nevertheless, to maximize engagement with DMHIs for psychosis or bipolar disorder per the cross-cutting themes outlined above, considerable financial support would be necessary, particularly to provide human support for DMHI use (theme 2), to source or develop well-designed DMHIs that are as low-effort as possible for users (theme 3), and to address practical needs, such as digital poverty (theme 3).
Discussion
Principal Results
This systematic review found that patient and staff engagement with DMHIs is likely to be strengthened by using DMHIs to meet specific needs, not replace human care; providing human support for DMHIs and considering compatibility with current workflows; making DMHIs as low effort as possible for patients and staff, inquiring about common practical barriers to use (eg, digital poverty or training needs) and addressing these where possible; acknowledging interactions between symptoms and DMHIs, including motivational barriers and fear of relapse; and ensuring appropriate financial provision to implement DMHIs in line with these recommendations.
Comparison With Prior Work
Similar to the review by Aref-Adib et al [] on DMHIs for psychosis or bipolar disorder, most data were coded into CFIR Intervention (149/175, 85.1% papers) or Individuals (137/175, 78.3% papers) domains, with very little covering Outer setting (9/175, 5.1% papers) or Implementation process (5/175, 2.9% papers). We found more data related to the Inner setting (20/175, 11.4% papers) than the previous review, with studies highlighting a lack of compatibility with clinical workflows (17/175, 9.7% papers) and lack of alignment with staff perceptions of their clinical “mission” (5/175, 2.9% papers) as important barriers to DMHI implementation. This difference appears to reflect progress in the psychosis and bipolar disorder DMHI literature over the intervening years (Figure 1). Workflow-related barriers are mentioned in reviews covering DMHIs for common mental health disorders [,,], a literature which has a chronological head-start on the equivalent literature for psychosis and bipolar disorder.
Our finding that human support was a key facilitator of DMHI engagement is consistent with previous reviews of factors affecting the implementation of DMHIs for psychosis or bipolar disorder [,], common mental health problems [,,,,], and a meta-synthesis of staff views []. Relatedly, our review and others reported that clinicians’ negative attitudes to DMHIs, resistance to change, and lack of digital skills were sometimes barriers to DMHI implementation [,,]. Nevertheless, such considerations are likely eclipsed by a need for compatibility with clinical workflows (which our findings suggest is currently absent; see Inner setting: Compatibility). Restructuring services to include new “digital navigator” roles, alongside traditional clinical roles, has been suggested as a model for implementing DMHIs in mental health services [,], and meta-analysis shows similar effectiveness across DMHIs guided by clinicians and nonclinicians []. Whether this model or any other service model is used, staff members (clinicians and nonclinicians) would need allocated time, training, and supervision to support patients’ DMHI use [], all of which have financial implications [].
DMHIs are frequently cited as potential methods of extending care in a cost-effective manner []. There is some empirical evidence supporting this, albeit from DMHIs tested in a research context [], but economic evaluations of mental health DMHIs are rare []. Nevertheless, up-front costs of restructuring services and ongoing costs of delivering DMHIs will likely be substantial []. Few reviewed papers (5/175, 2.9%) directly evaluated financial factors, but our findings highlight the importance of considering staff-related costs and costs of addressing engagement barriers (eg, dated-looking digital interfaces; patients lacking suitable devices or skills). Without financial investment, DMHI engagement will likely be poor, potentially perpetuating existing treatment disparities [,]. More broadly, reviewed studies highlighted the difficulties of developing evidence-based DMHIs against a background of rapid technological advances and attractive but poorly evidenced commercial apps. Hill et al [] outlined similar issues in relation to developing evidence-based DMHIs for common mental health problems; they recommended that researchers collaborate with industry partners to commercialize well-evidence DMHIs, generating funds to cover ongoing maintenance and development, and enhancing long-term sustainability.
We found that mental health symptoms and DMHIs sometimes interacted in ways that prevented DMHI engagement, which echoes and expands on findings from the review by Aref-Adib et al []. We found more evidence of motivational barriers relating to increased rumination or fear of relapse (26/175, 14.9% papers) than that found in the previous review (1/26, 4%). We also found that patients disengaged with DMHIs because they found increased self-reflection and attention toward negative emotional states too “confronting,” which has not been reported in previous reviews. Evidence from reviewed studies did not allow an estimate of how many patients are affected by these barriers. Future studies should consider examining such reactions quantitatively, perhaps as part of formal AE monitoring, to help future users make informed choices about DMHI risks and benefits. More generally, our supplementary analysis of safety reporting suggested that AE reporting warrants considerably more attention, in line with recent findings from other reviews [,,] and recommendations regarding AE monitoring and reporting for DMHIs for psychosis [].
Strengths and Limitations of the Evidence Base
The reviewed literature contained a wealth of qualitative and quantitative data outlining factors that may affect staff and patient engagement with DMHIs for psychosis or bipolar disorder. However, the current evidence base has important limitations. First, most studies were conducted in traditional research settings rather than evaluating the process of implementing DMHIs directly in health services. Consequently, identified barriers and facilitators may not all translate and generalize to care provision settings. Nevertheless, the review provides an early indication of likely barriers and facilitators, across the CFIR domains, enabling these to be considered when designing future DMHIs and DMHI implementation strategies. This information will also be valuable in highlighting key implementation factors to prospectively examine in future implementation studies. Second, most studies seeking health professional views sought this information hypothetically (ie, they asked staff, who had not necessarily already used DMHIs, what they thought about using DMHIs in their clinical practice, in theory), whereas studies seeking information from patients more often asked about actual experiences of DMHI use (ie, they asked patients, who had already used a specific DMHI, about what it was like to use that DMHI). Berry et al [] suggested that hypothetical views and actual experiences of DMHIs do not always align. Therefore, the difference in methods between patient and health professional studies must be kept in mind when considering the findings of this review.
Strengths and Limitations of This Review
A key strength was the use of CFIR as a framework, allowing examination of interactions between staff and patient barriers and facilitators and wider contextual factors, and sensitizing us to implementation factors that may not otherwise have been identified in reviewed studies. Studies were identified using an inclusive search strategy, with intentionally wide inclusion criteria (eg, DMHI types, settings, and engagement indicators) to allow examination of a broad range of barriers and facilitators. However, the resulting studies were extremely heterogeneous, precluding meta-analysis and potentially conflating results from dissimilar studies. Likewise, the review covered a 21-year period (2010 to 2021), spanning a time of great technological advancement during which societal acceptance of DMHIs may have changed considerably, adding further heterogeneity. Finally, we limited our search to the period before the COVID-19 pandemic. Although this avoided COVID-19 pandemic–related heterogeneity, if the search had been conducted later, findings related to the impacts of the COVID-19 pandemic on digital uptake (Outer setting: Critical incidents) would likely have been prominent. We recommend that a future review should focus on mixed methods studies published since the pandemic, to examine the shifting views of staff and patients regarding potential barriers and facilitators of digital health tools for psychosis and bipolar disorder in the intervening period.
Conclusions
While DMHIs have a role in supporting those with psychosis or bipolar disorder, intervention, individual, and contextual factors all impact uptake. To enhance engagement, DMHIs must be simple, low effort, meet specific needs, and not be heralded as a replacement for face-to-face care. Human support can overcome engagement barriers related to motivation and practicalities of DMHI engagement. Financial provision is likely to be key to sustainable DMHI implementation.
Acknowledgments
The authors wish to thank Oshin Jacob, who assisted with data extraction. This research was funded by the UK National Institute for Health (NIHR) and Care Research (grant NIHR300794) and supported by the NIHR Manchester Biomedical Research Center (grant NIHR203308). There was no editorial direction or censorship from the study funder or the study sponsor. The study funder and study sponsor had no role in the study design, data collection, data analysis, data interpretation, or writing the report. The views expressed are those of the authors and not necessarily those of the NIHR or the Department of Health and Social Care.
Authors' Contributions
EE analyzed the data and drafted the paper as the first author, with support from SB and SF. All coauthors had the opportunity to contribute to drafts of the paper, to critically review the manuscript, and to approve the final version. EE and SB devised the study aims and methods and supervised the researchers during data collection and analysis. PW and JN provided methodological advice. SA, HB, DDB, JN, XZ and AP contributed to data collection.
Conflicts of Interest
SB is a director and shareholder of CareLoop Health Ltd, which develops and markets digital therapeutics for schizophrenia and a digital screening app for postnatal depression. All other authors declare no conflicts of interest.
Multimedia Appendix 1
PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses) checklist.
DOCX File , 33 KBMultimedia Appendix 2
Systematic review search terms and inclusion and exclusion criteria, detailed analysis of adverse event reports, list of excluded studies, and supplementary tables detailing characteristics and quality of included studies.
DOCX File , 402 KBReferences
- Dembek C, Mackie D, Modi K, Zhu Y, Niu X, Grinnell T. The economic and humanistic burden of bipolar disorder in adults in the United States. Ann Gen Psychiatry. Mar 24, 2023;22(1):13. [FREE Full text] [CrossRef] [Medline]
- Jin H, Mosweu I. The societal cost of schizophrenia: a systematic review. Pharmacoeconomics. Jan 2017;35(1):25-42. [CrossRef] [Medline]
- Torous J, Bucci S, Bell IH, Kessing LV, Faurholt-Jepsen M, Whelan P, et al. The growing field of digital psychiatry: current evidence and the future of apps, social media, chatbots, and virtual reality. World Psychiatry. Oct 2021;20(3):318-335. [FREE Full text] [CrossRef] [Medline]
- Chivilgina O, Wangmo T, Elger BS, Heinrich T, Jotterand F. mHealth for schizophrenia spectrum disorders management: a systematic review. Int J Soc Psychiatry. Nov 23, 2020;66(7):642-665. [CrossRef] [Medline]
- Sawyer C, McKeon G, Hassan L, Onyweaka H, Martinez Agulleiro L, Guinart D, et al. Digital health behaviour change interventions in severe mental illness: a systematic review. Psychol Med. Sep 28, 2023;53(15):6965-7005. [CrossRef]
- Jameel L, Valmaggia L, Barnes G, Cella M. mHealth technology to assess, monitor and treat daily functioning difficulties in people with severe mental illness: a systematic review. J Psychiatr Res. Nov 24, 2021;145:35-49. [CrossRef] [Medline]
- Morales-Pillado C, Fernández-Castilla B, Sánchez-Gutiérrez T, González-Fraile E, Barbeito S, Calvo A. Efficacy of technology-based interventions in psychosis: a systematic review and network meta-analysis. Psychol Med. Dec 06, 2022;53(13):6304-6315. [CrossRef]
- Camacho E, Levin L, Torous J. Smartphone apps to support coordinated specialty care for prodromal and early course schizophrenia disorders: systematic review. J Med Internet Res. Nov 12, 2019;21(11):e16393. [FREE Full text] [CrossRef] [Medline]
- Clarke S, Hanna D, Mulholland C, Shannon C, Urquhart C. A systematic review and meta-analysis of digital health technologies effects on psychotic symptoms in adults with psychosis. Psychosis. Aug 12, 2019;11(4):362-373. [CrossRef]
- Smith KA, Blease C, Faurholt-Jepsen M, Firth J, Van Daele T, Moreno C, et al. Digital mental health: challenges and next steps. BMJ Ment Health. Feb 17, 2023;26(1):e300670. [FREE Full text] [CrossRef] [Medline]
- Borghouts J, Eikey E, Mark G, De Leon C, Schueller SM, Schneider M, et al. Barriers to and facilitators of user engagement with digital mental health interventions: systematic review. J Med Internet Res. Mar 24, 2021;23(3):e24387. [FREE Full text] [CrossRef] [Medline]
- Graham AK, Lattie EG, Powell BJ, Lyon AR, Smith JD, Schueller SM, et al. Implementation strategies for digital mental health interventions in health care settings. Am Psychol. Nov 2020;75(8):1080-1092. [FREE Full text] [CrossRef] [Medline]
- Mohr DC, Lyon AR, Lattie EG, Reddy M, Schueller SM. Accelerating digital mental health research from early design and creation to successful implementation and sustainment. J Med Internet Res. May 10, 2017;19(5):e153. [FREE Full text] [CrossRef] [Medline]
- Forbes A, Keleher MR, Venditto M, DiBiasi F. Assessing patient adherence to and engagement with digital interventions for depression in clinical trials: systematic literature review. J Med Internet Res. Aug 11, 2023;25:e43727. [FREE Full text] [CrossRef] [Medline]
- Gilbody S, Littlewood E, Hewitt C, Brierley G, Tharmanathan P, Araya R, et al. Computerised cognitive behaviour therapy (cCBT) as treatment for depression in primary care (REEACT trial): large scale pragmatic randomised controlled trial. BMJ. Nov 11, 2015;351:h5627. [FREE Full text] [CrossRef] [Medline]
- Bauer MS, Kirchner J. Implementation science: what is it and why should I care? Psychiatry Res. Jan 2020;283:112376. [FREE Full text] [CrossRef] [Medline]
- Aref-Adib G, McCloud T, Ross J, O'Hanlon P, Appleton V, Rowe S, et al. Factors affecting implementation of digital health interventions for people with psychosis or bipolar disorder, and their family and friends: a systematic review. The Lancet Psychiatry. Mar 2019;6(3):257-266. [CrossRef]
- Carroll C, Booth A, Cooper K. A worked example of "best fit" framework synthesis: a systematic review of views concerning the taking of some potential chemopreventive agents. BMC Med Res Methodol. Mar 16, 2011;11(1):29. [FREE Full text] [CrossRef] [Medline]
- Carroll C, Booth A, Leaviss J, Rick J. "Best fit" framework synthesis: refining the method. BMC Med Res Methodol. Mar 13, 2013;13(1):37. [CrossRef] [Medline]
- Damschroder LJ, Reardon CM, Widerquist MA, Lowery J. The updated Consolidated Framework for Implementation Research based on user feedback. Implement Sci. Oct 29, 2022;17(1):75. [CrossRef] [Medline]
- Ross J, Stevenson F, Lau R, Murray E. Factors that influence the implementation of e-health: a systematic review of systematic reviews (an update). Implement Sci. Oct 26, 2016;11(1):146. [FREE Full text] [CrossRef] [Medline]
- Taher R, Hsu CW, Hampshire C, Fialho C, Heaysman C, Stahl D, et al. The safety of digital mental health interventions: systematic review and recommendations. JMIR Ment Health. Oct 09, 2023;10:e47433. [FREE Full text] [CrossRef] [Medline]
- Gómez Bergin AD, Valentine AZ, Rennick-Egglestone S, Slade M, Hollis C, Hall CL. Identifying and categorizing adverse events in trials of digital mental health interventions: narrative scoping review of trials in the International Standard Randomized Controlled Trial Number Registry. JMIR Ment Health. Feb 22, 2023;10:e42501. [FREE Full text] [CrossRef] [Medline]
- Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. Mar 29, 2021;372:n71. [FREE Full text] [CrossRef] [Medline]
- Batra S, Baker R, Wang T, Forma F, DiBiasi F, Peters-Strickland T. Digital health technology for use in patients with serious mental illness: a systematic review of the literature. Med Devices (Auckl). Oct 2017;10:237-251. [CrossRef] [Medline]
- Koh D. COVID-19 lockdowns throughout the world. Occup Med. Jul 2020;70(5):322. [CrossRef]
- Nvivo. Lumivero. URL: https://lumivero.com/ [accessed 2024-12-05]
- Hong QN, Fàbregues S, Bartlett G, Boardman F, Cargo M, Dagenais P, et al. The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers. Educ Inf. Dec 18, 2018;34(4):285-291. [CrossRef]
- Junqueira DR, Zorzela L, Golder S, Loke Y, Gagnier J, Julious SA, et al. CONSORT Harms 2022 statement, explanation, and elaboration: updated guideline for the reporting of harms in randomized trials. J Clin Epidemiol. Jun 2023;158:149-165. [FREE Full text] [CrossRef] [Medline]
- Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. Aug 07, 2009;4:50. [FREE Full text] [CrossRef] [Medline]
- Allan S, Bradstreet S, Mcleod H, Farhall J, Lambrou M, Gleeson J, EMPOWER Group, et al. Developing a hypothetical implementation framework of expectations for monitoring early signs of psychosis relapse using a mobile app: qualitative study. J Med Internet Res. Oct 24, 2019;21(10):e14366. [FREE Full text] [CrossRef] [Medline]
- Baumel A, Correll CU, Birnbaum M. Adaptation of a peer based online emotional support program as an adjunct to treatment for people with schizophrenia-spectrum disorders. Internet Interv. May 2016;4:35-42. [FREE Full text] [CrossRef] [Medline]
- Bucci S, Berry N, Morris R, Berry K, Haddock G, Lewis S, et al. "They are not hard-to-reach clients. We have just got hard-to-reach services." Staff views of digital health tools in specialist mental health services. Front Psychiatry. 2019;10:344. [FREE Full text] [CrossRef] [Medline]
- Steare T, Giorgalli M, Free K, Harju-Seppänen J, Akther S, Eskinazi M, et al. A qualitative study of stakeholder views on the use of a digital app for supported self-management in early intervention services for psychosis. BMC Psychiatry. Jun 19, 2021;21(1):311. [FREE Full text] [CrossRef] [Medline]
- Williams A, Fossey E, Farhall J, Foley F, Thomas N. Going online together: the potential for mental health workers to integrate recovery oriented e-mental health resources into their practice. Psychiatry. 2018;81(2):116-129. [CrossRef] [Medline]
- Ben-Zeev D, Scherer EA, Gottlieb JD, Rotondi AJ, Brunette MF, Achtyes ED, et al. mHealth for schizophrenia: patient engagement with a mobile phone intervention following hospital discharge. JMIR Ment Health. Jul 27, 2016;3(3):e34. [FREE Full text] [CrossRef] [Medline]
- Bonet L, Torous J, Arce D, Blanquer I, Sanjuan J. ReMindCare app for early psychosis: pragmatic real world intervention and usability study. JMIR Mhealth Uhealth. Nov 06, 2020;8(11):e22997. [FREE Full text] [CrossRef] [Medline]
- Ryan KA, Babu P, Easter R, Saunders E, Lee AJ, Klasnja P, et al. A smartphone app to monitor mood symptoms in bipolar disorder: development and usability study. JMIR Ment Health. Sep 22, 2020;7(9):e19476. [FREE Full text] [CrossRef] [Medline]
- Hidalgo-Mazzei D, Reinares M, Mateu A, Nikolova VL, Bonnín CD, Samalin L, et al. OpenSIMPLe: a real-world implementation feasibility study of a smartphone-based psychoeducation programme for bipolar disorder. J Affect Disord. Dec 01, 2018;241:436-445. [CrossRef] [Medline]
- Lal S, Nguyen V, Theriault J. Seeking mental health information and support online: experiences and perspectives of young people receiving treatment for first-episode psychosis. Early Interv Psychiatry. Jun 2018;12(3):324-330. [CrossRef] [Medline]
- Bucci S, Morris R, Berry K, Berry N, Haddock G, Barrowclough C, et al. Early psychosis service user views on digital technology: qualitative analysis. JMIR Ment Health. Oct 31, 2018;5(4):e10091. [FREE Full text] [CrossRef] [Medline]
- Lobban F, Dodd AL, Sawczuk AP, Asar O, Dagnan D, Diggle PJ, et al. Assessing feasibility and acceptability of web-based enhanced relapse prevention for bipolar disorder (ERPonline): a randomized controlled trial. J Med Internet Res. Mar 24, 2017;19(3):e85. [FREE Full text] [CrossRef] [Medline]
- Ben-Zeev D, Brian RM, Aschbrenner KA, Jonathan G, Steingard S. Video-based mobile health interventions for people with schizophrenia: bringing the "pocket therapist" to life. Psychiatr Rehabil J. Mar 2018;41(1):39-45. [FREE Full text] [CrossRef] [Medline]
- Dodd AL, Mallinson S, Griffiths M, Morriss R, Jones SH, Lobban F. Users' experiences of an online intervention for bipolar disorder: important lessons for design and evaluation. Evid Based Ment Health. Nov 2017;20(4):133-139. [FREE Full text] [CrossRef] [Medline]
- Terp M, Jørgensen R, Laursen BS, Mainz J, Bjørnes CD. A smartphone app to foster power in the everyday management of living with schizophrenia: qualitative analysis of young adults' perspectives. JMIR Ment Health. Oct 01, 2018;5(4):e10157. [FREE Full text] [CrossRef] [Medline]
- Chiauzzi E, Newell A. Mental health apps in psychiatric treatment: a patient perspective on real world technology usage. JMIR Ment Health. Apr 22, 2019;6(4):e12292. [FREE Full text] [CrossRef] [Medline]
- Morton E, Nicholas J, Lapadat L, O'Brien HL, Barnes SJ, Poh C, et al. Use of smartphone apps in bipolar disorder: an international web-based survey of feature preferences and privacy concerns. J Affect Disord. Dec 01, 2021;295:1102-1109. [CrossRef] [Medline]
- Nicholas J, Boydell K, Christensen H. Beyond symptom monitoring: consumer needs for bipolar disorder self-management using smartphones. Eur Psychiatry. Jul 2017;44:210-216. [CrossRef] [Medline]
- Morton E, Torous J, Murray G, Michalak EE. Using apps for bipolar disorder - an online survey of healthcare provider perspectives and practices. J Psychiatr Res. May 2021;137:22-28. [CrossRef] [Medline]
- Michalak EE, Morton E, Barnes SJ, Hole R, CREST.BD, Murray G. Supporting self-management in bipolar disorder: mixed-methods knowledge translation study. JMIR Ment Health. Apr 15, 2019;6(4):e13493. [FREE Full text] [CrossRef] [Medline]
- Arnold C, Williams A, Thomas N. Engaging with a web-based psychosocial intervention for psychosis: qualitative study of user experiences. JMIR Ment Health. Jun 19, 2020;7(6):e16730. [FREE Full text] [CrossRef] [Medline]
- Berry N, Lobban F, Bucci S. A qualitative exploration of service user views about using digital health interventions for self-management in severe mental health problems. BMC Psychiatry. Jan 21, 2019;19(1):35. [FREE Full text] [CrossRef] [Medline]
- Almeida RF, Sousa TJ, Couto AS, Marques AJ, Queirós CM, Martins CL. Development of weCope, a mobile app for illness self-management in schizophrenia. Arch Clin Psychiatry (São Paulo). Feb 2019;46(1):1-4. [CrossRef]
- Valentine L, McEnery C, O'Sullivan S, Gleeson J, Bendall S, Alvarez-Jimenez M. Young people's experience of a long-term social media-based intervention for first-episode psychosis: qualitative analysis. J Med Internet Res. Jun 26, 2020;22(6):e17570. [FREE Full text] [CrossRef] [Medline]
- Lederman R, Wadley G, Gleeson J, Bendall S, Álvarez-Jiménez M. Moderated online social therapy: designing and evaluating technology for mental health. ACM Trans Comput Hum Interact. Feb 01, 2014;21(1):1-26. [CrossRef]
- Aschbrenner KA, Naslund JA, Bartels SJ. A mixed methods study of peer-to-peer support in a group-based lifestyle intervention for adults with serious mental illness. Psychiatr Rehabil J. Dec 2016;39(4):328-334. [FREE Full text] [CrossRef] [Medline]
- Naslund JA, Aschbrenner KA, Marsch LA, Bartels SJ. Feasibility and acceptability of Facebook for health promotion among people with serious mental illness. Digit Health. Nov 01, 2016;2:2055207616654822. [FREE Full text] [CrossRef] [Medline]
- Huerta-Ramos E, Escobar-Villegas MS, Rubinstein K, Unoka ZS, Grasa E, Hospedales M, M-RESIST Group, et al. Measuring users' receptivity toward an integral intervention model based on mHealth solutions for patients with treatment-resistant schizophrenia (m-RESIST): a qualitative study. JMIR Mhealth Uhealth. Sep 28, 2016;4(3):e112. [FREE Full text] [CrossRef] [Medline]
- Huerta-Ramos E, Marcó-García S, Escobar-Villegas MS, Rubio-Abadal E, Ochoa S, Grasa Bello EM, Group M-Resist, et al. m-RESIST, a complete m-Health solution for patients with treatmentresistant schizophrenia: a qualitative study of user needs and acceptability in the Barcelona metropolitan area. Actas Esp Psiquiatr. Nov 2017;45(6):277-289. [FREE Full text] [Medline]
- Lal S, Dell'Elce J, Tucci N, Fuhrer R, Tamblyn R, Malla A. Preferences of young adults with first-episode psychosis for receiving specialized mental health services using technology: a survey study. JMIR Ment Health. 2015;2(2):e18. [FREE Full text] [CrossRef] [Medline]
- Ludwig KA, Browne JW, Nagendra A, Gleeson JF, D'Alfonso S, Penn DL, et al. Horyzons USA: a moderated online social intervention for first episode psychosis. Early Interv Psychiatry. Apr 2021;15(2):335-343. [CrossRef] [Medline]
- Medalia A, Saperstein AM, Stefancic A, Meyler S, Styke S, Qian M, et al. Feasibility and acceptability of remotely accessed cognitive remediation for schizophrenia in public health settings. Psychiatry Res. Jul 2021;301:113956. [FREE Full text] [CrossRef] [Medline]
- Jonathan G, Carpenter-Song EA, Brian RM, Ben-Zeev D. Life with FOCUS: a qualitative evaluation of the impact of a smartphone intervention on people with serious mental illness. Psychiatr Rehabil J. Jun 2019;42(2):182-189. [FREE Full text] [CrossRef] [Medline]
- Eisner E, Drake RJ, Berry N, Barrowclough C, Emsley R, Machin M, et al. Development and long-term acceptability of ExPRESS, a mobile phone app to monitor basic symptoms and early signs of psychosis relapse. JMIR Mhealth Uhealth. Mar 29, 2019;7(3):e11568. [FREE Full text] [CrossRef] [Medline]
- Valentine L, McEnery C, O'Sullivan S, D'Alfonso S, Gleeson J, Bendall S, et al. Young people's experience of online therapy for first-episode psychosis: a qualitative study. Psychol Psychother. Mar 2022;95(1):155-172. [CrossRef] [Medline]
- Saunders KE, Bilderbeck AC, Panchal P, Atkinson LZ, Geddes JR, Goodwin GM. Experiences of remote mood and activity monitoring in bipolar disorder: a qualitative study. Eur Psychiatry. Mar 27, 2017;41(1):115-121. [FREE Full text] [CrossRef] [Medline]
- Depp CA, Mausbach B, Granholm E, Cardenas V, Ben-Zeev D, Patterson TL, et al. Mobile interventions for severe mental illness: design and preliminary data from three approaches. J Nerv Ment Dis. Oct 2010;198(10):715-721. [FREE Full text] [CrossRef] [Medline]
- Valentine L, McEnery C, Bell I, O'Sullivan S, Pryor I, Gleeson J, et al. Blended digital and face-to-face care for first-episode psychosis treatment in young people: qualitative study. JMIR Ment Health. Jul 28, 2020;7(7):e18990. [FREE Full text] [CrossRef] [Medline]
- Murnane EL, Cosley D, Chang P, Guha S, Frank E, Gay G, et al. Self-monitoring practices, attitudes, and needs of individuals with bipolar disorder: implications for the design of technologies to manage mental health. J Am Med Inform Assoc. May 2016;23(3):477-484. [CrossRef] [Medline]
- Realpe A, Elahi F, Bucci S, Birchwood M, Vlaev I, Taylor D, et al. Co-designing a virtual world with young people to deliver social cognition therapy in early psychosis. Early Interv Psychiatry. Feb 2020;14(1):37-43. [CrossRef] [Medline]
- Bucci S, Barrowclough C, Ainsworth J, Machin M, Morris R, Berry K, et al. Actissist: proof-of-concept trial of a theory-driven digital intervention for psychosis. Schizophr Bull. Aug 20, 2018;44(5):1070-1080. [FREE Full text] [CrossRef] [Medline]
- Poole R, Simpson SA, Smith DJ. Internet-based psychoeducation for bipolar disorder: a qualitative analysis of feasibility, acceptability and impact. BMC Psychiatry. Sep 13, 2012;12:139. [CrossRef]
- Fletcher K, Murray G. Towards tailored psychosocial intervention for BD-II: lived experience perspectives on current and future management options. J Affect Disord. Jun 15, 2021;289:110-116. [CrossRef] [Medline]
- Puskar K, Schlenk EA, Callan J, Bickmore T, Sereika S. Relational agents as an adjunct in schizophrenia treatment. J Psychosoc Nurs Ment Health Serv. Aug 2011;49(8):22-29. [CrossRef] [Medline]
- Achtyes ED, Ben-Zeev D, Luo Z, Mayle H, Burke B, Rotondi AJ, et al. Off-hours use of a smartphone intervention to extend support for individuals with schizophrenia spectrum disorders recently discharged from a psychiatric hospital. Schizophr Res. Apr 2019;206:200-208. [CrossRef] [Medline]
- Bechtel JM, Lepoire E, Bauer AM, Bowen DJ, Fortney JC. Care manager perspectives on integrating an mHealth app system into clinical workflows: a mixed methods study. Gen Hosp Psychiatry. Jan 2021;68:38-45. [FREE Full text] [CrossRef] [Medline]
- Vilardaga R, Rizo J, Ries RK, Kientz JA, Ziedonis DM, Hernandez K, et al. Formative, multimethod case studies of learn to quit, an acceptance and commitment therapy smoking cessation app designed for people with serious mental illness. Transl Behav Med. Nov 25, 2019;9(6):1076-1086. [FREE Full text] [CrossRef] [Medline]
- Switsers L, Dauwe A, Vanhoudt A, Van Dyck H, Lombaerts K, Oldenburg J. Users' perspectives on mHealth self-management of bipolar disorder: qualitative focus group study. JMIR Mhealth Uhealth. May 02, 2018;6(5):e108. [FREE Full text] [CrossRef] [Medline]
- Röhricht F, Padmanabhan R, Binfield P, Mavji D, Barlow S. Simple mobile technology health management tool for people with severe mental illness: a randomised controlled feasibility trial. BMC Psychiatry. Jul 16, 2021;21(1):357. [FREE Full text] [CrossRef] [Medline]
- Ybarra ML, Rodriguez K, Madison H, Mojtabai R, Cullen BA. Developing texting for relapse prevention: a scalable mHealth program for people with schizophrenia and schizoaffective disorder. J Nerv Ment Dis. Oct 2019;207(10):854-862. [FREE Full text] [CrossRef] [Medline]
- Daus H, Kislicyn N, Heuer S, Backenstrass M. Disease management apps and technical assistance systems for bipolar disorder: investigating the patients´ point of view. J Affect Disord. Mar 15, 2018;229:351-357. [CrossRef] [Medline]
- Matthews M, Murnane E, Snyder J. Quantifying the changeable self: the role of self-tracking in coming to terms with and managing bipolar disorder. Hum Comput Interact. Apr 07, 2017;32(5-6):413-446. [CrossRef]
- Hatch A, Hoffman JE, Ross R, Docherty JP. Expert consensus survey on digital health tools for patients with serious mental illness: optimizing for user characteristics and user support. JMIR Ment Health. Jun 12, 2018;5(2):e46. [FREE Full text] [CrossRef] [Medline]
- Berry N, Bucci S, Lobban F. Use of the internet and mobile phones for self-management of severe mental health problems: qualitative study of staff views. JMIR Ment Health. Nov 01, 2017;4(4):e52. [FREE Full text] [CrossRef] [Medline]
- Palmier-Claus JE, Rogers A, Ainsworth J, Machin M, Barrowclough C, Laverty L, et al. Integrating mobile-phone based assessment for psychosis into people's everyday lives and clinical care: a qualitative study. BMC Psychiatry. Jan 23, 2013;13:34. [FREE Full text] [CrossRef] [Medline]
- Austin SF, Frøsig A, Buus N, Lincoln T, von Malachowski A, Schlier B, et al. Service user experiences of integrating a mobile solution (IMPACHS) into clinical treatment for psychosis. Qual Health Res. Apr 23, 2021;31(5):942-954. [CrossRef] [Medline]
- Thomas N, Farhall J, Foley F, Leitan ND, Villagonzalo KA, Ladd E, et al. Promoting personal recovery in people with persisting psychotic disorders: development and pilot study of a novel digital intervention. Front Psychiatry. 2016;7:196. [FREE Full text] [CrossRef] [Medline]
- van den Heuvel SC, Meije D, Regeer EJ, Sinnema H, Riemersma RF, Kupka RW. The user experiences and clinical outcomes of an online personal health record to support self-management of bipolar disorder: a pretest-posttest pilot study. J Affect Disord. Oct 01, 2018;238:261-268. [CrossRef] [Medline]
- Hardy A, Wojdecka A, West J, Matthews E, Golby C, Ward T, et al. How inclusive, user-centered design research can improve psychological therapies for psychosis: development of SlowMo. JMIR Ment Health. Dec 05, 2018;5(4):e11222. [FREE Full text] [CrossRef] [Medline]
- Morton E, Hole R, Murray G, Buzwell S, Michalak E. Experiences of a web-based quality of life self-monitoring tool for individuals with bipolar disorder: a qualitative exploration. JMIR Ment Health. Dec 04, 2019;6(12):e16121. [FREE Full text] [CrossRef] [Medline]
- Peters-Strickland T, Pestreich L, Hatch A, Rohatagi S, Baker RA, Docherty JP, et al. Usability of a novel digital medicine system in adults with schizophrenia treated with sensor-embedded tablets of aripiprazole. Neuropsychiatr Dis Treat. 2016;12:2587-2594. [FREE Full text] [CrossRef] [Medline]
- Bos FM, Snippe E, Bruggeman R, Doornbos B, Wichers M, van der Krieke L. Recommendations for the use of long-term experience sampling in bipolar disorder care: a qualitative study of patient and clinician experiences. Int J Bipolar Disord. Dec 01, 2020;8(1):38. [FREE Full text] [CrossRef] [Medline]
- Mueller NE, Panch T, Macias C, Cohen BM, Ongur D, Baker JT. Using smartphone apps to promote psychiatric rehabilitation in a peer-led community support program: pilot study. JMIR Ment Health. Aug 15, 2018;5(3):e10092. [FREE Full text] [CrossRef] [Medline]
- Wenze SJ, Armey MF, Weinstock LM, Gaudiano BA, Miller IW. An open trial of a smartphone-assisted, adjunctive intervention to improve treatment adherence in bipolar disorder. J Psychiatr Pract. Nov 2016;22(6):492-504. [FREE Full text] [CrossRef] [Medline]
- Bell IH, Fielding-Smith SF, Hayward M, Rossell SL, Lim MH, Farhall J, et al. Smartphone-based ecological momentary assessment and intervention in a blended coping-focused therapy for distressing voices: development and case illustration. Internet Interv. Dec 2018;14:18-25. [FREE Full text] [CrossRef] [Medline]
- Bell IH, Rossell SL, Farhall J, Hayward M, Lim MH, Fielding-Smith SF, et al. Pilot randomised controlled trial of a brief coping-focused intervention for hearing voices blended with smartphone-based ecological momentary assessment and intervention (SAVVy): feasibility, acceptability and preliminary clinical outcomes. Schizophr Res. Feb 2020;216:479-487. [CrossRef] [Medline]
- Carpenter-Song E, Jonathan G, Brian R, Ben-Zeev D. Perspectives on mobile health versus clinic-based group interventions for people with serious mental illnesses: a qualitative study. Psychiatr Serv. Jan 01, 2020;71(1):49-56. [CrossRef] [Medline]
- Moura BM, Avila A, Chendo I, Frade P, Barandas R, Vian J, et al. Facilitating the delivery of cognitive remediation in first-episode psychosis: pilot study of a home-delivered web-based intervention. J Nerv Ment Dis. Nov 2019;207(11):951-957. [CrossRef] [Medline]
- Moitra E, Gaudiano BA, Davis CH, Ben-Zeev D. Feasibility and acceptability of post-hospitalization ecological momentary assessment in patients with psychotic-spectrum disorders. Compr Psychiatry. Apr 2017;74:204-213. [FREE Full text] [CrossRef] [Medline]
- Brunette MF, Ferron JC, Gottlieb J, Devitt T, Rotondi A. Development and usability testing of a web-based smoking cessation treatment for smokers with schizophrenia. Internet Interv. May 2016;4:113-119. [FREE Full text] [CrossRef] [Medline]
- Forchuk C, Reiss JP, O'Regan T, Ethridge P, Donelle L, Rudnick A. Client perceptions of the mental health engagement network: a qualitative analysis of an electronic personal health record. BMC Psychiatry. Oct 14, 2015;15:250. [FREE Full text] [CrossRef] [Medline]
- Ainsworth J, Palmier-Claus JE, Machin M, Barrowclough C, Dunn G, Rogers A, et al. A comparison of two delivery modalities of a mobile phone-based assessment for serious mental illness: native smartphone application vs text-messaging only implementations. J Med Internet Res. Apr 05, 2013;15(4):e60. [FREE Full text] [CrossRef] [Medline]
- Prociow P, Wac K, Crowe J. Mobile psychiatry: towards improving the care for bipolar disorder. Int J Ment Health Syst. May 29, 2012;6(1):5. [FREE Full text] [CrossRef] [Medline]
- Berry A, Drake RJ, Butcher I, Yung AR. Examining the feasibility, acceptability, validity and reliability of physical activity, sedentary behaviour and sleep measures in people with schizophrenia. Ment Health Phys Activity. Oct 2021;21:100415. [CrossRef]
- Naslund JA, Aschbrenner KA, Bartels SJ. Wearable devices and smartphones for activity tracking among people with serious mental illness. Ment Health Phys Act. Mar 2016;10:10-17. [FREE Full text] [CrossRef] [Medline]
- Rodriguez-Villa E, Rozatkar AR, Kumar M, Patel V, Bondre A, Naik SS, et al. Cross cultural and global uses of a digital mental health app: results of focus groups with clinicians, patients and family members in India and the United States. Glob Ment Health (Camb). 2021;8:e30. [FREE Full text] [CrossRef] [Medline]
- Berry N, Machin M, Ainsworth J, Berry K, Edge D, Haddock G, et al. Developing a theory-informed smartphone app for early psychosis: learning points from a multidisciplinary collaboration. Front Psychiatry. 2020;11:602861. [FREE Full text] [CrossRef] [Medline]
- Laine A, Välimäki M, Pekurinen V, Löyttyniemi E, Marttunen M, Anttila M. Feasibility, acceptability, and preliminary impacts of web-based patient education on patients with schizophrenia spectrum disorder: quasi-experimental cluster study. J Med Internet Res. Oct 17, 2019;21(10):e13073. [FREE Full text] [CrossRef] [Medline]
- Kidd SA, Feldcamp L, Adler A, Kaleis L, Wang W, Vichnevetski K, et al. Feasibility and outcomes of a multi-function mobile health approach for the schizophrenia spectrum: App4Independence (A4i). PLoS One. Jul 15, 2019;14(7):e0219491. [FREE Full text] [CrossRef] [Medline]
- Lewandowski KE. Feasibility and tolerability of a cognitive remediation clinical service in first episode coordinated specialty care. Early Interv Psychiatry. Apr 08, 2021;15(2):391-396. [FREE Full text] [CrossRef] [Medline]
- Cai Y, Gong W, He H, Hughes JP, Simoni J, Xiao S, et al. Mobile texting and lay health supporters to improve schizophrenia care in a resource-poor community in rural China (LEAN Trial): randomized controlled trial extended implementation. J Med Internet Res. Dec 01, 2020;22(12):e22631. [FREE Full text] [CrossRef] [Medline]
- Välimäki M, Kannisto KA, Vahlberg T, Hätönen H, Adams CE. Short text messages to encourage adherence to medication and follow-up for people with psychosis (Mobile.Net): randomized controlled trial in Finland. J Med Internet Res. Jul 12, 2017;19(7):e245. [FREE Full text] [CrossRef] [Medline]
- Xu DR, Xiao S, He H, Caine ED, Gloyd S, Simoni J, et al. Lay health supporters aided by mobile text messaging to improve adherence, symptoms, and functioning among people with schizophrenia in a resource-poor community in rural China (LEAN): a randomized controlled trial. PLoS Med. Apr 2019;16(4):e1002785. [FREE Full text] [CrossRef] [Medline]
- Camacho E, Torous J. Interest and readiness for digital mental health in coordinate specialty care for early course psychosis: a survey study of 42 programs in 30 states. Early Interv Psychiatry. Oct 2021;15(5):1243-1255. [CrossRef] [Medline]
- McClelland GT, Fitzgerald M. A participatory mobile application (app) development project with mental health service users and clinicians. Health Educ J. Jun 05, 2018;77(7):815-827. [CrossRef]
- Mistler LA, Ben-Zeev D, Carpenter-Song E, Brunette MF, Friedman MJ. Mobile mindfulness intervention on an acute psychiatric unit: feasibility and acceptability study. JMIR Ment Health. Aug 21, 2017;4(3):e34. [FREE Full text] [CrossRef] [Medline]
- Williams A, Fossey E, Farhall J, Foley F, Thomas N. Impact of jointly using an e-mental health resource (self-management and recovery technology) on interactions between service users experiencing severe mental illness and community mental health workers: grounded theory study. JMIR Ment Health. Jun 16, 2021;8(6):e25998. [FREE Full text] [CrossRef] [Medline]
- Kreyenbuhl J, Record EJ, Himelhoch S, Charlotte M, Palmer-Bacon J, Dixon LB, et al. Development and feasibility testing of a smartphone intervention to improve adherence to antipsychotic medications. Clin Schizophr Relat Psychoses. 2019;12(4):152-167. [FREE Full text] [CrossRef] [Medline]
- Enrique A, Duffy D, Lawler K, Richards D, Jones S. An internet-delivered self-management programme for bipolar disorder in mental health services in Ireland: results and learnings from a feasibility trial. Clin Psychol Psychother. Nov 2020;27(6):925-939. [FREE Full text] [CrossRef] [Medline]
- Forchuk C, Donelle L, Ethridge P, Warner L. Client perceptions of the mental health engagement network: a secondary analysis of an intervention using smartphones and desktop devices for individuals experiencing mood or psychotic disorders in Canada. JMIR Ment Health. Jan 21, 2015;2(1):e1. [FREE Full text] [CrossRef] [Medline]
- Nicholas J, Proudfoot J, Parker G, Gillis I, Burckhardt R, Manicavasagar V, et al. The ins and outs of an online bipolar education program: a study of program attrition. J Med Internet Res. Dec 19, 2010;12(5):e57. [FREE Full text] [CrossRef] [Medline]
- Todd NJ, Jones SH, Lobban FA. What do service users with bipolar disorder want from a web-based self-management intervention? A qualitative focus group study. Clin Psychol Psychother. Jun 20, 2013;20(6):531-543. [CrossRef] [Medline]
- Vilardaga R, Rizo J, Kientz JA, McDonell MG, Ries RK, Sobel K. User experience evaluation of a smoking cessation app in people with serious mental illness. Nicotine Tob Res. May 2016;18(5):1032-1038. [FREE Full text] [CrossRef] [Medline]
- Brunette MF, Ferron JC, Geiger P, Guarino S, Pratt SI, Lord SE, et al. Pilot study of a mobile smoking cessation intervention for low-income smokers with serious mental illness. J Smok Cessat. Jun 05, 2019;14(4):203-210. [CrossRef]
- Hidalgo-Mazzei D, Reinares M, Mateu A, Juruena MF, Young AH, Pérez-Sola V, et al. Is a SIMPLe smartphone application capable of improving biological rhythms in bipolar disorder? J Affect Disord. Dec 01, 2017;223:10-16. [CrossRef] [Medline]
- Eisner E, Bucci S, Berry N, Emsley R, Barrowclough C, Drake RJ. Feasibility of using a smartphone app to assess early signs, basic symptoms and psychotic symptoms over six months: a preliminary report. Schizophr Res. Jun 2019;208:105-113. [FREE Full text] [CrossRef] [Medline]
- Faurholt-Jepsen M, Frost M, Christensen EM, Bardram JE, Vinberg M, Kessing LV. Validity and characteristics of patient-evaluated adherence to medication via smartphones in patients with bipolar disorder: exploratory reanalyses on pooled data from the MONARCA I and II trials. Evid Based Ment Health. Feb 2020;23(1):2-7. [FREE Full text] [CrossRef] [Medline]
- Lopez-Morinigo JD, Barrigón ML, Porras-Segovia A, Ruiz-Ruano VG, Escribano Martínez AS, Escobedo-Aedo PJ, et al. Use of ecological momentary assessment through a passive smartphone-based app (eB2) by patients with schizophrenia: acceptability study. J Med Internet Res. Jul 26, 2021;23(7):e26548. [FREE Full text] [CrossRef] [Medline]
- Thonon B, Levaux MN, van Aubel E, Larøi F. A group intervention for motivational deficits: preliminary investigation of a blended care approach using ambulatory assessment. Behav Modif. Sep 2022;46(5):1167-1197. [FREE Full text] [CrossRef] [Medline]
- Luther L, Fischer MW, Johnson-Kwochka AV, Minor KS, Holden R, Lapish CL, et al. Mobile enhancement of motivation in schizophrenia: a pilot randomized controlled trial of a personalized text message intervention for motivation deficits. J Consult Clin Psychol. Oct 2020;88(10):923-936. [FREE Full text] [CrossRef] [Medline]
- Gordon-Smith K, Saunders K, Geddes JR, Harrison PJ, Hinds C, Craddock N, et al. Large-scale roll out of electronic longitudinal mood-monitoring for research in affective disorders: report from the UK bipolar disorder research network. J Affect Disord. Mar 01, 2019;246:789-793. [CrossRef] [Medline]
- Ferreira-Correia A, Barberis T, Msimanga L. Barriers to the implementation of a computer-based rehabilitation programme in two public psychiatric settings. S Afr J Psychiatr. 2018;24:1163. [FREE Full text] [CrossRef] [Medline]
- Arnold C, Villagonzalo KA, Meyer D, Farhall J, Foley F, Kyrios M, et al. Predicting engagement with an online psychosocial intervention for psychosis: exploring individual- and intervention-level predictors. Internet Interv. Dec 2019;18:100266. [FREE Full text] [CrossRef] [Medline]
- Proudfoot J, Parker G, Manicavasagar V, Hadzi-Pavlovic D, Whitton A, Nicholas J, et al. Effects of adjunctive peer support on perceptions of illness control and understanding in an online psychoeducation program for bipolar disorder: a randomised controlled trial. J Affect Disord. Dec 15, 2012;142(1-3):98-105. [CrossRef] [Medline]
- Simon GE, Ludman EJ, Goodale LC, Dykstra DM, Stone E, Cutsogeorge D, et al. An online recovery plan program: can peer coaching increase participation? Psychiatr Serv. Jun 2011;62(6):666-669. [FREE Full text] [CrossRef] [Medline]
- Van Til K, McInnis MG, Cochran A. A comparative study of engagement in mobile and wearable health monitoring for bipolar disorder. Bipolar Disord. Mar 25, 2020;22(2):182-190. [FREE Full text] [CrossRef] [Medline]
- Swartz HA, Rollman BL, Mohr DC, Sadow S, Frank E. A randomized pilot study of Rhythms And You (RAY): an internet-based program for bipolar disorder administered with and without clinical helper support in primary care. J Affect Disord. Dec 01, 2021;295:183-191. [FREE Full text] [CrossRef] [Medline]
- Harris AW, Kosic T, Xu J, Walker C, Gye W, Redoblado Hodge A. Web-based cognitive remediation improves supported employment outcomes in severe mental illness: randomized controlled trial. JMIR Ment Health. Sep 20, 2017;4(3):e30. [FREE Full text] [CrossRef] [Medline]
- Bucci S, Ainsworth J, Barrowclough C, Lewis S, Haddock G, Berry K, et al. A theory-informed digital health intervention in people with severe mental health problems. Stud Health Technol Inform. Aug 21, 2019;264:526-530. [CrossRef] [Medline]
- Ben-Zeev D, Brenner CJ, Begale M, Duffecy J, Mohr DC, Mueser KT. Feasibility, acceptability, and preliminary efficacy of a smartphone intervention for schizophrenia. Schizophr Bull. Nov 2014;40(6):1244-1253. [FREE Full text] [CrossRef] [Medline]
- Steare T, O'Hanlon P, Eskinazi M, Osborn D, Lloyd-Evans B, Jones R, et al. Smartphone-delivered self-management for first-episode psychosis: the ARIES feasibility randomised controlled trial. BMJ Open. Aug 26, 2020;10(8):e034927. [FREE Full text] [CrossRef] [Medline]
- Lim MH, Gleeson JF, Rodebaugh TL, Eres R, Long KM, Casey K, et al. A pilot digital intervention targeting loneliness in young people with psychosis. Soc Psychiatry Psychiatr Epidemiol. Jul 2020;55(7):877-889. [CrossRef] [Medline]
- Cho CH, Lee T, Lee JB, Seo JY, Jee HJ, Son S, et al. Effectiveness of a smartphone app with a wearable activity tracker in preventing the recurrence of mood disorders: prospective case-control study. JMIR Ment Health. Aug 05, 2020;7(8):e21283. [FREE Full text] [CrossRef] [Medline]
- Hanssen E, Balvert S, Oorschot M, Borkelmans K, van Os J, Delespaul P, et al. An ecological momentary intervention incorporating personalised feedback to improve symptoms and social functioning in schizophrenia spectrum disorders. Psychiatry Res. Feb 2020;284:112695. [FREE Full text] [CrossRef] [Medline]
- Meyer N, Kerz M, Folarin A, Joyce DW, Jackson R, Karr C, et al. Capturing rest-activity profiles in schizophrenia using wearable and mobile technologies: development, implementation, feasibility, and acceptability of a remote monitoring platform. JMIR Mhealth Uhealth. Oct 30, 2018;6(10):e188. [FREE Full text] [CrossRef] [Medline]
- Hidalgo-Mazzei D, Mateu A, Reinares M, Murru A, Del Mar Bonnín C, Varo C, et al. Psychoeducation in bipolar disorder with a SIMPLe smartphone application: feasibility, acceptability and satisfaction. J Affect Disord. Aug 2016;200:58-66. [CrossRef] [Medline]
- Barnes E, Simpson S, Griffiths E, Hood K, Craddock N, Smith DJ. Developing an online psychoeducation package for bipolar disorder. J Ment Health. Feb 2011;20(1):21-31. [CrossRef] [Medline]
- Bopp JM, Miklowitz DJ, Goodwin GM, Stevens W, Rendell JM, Geddes JR. The longitudinal course of bipolar disorder as revealed through weekly text messaging: a feasibility study. Bipolar Disord. May 2010;12(3):327-334. [FREE Full text] [CrossRef] [Medline]
- McKnight RF, Bilderbeck AC, Miklowitz DJ, Hinds C, Goodwin GM, Geddes JR. Longitudinal mood monitoring in bipolar disorder: course of illness as revealed through a short messaging service. J Affect Disord. Dec 01, 2017;223:139-145. [CrossRef] [Medline]
- Thomas N, Foley F, Lindblom K, Lee S. Are people with severe mental illness ready for online interventions? Access and use of the internet in Australian mental health service users. Australas Psychiatry. Jun 2017;25(3):257-261. [CrossRef] [Medline]
- Ben-Zeev D, Kaiser SM, Brenner CJ, Begale M, Duffecy J, Mohr DC. Development and usability testing of FOCUS: a smartphone system for self-management of schizophrenia. Psychiatr Rehabil J. Dec 2013;36(4):289-296. [FREE Full text] [CrossRef] [Medline]
- Ferron JC, Brunette MF, McGurk SR, Xie H, Frounfelker R, Cook JA, et al. Do symptoms and cognitive problems affect the use and efficacy of a web-based decision support system for smokers with serious mental illness? J Dual Diagn. Nov 2012;8(4):315-325. [CrossRef]
- Bowden CL, Priesmeyer R, Tohen M, Singh V, Calabrese JR, Ketter T, et al. Development of a patient-centered software system to facilitate effective management of bipolar disorder. Psychopharmacol Bull. Mar 16, 2021;51(2):8-19. [FREE Full text] [Medline]
- Ben-Zeev D, Wang R, Abdullah S, Brian R, Scherer EA, Mistler LA, et al. Mobile behavioral sensing for outpatients and inpatients with schizophrenia. Psychiatr Serv. May 01, 2016;67(5):558-561. [FREE Full text] [CrossRef] [Medline]
- Torous J, Wisniewski H, Liu G, Keshavan M. Mental health mobile phone app usage, concerns, and benefits among psychiatric outpatients: comparative survey study. JMIR Ment Health. Nov 16, 2018;5(4):e11715. [FREE Full text] [CrossRef] [Medline]
- Röhricht F, Padmanabhan R, Binfield P, Mavji D, Barlow S. Simple mobile technology health management tool for people with severe mental illness: a randomised controlled feasibility trial. BMC Psychiatry. Jul 16, 2021;21(1):357. [FREE Full text] [CrossRef] [Medline]
- Bos FM, Snippe E, Bruggeman R, Doornbos B, Wichers M, van der Krieke L. Recommendations for the use of long-term experience sampling in bipolar disorder care: a qualitative study of patient and clinician experiences. Int J Bipolar Disord. Dec 01, 2020;8(1):38. [FREE Full text] [CrossRef] [Medline]
- Davies MJ, Gagliardino JJ, Gray LJ, Khunti K, Mohan V, Hughes R. Real-world factors affecting adherence to insulin therapy in patients with type 1 or type 2 diabetes mellitus: a systematic review. Diabet Med. May 2013;30(5):512-524. [CrossRef] [Medline]
- Granja C, Janssen W, Johansen MA. Factors determining the success and failure of eHealth interventions: systematic review of the literature. J Med Internet Res. May 01, 2018;20(5):e10235. [FREE Full text] [CrossRef] [Medline]
- Killikelly C, He Z, Reeder C, Wykes T. Improving adherence to web-based and mobile technologies for people with psychosis: systematic review of new potential predictors of adherence. JMIR Mhealth Uhealth. Jul 20, 2017;5(7):e94. [FREE Full text] [CrossRef] [Medline]
- Patel S, Akhtar A, Malins S, Wright N, Rowley E, Young E, et al. The acceptability and usability of digital health interventions for adults with depression, anxiety, and somatoform disorders: qualitative systematic review and meta-synthesis. J Med Internet Res. Jul 06, 2020;22(7):e16228. [FREE Full text] [CrossRef] [Medline]
- Vis C, Mol M, Kleiboer A, Bührmann L, Finch T, Smit J, et al. Improving implementation of eMental health for mood disorders in routine practice: systematic review of barriers and facilitating factors. JMIR Ment Health. Mar 16, 2018;5(1):e20. [FREE Full text] [CrossRef] [Medline]
- Titov N, Hadjistavropoulos HD, Nielssen O, Mohr DC, Andersson G, Dear BF. From research to practice: ten lessons in delivering digital mental health services. J Clin Med. Aug 17, 2019;8(8):1239. [FREE Full text] [CrossRef] [Medline]
- Connolly SL, Kuhn E, Possemato K, Torous J. Digital clinics and mobile technology implementation for mental health care. Curr Psychiatry Rep. May 07, 2021;23(7):38. [FREE Full text] [CrossRef] [Medline]
- Leung C, Pei J, Hudec K, Shams F, Munthali R, Vigo D. The effects of nonclinician guidance on effectiveness and process outcomes in digital mental health interventions: systematic review and meta-analysis. J Med Internet Res. Jun 15, 2022;24(6):e36004. [FREE Full text] [CrossRef] [Medline]
- Ganapathy A, Clough BA, Casey LM. Organizational and policy barriers to the use of digital mental health by mental health professionals. Telemed J E Health. Dec 01, 2021;27(12):1332-1343. [CrossRef] [Medline]
- Mohr DC, Burns MN, Schueller SM, Clarke G, Klinkman M. Behavioral intervention technologies: evidence review and recommendations for future research in mental health. Gen Hosp Psychiatry. 2013;35(4):332-338. [FREE Full text] [CrossRef] [Medline]
- Ben-Zeev D, Razzano LA, Pashka NJ, Levin CE. Cost of mHealth versus clinic-based care for serious mental illness: same effects, half the price tag. Psychiatr Serv. Apr 01, 2021;72(4):448-451. [CrossRef] [Medline]
- Iribarren SJ, Cato K, Falzon L, Stone PW. What is the economic evidence for mHealth? A systematic review of economic evaluations of mHealth solutions. PLoS One. 2017;12(2):e0170581. [FREE Full text] [CrossRef] [Medline]
- Spanakis P, Wadman R, Walker L, Heron P, Mathers A, Baker J, et al. Measuring the digital divide among people with severe mental ill health using the essential digital skills framework. Perspect Public Health. Jan 05, 2024;144(1):21-30. [FREE Full text] [CrossRef] [Medline]
- Spanakis P, Peckham E, Mathers A, Shiers D, Gilbody S. The digital divide: amplifying health inequalities for people with severe mental illness in the time of COVID-19. Br J Psychiatry. Oct 23, 2021;219(4):529-531. [FREE Full text] [CrossRef] [Medline]
- Hill C, Martin JL, Thomson S, Scott-Ram N, Penfold H, Creswell C. Navigating the challenges of digital health innovation: considerations and solutions in developing online and smartphone-application-based interventions for mental health disorders. Br J Psychiatry. Aug 2017;211(2):65-69. [CrossRef] [Medline]
- Allan S, Ward T, Eisner E, Bell IH, Cella M, Chaudhry IB, et al. Adverse events reporting in digital interventions evaluations for psychosis: a systematic literature search and individual level content analysis of adverse event reports. Schizophr Bull. Nov 08, 2024;50(6):1436-1455. [CrossRef] [Medline]
- Eisner E, Richardson C, Thomas N, Rus-Calafell M, Syrett S, Firth J, et al. Measurement of adverse events in studies of digital health interventions for psychosis: guidance and recommendations based on a literature search and framework analysis of standard operating procedures. Schizophr Bull. Nov 08, 2024;50(6):1456-1470. [CrossRef] [Medline]
- Berry N, Lobban F, Emsley R, Bucci S. Acceptability of interventions delivered online and through mobile phones for people who experience severe mental health problems: a systematic review. J Med Internet Res. May 31, 2016;18(5):e121. [FREE Full text] [CrossRef] [Medline]
Abbreviations
| AE: adverse event |
| cCBT: computerized cognitive behavior therapy |
| CFIR: Consolidated Framework for Implementation Research |
| CONSORT: Consolidated Standards of Reporting Trials |
| DMHI: digital mental health intervention |
| PRISMA: Preferred Reporting Items for Systematic Reviews and Meta-Analyses |
| RCT: randomized controlled trial |
Edited by J Torous; submitted 09.08.24; peer-reviewed by C Leung, I Starvaggi; comments to author 21.09.24; revised version received 02.10.24; accepted 13.10.24; published 20.01.25.
Copyright©Emily Eisner, Sophie Faulkner, Stephanie Allan, Hannah Ball, Daniela Di Basilio, Jennifer Nicholas, Aansha Priyam, Paul Wilson, Xiaolong Zhang, Sandra Bucci. Originally published in JMIR Mental Health (https://mental.jmir.org), 20.01.2025.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Mental Health, is properly cited. The complete bibliographic information, a link to the original publication on https://mental.jmir.org/, as well as this copyright and license information must be included.