Published on in Vol 11 (2024)

This is a member publication of University of Oxford (Jisc)

Preprints (earlier versions) of this paper are available at, first published .
Digital Mental Health for Schizophrenia and Other Severe Mental Illnesses: An International Consensus on Current Challenges and Potential Solutions

Digital Mental Health for Schizophrenia and Other Severe Mental Illnesses: An International Consensus on Current Challenges and Potential Solutions

Digital Mental Health for Schizophrenia and Other Severe Mental Illnesses: An International Consensus on Current Challenges and Potential Solutions

Original Paper

1Department of Psychiatry, University of Oxford, Oxford, United Kingdom

2Oxford Health NHS Foundation Trust, Warneford Hospital, Oxford, United Kingdom

3Oxford Precision Psychiatry Lab, NIHR Oxford Health Biomedical Research Centre, Oxford, United Kingdom

4Department of Psychology, Institute of Psychiatry, Psychology and Neuroscience, King’s College London, London, United Kingdom

5South London & Maudsley NHS Foundation Trust, London, United Kingdom

6MQ Mental Health Research, London, United Kingdom

7Participatory eHealth and Health Data Research Group, Department of Women's and Children's Health, Uppsala University, Uppsala, Sweden

8Division of Digital Psychiatry, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, MA, United States

9Department of Psychiatry and Psychology, Institute of Neuroscience, Hospital Clínic de Barcelona, Barcelona, Spain

10Bipolar and Depressive Disorders Unit, Digital Innovation Group, Institut d’Investigacions Biomèdiques August Pi i Sunyer (IDIBAPS), Barcelona, Spain

11Centro de Investigación Biomédica en Red de Salud Mental (CIBERSAM), Instituto de Salud Carlos III, Madrid, Spain

12Department of Medicine, School of Medicine and Health Sciences, Institute of Neurosciences, University of Barcelona, Barcelona, Spain

13Department of Experimental Psychology, University of Oxford, Oxford, United Kingdom

14School of Science and Technology, Middlesex University, London, United Kingdom

15Institute of Neuroscience and Psychology, University of Glasgow, Glasgow, United Kingdom

16Department of Child and Adolescent Psychiatry, Charité – Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany

17Psychiatry Unit, Department of Neurosciences and Mental Health, Ospedale Maggiore Policlinico Ca’ Granda, Fondazione Istituto di Ricovero e Cura a Carattere Scientifico, Milan, Italy

18Department of Pathophysiology and Transplantation, University of Milan, Milan, Italy


*these authors contributed equally

Corresponding Author:

Katharine A Smith, MD, PhD

Department of Psychiatry

University of Oxford

Warneford Hospital

Warneford Lane

Oxford, OX3 7JX

United Kingdom

Phone: 44 1865 902135


Background: Digital approaches may be helpful in augmenting care to address unmet mental health needs, particularly for schizophrenia and severe mental illness (SMI).

Objective: An international multidisciplinary group was convened to reach a consensus on the challenges and potential solutions regarding collecting data, delivering treatment, and the ethical challenges in digital mental health approaches for schizophrenia and SMI.

Methods: The consensus development panel method was used, with an in-person meeting of 2 groups: the expert group and the panel. Membership was multidisciplinary including those with lived experience, with equal participation at all stages and coproduction of the consensus outputs and summary. Relevant literature was shared in advance of the meeting, and a systematic search of the recent literature on digital mental health interventions for schizophrenia and psychosis was completed to ensure that the panel was informed before the meeting with the expert group.

Results: Four broad areas of challenge and proposed solutions were identified: (1) user involvement for real coproduction; (2) new approaches to methodology in digital mental health, including agreed standards, data sharing, measuring harms, prevention strategies, and mechanistic research; (3) regulation and funding issues; and (4) implementation in real-world settings (including multidisciplinary collaboration, training, augmenting existing service provision, and social and population-focused approaches). Examples are provided with more detail on human-centered research design, lived experience perspectives, and biomedical ethics in digital mental health approaches for SMI.

Conclusions: The group agreed by consensus on a number of recommendations: (1) a new and improved approach to digital mental health research (with agreed reporting standards, data sharing, and shared protocols), (2) equal emphasis on social and population research as well as biological and psychological approaches, (3) meaningful collaborations across varied disciplines that have previously not worked closely together, (4) increased focus on the business model and product with planning and new funding structures across the whole development pathway, (5) increased focus and reporting on ethical issues and potential harms, and (6) organizational changes to allow for true communication and coproduction with those with lived experience of SMI. This study approach, combining an international expert meeting with patient and public involvement and engagement throughout the process, consensus methodology, discussion, and publication, is a helpful way to identify directions for future research and clinical implementation in rapidly evolving areas and can be combined with measurements of real-world clinical impact over time. Similar initiatives will be helpful in other areas of digital mental health and similarly fast-evolving fields to focus research and organizational change and effect improved real-world clinical implementation.

JMIR Ment Health 2024;11:e57155




Addressing the shortfall in service provision in mental health is a key challenge [1] that has been brought into even sharper focus in the aftermath of the COVID-19 pandemic. The pandemic showed that a shift to digital platforms to deliver synchronous mental health services could be rapidly implemented [2] and can be an acceptable format for many clinicians, patients, and carers [3,4]. However, asynchronous digital approaches (eg, measuring symptoms using digital phenotyping or ecological momentary assessment or providing partially automated therapies using digital platforms) show even more potential to increase capacity and outcomes [5]. These innovations allow patients to undertake a variety of clinically relevant tasks (such as self-monitoring or therapy tasks) outside the in-person clinical encounter. This can be completed independently or with the support of digital navigators or technicians [6] and could reduce the need for specialist clinician support. However, despite their potential, these approaches are mostly still in development and have often proved to be challenging to implement in real-world clinical and community settings [6].

An additional challenge for mental health services is to ensure that the potential benefits of digital approaches are applied in the areas of greatest need, particularly because the pandemic has worsened preexisting disparities in mental health care [7]. Increasing access to care for people with severe mental illnesses (SMIs) such as psychosis and bipolar disorder is a priority worldwide for mental health, particularly in low- and middle-income countries (LMICs) [8]. Although people with SMI experience major health inequalities and have a life expectancy 10 to 25 years shorter than that of the general population [9,10], less research on digital interventions has been conducted in this patient group compared with other conditions [11]. Assumptions that people with SMI will not be able or willing to engage in technology-augmented assessment and treatment have combined with the known element of digital exclusion due to the intersection of SMI and socioeconomic inequities, resulting in a shortfall of development in this area [12,13].

Internet access among people with SMI is increasing [14,15], and ownership of smartphones is now more common than ownership of computers [16]. However, there are still barriers to digital access and digital literacy in this group [17,18]. Even with a device, many people with SMI have insufficient economic resources to maintain consistent access or may lack the confidence or skills to use it to its full ability [12,18,19]. Although there are examples of training programs in digital skills and confidence for people with SMI [20,21] and evidence that they are willing and able to engage effectively with digital mental health [22-25], this is often a neglected area [11].

Study Objectives

In this study, we used a consensus method to identify the challenges and potential solutions in the development and implementation of digital mental health in the care pathways of people with schizophrenia and other SMIs, and consider how new research could help fill the gap between need and service provision to improve patients’ outcomes. To explore this, we convened an expert international multidisciplinary group to focus specifically on the complexities of collecting data, delivering treatment, and the ethical challenges in this area. We focused on SMI as a group where there is arguably the greatest need. While SMI is a broad term covering mental illnesses causing serious functional impairment, we focused primarily on evidence and examples of digital mental health interventions for schizophrenia and psychosis, and related risks such as suicide.


We used the consensus development panel (or consensus development conference [CDC]) approach [26,27] and followed the methodology described and used by the US National Institutes of Health [28] and the World Health Organization [26,29]. The CDCs were developed by the National Institutes of Health [26] and we chose this as it is a particularly effective consensus method for identifying broad areas of challenge and potential solutions. This is in contrast to alternative consensus approaches (such as the Delphi or nominal group techniques) that aim to achieve specific criteria or protocols [30]. Therefore, the CDC is a particularly relevant method for a rapidly developing area such as digital mental health [6]. In addition, it enables a multidisciplinary approach and can moderate potential bias from a group of individual experts using several strategies, such as the inclusion of a separate panel of nonexpert participants (hereafter, “the panel”).

Central to the methodology of the CDC is a face-to-face meeting between the expert group and the panel involving an interactive method to develop a consensus. Panel members are provided with evidence by the expert group. The panel members ask questions to clarify and then deliberate on the issue directed by their chairperson in the process to reach a consensus [26].

In this study, the panel also increased their knowledge of the field in advance of the meeting by conducting a literature review using PubMed to search for terms relevant to the main themes identified by the experts (see Textbox 1 and Figure 1 for details). This preliminary work identified the areas of recent development, uncertainties, or challenges that formed the agenda for the questions to be addressed in the face-to-face meeting.

Textbox 1. Literature review.


  • The panel members conducted a literature review of the records published over the last 5 years (from January 1, 2018) on PubMed, and searched for articles relevant to the themes identified by the expert group in the area of digital technologies and severe mental illness (SMI; focusing for this review on schizophrenia and psychosis).
  • The seven identified topics suggested by the experts were (1) digital markers and personalized interventions, (2) patient and public involvement and engagement perspectives on SMI, (3) digital technologies and SMI, (4) virtual reality approaches for SMI, (5) technology and mobile health for suicide prevention, (6) digital approaches to empathy in clinician-patient interactions, and (7) the use of web-based screening to detect emerging psychosis.
  • The panel used a broad search strategy for papers relevant to schizophrenia, psychosis, and digital mental health. The search was conducted on September 17, 2023, using the following search strategy: (“Schizophrenia Spectrum and Other Psychotic Disorders” [MeSH Terms] OR (“schizophreni*”[Title/Abstract] OR “psychos*”[Title/Abstract] OR “psychotic”[Title/Abstract])) AND (“digital*”[Title/Abstract] OR “smartphone*”[Title/Abstract] OR “mobile*”[Title/Abstract] OR “virtual*”[Title/Abstract] OR “internet*”[Title/Abstract]) AND 2018/01/01:3000/12/31[Date – Publication].
  • The screening process was completed independently by the 7 panelists and any queries were resolved through team discussion. Papers were eligible for inclusion if they were relevant to schizophrenia or psychosis and digital mental health. The search resulted in 3517 records, 3170 (90.13%) of which were excluded after title and abstract screening.
  • At the full-text screening, 125 records met the inclusion criteria. Of these 125 records, 45 (36%) were further selected as essential reading by the panel and shared with all the panelists, including a selection of up to 5 articles suggested by each member of the expert group. The remaining 64% (80/125) of the records were listed as supplementary reading material for consultation purposes (see the PRISMA [Preferred Reporting Items for Systematic Reviews and Meta-Analyses] flow diagram [31] in Figure 1).
Figure 1. PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flowchart. *Two of the records were each allocated to two topics as they were relevant to both.

The Consensus Meeting

The meeting was held in Rome over 2 days in October 2023. The meeting was facilitated by JT and AC who led the expert group, and by the chair of the panel (KAS) who also recorded and summarized the meeting. Each expert gave a brief presentation, including shared slides, methodology, analysis of data, and relevant citations, followed by a whole-group discussion led by a different member of the panel for each talk. At the end of each day a summary was prepared in discussion with the whole group, and the structure of the consensus (challenges and potential solutions) was agreed upon at the end of the second day. Consensus was defined as either fully met or not met at all, with the outcome being transparently reported [32,33]. At the end of the meeting, the whole group engaged in plenary discussion to identify the key themes and structure the results.

The Expert Group

The 9 experts (AC, AH, CB, DH, SL, L Milligan, L Marzano, JT, and PU) encompassed expertise in a variety of specialist areas within SMI and digital mental health (including virtual reality, coproduction and co-design, suicide prevention, web-based screening and early intervention, digital approaches to empathy and the therapeutic relationship, ethical issues, and lived experience). The group composition was gender-balanced, and professional backgrounds included psychiatry, psychology, methodology, evidence synthesis, patient and public involvement (PPI), and ethics. The expert group was international (including members from Germany, Italy, Ireland, Spain, Sweden, the United Kingdom, and the United States). In advance of the meeting the experts were asked to provide information on their area of expertise as a short abstract supported by up to 5 references that they considered to be key in the last 2 to 3 years, and this was circulated to the expert group and panel. This was supplemented by a systematic search of the literature conducted by the panel (Textbox 1).

The Panel

The panel was composed of 7 members (GA, RA, BD, EGO, KAS, AV, and CZ), including early-career and more experienced clinicians and researchers (at a different level of expertise) and an individual with lived experience. The panel members were chosen because they were well informed or experienced in the field of SMI but had no particular expertise in any one area of digital psychiatry. The panel was also international (including members from Italy, Paraguay, Spain, the United Kingdom, and the United States).

In preparation for the meeting, the panel members summarized the best available recent evidence on the 7 topics to be discussed. This was achieved by completing a systematic search of the recent literature in the areas to be covered by the meeting (see Textbox 1 and Figure 1 for details). All panelists were asked to read the selected papers, and each panel member led the group discussion on one expert talk to facilitate equal contributions from members of the expert group and the panel.

Reflexivity Statement

The meeting was convened by JT and AC, who selected the expert group to represent a balance of professional backgrounds, areas of specialist digital mental health expertise, lived experience, and gender. Panel members were suggested by members of the expert group and through professional contacts. JT and AC were assisted by KAS in the organization and preparation of the meeting. The logistics of the meeting were supported externally by Angelini Pharma, but they did not have any input in the design of the meeting, identification or selection of the expert group or panel, agenda of the meeting, discussions, consensus, or output. We acknowledge that the shared knowledge and experiences of the expert group and panel may have had an impact on the interpretation of the data.

Ethical Considerations

Ethical approval was not sought for this study as it did not involve research on human participants. The consensus was conference based, and all attendees offered contributions to the research topic in an open environment where talks were voluntary. No personal details were solicited or reported, and all presentations were based on expertise, including the lived experience expert coinvestigator who spoke in her capacity as an expert in how patients with SMI navigate health systems.


The group achieved full consensus on all the outputs of the meeting, identified 4 broad themes, and divided each consensus theme into the current challenges and potential solutions, with examples from existing projects for SMI. A summary of the results is presented in Textbox 2.

Textbox 2. Summary of consensus themes, challenges, and potential solutions.

Theme 1: user involvement for real coproduction


  • User engagement:
    • Most apps are discarded within days
    • There is a lack of consensus or standards for assessing and monitoring engagement and attrition
    • Most of the research and development of digital tools focuses on conditions outside severe mental illness and is conducted and published in higher-income countries (not low- and middle-income countries)
    • Interventions need to be at a younger age and across the life span

Potential solutions:

  • Patient and public involvement and active engagement in human-centered design:
    • True representation of diverse populations
    • Coproduction
    • Patient and public involvement and engagement which is integral within the study
  • Personalized approaches
  • Early interventions

Theme 2: research methodology


  • There is a lack of validated and agreed standards for digital biomarkers or behavioral data

Potential solutions:

  • Agreed standards for digital mental health research
  • Transparent data processing pipelines and data sharing
  • Measuring harms
  • A focus on prevention and longer-term strategies
  • Mechanistic research in digital mental health interventions

Theme 3: regulation, governance, and funding


  • Achieving sustainable funding
  • Unstable regulation

Potential solutions:

  • Sustainability planning in funding calls and applications
  • Governance change
  • Guidelines for best practice and research standards

Theme 4: implementation in real-world clinical settings


  • Translating research findings into the clinical setting

Potential solutions:

  • Collaboration between multidisciplinary groups
  • Training and assessment in clinics

Theme 1: User Involvement for Real Coproduction

Current Challenges
User Engagement

User engagement is a key issue as it represents a potential mechanism of change for improving clinical outcomes in digital health interventions [34,35]. Even in the general population, most apps are discarded within days of first use [36], and recently some have labeled engagement as the “Achilles Heel” of digital therapeutics for mental health, with attrition also of concern [37]. An additional complication is that engagement and adherence are often conflated; participants may disengage for multiple reasons (including the attainment of their goals rather than lack of adherence). The concept of optimal use may be a more representative indicator but needs to be defined a priori in research studies [38]. However, there is also a lack of consensus or standards for assessing and monitoring engagement with and attrition from digital tools, limiting progress in the field [39]. In addition, most of the research on and development of digital tools focuses on conditions outside SMI [11]. Digital technologies, particularly smartphone apps, have the potential to be novel tools for managing SMIs, especially in LMICs [40]. However, most research studies in this area are conducted and published in higher-income countries, and samples may not be representative of the wider SMI population [41]. Individual reports from researchers in LMICs suggest that engagement is also challenging in these settings [42].

Interventions at a Younger Age and Across the Life Span

Most mental disorders begin during adolescence [43] and are frequently preceded by subthreshold symptoms, suggesting that this period is critical for early intervention and diagnosis [44]. While this age group may be particularly suited to digital interventions (eg, 46% of individuals aged 13-17 years in the United States reported “almost constant” use of social networking services [45]), effective digital interventions may need to be targeted earlier than in previous studies and tailored to the needs of young people [46,47]. Youth is not the only stage of the life span in which focus is needed. In general, older people are most likely to be digitally excluded, and enduring or treatment-resistant psychosis has received less focus, funding, and innovation for digital approaches [18].

Potential Solutions
PPI and Engagement in Human-Centered Design

A potential solution to the challenges of user engagement with digital tools is to actively involve patients and participants in the design and development of digital tools and all stages of the research process (see Textbox 3 for further information from a lived experience perspective).

Not only does this active engagement and involvement result in research that is more relevant and useful for users, but it also meets the essential rights of users to be included in the development of interventions that will affect their lives and those closest to them [49]. However, even when users have been included, there is often a lack of true representation of diverse populations [50], resulting in digital solutions that do not provide sufficient options for different individual preferences and life circumstances. PPI and engagement (PPIE) within digital development and research studies can be highly variable, and although the aim is coproduction, for true design with and not for participants, PPIE needs to be integral within the structures, policies, and processes of the study team so that power is shared [51-54]. This will only occur by building on the existing methods for public engagement using innovative new models encompassing the breadth of patient voices alongside industry, regulators, and academics [55,56].

While “ticking the box” of PPIE is relatively easy, PPIE that is truly representative of the target population and integrated into all phases of the design and research process is much more challenging and therefore costly. However, there are examples of integration of a wider range of patient populations throughout the design, prototype, and evaluation phases (eg, SlowMo and gameChange [see Textbox 4 for details] and mindLAMP [57]) and examples of global applications across LMICs as well as higher-income countries [58]. Textbox 4 provides some more details on 2 examples.

Consideration of the specific characteristics of involved PPIE members is also important. In the move toward an improved user-centered PPIE involvement, researchers should consider involving a broader range of patient experiences. PPIE involvement in product and study development often comes from “expert” PPIE members. While this is helpful (“expert” PPIE members have often had experience with other interventions and can contribute specialist knowledge), input from diverse experiences enhances the breadth of perspectives and understanding throughout the development phases. A risk inherent in participatory design is that the voices of seldom-heard groups are neglected, and so purposive recruitment of PPIE to ensure representation is recommended. This addresses potential hurdles such as digital exclusion (ie, in terms of access and skills) and the monitoring of the negative and positive effects of proposed design solutions for a range of stakeholders [6,41].

Textbox 3. Lived experience perspectives.


  • The inclusion of an individual with lived experience as part of the multidisciplinary panel approach was an important way to ensure that the patient voice was heard and included during the meeting and interwoven throughout the discussions. It was also an opportunity to provide an alternative perspective beyond the parameters of academic research and study.
  • The user perspective shared in the meeting itself drew upon a number of key challenges in the current model of mental health care (in the United Kingdom), namely, (1) respect for patient autonomy and voice, (2) sensitivity to cultural barriers, and (3) accessibility of interventions and their adaptability to patients’ lives. While this is the experience of only one individual, it does raise important considerations and can inform 3 key user perspective takeaways from the meeting to guide further work in the study of digital mental health for severe mental illness (SMI).

Patient autonomy and voice

  • Respect for patient autonomy and listening to the voices of patients are crucially important in the research and development of digital mental health interventions, just as they are in the delivery of care. The lived experience shared during the meeting indicated that there was a feeling of a lack of control and respect for patient decision-making capacity in the provision of care. There was a sentiment expressed that, by stressing the importance of the patient’s voice in the research and production stage of mental health interventions, this may subsequently set the tone for real-world application.
  • As discussed in this paper, there is already an identified greater need for genuine coproduction to facilitate this, with coproduction being defined as an equal weight placed on the involvement and accountability of those with lived experience and of academics and experts [48]. A challenge in coproduction continues in current methodologies, language, and ways of working in academia that are not always easily translated or transferred to those outside this field. There needs to be more to ensure that the platform for shared work is an equal one, with opportunities for both parties to learn from one another and ensure that coproduction is suitably accessible for genuine equal input in knowledge production. During the meeting, it was considered from a user perspective that, when principles of coproduction in research are fully realized, this may improve patient uptake and also strengthen patient autonomy. With respect to the latter, this can arise because greater insight into the needs of patients among digital health designers and clinician researchers should be much clearer as a result of patient input as well as shifting standards in mental health care more broadly.

Cultural barriers

  • Related to the aforementioned, there is a need for understanding regarding cultural barriers and stigma that act as a deterrent to access to mental health interventions and care for different users. There is a plethora of reasons why culture can be a barrier, from traditional roles and responsibilities of an individual that draw on available time, to education on mental illness (or a lack thereof) and particularly negative stigmatization. The user perspective shared during the meeting indicated that many currently available interventions do not offer sufficient nuance or flexibility in recognizing the challenges to access faced by many communities. This is linked to existing literature that often acknowledges the limitation of research being undertaken with a focus on Western-centric, middle-class, digitally literate populations despite high proportions of global mental illness being found in low- and middle-income countries. Both in the early stages of research and development and in the deployment of existing interventions, it is necessary to reflect on which communities have the greatest needs to be heard and understood.

Accessibility and adaptability

  • For a service user or individual with lived experience of SMI, a best-case scenario is one in which all barriers to accessing an intervention have been removed. An important barrier for many will be flexibility of access. Converging both the points on patient autonomy and overcoming cultural barriers, increased flexibility in access to interventions takes away a number of crucial obstacles. It is here that digital mental health has an opportunity to have a significant impact. Whether owing to the need for discretion, the weight of other time-consuming responsibilities (particularly in the case of high-functioning individuals with SMI, as shared during the discussion), or cultural stigma, the opportunity for adaptability not only serves a practical purpose but also provides a notion of empowerment and a sense of control for the individual.
Textbox 4. Examples of human-centered design in the development and evaluation of digital therapeutics for psychosis.

SlowMo: integrating an interventionist causal approach and inclusive human-centered design to develop a next-generation cognitive behavioral therapy for psychosis (CBTp)

  • Background
    • An evidence-based causal reasoning mechanism in paranoia (“fast thinking”) has an antidote in slow thinking (ie, belief flexibility) to improve paranoia and promote living well [59].
    • Proof of concept randomized experiments, and a feasibility randomized controlled trial (RCT) showed CBTp targets fast thinking and promotes slow thinking and improved paranoia [60-63].
  • Patient and public involvement and engagement (PPIE) and an inclusive, human-centered design
    • Problem: the need to improve access, experience, and outcomes for the National Institute for Health and Care Excellence (NICE) recommended CBTp, particularly among marginalized groups [64].
    • Method: UK Design Council’s Double Diamond, using ethnographic methods to define the design problem and iteratively co-designing solutions and testing prototypes with purposive sampling of users (n=18). Interdisciplinary collaboration with the Helen Hamlyn Centre for Design, the Royal College of Art, King’s College, London, software developers, and National Health Service (NHS) Trusts.
    • Solution: a redesigned version of the therapy, SlowMo, tailored to users’ needs which supports self-monitoring, provides accessible and memorable information, is enjoyable and trustworthy, promotes personalization, and provides flexible interpersonal support.
      • A web application supports the delivery of sessions with a therapist, which is synchronized with a native mobile app for use in daily life, addressing access and data protection concerns.
      • Responsive, touch screen technology supports personalization and visualization of thoughts and thinking habits.
      • Audiovisual lived experience stories provide engaging interpersonal support.
  • Multisite RCT and process evaluation (SlowMo1 [25]; N=361)
    • This was the first digitally supported therapy for paranoia to demonstrate efficacy (improved effect size compared to conventional CBTp was achieved in half of the minimum number of recommended sessions [65]), and the mechanism of change. A process evaluation of therapy experience demonstrated high rates of therapy uptake and adherence [66], a coproduced qualitative interview study supported the user experience and mechanism of change [67], and a user experience study showed that SlowMo bridged the “digital divide” as poorer digital literacy in Black people and older people did not translate to the user experience of SlowMo [19].
    • A PPIE evaluation indicated valued outcomes in the RCT [68].
  • Future research includes implementation, effectiveness, and cost-effectiveness study (SlowMo2; ongoing)
    • Interdisciplinary co-design of software for implementation incorporating lived experience findings from the RCT.
    • Lived experience coapplicant and leadership.
    • Substantive posts for lived experienced researchers and purposive recruitment of a representative lived experience advisory panel.
    • The NICE early value assessment has recommended SlowMo for use in the NHS for the treatment of paranoia in adults with psychosis while more evidence is generated [69].

gameChange: automated virtual reality (VR) cognitive therapy for treating agoraphobic avoidance and distress in patients with psychosis

  • Target
    • Agoraphobic avoidance in psychosis
  • Defining the clinical problem
    • A survey of 1809 people with psychosis [70] showed two-thirds had levels of anxious avoidance comparable to agoraphobia.
  • Steps of user involvement
    • Aimed to create a VR therapy to help people with psychosis feel safer, more confident, and in control in everyday situations.
    • The design brief: 3 hours of novel VR experiences with graded levels of difficulty. Automated through a virtual coach, users would be guided through 6 scenarios and given opportunities to drop their defenses and test their fear beliefs.
    • A person-centered design approach was used involving people with psychosis at each stage of development, including choosing scenarios, selecting tasks, testing prototypes, and user testing [71].
    • In total, >100 people with psychosis provided ≥500 hours of input.
    • gameChange is Conformité Européenne (CE) marked as a Class I active medical device.
  • Clinical testing and implementation
    • Clinical testing in a multisite RCT [72].
    • Health economics evaluation was embedded in the clinical trial [73].
    • gameChange is approved for use in the NHS while more evidence is generated to treat severe agoraphobic avoidance in people with psychosis aged ≥16 years (with the support of a mental health professional) [74].
Personalized Approaches

Facilitating personalization of digital approaches increases their ability to meet the diverse needs of people in real-world clinical populations [10]. Incorporating patient preferences in the broad development of tools is essential, but integrating patient preferences into the operation of the digital tool also enables genuine personalization so that the tool is closely matched to the individual requirements and preferences of the user [75]. A concrete step toward personalized approaches can include increasing the pool of data that informs the use of an intervention. One route to this increased body of data would be to facilitate data sharing from all studies in a specific area (see also the Theme 2: Research Methodology section). While there are specific ethical challenges in data sharing, particularly in digital mental health studies where sensitive personal data are often collected, there are well-defined principles to guide best practice [76]. Despite this, digital health data sharing remains less common, although there is a clear recognition of the need for this [76].

Early Interventions

Early intervention for psychosis has the potential to identify and improve the outcome for individuals who meet clinical high-risk state for psychosis (CHR-P) or first-episode psychosis (FEP) criteria, but most individuals with CHR-P who later develop psychosis are not currently detected during the prodromal phase [77]. Most individuals with FEP use web-based resources, and 76% responded favorably to the possibility of receiving web-based mental health support [78] and so could be identified through web-based approaches [79]. However, such tools need to be appropriate and appealing for this age group—social media and low-threshold entry points may be useful to extend the early intervention approach outside established care pathways [80]. In addition, evidence-based digital tools and resources are needed to guide parents, carers, and the young person’s wider support networks (such as teachers or social workers). Young people also need to be actively involved in coproduction to ensure the highest-quality equitable outcomes [81].

Theme 2: Research Methodology

Current Challenges

A key challenge for all areas of digital mental health is the lack of validated and internationally agreed standards for digital biomarkers or objective behavioral data obtained from patients’ personal devices [6,82]. This means that studies vary in how they assess digital biomarkers and interventions, and in what and how they measure change, engagement, adherence, improvement, or potential harms. For instance, with some exceptions, studies have not examined adverse events [38]. This is needed for safety measurements in digital interventions not only among people with SMI but also in health care more broadly. In 2023, regulatory bodies such as the US Food and Drug Administration (FDA) issued draft guidance outlining how verification and validation of digital health technologies should be approached [83], including some challenges that are more prominent in digital health, such as data privacy and confidentiality. However, a 2023 review of FDA approvals across all domains of health care, including digital health technologies, suggested a lack of scientific rigor across studies and the need for higher-quality research [84]. Because there is significant variability between studies, current research is not easy to replicate or validate, leading to reduced confidence in the results and the robustness of the evidence base supporting digital interventions. In addition, digital mental health research in recent years has focused mainly on reactive interventions that address immediate or short-term needs, whereas longer-term and preventative approaches or time series–aware methods often remain lacking [85].

Potential Solutions
Agreed Standards for Digital Mental Health Research

Differences regarding study standards and data quality need to be addressed at an international level with agreed standards for studies and their subsequent publication. The homogenization of standards could mitigate fragmented approaches, and regulatory agencies and funders could insist on such standards [86]. This could prompt researchers to align research protocols in digital mental health so that there is transparent reporting of the data collected and the frequency with which they are collected (which would allow for comparison and combination of data sets). This would also facilitate the understanding of the relative usefulness of, for example, particular machine learning paradigms (as these are most accurate on data sets that are similar to those they have been trained on), and this would require separate data processing standards [87,88].

Data Sharing

To enable the comparison and combination of data sets, standards should require the sharing of data via open-source documentation (eg, the study by Bent et al [89]). While this can generate some potential ethical challenges (Textbox 5), there are already international examples demonstrating data sharing [90,91]. The older 2016 CrossCheck study of people with schizophrenia using a smartphone app to collect digital phenotyping data also offers an open data set that has enabled numerous publications advancing computational methods of symptom and relapse prediction [92].

Textbox 5. Applying a biomedical ethics framework in the use of digital tools for severe mental illness (SMI).

Respect for autonomy

  • Digital phenotyping involves continuous, real-time multimodal streaming of data from smartphones and other internet-enabled devices. As noted, the insights offered by these data may be highly sensitive, uncovering, for example, an internet browsing history, purchasing patterns, or levels of social contact. Conceivably, such data may also create legal exposures. For example, an estimated 1 in 4 people with SMI also experience substance use disorders [93], which, in some circumstances, could heighten risks of criminal behavior or illicit drug purchasing or use. For clinicians to access these digital data streams to yield benefits for patients, health care users must provide fully informed consent regarding the whys, hows, and consequences of sharing digital information, including which data may yield the most valuable insights and how health care providers will store and use this information.


  • Digital tools have the potential to offer faster, less expensive, and more accessible care at scale. Many digital interventions, such as apps, also afford patients with SMI opportunities to engage with technology without the fear or risks of stigmatization that may arise in clinic visits. While some tools already offer considerable promise, for the benefits of these applications to be optimized, greater research is needed to explore implementation to increase uptake (themes 1, 2, and 4).


  • Clinicians are duty bound to “first, do no harm,” yet, currently, there is a deficit of research into the potential adverse effects of digital health tools for SMI (theme 2). Identifying when tools lead to incidents of harm, including self-harm or discontinuation of treatment is critical to ensure safety. For example, preoccupation with checking health-tracking data via downloadable apps and wearable devices might increase anxiety, especially among some subpopulations of patients with SMI.


  • Artificial intelligence–powered digital tools rely on patient involvement, and if data sets are not representative of the populations in which they will be used, these technologies may not be as useful or could be harmful for these groups. Furthermore, to benefit all patients, digital tools need to be accessible to everyone [94]. While digital divides are narrowing, the most vulnerable patients—including those with SMI—are often more likely to live with lower incomes, meaning that they are still less likely to own digital devices, to have access to broadband, or have acquired the digital literacy skills necessary to partake in technology use and reap the benefits [95]. In many countries, advancing digital health research is now a priority [96]; however, without concerted efforts to improve the distribution and access to these tools, inequities will persist or potentially increase. Aimed at improving digital literacy among patients, including those with SMI, the Digital Opportunities for Outcomes in Recovery Services program has been deployed in many community settings, including in the United Kingdom and the United States [20].

Professional-patient relationships

  • Used effectively, digital tools are unlikely to replace human relationships, including in care settings. Moreover, for data to be effectively interpreted and understood, a deeper and more honest and trusting partnership between patients and clinicians is imperative. For example, the insights gleaned from digital phenotyping are fallible and require context—only patients with lived experience can assist in offering the situational knowledge needed to furnish a deeper understanding of what the data show and how they might be harnessed in preventative care [94]. Conversely, as noted previously, owing to the nature of data gathering, there are also multiple new opportunities for these technologies to undermine trust in the fiduciary clinician-patient relationship (see the previous points). Regulatory policies, civic debate and patient involvement, and health professional ethical awareness must strive to keep abreast of advances (themes 1 and 3).
Measuring Harms

For transparency, standards should also require that potential adverse events or harms, as well as benefits, are identified and measured (eg, the International Collaboration for Harmonising Adverse Events Reporting in Technology for Schizophrenia (iCHARTS) network by Bucci et al [97]). In addition, preregistration of protocols [98] should also be implemented to increase the publication of “negative” findings. This would balance the known bias toward the publication of “positive results”—a trend already noted in biomarker research for bipolar disorder [99]. Such approaches could better inform researchers about when to build interventions using existing platforms and when novel platforms are needed. Harm need not be limited to classic symptom exacerbation and can also include loss of privacy, inequality or discrimination, social or personal loss, and even physical injury. These could be measured using a combination of both self-report and investigator-assessed scales.

Prevention and Longer-Term Strategies

Digital mental health research in recent years has focused mainly on reactive interventions in the short term, but longer-term or preventative interventions may be equally or more important, particularly in areas such as SMI and suicide prevention. Examples include the use of web-based platforms in suicide prevention while tackling harmful content that could promote or encourage suicide and self-harm [100-103], and web-based screening in youth mental health to support the identification of high-risk individuals or groups.

Mechanistic Research in Digital Mental Health

Research on the mechanisms of digital mental health is needed, specifically more focused research on the potential moderators or mediators of effects. Even engagement itself may require mechanistic research as it has proven to be a challenging construct to improve upon. For example, a recent review of digital therapeutics suggested that the field could benefit from the application of clinical pharmacology principles from the drug development field, such as a stepwise and progressive focus on engagement and adherence, proxy of effects, and clinical end points [104]. This would need funders to launch specific calls for this type of research (as this is intricate and costly) and is explored further in theme 3 in the following section.

Theme 3: Regulation, Governance, and Funding

Current Challenges

Funding is a major issue specifically in digital mental health as the process is often fragmented [105] and may only cover the initial development of digital interventions. The rest of the pathway is also often lengthy and costly, but is critical in the road toward implementation and needs to incorporate the processes of regulation and market transfer. A particularly important area to assess is the implementation of a mental health digital product in the real-world health care setting. This will involve a comprehensive analysis of implementation costs, integration into the existing IT infrastructures of the institution, and regulatory and privacy compliance, but this may not have been adequately assessed at the beginning of the process. A result is that many digital health technologies are not iterated and sustained, with a 2022 review finding that nearly half of apps created for schizophrenia research in the last decade are no longer accessible or supported [11]. These challenges in funding impede replication and impair the research-to-clinical translation of digital mental health tools (see further discussion in theme 4). An area of additional challenge is that research funding has usually not considered the complexity of science (including the need for significant user input throughout the research cycle, as explored in theme 1). In addition, digital interventions have technological complexities that have a significant impact on costs, such as the need to maintain apps on an ongoing basis, provide updates, and implement new and regular cybersecurity and data protection measures.


These challenges are complicated by a lack of health care regulation specifically tailored for the digital space. For example, in the United States, many digital interventions fall within FDA regulations that are challenging when applied to this area, and so many interventions do not come to market. There are also other extra regulatory issues that need to be formally addressed regarding safety and data protection, including privacy and security [106]. For example, data collection in digital phenotyping involves tracking patients beyond traditional health information and can involve data such as social media posts, geolocation, and telephone and SMS text message traffic (among other data) that can provide revealing insights about the daily lives of individuals [107,108]. Studies also show that there are significant limitations with clinicians’ awareness of the ethical considerations regarding artificial intelligence–powered innovations in health care [109-113] (Textbox 5). Furthermore, health laws have not kept up with digital technologies, although authorities have recently made efforts to regulate technology while also aiming to protect civil liberties and rights to privacy [107,108].

Potential Solutions

Solutions include advocating for governance change, but this would need to ensure the involvement of and contributions from multiple stakeholders (patients and carers, technology experts, researchers, clinicians, health technology assessment agencies, and regulators). One option as a model might be a roundtable discussion with all relevant stakeholders, such as that at the recent UK AI Safety Summit [114,115]. Governance change could also encompass standards for the level of evidence required, including controlling for digital placebo effects and demonstrating savings in cost [116]. In line with governance change, funding in this area needs to be more adaptive to recognize the particular needs in this space (eg, commitments to funding until completion or funding the people rather than the project [117]). As an intermediate step, guidelines for best practices and research standards that funders and journals may enforce would also shift the field in a positive direction. In terms of ethical concerns, including how to navigate evolving regulations, clinicians will require greater training, and patients will need more guidance and advice on the benefits and potential risks of these digital tools in health care settings (Textbox 5). Brief training interventions should be considered for all stakeholders. For example, among patients, the use of “digital navigators”—peer supporters who can offer patients advice on how to download and use apps, where to find information about privacy considerations, and the evidence base for these tools—has been pioneered in outpatient psychiatry with notable success [118].

Theme 4: Implementation in Real-World Clinical Settings

Current Challenges

Translation of research findings into the clinical setting is a significant challenge. In general, many digital health studies fail to reach the market or translate into real-world clinical care, and even when they do, rates of adoption can be low [119]. For example, in a study of smartphone apps for schizophrenia, <10 of those identified from a search of interventions from the research literature were easily accessible to the population, and the picture was similar in a parallel search of marketplace apps [11]. These were also few in number and lacked frequent updates (average time since last update 1121 days). Even where there is engagement in the research setting, sustaining this in the clinic, outside of the constraints and incentives of clinical trials, is challenging, particularly over the longer term. For example, digital health tools with 44% to 99% completion rates in research studies translate to only 1% to 28% in actual clinical use [120].

There are several potential reasons for this:

  1. Researchers are often not trained in the technology skills and application of the implementation science methods required to ensure the tailoring and ongoing development of interventions for real-world uptake.
  2. Even once digital interventions have been fully developed, individual clinicians may be reluctant to implement them in the clinic. Uptake requires behavior change from clinicians, who may be resistant due to lack of training, skills, or confidence in digital interventions, regulatory issues, and cautions and anxieties about adopting commercial therapy products.
  3. Patients, carers, and the public may also be reluctant due to digital inclusion issues such as digital literacy and confidence (especially for some groups, such as older patients or patients who are severely ill [121]) as well as concerns about privacy and digital coercion [122].

In addition, the focus of digital mental health research may contribute to difficulties in translation into real-world settings:

  1. Many research studies focus on “replacing” the clinician, with apps designed for stand-alone use, whereas the research base suggests that blended or augmented care, in line with patient preference, is more beneficial [16,118,123].
  2. Many studies focus on general outcomes rather than the mechanisms of change, and without registration of studies, there is a risk of inefficient use of funding resources.
  3. The focus of digital mental health research also tends to be on biological and psychological parameters, whereas social mechanisms are equally important [10].
  4. For the user, there is a lack of guidance on which interventions might be better or worse. This is a long-standing problem [124] and reflects the lack of evidence base. For example, assessments of smartphone tools for suicide prevention have identified a wide variety of approaches but also include apps with potentially harmful content [125,126].
Potential Solutions

Successful development, implementation, and application in real-world clinical settings will require collaboration between multidisciplinary groups who have not traditionally been brought together, for example, a wider range of academics with complementary expertise, clinicians, user groups and industrial designers, software developers, commercial partners, and regulatory specialists. Skills and confidence in using digital interventions will be a key element. Training schedules for clinicians in using and integrating digital tools into their clinical practice have already been defined, and these need to be integrated into clinical training at all levels from core training to specialist academic programs [127], with health care organizations creating sustained budgets to fund digital inclusion schemes. Patient and user confidence is also important, and there are examples of effective training schemes for SMI (eg, the study by Hoffman et al [20]) and of interventions to help bridge the technological divide, such as digital navigators [118] and specialized youth mental health workers [44].

Workable and scalable solutions will rest in augmenting the in-person consultation using a blended approach. In addition, the focus needs to be on increasing the capacity of clinical services, such as by augmenting service provision with support workers who can deliver protocolized interventions. Implementation science frameworks can be used to guide the assessment of site readiness and evaluate their ability to successfully introduce, implement, and sustain digital technology use [119,128].

Solutions will also depend on looking at social interventions, which may be population focused as well as targeted at the individual or specific group level. Examples include wide-reaching digital training, awareness and antistigma campaigns [129], and automated solutions to reduce access and exposure to lethal means of suicide [130-132]. Although in the past there have been difficulties in formal guidance for the user [121], the American Psychiatric Association app evaluation framework offers a viable alternative [133].


In this paper, we have illustrated and discussed the complexities of collecting data, delivering treatment, and the ethical challenges of digital mental health in the care pathways of people with SMI. During the process of the consensus meeting and the consensus recommendations, we implemented a thematic approach focusing broadly on digital interventions for psychosis. This enabled us to concentrate on an area of significant need where digital health innovation has the potential to be safe and effective [134]. However, we also found that, in taking this thematic approach, we identified broader issues, and the solutions proposed can apply to other fields of mental health.

Potential Limitations

There are some potential limitations to our approach. While we conducted a systematic review of the literature, we restricted this to one source (PubMed) and to articles published in the last 5 years on psychosis and schizophrenia. The primary aim of this review was to ensure that the panel was informed before the meeting, but it is possible that this approach may have missed some relevant publications (eg, on other diagnoses within SMI, such as bipolar disorder). In addition, as with all consensus meetings, there are no standard guidelines for identifying expertise. Although we selected participants to represent a diverse spectrum of views, the reliability of consensus opinions is dependent on the specialist knowledge and experiences of those who participated. We aimed to encompass a wide variety of expertise in the expert group and panel to ensure that many different perspectives were heard. These included clinical psychology, psychotherapy, psychiatry, philosophy of medicine and health care ethics, health services research, social sciences, health informatics, digital health care, and mental health charity management. We included an expert with lived experience of SMI in the panel but recognize that we could have included more, which we take on board for future consensus studies. The structured in-person nature of the meeting may also have unintentionally excluded the opinions of experts, particularly those with lived experience of SMI, who are not willing or able to engage in that format. However, we had 2 PPI contributors within the process (one in the panel and one in the expert group) who made material contributions throughout; to the literature review, presentation and discussion of the evidence, formation of consensus, and coproduction of the paper. In this way, we aimed to engage a high-quality PPI coproduction rather than focusing purely on the number of PPI members involved. In addition, by adopting and building on new features of the consensus method used in our previous work [6] (eg, including a panel separate from the expert group and including PPIE members in the expert group and panel), we have strengthened international and multidisciplinary discussion on this important but often overlooked area [134].

Principal Findings

The consensus meeting identified a number of broad recommendations in this field:

  1. A new approach to research in digital mental health is needed that is different from the standard pathways used in pharmacological and psychological intervention research.
  2. This new approach requires internationally agreed standards for reporting research and open data access to allow for true collaboration and enable easy validation of biomarkers and replication of interventions.
  3. This could be facilitated by the development of shared protocols for research to be carried out in multiple recruiting sites.
  4. Research should place an equal emphasis on social and population factors [135] as well as biological and social factors in the etiology and maintenance of symptoms and risk in SMI.
  5. Successful implementation and application of digital mental health in real-world clinical settings will require new and evolving collaborations between academics, clinicians, people with lived experience of SMI, industrial designers, software developers, and regulatory specialists.
  6. The uniqueness of the digital space in clinical research means that it requires a different approach, focusing not just on the translational pathway of research in isolation but also on the business model; the “product”; and the impact or value of the intervention for all relevant stakeholders, from patients to clinicians, health care organizations, and society at large.
  7. This new approach may prompt a possible conflict of values as the “product” (the digital mental health intervention) needs to be economically viable so that it can be scaled and sustained while also ensuring effectiveness, safety, and compliance with medical technology quality standards and health care regulation.
  8. Potential harms are just as important to record as in other areas of research (such as with pharmacological or psychological interventions) but may be more hidden and need to be actively sought out and logged.
  9. Funding structures need to be adjusted to the new elements required within digital mental health research and must be sufficient to support ongoing product development in line with regulatory requirements and allow for representative PPIE.
  10. Funding streams will need to recognize that not all the tasks required can be performed by a single person or group and they may need a “relay” approach between stakeholders (ie, projects led by clinician academics at the proof-of-concept, feasibility, and efficacy stages, with increased commercialization and regulatory input as products move into implementation and market).
  11. In addition, more fundamental organizational changes are needed to underpin necessary changes to funding and research study approaches. Participants with lived experience and academic experts do not always “speak the same language” [136], so awareness of differences in expression and the need to work together to solve health problems is necessary to minimize the impact of power imbalances and promote coproduction [137]. This can be achieved by developing safe spaces to create and share knowledge [138] and allow for opportunities for researchers and participants with lived experience to learn and enrich their own expertise from the experience of informed participation and collaboration [49]. This improved communication could provide the platform to create new models of care to deliver digital services, which will also require adjustments to organizational structure, policies, and membership.
  12. The ethical components of digital mental health are also crucial. This is not just in terms of trust and trustworthiness in digital mental health but also in managing patient expectations, “ownership of their own health,” and future developments.


In this study, our approach, which combined an international expert meeting with PPIE throughout the process, consensus methodology, discussion, and publication, was a fruitful way to reach expert consensus and focus directions for future research and clinical implementation, especially in rapidly evolving fields. We improved and expanded our approach and showed how to integrate research evidence with a process of measuring real-world clinical impact over time [139]. To enhance our scope, future meetings should directly involve stakeholders in health technology assessment and representatives from regulatory agencies and industry and encompass researchers working in health care ethics and policy in different countries, including from the Global South. Similar initiatives should be repeated regularly in digital mental health and adopted also by researchers in other fields to focus research and organizational change to effect real-world clinical implementation.


We acknowledge especially the contribution of our co-author and colleague, Lea Milligan, CEO of MQ Mental Health Research, who sadly passed away unexpectedly on the 15th of April 2024 following a sudden illness. He made a significant and much-valued contribution to this research project as an active member of the expert panel and co-author, and will be hugely missed

KAS, CZ, EGO, and AC are supported by the National Institute for Health and Care Research (NIHR) Oxford Health Clinical Research Facility. AC and EGO are supported by an NIHR Research Professorship (grant RP-2017-08-ST2-006), the NIHR Oxford and Thames Valley Applied Research Collaboration, and the NIHR Oxford Health Biomedical Research Centre (grant BRC-1215-20005). AC is supported by Wellcome (Global Alliance for Living Evidence on Anxiety, Depression, and Psychosis [GALENOS] project). EGO is supported by the Brasenose College Senior Hulme Scholarship. AH is partly funded by the NIHR Maudsley Biomedical Research Centre at the South London and Maudsley National Health Service Foundation Trust and King’s College London. LM is partly funded by Wellcome and the Medical Research Council Health Data Research United Kingdom. SL is supported by an NIHR Doctoral Fellowship (NIHR301483) and the NIHR Oxford Health Biomedical Research Centre (BRC-1215-20005). RA is supported by the Italian Ministry of University and Research via the Residency Training Programme in Psychiatry. GA is supported by a Juan Rodés 2023 grant (JR23/00050), a Rio Hortega 2021 grant (CM21/00017) and Acción Estratégica en Salud - Mobility (M-AES) fellowship (MV22/00058), from the Spanish Ministry of Health financed by the Instituto de Salud Carlos III (ISCIII) and co-financed by the Fondo Social Europeo Plus (FSE+). GA thanks the support of the Spanish Ministry of Health financed by the Instituto de Salud Carlos III (ISCIII) and co-financed by the European Social Fund+ (ESF+) (JR23/00050, MV22/00058, CM21/00017); the ISCIII (PI21/00340, PI21/00169); the Milken Family Foundation (PI046998); the Fundació Clínic per a la Recerca Biomèdica (FCRB) - Pons Bartan 2020 grant (PI04/6549), the Sociedad Española de Psiquiatría y Salud Mental (SEPSM); the Fundació Vila Saborit; and the Societat Catalana de Psiquiatria i Salut Mental (SCPiSM). PJU was supported by the Wellcome Trust Innovator Grant 215793/Z/19/Z. This paper is based on a meeting held in Rome in October 2023. The meeting and the logistics of the event were supported by Angelini Pharma, who did not have any influence on the content of the discussion, the outcome, or the preparation of this manuscript. The views expressed are those of the authors and not necessarily those of the UK National Health Service, the NIHR, or the UK Department of Health and Social Care.

Data Availability

All data generated or analyzed during this study are included in this published article.

Authors' Contributions

AC and JT developed the concept for the study and the format of the consensus meeting and chaired the expert group. KAS assisted with the consensus meeting, chaired the panel, wrote the first draft of the manuscript, and coordinated the editing and updates of subsequent drafts. All coauthors attended and actively contributed to the meeting and the process of consensus formation, agreed on consensus statements, and reviewed and approved the final draft of the manuscript.

Conflicts of Interest

KAS, AH, CZ, RA, and LM have no conflicts of interest to declare. AC has received research and consultancy fees from the Italian Network for Pediatric Clinical Trials, the Cariplo Foundation, Lundbeck, and Angelini Pharma outside the submitted work. EGO has received research and consultancy fees from Angelini Pharma outside the submitted work. GA has received honoraria or consulting fees from Angelini Pharma, Casen Recordati, Janssen-Cilag, Lundbeck, Lundbeck-Otsuka, Rovi, and Viatris outside the submitted work. PJU has received honoraria from Boehringer Ingelheim and research funding from Lundbeck and Lilly United Kingdom. SL has received consultancy fees from OxfordVR, a University of Oxford spin-out company that is commercializing the gameChange treatment. JT is cofounder of a mental health company called Precision Mental Wellness and is the editor-in-chief of JMIR Mental Health at the time of publication. DH has received CME-related honoraria and served as consultant for Abbott, Angelini, Ethypharm Digital Therapy, Janssen-Cilag, Lundbeck and Viatris outside the submitted work.

  1. Mahase E. Government must get a "firmer grip" on mental health crisis, says watchdog. BMJ. Feb 09, 2023;380:324. [FREE Full text] [CrossRef] [Medline]
  2. Hong JS, Sheriff R, Smith K, Tomlinson A, Saad F, Smith T, et al. Impact of COVID-19 on telepsychiatry at the service and individual patient level across two UK NHS mental health trusts. Evid Based Ment Health. Nov 2021;24(4):161-166. [FREE Full text] [CrossRef] [Medline]
  3. Zangani C, Ostinelli EG, Smith KA, Hong JS, Macdonald O, Reen G, et al. Impact of the COVID-19 pandemic on the global delivery of mental health services and telemental health: systematic review. JMIR Ment Health. Aug 22, 2022;9(8):e38600. [FREE Full text] [CrossRef] [Medline]
  4. Sheriff R, Hong JS, Henshall C, D'Agostino A, Tomassi S, Stein HC, et al. Evaluation of telepsychiatry during the COVID-19 pandemic across service users, carers and clinicians: an international mixed-methods study. BMJ Ment Health. Aug 2023;26(1):e300646. [FREE Full text] [CrossRef] [Medline]
  5. Currey D, Torous J. Increasing the value of digital phenotyping through reducing missingness: a retrospective review and analysis of prior studies. BMJ Ment Health. Feb 2023;26(1):e300718. [FREE Full text] [CrossRef] [Medline]
  6. Smith KA, Blease C, Faurholt-Jepsen M, Firth J, Van Daele T, Moreno C, et al. Digital mental health: challenges and next steps. BMJ Ment Health. Feb 2023;26(1):e300670. [FREE Full text] [CrossRef] [Medline]
  7. Patel V, Saxena S, Lund C, Kohrt B, Kieling C, Sunkel C, et al. Transforming mental health systems globally: principles and policy recommendations. Lancet. Aug 2023;402(10402):656-666. [FREE Full text] [CrossRef]
  8. Asher L, Fekadu A, Hanlon C. Global mental health and schizophrenia. Curr Opin Psychiatry. May 2018;31(3):193-199. [CrossRef] [Medline]
  9. Fiorillo A, de Girolamo G, Simunovic IF, Gureje O, Isaac M, Lloyd C, et al. The relationship between physical and mental health: an update from the WPA working group on managing comorbidity of mental and physical health. World Psychiatry. Feb 2023;22(1):169-170. [FREE Full text] [CrossRef] [Medline]
  10. O'Connor RC, Worthman CM, Abanga M, Athanassopoulou N, Boyce N, Chan LF, et al. Gone Too Soon: priorities for action to prevent premature mortality associated with mental illness and mental distress. Lancet Psychiatry. Jun 2023;10(6):452-464. [FREE Full text] [CrossRef]
  11. Kwon S, Firth J, Joshi D, Torous J. Accessibility and availability of smartphone apps for schizophrenia. Schizophrenia (Heidelb). Nov 16, 2022;8(1):98. [FREE Full text] [CrossRef] [Medline]
  12. Kozelka EE, Acquilano SC, Al-Abdulmunem M, Guarino S, Elwyn G, Drake RE, et al. Documenting the digital divide: identifying barriers to digital mental health access among people with serious mental illness in community settings. SSM Mental Health. Dec 2023;4:100241. [FREE Full text] [CrossRef]
  13. Spanakis P, Lorimer B, Newbronner E, Wadman R, Crosland S, Gilbody S, et al. Digital health literacy and digital engagement for people with severe mental ill health across the course of the COVID-19 pandemic in England. BMC Med Inform Decis Mak. Sep 26, 2023;23(1):193. [FREE Full text] [CrossRef] [Medline]
  14. Firth J, Cotter J, Torous J, Bucci S, Firth JA, Yung AR. Mobile phone ownership and endorsement of "mHealth" among people with psychosis: a meta-analysis of cross-sectional studies. Schizophr Bull. Mar 22, 2016;42(2):448-455. [FREE Full text] [CrossRef] [Medline]
  15. Iliescu R, Kumaravel A, Smurawska L, Torous J, Keshavan M. Smartphone ownership and use of mental health applications by psychiatric inpatients. Psychiatry Res. May 2021;299:113806. [FREE Full text] [CrossRef] [Medline]
  16. Eisner E, Berry N, Bucci S. Digital tools to support mental health: a survey study in psychosis. BMC Psychiatry. Oct 07, 2023;23(1):726. [FREE Full text] [CrossRef] [Medline]
  17. Middle R, Welch L. Experiences of digital exclusion and the impact on health in people living with severe mental illness. Front Digit Health. Nov 22, 2022;4:1004547. [FREE Full text] [CrossRef] [Medline]
  18. Spanakis P, Wadman R, Walker L, Heron P, Mathers A, Baker J, et al. Measuring the digital divide among people with severe mental ill health using the essential digital skills framework. Perspect Public Health. Jan 2024;144(1):21-30. [FREE Full text] [CrossRef] [Medline]
  19. Hardy A, Ward T, Emsley R, Greenwood K, Freeman D, Fowler D, et al. Bridging the digital divide in psychological therapies: observational study of engagement with the SlowMo mobile app for paranoia in psychosis. JMIR Hum Factors. Jul 01, 2022;9(3):e29725. [FREE Full text] [CrossRef] [Medline]
  20. Hoffman L, Wisniewski H, Hays R, Henson P, Vaidyam A, Hendel V, et al. Digital opportunities for outcomes in recovery services (DOORS): a pragmatic hands-on group approach toward increasing digital health and smartphone competencies, autonomy, relatedness, and alliance for those with serious mental illness. J Psychiatr Pract. Mar 2020;26(2):80-88. [FREE Full text] [CrossRef] [Medline]
  21. Camacho E, Torous J. Impact of digital literacy training on outcomes for people with serious mental illness in community and inpatient settings. Psychiatr Serv. May 01, 2023;74(5):534-538. [FREE Full text] [CrossRef] [Medline]
  22. García-Estela A, Cantillo J, Angarita-Osorio N, Mur-Milà E, Anmella G, Pérez V, et al. Real-world implementation of a smartphone-based psychoeducation program for bipolar disorder: observational ecological study. J Med Internet Res. Feb 02, 2022;24(2):e31565. [FREE Full text] [CrossRef] [Medline]
  23. Hidalgo-Mazzei D, Nikolova VL, Kitchen S, Young AH. Internet-connected devices ownership, use and interests in bipolar disorder: from desktop to mobile mental health. Digit Psychiatry. May 28, 2019;2(1):1-7. [FREE Full text] [CrossRef]
  24. Green JB, Rodriguez J, Keshavan M, Lizano P, Torous J. Implementing technologies to enhance coordinated specialty care framework: implementation outcomes from a development and usability study. JMIR Form Res. Oct 03, 2023;7:e46491. [FREE Full text] [CrossRef] [Medline]
  25. Garety P, Ward T, Emsley R, Greenwood K, Freeman D, Fowler D, et al. Effects of SlowMo, a blended digital therapy targeting reasoning, on paranoia among people with psychosis: a randomized clinical trial. JAMA Psychiatry. Jul 01, 2021;78(7):714-725. [FREE Full text] [CrossRef] [Medline]
  26. Arakawa N, Bader LR. Consensus development methods: considerations for national and global frameworks and policy development. Res Social Adm Pharm. Jan 2022;18(1):2222-2229. [FREE Full text] [CrossRef] [Medline]
  27. Fouad Y, Dufour JF, Zheng MH, Bollipo S, Desalegn H, Grønbaek H, et al. The NAFLD-MAFLD debate: is there a consensus-on-consensus methodology? Liver Int. Apr 2022;42(4):742-748. [FREE Full text] [CrossRef] [Medline]
  28. Ferguson JH. The NIH consensus development program. the evolution of guidelines. Int J Technol Assess Health Care. 1996;12(3):460-474. [Medline]
  29. WHO handbook for guideline development. World Health Organization. 2014. URL: [accessed 2024-04-15]
  30. Waggoner J, Carline JD, Durning SJ. Is there a consensus on consensus methodology? Descriptions and recommendations for future consensus research. Acad Med. May 2016;91(5):663-668. [CrossRef] [Medline]
  31. Page MJ, Moher D, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews. BMJ. Mar 29, 2021;372:n160. [FREE Full text] [CrossRef] [Medline]
  32. Gattrell W, Logullo P, van Zuuren EJ, Price A, Hughes EL, Blazey P, et al. ACCORD (ACcurate COnsensus Reporting Document): a reporting guideline for consensus methods in biomedicine developed via a modified Delphi. PLoS Med. Jan 2024;21(1):e1004326. [FREE Full text] [CrossRef] [Medline]
  33. Humphrey-Murto S, Varpio L, Gonsalves C, Wood TJ. Using consensus group methods such as Delphi and nominal group in medical education research. Med Teach. Nov 12, 2016;39(1):14-19. [FREE Full text] [CrossRef]
  34. Graham A, Lattie EG, Mohr DC. Experimental therapeutics for digital mental health. JAMA Psychiatry. Dec 01, 2019;76(12):1223-1224. [FREE Full text] [CrossRef] [Medline]
  35. Perski O, Blandford A, West R, Michie S. Conceptualising engagement with digital behaviour change interventions: a systematic review using principles from critical interpretive synthesis. Transl Behav Med. Jun 2017;7(2):254-267. [FREE Full text] [CrossRef] [Medline]
  36. Torous J, Nicholas J, Larsen ME, Firth J, Christensen H. Clinical review of user engagement with mental health smartphone apps: evidence, theory and improvements. Evid Based Ment Health. Aug 2018;21(3):116-119. [FREE Full text] [CrossRef] [Medline]
  37. Nwosu A, Boardman S, Husain MM, Doraiswamy PM. Digital therapeutics for mental health: is attrition the Achilles heel? Front Psychiatry. 2022;13:900615. [FREE Full text] [CrossRef] [Medline]
  38. Torous J, Bucci S, Bell IH, Kessing LV, Faurholt-Jepsen M, Whelan P, et al. The growing field of digital psychiatry: current evidence and the future of apps, social media, chatbots, and virtual reality. World Psychiatry. Oct 2021;20(3):318-335. [FREE Full text] [CrossRef] [Medline]
  39. Ng MM, Firth J, Minen M, Torous J. User engagement in mental health apps: a review of measurement, reporting, and validity. Psychiatr Serv. Jul 01, 2019;70(7):538-544. [FREE Full text] [CrossRef] [Medline]
  40. Merchant R, Torous J, Rodriguez-Villa E, Naslund JA. Digital technology for management of severe mental disorders in low-income and middle-income countries. Curr Opin Psychiatry. Sep 2020;33(5):501-507. [FREE Full text] [CrossRef] [Medline]
  41. Aref-Adib G, McCloud T, Ross J, O'Hanlon P, Appleton V, Rowe S, et al. Factors affecting implementation of digital health interventions for people with psychosis or bipolar disorder, and their family and friends: a systematic review. Lancet Psychiatry. Mar 2019;6(3):257-266. [FREE Full text] [CrossRef]
  42. Singh SR, Sagar R. Time to have effective regulation of the mental health apps market: maximize gains and minimize harms. Indian J Psychol Med. Jul 2022;44(4):399-404. [FREE Full text] [CrossRef] [Medline]
  43. Solmi M, Radua J, Olivola M, Croce E, Soardo L, Salazar de Pablo G, et al. Age at onset of mental disorders worldwide: large-scale meta-analysis of 192 epidemiological studies. Mol Psychiatry. Jan 02, 2022;27(1):281-295. [FREE Full text] [CrossRef] [Medline]
  44. Uhlhaas PJ, Davey CG, Mehta UM, Shah J, Torous J, Allen NB, et al. Towards a youth mental health paradigm: a perspective and roadmap. Mol Psychiatry. Aug 2023;28(8):3171-3181. [FREE Full text] [CrossRef] [Medline]
  45. Teens, social media and technology. Pew Research Center. 2023. URL: [accessed 2024-04-15]
  46. Wong CA, Madanay F, Ozer EM, Harris SK, Moore M, Master SO, et al. Digital health technology to enhance adolescent and young adult clinical preventive services: affordances and challenges. J Adolesc Health. Aug 2020;67(2S):S24-S33. [FREE Full text] [CrossRef] [Medline]
  47. Bergin AD, Vallejos EP, Davies EB, Daley D, Ford T, Harold G, et al. Preventive digital mental health interventions for children and young people: a review of the design and reporting of research. NPJ Digit Med. 2020;3:133. [FREE Full text] [CrossRef] [Medline]
  48. Doughty B. What is co-production? MQ Mental Health Research. 2021. URL: [accessed 2024-04-05]
  49. Richmond L, Caton N, Downs J, Newton A, Devereux-Fitzgerald A, Brooks H. Creating positive experiences of involvement in mental health research. Lancet Psychiatry. Mar 2023;10(3):220-227. [CrossRef]
  50. Landscaping international longitudinal datasets: theory of change process and a deeper dive into lived experience input. MQ Mental Health Research. URL: [accessed 2024-04-15]
  51. Guidance on co-producing a research project. National Institutes for Health and Care Research. URL: https:/​/www.​learning​content/​resource/​nihr-guidance-on-co-producing-a-research-project/​ [accessed 2024-04-15]
  52. Sartor C. Mental health and lived experience: the value of lived experience expertise in global mental health. Glob Ment Health (Camb). 2023;10:e38-e35. [FREE Full text] [CrossRef] [Medline]
  53. Riggare S. E-patients hold key to the future of healthcare. BMJ. Feb 26, 2018;360:k846. [FREE Full text] [CrossRef] [Medline]
  54. Price A, Clarke M, Staniszewska S, Chu L, Tembo D, Kirkpatrick M, et al. Patient and public involvement in research: a journey to co-production. Patient Educ Couns. Apr 2022;105(4):1041-1047. [FREE Full text] [CrossRef] [Medline]
  55. Byrne L, Wykes T. A role for lived experience mental health leadership in the age of COVID-19. J Ment Health. Jun 2020;29(3):243-246. [FREE Full text] [CrossRef] [Medline]
  56. Holmes EA, O'Connor RC, Perry VH, Tracey I, Wessely S, Arseneault L, et al. Multidisciplinary research priorities for the COVID-19 pandemic: a call for action for mental health science. Lancet Psychiatry. Jun 2020;7(6):547-560. [FREE Full text] [CrossRef] [Medline]
  57. Vaidyam A, Halamka J, Torous J. Enabling research and clinical use of patient-generated health data (the mindLAMP platform): digital phenotyping study. JMIR Mhealth Uhealth. Jan 07, 2022;10(1):e30557. [FREE Full text] [CrossRef] [Medline]
  58. Rodriguez-Villa E, Rozatkar AR, Kumar M, Patel V, Bondre A, Naik SS, et al. Cross cultural and global uses of a digital mental health app: results of focus groups with clinicians, patients and family members in India and the United States. Glob Ment Health (Camb). 2021;8:e30. [FREE Full text] [CrossRef] [Medline]
  59. Ward T, Hardy A, Holm R, Collett N, Rus-Calafell M, Sacadura C, et al. SlowMo therapy, a new digital blended therapy for fear of harm from others: an account of therapy personalisation within a targeted intervention. Psychol Psychother. Jun 12, 2022;95(2):423-446. [FREE Full text] [CrossRef] [Medline]
  60. Garety P, Waller H, Emsley R, Jolley S, Kuipers E, Bebbington P, et al. Cognitive mechanisms of change in delusions: an experimental investigation targeting reasoning to effect change in paranoia. Schizophr Bull. Mar 2015;41(2):400-410. [FREE Full text] [CrossRef] [Medline]
  61. Ross K, Freeman D, Dunn G, Garety P. A randomized experimental investigation of reasoning training for people with delusions. Schizophr Bull. Mar 2011;37(2):324-333. [FREE Full text] [CrossRef] [Medline]
  62. Waller H, Emsley R, Freeman D, Bebbington P, Dunn G, Fowler D, et al. Thinking well: a randomised controlled feasibility study of a new CBT therapy targeting reasoning biases in people with distressing persecutory delusional beliefs. J Behav Ther Exp Psychiatry. Sep 2015;48:82-89. [FREE Full text] [CrossRef] [Medline]
  63. Waller H, Freeman D, Jolley S, Dunn G, Garety P. Targeting reasoning biases in delusions: a pilot study of the Maudsley review training programme for individuals with persistent, high conviction delusions. J Behav Ther Exp Psychiatry. Sep 2011;42(3):414-421. [FREE Full text] [CrossRef] [Medline]
  64. Morris RM, Sellwood W, Edge D, Colling C, Stewart R, Cupitt C, et al. Ethnicity and impact on the receipt of cognitive-behavioural therapy in people with psychosis or bipolar disorder: an English cohort study. BMJ Open. Dec 15, 2020;10(12):e034913. [FREE Full text] [CrossRef] [Medline]
  65. Turner DT, Burger S, Smit F, Valmaggia LR, van der Gaag M. What constitutes sufficient evidence for case formulation-driven CBT for psychosis? Cumulative meta-analysis of the effect on hallucinations and delusions. Schizophr Bull. Sep 21, 2020;46(5):1072-1085. [FREE Full text] [CrossRef] [Medline]
  66. Ward T, Garety PA. Fast and slow thinking in distressing delusions: a review of the literature and implications for targeted therapy. Schizophr Res. Jan 2019;203:80-87. [FREE Full text] [CrossRef] [Medline]
  67. Greenwood KE, Gurnani M, Ward T, Vogel E, Vella C, McGourty A, et al. SlowMo Patient‚ Public Involvement (PPI) team. The service user experience of SlowMo therapy: a co-produced thematic analysis of service users' subjective experience. Psychol Psychother. Sep 20, 2022;95(3):680-700. [FREE Full text] [CrossRef] [Medline]
  68. Greenwood K, Robertson S, Vogel E, Vella C, Ward T, McGourty A, et al. SlowMo PatientPublic Involvement (PPI) team. The impact of patient and public involvement in the SlowMo study: reflections on peer innovation. Health Expect. Feb 2022;25(1):191-202. [FREE Full text] [CrossRef] [Medline]
  69. Guidance assessment consultation document for GID-HTE10020 Digital health technologies to help manage symptoms of psychosis and prevent relapse: early value assessment. National Institute for Health and Care Excellence. Jan 18, 2024. URL: [accessed 2024-04-05]
  70. Freeman D, Taylor KM, Molodynski A, Waite F. Treatable clinical intervention targets for patients with schizophrenia. Schizophr Res. Sep 2019;211:44-50. [FREE Full text] [CrossRef] [Medline]
  71. Lambe S, Knight I, Kabir T, West J, Patel R, Lister R, et al. Developing an automated VR cognitive treatment for psychosis: gameChange VR therapy. J Behav Cogn Ther. Apr 2020;30(1):33-40. [FREE Full text] [CrossRef]
  72. Freeman D, Lambe S, Kabir T, Petit A, Rosebrock LM, Yu L, et al. Automated virtual reality therapy to treat agoraphobic avoidance and distress in patients with psychosis (gameChange): a multicentre, parallel-group, single-blind, randomised, controlled trial in England with mediation and moderation analyses. Lancet Psychiatry. May 2022;9(5):375-388. [FREE Full text] [CrossRef]
  73. Altunkaya J, Craven M, Lambe S, Beckley A, Rosebrock L, Dudley R, et al. Estimating the economic value of automated virtual reality cognitive therapy for treating agoraphobic avoidance in patients with psychosis: findings from the gameChange randomized controlled clinical trial. J Med Internet Res. Nov 18, 2022;24(11):e39248. [FREE Full text] [CrossRef] [Medline]
  74. Virtual reality technologies for treating agoraphobia or agoraphobic avoidance: early value assessment. National Institute for Health and Care Excellence. Nov 15, 2023. URL: [accessed 2024-04-05]
  75. Tomlinson A, Furukawa TA, Efthimiou O, Salanti G, De Crescenzo F, Singh I, et al. Personalise antidepressant treatment for unipolar depression combining individual choices, risks and big data (PETRUSHKA): rationale and protocol. Evid Based Ment Health. May 2020;23(2):52-56. [FREE Full text] [CrossRef] [Medline]
  76. Sadeh Y, Denejkina A, Karyotaki E, Lenferink LI, Kassam-Adams N. Erratum: opportunities for improving data sharing and FAIR data practices to advance global mental health-corrigendum. Glob Ment Health (Camb). 2023;10:e20. [FREE Full text] [CrossRef] [Medline]
  77. Fusar-Poli P. Integrated mental health services for the developmental period (0 to 25 years): a critical review of the evidence. Front Psychiatry. Jun 7, 2019;10:355. [FREE Full text] [CrossRef] [Medline]
  78. Birnbaum ML, Rizvi AF, Faber K, Addington J, Correll CU, Gerber C, et al. Digital trajectories to care in first-episode psychosis. Psychiatr Serv. Dec 01, 2018;69(12):1259-1263. [FREE Full text] [CrossRef] [Medline]
  79. McDonald M, Christoforidou E, Van Rijsbergen N, Gajwani R, Gross J, Gumley AI, et al. Using online screening in the general population to detect participants at clinical high-risk for psychosis. Schizophr Bull. Apr 25, 2019;45(3):600-609. [FREE Full text] [CrossRef] [Medline]
  80. Reilly T, Mechelli A, McGuire P, Fusar-Poli P, Uhlhaas PJ. E-clinical high risk for psychosis: viewpoint on potential of digital innovations for preventive psychiatry. JMIR Ment Health. Oct 03, 2019;6(10):e14581. [FREE Full text] [CrossRef] [Medline]
  81. Tiwari M. Young people can help to ease the global mental health crisis. Nat Hum Behav. Jun 2023;7(6):831-832. [FREE Full text] [CrossRef] [Medline]
  82. Torous J, Rodriguez J, Powell A. The new digital divide for digital biomarkers. Digit Biomark. Sep 2017;1(1):87-91. [FREE Full text] [CrossRef] [Medline]
  83. Framework for the use of digital health technologies in drug and biological product development. US Food and Drug Administration. URL: [accessed 2024-04-15]
  84. Kumar A, Ross JS, Patel NA, Rathi V, Redberg RF, Dhruva SS. Studies of prescription digital therapeutics often lack rigor and inclusivity. Health Aff (Millwood). Nov 2023;42(11):1559-1567. [FREE Full text] [CrossRef] [Medline]
  85. Jenkins DA, Martin GP, Sperrin M, Riley RD, Debray TP, Collins GS, et al. Continual updating and monitoring of clinical prediction models: time for dynamic prediction systems? Diagn Progn Res. Jan 11, 2021;5(1):1. [FREE Full text] [CrossRef] [Medline]
  86. Brown P, Prest B, Miles P, Rossi V. The development of national safety and quality digital mental health standards. Australas Psychiatry. Apr 2022;30(2):154-157. [FREE Full text] [CrossRef] [Medline]
  87. Vollmer S, Mateen BA, Bohner G, Király FJ, Ghani R, Jonsson P, et al. Machine learning and artificial intelligence research for patient benefit: 20 critical questions on transparency, replicability, ethics, and effectiveness. BMJ. Mar 20, 2020;368:l6927. [FREE Full text] [CrossRef] [Medline]
  88. Radua J, Koutsouleris N. Ten simple rules for using machine learning in mental health research. Biol Psychiatry (Forthcoming). Nov 18, 2023. [FREE Full text] [CrossRef] [Medline]
  89. Bent B, Wang K, Grzesiak E, Jiang C, Qi Y, Jiang Y, et al. The digital biomarker discovery pipeline: an open-source software platform for the development of digital biomarkers using mHealth and wearables data. J Clin Transl Sci. Jul 14, 2020;5(1):e19. [FREE Full text] [CrossRef] [Medline]
  90. Accelerating medicines Partnership® program for schizophrenia (AMP® SCZ). National Institute of Mental Health. 2023. URL: https:/​/www.​​research/​research-funded-by-nimh/​research-initiatives/​accelerating-medicines-partnershipr -program-schizophrenia-ampr-scz [accessed 2024-04-15]
  91. Constantinides C, Han LK, Alloza C, Antonucci LA, Arango C, Ayesa-Arriola R, ENIGMA Schizophrenia Consortium, et al. Brain ageing in schizophrenia: evidence from 26 international cohorts via the ENIGMA Schizophrenia consortium. Mol Psychiatry. Mar 2023;28(3):1201-1209. [FREE Full text] [CrossRef] [Medline]
  92. Ben-Zeev D, Brian R, Wang R, Wang W, Campbell AT, Aung MS, et al. CrossCheck: integrating self-report, behavioral sensing, and smartphone use to identify digital indicators of psychotic relapse. Psychiatr Rehabil J. Sep 2017;40(3):266-275. [FREE Full text] [CrossRef] [Medline]
  93. Part 1: the connection between substance use disorders and mental illness. National Institute on Drug Abuse. 2022. URL: https:/​/nida.​​publications/​research-reports/​common-comorbidities-substance-use-disorders/​part-1-connection-between -substance-use-disorders-mental-illness [accessed 2024-04-05]
  94. Torous J, Blease C. Return of results in digital phenotyping: ethical considerations for real-world use cases. Am J Bioeth. Feb 31, 2024;24(2):91-93. [CrossRef] [Medline]
  95. Sieck CJ, Sheon A, Ancker JS, Castek J, Callahan B, Siefer A. Digital inclusion as a social determinant of health. NPJ Digit Med. Mar 17, 2021;4(1):52-53. [FREE Full text] [CrossRef] [Medline]
  96. Report on mental health research priorities. The White House. 2023. URL: https:/​/www.​​wp-content/​uploads/​2023/​02/​White-House-Report-on-Mental-Health-Research-Priorities.​pdf [accessed 2024-04-05]
  97. Bucci S, Richardson C, Allan S. iCHARTS. Schizophrenia International Research Society. URL: https://schizophreniare [accessed 2024-04-15]
  98. Cashin AG, Richards GC, DeVito NJ, Mellor DT, Lee H. Registration of health and medical research. BMJ Evid Based Med. Feb 2023;28(1):68-72. [FREE Full text] [CrossRef] [Medline]
  99. Carvalho AF, Köhler CA, Fernandes BS, Quevedo J, Miskowiak KW, Brunoni AR, et al. Bias in emerging biomarkers for bipolar disorder. Psychol Med. May 19, 2016;46(11):2287-2297. [FREE Full text] [CrossRef]
  100. La Sala L, Teh Z, Lamblin M, Rajaram G, Rice S, Hill N, et al. Can a social media intervention improve online communication about suicide? A feasibility study examining the acceptability and potential impact of the #chatsafe campaign. PLoS One. 2021;16(6):e0253278. [FREE Full text] [CrossRef] [Medline]
  101. Online safety act 2023. UK Government. URL: [accessed 2024-04-05]
  102. Kirtley OJ, O'Connor RC. Suicide prevention is everyone's business: challenges and opportunities for Google. Soc Sci Med. Oct 2020;262:112691. [FREE Full text] [CrossRef] [Medline]
  103. Self-harm and the internet: a user-led critique of the proposed online safety bill. National Survivor User Network. 2023. URL: https:/​/www.​​self-harm-and-the-internet-a-user-led-critique-of-the-proposed-online-safety-bill/​ [accessed 2024-04-05]
  104. Ribba B, Peck R, Hutchinson L, Bousnina I, Motti D. Digital therapeutics as a new therapeutic modality: a review from the perspective of clinical pharmacology. Clin Pharmacol Ther. Sep 10, 2023;114(3):578-590. [FREE Full text] [CrossRef] [Medline]
  105. Titov N, Hadjistavropoulos HD, Nielssen O, Mohr DC, Andersson G, Dear BF. From research to practice: ten lessons in delivering digital mental health services. J Clin Med. Aug 17, 2019;8(8):1239. [FREE Full text] [CrossRef] [Medline]
  106. Huckvale K, Torous J, Larsen ME. Assessment of the data sharing and privacy practices of smartphone apps for depression and smoking cessation. JAMA Netw Open. Apr 05, 2019;2(4):e192542. [FREE Full text] [CrossRef] [Medline]
  107. Executive order on the safe, secure, and trustworthy development and use of artificial intelligence. The White House. Oct 2023. URL: https:/​/www.​​briefing-room/​presidential-actions/​2023/​10/​30/​executive-order-on-the-safe-secure -and-trustworthy-development-and-use-of-artificial-intelligence/​ [accessed 2024-04-05]
  108. Artificial intelligence act: council and parliament strike a deal on the first rules for AI in the world. Council of the European Union. 2023. URL: https:/​/www.​​en/​press/​press-releases/​2023/​12/​09/​artificial-intelligence-act-council-and -parliament-strike-a-deal-on-the-first-worldwide-rules-for-ai/​ [accessed 2023-12-11]
  109. Blease C, Kaptchuk TJ, Bernstein MH, Mandl KD, Halamka JD, DesRoches CM. Artificial intelligence and the future of primary care: exploratory qualitative study of UK general practitioners' views. J Med Internet Res. Mar 20, 2019;21(3):e12802. [FREE Full text] [CrossRef] [Medline]
  110. Blease C, Kharko A, Bernstein M, Bradley C, Houston M, Walsh I, et al. Machine learning in medical education: a survey of the experiences and opinions of medical students in Ireland. BMJ Health Care Inform. Feb 2022;29(1):e100480. [FREE Full text] [CrossRef] [Medline]
  111. Blease C, Torous J. ChatGPT and mental healthcare: balancing benefits with risks of harms. BMJ Ment Health. Nov 2023;26(1):e300884. [FREE Full text] [CrossRef] [Medline]
  112. Hswen Y, Abbasi J. AI will-and should-change medical school, says Harvard's dean for medical education. JAMA. Nov 21, 2023;330(19):1820-1823. [FREE Full text] [CrossRef] [Medline]
  113. Torous J, Blease C. Generative artificial intelligence in mental health care: potential benefits and current challenges. World Psychiatry. Feb 2024;23(1):1-2. [FREE Full text] [CrossRef] [Medline]
  114. Expert comment: leading AI nations convene for day one of the UK AI summit. University of Oxford. 2023. URL: [accessed 2024-04-05]
  115. Expert comment: Oxford AI experts comment on the outcomes of the UK AI safety summit. University of Oxford. 2023. URL: https:/​/www.​​news/​2023-11-03-expert-comment-oxford-ai-experts-comment-outcomes-uk-ai-safety-summit [accessed 2024-04-05]
  116. Powell A, Torous J. A patient-centered framework for measuring the economic value of the clinical benefits of digital health apps: theoretical modeling. JMIR Ment Health. Oct 30, 2020;7(10):e18812. [FREE Full text] [CrossRef] [Medline]
  117. Ioannidis J. More time for research: fund people not projects. Nature. Sep 28, 2011;477(7366):529-531. [FREE Full text] [CrossRef] [Medline]
  118. Perret S, Alon N, Carpenter-Song E, Myrick K, Thompson K, Li S, et al. Standardising the role of a digital navigator in behavioural health: a systematic review. Lancet Digit Health. Dec 2023;5(12):e925-e932. [FREE Full text] [CrossRef]
  119. Greenhalgh T, Wherton J, Papoutsi C, Lynch J, Hughes G, A'Court C, et al. Beyond adoption: a new framework for theorizing and evaluating nonadoption, abandonment, and challenges to the scale-up, spread, and sustainability of health and care technologies. J Med Internet Res. Nov 01, 2017;19(11):e367. [FREE Full text] [CrossRef] [Medline]
  120. Fleming T, Bavin L, Lucassen M, Stasiak K, Hopkins S, Merry S. Beyond the trial: systematic review of real-world uptake and engagement with digital self-help interventions for depression, low mood, or anxiety. J Med Internet Res. Jun 06, 2018;20(6):e199. [FREE Full text] [CrossRef] [Medline]
  121. Noori S, Jordan A, Bromage W, Fineberg S, Cahill J, Mathis WS. Navigating the digital divide: providing services to people with serious mental illness in a community setting during COVID-19. SN Soc Sci. 2022;2(8):160-161. [FREE Full text] [CrossRef] [Medline]
  122. Hardy A, Wojdecka A, West J, Matthews ED, Golby C, Ward T, et al. How inclusive, user-centered design research can improve psychological therapies for psychosis: development of SlowMo. JMIR Ment Health. Dec 05, 2018;5(4):e11222. [FREE Full text] [CrossRef] [Medline]
  123. Allan S, Beedie S, McLeod HJ, Farhall J, Gleeson J, Bradstreet S, et al. Using EMPOWER in daily life: a qualitative investigation of implementation experiences. BMC Psychiatry. Aug 17, 2023;23(1):597. [FREE Full text] [CrossRef] [Medline]
  124. Leigh S, Flatt S. App-based psychological interventions: friend or foe? Evid Based Ment Health. Nov 2015;18(4):97-99. [CrossRef] [Medline]
  125. Larsen ME, Nicholas J, Christensen H. A systematic assessment of smartphone tools for suicide prevention. PLoS One. 2016;11(4):e0152285. [FREE Full text] [CrossRef] [Medline]
  126. Wilks CR, Chu C, Sim D, Lovell J, Gutierrez P, Joiner T, et al. User engagement and usability of suicide prevention apps: systematic search in app stores and content analysis. JMIR Form Res. Jul 14, 2021;5(7):e27018. [FREE Full text] [CrossRef] [Medline]
  127. Smith K, Torous J, Cipriani A. Teaching telepsychiatry skills: building on the lessons of the COVID-19 pandemic to enhance mental health care in the future. JMIR Ment Health. Oct 14, 2022;9(10):e37939. [FREE Full text] [CrossRef] [Medline]
  128. Camacho E, Torous J. Introducing an implementation framework for augmenting care with digital technology for early psychosis patients: theory and motivation. J Ment Health. Dec 2022;31(6):816-824. [FREE Full text] [CrossRef] [Medline]
  129. Evans-Lacko S, Corker E, Williams P, Henderson C, Thornicroft G. Effect of the time to change anti-stigma campaign on trends in mental-illness-related public stigma among the English population in 2003–13: an analysis of survey data. Lancet Psychiatry. Jul 2014;1(2):121-128. [FREE Full text] [CrossRef]
  130. Lee J, Lee CM, Park NK. Application of sensor network system to prevent suicide from the bridge. Multimed Tools Appl. Dec 16, 2015;75(22):14557-14568. [FREE Full text] [CrossRef]
  131. Looi M. The man helping prevent suicide with Google adverts. BMJ. Dec 13, 2023;383:2847. [FREE Full text] [CrossRef] [Medline]
  132. Mukherjee A, Ghosh B. An automated approach to prevent suicide in metro stations. In: Proceedings of the 5th International Conference on Frontiers in Intelligent Computing: Theory and Applications. 2016. Presented at: FICTA '16; September 16-17, 2016;753-765; Bhubaneswar, India. URL: [CrossRef]
  133. Lagan S, Sandler L, Torous J. Evaluating evaluation frameworks: a scoping review of frameworks for assessing health apps. BMJ Open. Mar 19, 2021;11(3):e047001. [FREE Full text] [CrossRef] [Medline]
  134. Torous J, Smith KA, Hardy A, Vinnikova A, Blease C, Milligan L, et al. Digital health interventions for schizophrenia: setting standards for mental health. Schizophr Res. Apr 18, 2024;267:392-395. [CrossRef] [Medline]
  135. Kozelka EE, Acquilano SC, Al-Abdulmunem M, Guarino S, Elwyn G, Drake RE, et al. Digital mental health and its discontents: assumptions about technology that create barriers to equitable access. Psychiatr Serv. Mar 01, 2024;75(3):299-302. [FREE Full text] [CrossRef] [Medline]
  136. Fricker M. Epistemic Injustice: Power and the Ethics of Knowing. Oxfordshire, UK. Oxford University Press; 2007.
  137. Hawke LD, Sheikhan NY, Jones N, Slade M, Soklaridis S, Wells S, et al. Embedding lived experience into mental health academic research organizations: critical reflections. Health Expect. Oct 2022;25(5):2299-2305. [FREE Full text] [CrossRef] [Medline]
  138. PARTNERS2 writing collective. Exploring patient and public involvement (PPI) and co-production approaches in mental health research: learning from the PARTNERS2 research programme. Res Involv Engagem. 2020;6:56. [FREE Full text] [CrossRef] [Medline]
  139. Smith KA, Ostinelli EG, Ede R, Allard L, Thomson M, Hewitt K, et al. Assessing the impact of evidence-based mental health guidance during the COVID-19 pandemic: systematic review and qualitative evaluation. JMIR Ment Health. Dec 22, 2023;10:e52901. [FREE Full text] [CrossRef] [Medline]

CDC: consensus development conference
CHR-P: clinical high-risk state for psychosis
FDA: Food and Drug Administration
FEP: first-episode psychosis
GALENOS: Global Alliance for Living Evidence on Anxiety, Depression, and Psychosis
LMIC: low- and middle-income country
PPI: patient and public involvement
PPIE: patient and public involvement and engagement
SMI: severe mental illness

Edited by T de Azevedo Cardoso; submitted 07.02.24; peer-reviewed by K Ludlow, S Stroup, N Josifovski; comments to author 23.02.24; revised version received 08.03.24; accepted 21.03.24; published 08.05.24.


©Katharine A Smith, Amy Hardy, Anastasia Vinnikova, Charlotte Blease, Lea Milligan, Diego Hidalgo-Mazzei, Sinéad Lambe, Lisa Marzano, Peter J Uhlhaas, Edoardo G Ostinelli, Gerard Anmella, Caroline Zangani, Rosario Aronica, Bridget Dwyer, John Torous, Andrea Cipriani. Originally published in JMIR Mental Health (, 08.05.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Mental Health, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.