This paper is in the following e-collection/theme issue:

Methods and New Tools in Mental Health Research (425) Web-based and Mobile Health Interventions (3119) Ecological Momentary Assessment (EMA) (297) Anxiety and Stress Disorders (1006) Telepsychiatry (146) Personal Health Records, Patient-Accessible Electronic Health Records, Patient Portals (542) Electronic Health Records (967)

Published on in Vol 9, No 8 (2022): August

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/36430, first published .
A Web-Based Application for Personalized Ecological Momentary Assessment in Psychiatric Care: User-Centered Development of the PETRA Application

A Web-Based Application for Personalized Ecological Momentary Assessment in Psychiatric Care: User-Centered Development of the PETRA Application

A Web-Based Application for Personalized Ecological Momentary Assessment in Psychiatric Care: User-Centered Development of the PETRA Application

Authors of this article:

Fionneke M Bos1, 2 Author Orcid Image ;   Lino von Klipstein2 Author Orcid Image ;   Ando C Emerencia3 Author Orcid Image ;   Erwin Veermans1 Author Orcid Image ;   Tom Verhage2 Author Orcid Image ;   Evelien Snippe2 Author Orcid Image ;   Bennard Doornbos4 Author Orcid Image ;   Grietje Hadders-Prins1 Author Orcid Image ;   Marieke Wichers2 Author Orcid Image ;   Harriëtte Riese2 Author Orcid Image
Article Authors Cited by (10) Tweetations (16) Metrics

    Original Paper

    1Rob Giel Research Center, Department of Psychiatry, University Medical Center Groningen, University of Groningen, Groningen, Netherlands

    2Interdisciplinary Center Psychopathology and Emotion Regulation, Department of Psychiatry, University Medical Center Groningen, University of Groningen, Groningen, Netherlands

    3Research Support, Faculty of Behavioral and Social Sciences, University of Groningen, Groningen, Netherlands

    4Lentis Research, Groningen, Netherlands

    Corresponding Author:

    Fionneke M Bos, PhD

    Rob Giel Research Center

    Department of Psychiatry

    University Medical Center Groningen, University of Groningen

    Hanzeplein 1

    Groningen, 9700 RB

    Netherlands

    Phone: 31 50 361 4860

    Email: f.m.bos01@umcg.nl


    Background: Smartphone self-monitoring of mood, symptoms, and contextual factors through ecological momentary assessment (EMA) provides insights into the daily lives of people undergoing psychiatric treatment. Therefore, EMA has the potential to improve their care. To integrate EMA into treatment, a clinical tool that helps clients and clinicians create personalized EMA diaries and interpret the gathered data is needed.

    Objective: This study aimed to develop a web-based application for personalized EMA in specialized psychiatric care in close collaboration with all stakeholders (ie, clients, clinicians, researchers, and software developers).

    Methods: The participants were 52 clients with mood, anxiety, and psychotic disorders and 45 clinicians (psychiatrists, psychologists, and psychiatric nurses). We engaged them in interviews, focus groups, and usability sessions to determine the requirements for an EMA web application and repeatedly obtained feedback on iteratively improved high-fidelity EMA web application prototypes. We used human-centered design principles to determine important requirements for the web application and designed high-fidelity prototypes that were continuously re-evaluated and adapted.

    Results: The iterative development process resulted in Personalized Treatment by Real-time Assessment (PETRA), which is a scientifically grounded web application for the integration of personalized EMA in Dutch clinical care. PETRA includes a decision aid to support clients and clinicians with constructing personalized EMA diaries, an EMA diary item repository, an SMS text message–based diary delivery system, and a feedback module for visualizing the gathered EMA data. PETRA is integrated into electronic health record systems to ensure ease of use and sustainable integration in clinical care and adheres to privacy regulations.

    Conclusions: PETRA was built to fulfill the needs of clients and clinicians for a user-friendly and personalized EMA tool embedded in routine psychiatric care. PETRA is unique in this codevelopment process, its extensive but user-friendly personalization options, its integration into electronic health record systems, its transdiagnostic focus, and its strong scientific foundation in the design of EMA diaries and feedback. The clinical effectiveness of integrating personalized diaries via PETRA into care requires further research. As such, PETRA paves the way for a systematic investigation of the utility of personalized EMA for routine mental health care.

    JMIR Ment Health 2022;9(8):e36430

    doi:10.2196/36430

    Keywords

    eHealth; clinical implementation; ecological momentary assessment; experience sampling method; smartphone; mobile health; mHealth; personalized diaries; personalized psychiatry; client-tailored; cocreation; shared decision-making; mobile phone 


    Background

    Ecological momentary assessment (EMA; also referred to as the experience sampling method or ambulatory monitoring) [1], is an increasingly popular self-monitoring method to gain insight into daily life experiences relevant to mental health problems. With EMA, clients monitor their daily life as it is lived via short questionnaires (ie, diaries) on their smartphones. Several times per day, they rate their experienced symptoms, affect, thoughts, activities, and (social) contexts [2]. In research, EMA has already been applied to improve our understanding of mental disorders by examining how symptoms fluctuate in daily life, in relation to stressful and rewarding events, and in a diverse range of mental disorders [3-6]. Increasingly, researchers and scientists are recognizing the potential of EMA as a tool for improving mental health care [7-10]. Self-monitoring through EMA and visualizing the data in EMA feedback may offer clients and clinicians relevant insights into the person-specific mechanisms that contribute to clients’ mental health problems [8].

    EMA as a Self-monitoring Tool

    In many medical disciplines, self-monitoring of, for example, daily blood pressure or glucose levels, is already a common practice and is informative for diagnosis and treatment planning [11,12]. Similarly, mental health treatment may benefit from the self-monitoring of daily symptoms, (social) contextual experiences, and treatment-related factors [13]. Indeed, for this reason, many mental health treatments such as cognitive behavioral therapy already use some form of (paper-and-pencil) self-monitoring [14], and routine outcome monitoring is well established [15]. However, EMA is both more detailed and more frequent than existing monitoring methods. Typically, EMA assesses momentary experiences (“right now, I feel cheerful”), which can be rated on a visual analog scale ranging from 0 to 100 (“not at all” to “very much,” respectively). In mental health care, momentary experiences can pertain to symptoms (eg, sadness, restlessness, or hearing voices) or other factors relevant to treatment (eg, harmful or helpful thoughts and behaviors). The number of EMA assessments per day, as well as the timing and duration of the assessments, depend on the goal of the EMA diary and the experienced burden of completing the diaries. Most EMA studies thus far have used 3 to 10 assessments per day for 1 to 2 weeks [16,17], whereas some have monitored for up to 4 months [18].

    Clinical Promise of EMA

    Self-monitoring through EMA is suggested to improve clients’ self-management by increasing their insights into their well-being. By frequently reflecting on one’s symptoms and learning what types of activities or situations positively or negatively influence well-being, clients may become more in control [19]. As such, EMA self-monitoring may be an intervention in itself. In addition, EMA data can be visualized to enhance the understanding of clients’ personal mechanisms that contribute to mental health problems. Such EMA feedback could be as simple as demonstrating variability in affect or showing contexts in which clients have complaints. As EMA is suggested to be unaffected by memory biases [20], such feedback could provide a more reliable overview of how clients fared between treatment sessions. This could improve shared decision-making and the therapeutic alliance (ie, the client-clinician relationship, involving an agreement on the goals and tasks of treatment, as well as their interpersonal bonds [21]) [10,19]. Therefore, EMA feedback could form the basis of a more collaborative approach to diagnosis and intervention, where clients and therapists use EMA data to decide together on the next treatment step.

    Empirical Evidence for Clinical Effectiveness of EMA

    The first empirical investigations into the clinical effectiveness of EMA are promising. In qualitative research, clients and clinicians describe the beneficial effects of EMA on client self-management, therapeutic alliance, and treatment effectiveness [22-25]. This is supported by a clinical trial demonstrating that adding EMA monitoring and feedback to antidepressant treatment was more effective in reducing depressive symptoms than using antidepressant medication alone [26], and it improved clients’ feelings of empowerment [27], emotion differentiation [28], and behavioral change [29]. However, 2 other clinical trials did not replicate these beneficial effects in individuals with depression in routine care [30] and individuals reporting a loss of interest and pleasure (anhedonia) [31]. This discrepancy may be because in clinical trials thus far, EMA diaries have been standardized for all clients. This insufficiently relates EMA feedback to clients’ treatment goals, making the EMA less effective, and may increase the burden for clients, who have to frequently answer questions that are irrelevant to their current situation [32]. Therefore, personalization of the EMA seems to be a core requirement for successful clinical implementation.

    Developing Web-Based EMA Technology

    Given that clients, clinicians, and researchers align in their positive evaluation of the clinical utility of EMA, the field needs a flexible, user-friendly, and evidence-based application for the use of personalized EMA diaries in mental health care. More specifically, this entails the development of a digital infrastructure that enables clients and clinicians to intuitively construct personalized EMA diaries, deliver them to clients’ smartphones, and dynamically visualize the gathered EMA data.

    In this paper, we describe the development of Personalized Treatment by Real-time Assessment (PETRA), a web-based application that enables the use of personalized EMA diaries in Dutch mental health care. PETRA was built into electronic health record (EHR) systems to ensure sustainability and easy access. The web application comprises a decision aid to help clients and clinicians construct personalized EMA diaries, an EMA diary item repository, a diary delivery system, and a feedback module to visualize the gathered EMA data. PETRA was developed in close collaboration with clients and clinicians as they will be the primary users. Involving them early in the development process is crucial to ensure the uptake of PETRA.


    Study Design

    PETRA was developed using the road map of the Center for eHealth Research (CeHRes) of the University of Twente, the Netherlands [33]. The road map comprises 5 iterative phases. In the first 2 phases of the CeHRes road map (contextual inquiry and value specification), we conducted focus groups and interviews to identify the needs and requirements of the relevant stakeholders, clients and clinicians, for a personalized diary web application [34]. The next phase (design) comprised an iterative design process in which prototypes were developed, evaluated, and adapted frequently based on feedback from clients and clinicians [35]. In this paper, we describe the results of these 3 phases and the resulting PETRA web application. The final 2 phases of the CeHRes road map (operationalization and summative evaluation) will involve implementing the web application and evaluating its use in practice.

    During each of these phases, stakeholders were actively involved to ensure their perspectives are integrated into PETRA. The CeHRes road map follows human-centered design principles, attempting to understand the needs of users and actively involving them in designing solutions to meet these needs [36]. Therefore, the PETRA project team was multidisciplinary and comprised the core scientific team; a scientific programmer; the RoQua development team to ensure sustainable embedding in existing mental health care software; a user experience (UX) designer to integrate the needs of all stakeholders; and client and clinician representatives engaged in interviews, focus groups, and usability sessions. All team members critically reviewed the prototypes, which were then continuously adapted and re-evaluated (Figure 1).

    ‎
    Figure 1. Overview of the developmental phases of PETRA (Personalized Treatment by Real-time Assessment). EMA: ecological momentary assessment.
    View this figure

    Setting and Participants

    The PETRA web application was developed from 2016 to 2022 at the Department of Psychiatry, University Medical Center Groningen (the Netherlands), facilitated by iLab, a collaborative initiative to implement scientific innovations into clinical practice [37]. PETRA’s software was built into RoQua [38], a nonprofit web-based questionnaire system for research and care. RoQua is fully integrated into the EHR systems of several mental health care facilities in the Netherlands, which are part of the Rob Giel Research Center (RGOc) [39]. The facilities participating in this collective mostly provide secondary or tertiary (specialist) mental health care services to clients presenting with a diverse range of mental disorders, such as major depression, bipolar disorder, anxiety disorders, and psychosis.

    Participant Recruitment

    Participants comprised a convenience sample of clients and clinicians recruited from RGOc facilities, who took part in interviews, focus groups, and usability sessions. All clients with a (self-reported) history of mental health problems (regardless of type) were eligible to participate in the study. Similarly, all clinicians working in an RGOc facility were eligible. Clinicians could be psychiatrists, psychologists, or psychiatric nurses. Participant characteristics for the first 2 phases can be found in the studies by Bos et al [22,23] but were not systematically collected in the design phase. The clients were informed of the PETRA project by the RGOc coordinator of the client representatives or their clinician. If interested, clients received a flyer and, after providing consent, were invited for an interview, focus group, or usability session. The clients were reimbursed for their time (€10 [US $10.18] per hour). Clinicians were invited based on prior participation in ongoing diary studies or their expressed interest in PETRA during (department) presentations. Both clients and clinicians were invited to join multiple sessions to offer feedback on the adaptations that were made.

    Ethics Approval

    The University Medical Center Groningen Medical Ethics Committee exempted this research from a full review (reference number 201900401). Clients signed a written informed consent form to participate in the focus groups, interviews, and usability sessions.

    Phases 1 and 2: Identifying the Requirements of Clients and Clinicians

    Phases 1 and 2 aimed to identify perceived problems in the status quo of mental health treatment, perceived advantages and challenges of personalized electronic diaries, and core requirements for the EMA web application to be developed. In-depth qualitative interviews and focus groups were conducted with 40 clients and 27 clinicians between June 2016 and March 2018 on the perceived utility of EMA for psychiatric care and perceived important requirements for a clinical EMA tool. Focus groups with clients or clinicians were conducted in groups of 3 to 7 participants until data saturation was reached (ie, no new themes emerged). The focus groups and interviews were part of 2 qualitative studies that broadly focused on identifying applications of EMA in diverse clinical contexts and implementation requirements. For more details, we refer the reader to previous studies [22,23] (for interview guides, see Table S1 in Multimedia Appendix 1). In this paper, we specifically discuss the resulting recommendations for the development of an EMA tool on which PETRA was based. Furthermore, in this phase, we compiled a list of potentially relevant constructs to assess personalized EMA diaries. On the basis of previous EMA research and subsequent suggestions by clients and clinicians, each construct was linked to related EMA diary items (eg, “I feel anxious” for the construct “Anxiety”).

    Phase 3: Development of PETRA Prototypes

    Overview

    On the basis of the recommendations in phases 1 and 2, we designed prototypes for the web-based personalized EMA tool PETRA. These were continuously tested and redesigned in usability sessions with clients and clinicians held from May 2019 to August 2021. Owing to the COVID-19 pandemic, all usability sessions after March 2020 were conducted via video calls (for interview guides, see Table S1 in Multimedia Appendix 1). At each session, the researcher FB and UX designer TV were present. After each session, they wrote a report with the main recommendations. The participants were invited to provide feedback on this report.

    Sessions With Clinicians

    On the basis of phases 1 and 2, clinicians were considered the primary users of the decision aid and feedback module: they would need to integrate PETRA into their working routines and introduce it to clients. Therefore, most of the usability sessions were held with clinicians. A total of 18 clinicians participated in ≥1 of the 31 usability sessions (5/18, 28% were men). Approximately 77% (24/31) of the sessions were held individually, and 23% (7/31) of the sessions took place in groups of 2 to 3 clinicians. Most clinicians were psychologists (10/18, 56%), followed by psychiatrists (4/18, 22%) and psychiatric nurses (4/18, 22%). Of the 18 clinicians, 7 (39%) participated once, 8 (44%) participated twice, and 3 (17%) participated thrice. During each session, clinicians were asked to describe their thoughts and expectations regarding EMA and their requirements for a diary web application. Subsequently, the clinicians were walked through a digital prototype of the PETRA web application. They were encouraged to describe what they saw on the screen and provide their initial thoughts on each page (the think-aloud method). When the clinicians offered suggestions for improvement, follow-up questions were asked until an in-depth understanding of the proposed adaptation was reached. The clinicians were also invited to explain how they would use PETRA in treatment and to suggest new features or feedback types.

    Sessions With Clients

    A total of 12 clients participated in ≥1 of the 4 usability sessions, which were similar to those with the clinicians. Clients were currently or had been previously in treatment for mood, psychotic, or anxiety disorders (8/12, 67% were men). Of the 12 clients, 9 (75%) participated in 1 usability session, and 3 (25%) participated in 2 sessions. The sessions took place in groups of 3 to 7 clients. One of the sessions specifically focused on designing low-fidelity (paper-and-pencil) prototypes for EMA diary feedback. The other usability sessions were used to obtain feedback on the digital high-fidelity prototypes of the PETRA web application. Feedback was sought on the workflow and its intended use in treatment.

    Data Analysis

    The interviews and focus groups were transcribed and analyzed thematically using the Qualitative Analysis Guide of Leuven [40]. Briefly, this approach involves the identification of important themes within qualitative data, which are then iteratively verified against the transcripts. Further details are provided elsewhere [22,23]. The resulting themes regarding the requirements of a clinical EMA tool are described in the Results section.

    After each usability session, wireframes were created with the proposed changes to the PETRA web application (see Figures 2-4 for example wireframes or a slide-by-slide overview on the PETRA website [41]). A wireframe is a blueprint that explicates the flow throughout the web application. In total, 9 low-fidelity wireframes of the overarching structure and workflow of PETRA were designed, each version improving upon the previous one. The final ninth wireframe was used for the next stage, which focused on determining the types of content, visuals, and graphs depicted on the screens. This stage comprised 15 high-fidelity wireframes, which were again improved iteratively.

    ‎
    Figure 2. The first wireframe that was designed of the first page of the decision aid, on which participants select the goal of the EMA diary. The usability sessions indicated a need for an introduction and a more detailed progress menu. More detailed wireframes can be found here [41]. EMA: ecological momentary assessment.
    View this figure
    ‎
    Figure 3. The ninth wireframe that was designed of the first page of the decision aid, on which participants select the goal of the EMA diary. As suggested in usability sessions, it incorporates an introduction and a more detailed progress menu. More detailed wireframes can be found here [41]. EMA: ecological momentary assessment.
    View this figure
    ‎
    Figure 4. The final (visual) design of the first page of the decision aid, on which participants select the goal of the EMA diary. More detailed wireframes can be found here [41]. EMA: ecological momentary assessment.
    View this figure

    Phases 1 and 2: Core Requirements of PETRA

    Overview

    The first 2 phases of the CeHRes road map resulted in an in-depth overview of clients’ and clinicians’ perceptions of perceived problems in the status quo of mental health treatment, perceived advantages and challenges of personalized electronic diaries, and core requirements for the EMA web application to be developed (Table 1). The core requirements were (1) extensive possibilities for personalization, (2) resource efficiency, (3) scientifically grounded and provided by a trusted source, and (4) meeting the demands regarding privacy and safe data storage. Relevant quotes can be found in Table S2 in Multimedia Appendix 1.

    Table 1. Results of phases 1 and 2 of the Center for eHealth Research road map in the development of the PETRA (Personalized Treatment by Real-time Assessment) application.
    ThemeDescription
    Problems clients and clinicians identified in the status quo of mental health treatment

    		Reliability or efficiency of current monitoring instruments
    • Often paper-and-pencil based and deemed less reliable and efficient
      • ‎
    • Often focus too much on symptoms and fail to take important contextual factors and strengths into account
      • ‎
    • Usually only administered 1 to 2 times per month; missing relevant information about clients’ daily lives throughout the day
      • ‎
    • Usually not person-specific enough to be directly relevant to clients
      • ‎

    		Insights
    • Limited insight into overall progress and treatment outcomes
      • ‎
    • Limited insight into the effects of treatment or lifestyle adaptations
      • ‎
    • Limited insight into the frequency and severity of symptoms and when they occur
      • ‎
    • Limited insight into triggers and relapse signals
      • ‎
    • Recall of well-being in between sessions is biased by current mood or otherwise difficult to recall for clients (recall bias)
      • ‎

    		Therapeutic alliance
    • Knowledge imbalance between client and clinician, weakening the therapeutic alliance
      • ‎
    • Current monitoring instruments are often not discussed in treatment, weakening the therapeutic alliance
      • ‎
    • Limited contact between client and clinician in between sessions
      • ‎
    • Client does not believe in the current treatment approach as the effects are not clearly visible
      • ‎
    Perceived advantages of personalized electronic diaries

    		Reliability or efficiency
    • More reliable and efficient as assessments take place multiple times per day via smartphone and are less easily forgotten or ignored
      • ‎

    		Insight
    • Offers insights into progress, treatment effects, and the flow of symptoms throughout daily life, thereby increasing client self-management
      • ‎
    • Offers insights into contextual factors and personal strengths
      • ‎

    		Therapeutic alliance
    • Client and clinicians share the same information and can, therefore, collaborate more easily
      • ‎
    • Relevant diary feedback can more easily be integrated in treatment, strengthening the therapeutic alliance and clients’ trust in their clinician
      • ‎
    Perceived challenges in constructing personalized electronic diaries

    		Diary construction
    • What kind of clinical questions can be answered with electronic diaries?
      • ‎
    • How to formulate or select relevant and valid diary items?
      • ‎
    • How to determine the number of assessments per day, balancing client burden and the number needed to answer a clinical question?
      • ‎
    • How to determine the necessary diary schedule (eg, time-contingent or event-contingent) for a clinical question?
      • ‎
    • How to make sure the diary maximizes its advantages and minimizes disadvantages?
      • ‎

    		Diary feedback
    • How to automatically analyze and visualize the diary data without the need for statistical knowledge of the clinician?
      • ‎
    • How to interpret the diary feedback in a clinical context?
      • ‎
    Personalization

    All clients and clinicians stressed the necessity for extensive personalization of the diary content, schedule, and duration. They suggested that clients and clinicians should be able to compile different EMA items based on the clients’ current care needs in a flexible manner. This means that EMA diaries should be transdiagnostic and adaptable in diverse stages of treatment. The alternative, where diaries are constructed based on client diagnosis, was viewed as limiting, more burdensome, and not in line with the real-world situation where many clients present with symptoms of multiple diagnoses. Furthermore, EMA diaries should not only focus on symptoms but also on personal strengths and (social) contextual factors. This need for personalization has also been highlighted in research on monitoring in general [32,42,43].

    Resource Efficiency

    Both clients and clinicians emphasized that the use of EMA in treatment should be easy, time efficient, and fit with the existing workflow of clinicians. This means that constructing personalized EMA diaries and interpreting the resulting feedback should be intuitive and cost clients and clinicians limited time, effort, and resources. This corresponds with research demonstrating that eHealth is most useful if it matches client and clinician expectations [44]. Thus, the web application should be largely automated and integrated into existing digital infrastructures used in mental health care, preferably the clinician’s EHR system.

    Scientific Background

    Clients and clinicians indicated that they found it difficult to assess the effectiveness of eHealth applications, which corresponds to eHealth research [45,46]. Therefore, they stressed the need for a web application that is scientifically grounded so that the resulting diary and feedback are valid, trustworthy, and according to scientific standards for EMA [16]. Furthermore, as clinicians are not primarily trained in the innovative methods and statistics necessary for constructing and analyzing diary data, they indicated that the web application should help them with constructing diaries and interpreting feedback.

    Privacy

    The final core requirement clients and clinicians mentioned was that the gathered diary data should be safely stored, protecting the privacy of clients, according to data protection laws (eg, the General Data Protection Regulation [GDPR] law of the European Union). Given the far-reaching personalization of the diaries, the data should be treated accordingly.

    Phase 3: Design of PETRA

    Overview

    The core requirements for a web application for personalized EMA diaries were translated into the current design of PETRA. PETRA was developed as a web-based application, built into EHR systems via RoQua. As per the suggestion of clinicians, this means that clients and clinicians do not need additional log-in data, and clients do not need to install an app on their phones, allowing easy access for both. PETRA was developed for adult clients diagnosed with mood, anxiety, or psychotic disorders receiving treatment in specialized mental health care. Proficiency in the Dutch language and client possession of a smartphone with internet access are required to use PETRA.

    PETRA comprises four main parts: (1) a decision aid, (2) an item repository of diary items, (3) an SMS text message–based diary delivery system, and (4) a feedback module. An overview of the various parts of PETRA is shown in Figure 5. Each part was iteratively evaluated and adapted based on the feedback of clients and clinicians, who mostly agreed on the design of PETRA’s features. PETRA was designed to be used collaboratively: clients and clinicians together decide on the goal, content, and schedule of the EMA diary and interpret the feedback together. Clients have access to the decision aid and feedback modules via their clinician. PETRA’s steps are outlined in more detail in the following sections.

    ‎
    Figure 5. Overview of PETRA.
    View this figure
    Decision Aid

    A decision aid was developed to assist clients and clinicians in constructing personalized EMA diaries. This decision aid is intended to be used collaboratively by clients and clinicians. The following specifications can be personalized via the decision aid: (1) the content (or diary items), (2) the number of assessments per day, (3) the diary schedule (timing of the assessments), and (4) the length of the EMA monitoring period. These specifications depend on the intended goal of the diary, anticipated burden for the client, and the client’s main symptom profile.

    PETRA’s decision aid was developed based on the results of phases 1 and 2 and guidelines for designing EMA studies [16,47]. First, we developed a list of clinical goals for which personalized EMA may be insightful. Such goals include, for example, gaining insights into the effects of lifestyle on well-being or monitoring the effects of medication. Each goal is linked to potentially relevant EMA items, a preferred number of assessments, a preferred diary schedule, and a preferred diary period (Tables S3-S5 in Multimedia Appendix 1). This matching process was based on clinical EMA research demonstrating the requirements for different types of research questions. For example, if gaining insights into contextual factors is the primary goal, the best practice in EMA research is to opt for a semirandom diary schedule, in which the assessments are randomly generated within multiple predefined time blocks. This is generally considered the most ecologically valid way of gaining an overview of someone’s activities and events during the day [48].

    In addition to the goal, clients and clinicians select the symptom profile they wish to focus on. We developed a list of prevalent symptom profiles in specialist mental health care (eg, depression, anxiety, and psychosis). Each symptom profile was linked to potentially relevant EMA items. For example, relevant constructs for clients in depression treatment could be mood, anhedonia, worry, self-esteem, and resilience.

    The decision aid first prompts clients and clinicians to decide on the clinical goal, symptom profile, and preferred diary schedule (see Figure 5 or the PETRA website for a video [49]). In the next 2 steps, based on these decisions, PETRA automatically preselects the relevant EMA items and proposes a diary schedule. This preselection can be adjusted according to personal preferences while remaining within valid boundaries (eg, if the goal is to assess the relationship between activities and symptoms, only a semirandom schedule of at least five assessments per day can be selected [48]). Clients and clinicians can deselect irrelevant items or select additional items. The PETRA decision aid further includes a preview function, which visualizes the diary as the patient views it on their phone, and a burden indicator, which is a gradient bar that changes color according to the number of items in the diary to remind clients and clinicians to discuss the burden when constructing diaries (S6 and S7 in Multimedia Appendix 1 [16,17,47,50,51]). Although research currently offers no clear-cut guidelines on the diary length and associated burden [17,50], a study demonstrated that 60 items per diary were perceived as more burdensome than 30 [51], and another study showed that EMA studies, on average, comprised 30 items [16]. It is likely that the number of items and associated burden differs between, as well as within, individuals and across populations and treatment stages. Therefore, the burden indicator was based on our experience in designing EMA studies and will be updated based on the UXs and emerging research.

    EMA Item Repository

    The EMA item repository was developed as clients and clinicians indicated a preference for a menu in which they could select clinically relevant constructs (eg, mood or hearing voices) that automatically tie into preworded items. Although the EMA field is yet to establish the validity of EMA items, best practices have been developed in EMA research on the wording of EMA items [16,47,50]. The PETRA items were developed based on these best practices.

    The EMA item repository was developed by the scientific team based on extensive experience with the items used in EMA studies and subsequent evaluations and suggestions by clients and clinicians. First, we determined potentially relevant constructs to be assessed; that is, constructs pertaining to symptoms, strengths, and contextual factors. For each construct, we formulated 4 to 10 diary items.

    This resulted in four item sets based on the EMA assessment schedule: (1) at semirandom moments (128 items), (2) at fixed moments (126 items), (3) once daily in the evening (62 items), and (4) once daily in the morning (18 items). EMA assessments can occur either at fixed time points (eg, at noon) or at random points in predefined intervals (eg, somewhere between 10:30 AM and noon), termed fixed or semirandom EMA designs, respectively. Fixed designs are believed to be less burdensome for clients, whereas semirandom designs supposedly provide a more representative overview of daily experiences [2,48]. The item sets were developed as experiences can be expected to operate on different timescales and, therefore, require different wording. For example, mood usually fluctuates throughout the day and requires multiple assessments for valid estimation. However, excessive buying, a symptom of mania, may only vary from day to day and can be sufficiently assessed only in the evening. Most items are answered on 0 to 100 visual analog scales (ranging from “not at all” to “very much”), some via selecting categorical multiple-choice checkboxes (eg, indicating activities, coping strategies, or locations), and some via entering text (eg, descriptions of events). The full lists of EMA items are available in item S9 in Multimedia Appendix 1 [16-18,22,23,26,37,42,47,50-72] (Bos, F, unpublished data, May 2022; Jenner, JGL, unpublished data, October 2006; Bringmann, L, unpublished data, January 2021). In addition to the item sets, clients and clinicians can also formulate a limited number of person-specific items.

    Semirandom and fixed diary schedules are mutually exclusive. Once-daily items are often an informative addition to these more frequent assessment schedules. Therefore, in the decision aid, the constructs of the once-per-day schedules are integrated within the construct structure of the semirandom and fixed schedules. This ensures that clients and clinicians can select items based on the construct. For example, the construct (hypo)mania provides items assessed in the moment (semirandom or fixed), as well as morning and evening items. Although the EMA items are grouped into constructs in the decision aid, they are reordered in the actual diary to balance positive and negative items and ensure that more fleeting momentary emotions are assessed before contextual experiences [16,50].

    EMA Delivery System

    Once clients and clinicians have constructed the EMA diary, clients receive SMS text messages on their personal smartphones. The messages are sent according to the chosen diary schedules and contain a link to the EMA diary items, which are presented and filled in via the browser. Clients receive reminders if the diary has not been completed within 20 minutes, and the link is disabled after 30 minutes (semirandom and fixed schedules) or 3 hours (once-daily schedules).

    Feedback Module

    The gathered EMA data for each client are visualized in the PETRA feedback module, which clients and their clinicians can discuss during a regular treatment session (see Figures 6-8 or the PETRA website for a video [49]). This module is grouped around three themes identified by clients and clinicians as relevant to psychiatric treatment: the fluctuations and changes in symptoms and strengths over time, associations between (social) contexts and symptoms and strengths, and the impact of pleasant and unpleasant events (see item S8 in Multimedia Appendix 1 for more details). The early versions of the feedback module were based on the feedback module developed in the Therap-i trial [52]. The PETRA feedback module is updated in real time and offers dynamic feedback options, and each graph can be adapted and can display any of the assessed EMA items.

    Clients and clinicians indicated that the understanding of EMA data is greatly facilitated by qualitative descriptions of the context provided by clients. Therefore, clients are actively encouraged to provide such descriptions. In the feedback module, these qualitative evaluations can be linked to high and low scores to provide contextual information on moments the client experiences high or low symptoms. Furthermore, all entered text is summarized in word clouds to intuitively summarize the main themes that emerged from the qualitative data and provide context for when these themes emerged. The PETRA feedback is descriptive as statistical analysis of diary data was considered too complex, unclear, and vulnerable to misinterpretation [73,74]. However, the modular structure of the feedback module ensures that novel feedback can be easily added in later stages. For example, if empirical evidence shows that certain patterns in EMA data can be linked to treatment response or outcome, PETRA can offer this information directly in its feedback module [75-77]. Finally, the module has a report function, which enables users to save graphs and comments (eg, therapy notes) for later use.

    ‎
    Figure 6. An interactive graph on the variation in mood and symptoms in the feedback module. All continuous ecological momentary assessment (EMA) diary items can be selected from the menu on the right side of the screen. Clicking on any of the assessment points will activate a sliding pane depicted in Figure 7. For a video, see here [49].
    View this figure
    ‎
    Figure 7. Sliding panel in the feedback module that provides relevant contextual information for a specific moment in the time series and surrounding moments. Moments are selected by clicking on assessment points in the graph depicted in Figure 6. For a video, see here [49].
    View this figure
    ‎
    Figure 8. A word cloud of all freely entered text in the feedback module. Clicking on a word provides more context on when this word was used. For a video, see here [49].
    View this figure

    Principal Findings

    In this paper, we outline the development of PETRA, a web-based application for personalized EMA in Dutch psychiatric care. Interviews, focus groups, and usability sessions with clients and clinicians demonstrated that they expected an added value of integrating personalized EMA into treatment. Crucial requirements entailed a personalized and user-friendly diary tool built on scientific foundations, which adheres to privacy regulations. The subsequent thorough cocreation design process ensured that PETRA was considered intuitive, user-friendly, and useful for clients and clinicians.

    By involving clients and clinicians from the start of the development, PETRA can be considered a significant advancement of current EMA (web-based) applications that are mostly targeted at researchers. By systematically integrating the perspectives of clients and clinicians with those of researchers and software developers, the PETRA web application has promise as a tool for assisting both clients and clinicians in the personalized treatment of mental health problems.

    Limitations

    In this project, we attempted to include a diverse group of clients and clinicians with differing levels of interest in using personalized EMA for treatment. However, this was a relatively small sample, and participants were selected based on their interest in EMA. Therefore, it cannot be ruled out that we overestimated the eventual uptake in clinical practice. As EMA requires significant time and resources from both clients and clinicians, it may not appeal to everyone [78,79], which was also highlighted in the focus groups and interviews. However, research indicates that EMA is feasible for diverse psychiatric populations [47,80,81], including individuals diagnosed with depression [26,27,82], bipolar disorder [83,84], anxiety [85], and psychotic [53,86,87] disorders, from youth [88,89] to older adult populations [90]. Feasibility will likely not depend on diagnosis but rather on the intrinsic motivation of clients, symptom severity, and stage of care [22,23].

    Furthermore, there are some limitations that pertain specifically to the PETRA tool. First, PETRA is only accessible to clients via their clinicians via their EHRs. This can be a psychiatrist, psychologist, or psychiatric nurse affiliated with a RoQua mental health care facility. We are currently developing client access to PETRA directly via their personal health record system to enable clients to interpret feedback independently of their clinician. This is in line with clients’ desire to have access to their own data [23]. Second, proficiency in the Dutch language and client possession of a smartphone with internet access are required to use PETRA. This implies that PETRA is currently unavailable in other languages.

    Comparisons With Prior Studies

    Thus far, EMA software development has mainly focused on researchers [91-94], who have different needs from clients and clinicians. However, there are three known EMA platforms for clinical purposes: MindLogger [94], m-Path [95], and PsyMate [96,97]. MindLogger does not solely focus on EMA but also on other self-monitoring techniques and mobile interventions. As m-Path and PsyMate are most similar to PETRA, we will briefly review them here. Both platforms offer a web environment for clinicians and clients and a smartphone app that delivers EMA diaries to clients. PsyMate provides a standardized EMA diary: a set list of EMA items (which may differ slightly according to the target group) and 10 assessments per day on a semirandom schedule. m-Path is an open-source platform that targets both researchers and clinicians. It offers full freedom with creating a diary but no decision aid, requiring knowledge of constructing EMA diaries. Similar to PETRA, PsyMate and m-Path were developed based on best practices for EMA and GDPR compliance. Their smartphone apps enable client access to EMA data, which PETRA is still developing. PETRA differs in that it was codeveloped with clients and clinicians from the beginning, balancing personalization with user-friendly diary construction and feedback. Furthermore, the PETRA decision aid and feedback module were specifically developed for use as shared decision-making tools, integrating EMA with contextual information. Finally, its integration into EHR systems via RoQua [38] ensures that PETRA is sustainably embedded in clinical practice. As such, each platform has a different focus, and its concurrent development can advance our understanding of how to optimally implement EMA in clinical practice.

    Qualitative work demonstrates that clients and clinicians consider personalized EMA a useful add-on tool for diagnosis and treatment [22-24,78,98]. Similarly, self-monitoring of symptoms is considered safe and acceptable in diverse psychiatric populations [32,99,100]. Our results suggest the need for a collaborative approach to integrating personalized diaries in care, where clients and clinicians decide together on the focus of the EMA and the interpretation of feedback [22]. This corresponds to research demonstrating the importance of shared decision-making for psychiatric outcomes [10,101-103]. Throughout the development process, we realized the importance of tailoring PETRA’s EMA diaries and feedback to diverse clinical settings and client populations. However, much is unknown regarding the optimal type and timing of EMA feedback and how this potentially varies across diagnoses and treatment strategies. This means that PETRA will need to be continuously re-evaluated and adapted based on the diverse needs of clients and clinicians in psychiatric care. This iterative process has also been highlighted in other recent publications on the co-design of self-monitoring technology with users [42,97,104].

    Ethical Considerations

    Several ethical considerations arise when developing tools for highly personalized self-monitoring. First, the (sometimes qualitative) data are sensitive, and the privacy of the client should be well protected [105]. For PETRA, software and data transmission security are guaranteed by RoQua [38] and are in compliance with the GDPR. Second, clients may expect that the self-monitoring application alerts them or their clinician in the case of elevated EMA scores, which PETRA currently does not provide. Research indicates that this feature is desired by clients [23]; however, scientific evidence is lacking on how to provide EMA-based alerts with adequate sensitivity and specificity. Therefore, it is important to manage the expectations of clients and clinicians through training. Third, self-monitoring can be burdensome: clients can feel disturbed in daily activities or even experience symptom worsening [22,106]. Clients and clinicians will have to decide together whether the benefits of EMA outweigh this burden. The PETRA application helps clients and clinicians with this decision by providing a burden indicator, which helps them make informed decisions on the number of items. Similarly, the burden may be reduced by placing equal emphasis on symptoms and strengths in personalized diaries, in line with other studies on self-monitoring technology [104,107].

    Finally, research suggests that clinician training and support via a helpdesk are important requirements for the implementation of EMA in clinical settings [25,97]. The clinicians in this study also expressed these needs. Training is important to minimize potential health risks and help clinicians construct valid diaries, accurately interpret EMA feedback, and integrate EMA into treatment. Therefore, in the PETRA project, clinicians receive training on the valid use of EMA and the integration of PETRA in treatment, and they have access to a helpdesk. Clinicians are made aware that the feedback should be treated as an additional perspective that is equally (not more) valuable compared with the perspectives of clients and clinicians themselves. Furthermore, short instructive videos are included in PETRA’s various steps, explaining (1) how the decision aid works, (2) what clients should know when completing a diary, and (3) how to interpret the feedback [49].

    Directions for Future Research

    Although self-monitoring is already a prominent feature in several treatment modalities and protocols [14,108], an important next step is to determine the clinical effectiveness of adding personalized EMA to the treatment. This is especially relevant as only standardized diaries have been investigated in randomized controlled trials [26,27,29-31]. Extensive personalization options are clearly desired in clinical practice but may mean that effective components may differ across clients, which may complicate research into the effectiveness of personalized EMA [105]. Furthermore, future trials should not only solely focus on symptom improvement but also on more proximate outcomes relevant to the mechanisms of EMA, such as self-management and the therapeutic alliance [52]. Finally, future research will have to investigate the clinical populations and treatment settings for which personalized EMA can be effective. PETRA enables research on these questions. Within the PETRA project, the effectiveness of personalized EMA is tested both qualitatively and quantitatively. PETRA is currently being tested in pilot sessions with clients and clinicians and is continuously updated. Analytics are collected to assess which features of PETRA are used and how, when, and by whom. Furthermore, clients and clinicians are invited to fill out questionnaires on their positive and negative experiences with PETRA, as well as their effects on self-management [109,110] and the therapeutic alliance [111]. This will allow us to examine how personalized EMA influences psychiatric care and to improve PETRA accordingly. These steps can help to set personalized EMA, such as PETRA, as an eHealth technology apart from the myriad of apps available to clients and clinicians, who will be more inclined to use EMA if it is supported by scientific evidence [112].

    The second step concerns further scientific validation of the EMA items [113,114] and the development of more innovative EMA feedback. Such endeavors should explore more descriptive visualizations [115] and further develop statistical methods to analyze EMA data. Previous publications on EMA have often highlighted the potential of person-specific statistical models for informing treatment. These models include, for example, network analysis to demonstrate associations between symptoms [54,116,117], early warning signals to alert clients to impending relapse [118,119], and machine learning to differentiate between diagnoses [120]. However, since then, several publications have shown that the outcomes of these models depend on methodological and statistical choices [73] and that researchers disagree on their implications for treatment [74]. Therefore, PETRA focuses on descriptive feedback, with a large role for qualitative descriptions until more consensus on statistical models has been reached.

    Conclusions

    PETRA was developed to meet the demands of clients and clinicians for a personalized and user-friendly EMA tool embedded in routine psychiatric care. By collaboratively constructing EMA diaries and interpreting the resulting dynamic feedback, PETRA offers clients and clinicians a new tool to illuminate daily life processes that worsen or alleviate mental health problems. This approach may have beneficial effects on client self-management and the therapeutic alliance and, thus, has the potential to significantly improve clinical care. Our findings demonstrate the importance of a multidisciplinary approach to the development of personalized EMA tools, including clients, clinicians, researchers, and software developers, to ensure that their needs are sufficiently addressed in the design of the tool. PETRA is unique in its codevelopment process, extensive but user-friendly personalization options, integration into EHR systems, transdiagnostic focus, and strong scientific foundation in the design of EMA diaries and feedback. As such, PETRA paves the way for a systematic investigation of the effect of personalized EMA on specialized mental health care.

    Acknowledgments

    The authors thank all clients and clinicians for their invaluable insights. The authors also thank the RoQua team, in particular, Henk van der Veen and Inge ten Vaarwerk, for integrating the PETRA (Personalized Treatment by Real-time Assessment) web application within RoQua. The authors gratefully acknowledge Meike Bak for assisting with client recruitment and our research assistants Jannie Brouwer and Roan Carlier. The authors thank Judith Rosmalen for her insightful inputs on relevant ecological momentary assessment items and for her help in setting up the project. Finally, the authors thank Anoek Houben for designing the manual and the animation videos for the PETRA web application.

    This study was financially supported by Innovation Fund “Stichting De Friesland” (grant DS81 to HR and MW), the charitable foundation “Stichting tot Steun VCVGZ” (grant 239 to HR and Dr Jojanneke Bastiaansen), the European Research Council under the European Union’s Horizon 2020 research and innovation program (ERC-CoG-2015; grant 681466 to MW), and the Rob Giel Research Center. The funding agencies had no role in the interpretation of the results and the drafting of this manuscript.

    Authors' Contributions

    FMB, HR, and LvK conceptualized the study, with assistance from the scientific advisors MW, ES, client representative GHP, and clinician representative BD.

    HR and MW contributed to funding acquisition. FMB contributed to the data acquisition, formal analysis, project coordination, and writing of the original paper. ACE, TV, EV, and FMB contributed to software development. HR contributed to study supervision. FMB, HR, LvK, ACE, TV, EV, ES, BD, and GHP reviewed and edited the manuscript.

    Conflicts of Interest

    None declared.

    ‎
    Multimedia Appendix 1

    Supplementary Materials.

    PDF File (Adobe PDF File), 750 KB
    1. Shiffman S, Stone AA, Hufford MR. Ecological momentary assessment. Annu Rev Clin Psychol 2008 Apr 01;4(1):1-32. [CrossRef] [Medline]
    2. Delespaul P. Assessing Schizophrenia in Daily Life : The Experience Sampling Method. Maastricht: Universitaire Pers Maastricht; 1995.
    3. aan het Rot M, Hogenelst K, Schoevers RA. Mood disorders in everyday life: a systematic review of experience sampling and ecological momentary assessment studies. Clin Psychol Rev 2012 Aug;32(6):510-523. [CrossRef] [Medline]
    4. Walz LC, Nauta MH, Aan Het Rot M. Experience sampling and ecological momentary assessment for studying the daily lives of patients with anxiety disorders: a systematic review. J Anxiety Disord 2014 Dec;28(8):925-937. [CrossRef] [Medline]
    5. Oorschot M, Kwapil T, Delespaul P, Myin-Germeys I. Momentary assessment research in psychosis. Psychol Assess 2009 Dec;21(4):498-505. [CrossRef] [Medline]
    6. Shiffman S. Ecological momentary assessment (EMA) in studies of substance use. Psychol Assess 2009 Dec;21(4):486-497 [FREE Full text] [CrossRef] [Medline]
    7. Wichers M, Hartmann J, Kramer I, Lothmann C, Peeters F, van Bemmel L, et al. Translating assessments of the film of daily life into person-tailored feedback interventions in depression. Acta Psychiatr Scand 2011 May;123(5):402-403. [CrossRef] [Medline]
    8. Wichers M, Simons CJ, Kramer IM, Hartmann JA, Lothmann C, Myin-Germeys I, et al. Momentary assessment technology as a tool to help patients with depression help themselves. Acta Psychiatr Scand 2011 Oct;124(4):262-272. [CrossRef] [Medline]
    9. van Os J, Delespaul P, Wigman J, Myin-Germeys I, Wichers M. Beyond DSM and ICD: introducing "precision diagnosis" for psychiatry using momentary assessment technology. World Psychiatry 2013 Jun 04;12(2):113-117 [FREE Full text] [CrossRef] [Medline]
    10. van Os J, Verhagen S, Marsman A, Peeters F, Bak M, Marcelis M, ESM-MERGE Investigators PhD, et al. The experience sampling method as an mHealth tool to support self-monitoring, self-insight, and personalized health care in clinical practice. Depress Anxiety 2017 Jun 23;34(6):481-493. [CrossRef] [Medline]
    11. Kirk JK, Stegner J. Self-monitoring of blood glucose: practical aspects. J Diabetes Sci Technol 2010 Mar 01;4(2):435-439 [FREE Full text] [CrossRef] [Medline]
    12. Pickering TG, White WB, American Society of Hypertension Writing Group. When and how to use self (home) and ambulatory blood pressure monitoring. J Am Soc Hypertens 2008 May;2(3):119-124. [CrossRef] [Medline]
    13. Torous J, Bucci S, Bell IH, Kessing LV, Faurholt-Jepsen M, Whelan P, et al. The growing field of digital psychiatry: current evidence and the future of apps, social media, chatbots, and virtual reality. World Psychiatry 2021 Oct 09;20(3):318-335 [FREE Full text] [CrossRef] [Medline]
    14. Cohen JS, Edmunds JM, Brodman DM, Benjamin CL, Kendall PC. Using self-monitoring: implementation of collaborative empiricism in cognitive-behavioral therapy. Cognit Behav Pract 2013 Nov;20(4):419-428. [CrossRef]
    15. Lambert MJ, Whipple JL, Kleinstäuber M. Collecting and delivering progress feedback: a meta-analysis of routine outcome monitoring. Psychotherapy (Chic) 2018 Dec;55(4):520-537. [CrossRef] [Medline]
    16. Janssens KA, Bos EH, Rosmalen JG, Wichers MC, Riese H. A qualitative approach to guide choices for designing a diary study. BMC Med Res Methodol 2018 Nov 16;18(1):140 [FREE Full text] [CrossRef] [Medline]
    17. Vachon H, Viechtbauer W, Rintala A, Myin-Germeys I. Compliance and retention with the experience sampling method over the continuum of severe mental disorders: meta-analysis and recommendations. J Med Internet Res 2019 Dec 06;21(12):e14475 [FREE Full text] [CrossRef] [Medline]
    18. Helmich M, Snippe E, Kunkels Y, Riese H, Smit A, Wichers M. Transitions in Depression (TRANS-ID) Recovery: study protocol for a repeated intensive longitudinal n= 1 study design to search for personalized early warning signals of critical transitions towards improvement in depression. PsyArXiv 2020. [CrossRef]
    19. Alpay L, van der Boog P, Dumaij A. An empowerment-based approach to developing innovative e-health tools for self-management. Health Informatics J 2011 Dec 21;17(4):247-255 [FREE Full text] [CrossRef] [Medline]
    20. Reis H. Why researchers should think "real-world": a conceptual rationale. In: Handbook of Research Methods for Studying Daily Life. New York, NY United States: Guilford Press; 2012.
    21. Hatcher RL, Gillaspy JA. Development and validation of a revised short version of the working alliance inventory. Psychotherapy Res 2006 Jan;16(1):12-25. [CrossRef]
    22. Bos FM, Snippe E, Bruggeman R, Doornbos B, Wichers M, van der Krieke L. Recommendations for the use of long-term experience sampling in bipolar disorder care: a qualitative study of patient and clinician experiences. Int J Bipolar Disord 2020 Dec 01;8(1):38 [FREE Full text] [CrossRef] [Medline]
    23. Bos FM, Snippe E, Bruggeman R, Wichers M, van der Krieke L. Insights of patients and clinicians on the promise of the experience sampling method for psychiatric care. Psychiatr Serv 2019 Nov 01;70(11):983-991. [CrossRef] [Medline]
    24. Folkersma W, Veerman V, Ornée DA, Oldehinkel AJ, Alma MA, Bastiaansen JA. Patients' experience of an ecological momentary intervention involving self-monitoring and personalized feedback for depression. Internet Interv 2021 Dec;26:100436 [FREE Full text] [CrossRef] [Medline]
    25. Piot M, Mestdagh M, Riese H, Brouwer J, Weermeier J, Dejonckheere E. Practitioner and researcher perspectives on the utility of ecological momentary assessment in clinical practice: A survey study. OSF.   URL: https://osf.io/v9rtd/ [accessed 2022-07-08]
    26. Kramer I, Simons CJ, Hartmann JA, Menne-Lothmann C, Viechtbauer W, Peeters F, et al. A therapeutic application of the experience sampling method in the treatment of depression: a randomized controlled trial. World Psychiatry 2014 Mar 04;13(1):68-77 [FREE Full text] [CrossRef] [Medline]
    27. Simons C, Hartmann J, Kramer I, Menne-Lothmann C, Höhn P, van Bemmel A, et al. Effects of momentary self-monitoring on empowerment in a randomized controlled trial in patients with depression. Eur Psychiatry 2015 Nov 06;30(8):900-906. [CrossRef] [Medline]
    28. Widdershoven RL, Wichers M, Kuppens P, Hartmann JA, Menne-Lothmann C, Simons CJ, et al. Effect of self-monitoring through experience sampling on emotion differentiation in depression. J Affect Disord 2019 Feb 01;244:71-77. [CrossRef] [Medline]
    29. Snippe E, Simons CJ, Hartmann JA, Menne-Lothmann C, Kramer I, Booij SH, et al. Change in daily life behaviors and depression: within-person and between-person associations. Health Psychol 2016 May;35(5):433-441. [CrossRef] [Medline]
    30. Bastiaansen JA, Ornée DA, Meurs M, Oldehinkel AJ. An evaluation of the efficacy of two add-on ecological momentary intervention modules for depression in a pragmatic randomized controlled trial (ZELF-i). Psychol Med 2020 Dec 14:1-10. [CrossRef]
    31. Van Roekel E, Vrijen C, Heininga VE, Masselink M, Bos EH, Oldehinkel AJ. An exploratory randomized controlled trial of personalized lifestyle advice and tandem skydives as a means to reduce anhedonia. Behav Ther 2017 Jan;48(1):76-96. [CrossRef] [Medline]
    32. Murnane E, Cosley D, Chang P, Guha S, Frank E, Gay G, et al. Self-monitoring practices, attitudes, and needs of individuals with bipolar disorder: implications for the design of technologies to manage mental health. J Am Med Inform Assoc 2016 May;23(3):477-484. [CrossRef] [Medline]
    33. van Gemert-Pijnen JE, Nijland N, van Limburg M, Ossebaard HC, Kelders SM, Eysenbach G, et al. A holistic framework to improve the uptake and impact of eHealth technologies. J Med Internet Res 2011 Dec 05;13(4):e111 [FREE Full text] [CrossRef] [Medline]
    34. Kip H, Beerlage-de JN, Wentzel J. The contextual inquiry. In: eHealth Research, Theory and Development A Multidisciplinary Approach. London: Routledge; 2018.
    35. Holistic development of eHealth technology. In: eHealth Research, Theory and Development A Multidisciplinary Approach. London: Routledge; 2018.
    36. Van De Belt TH, Engelen LJ, Berben SA, Schoonhoven L. Definition of health 2.0 and medicine 2.0: a systematic review. J Med Internet Res 2010 Jun 11;12(2):e18 [FREE Full text] [CrossRef] [Medline]
    37. Home. iLab.   URL: https://ilab-psychiatry.nl/en_US/ [accessed 2021-12-20]
    38. roQua Routine Outcome Monitoring. RoQua.   URL: https://www.roqua.nl/ [accessed 2021-12-10]
    39. The Rob Giel Research center (RGOc) homepage. The Rob Giel Research center (RGOc).   URL: https://www.rgoc.nl/english-pages/ [accessed 2021-10-10]
    40. Dierckx de Casterlé B, Gastmans C, Bryon E, Denier Y. QUAGOL: a guide for qualitative data analysis. Int J Nurs Stud 2012 Mar;49(3):360-371. [CrossRef] [Medline]
    41. Development. PETRA.   URL: https://www.petrapsy.nl/en/ontwikkeling/ [accessed 2021-12-09]
    42. Rosmalen JG, van Gils A, Acevedo Mesa MA, Schoevers RA, Monden R, Hanssen DJ. Development of grip self-help: an online patient-tailored self-help intervention for functional somatic symptoms in primary care. Internet Interv 2020 Mar;19:100297 [FREE Full text] [CrossRef] [Medline]
    43. Wright AG, Woods WC. Personalized models of psychopathology. Annu Rev Clin Psychol 2020 May 07;16(1):49-74. [CrossRef] [Medline]
    44. Nijland N, van Gemert-Pijnen J, Boer H, Steehouder MF, Seydel ER. Evaluation of internet-based technology for supporting self-care: problems encountered by patients and caregivers when using self-care applications. J Med Internet Res 2008 May 15;10(2):e13 [FREE Full text] [CrossRef] [Medline]
    45. BinDhim NF, Hawkey A, Trevena L. A systematic review of quality assessment methods for smartphone health apps. Telemed J E Health 2015 Mar;21(2):97-104. [CrossRef] [Medline]
    46. Marshall JM, Dunstan DA, Bartik W. The digital psychiatrist: in search of evidence-based apps for anxiety and depression. Front Psychiatry 2019 Nov 15;10:831 [FREE Full text] [CrossRef] [Medline]
    47. Palmier-Claus JE, Myin-Germeys I, Barkus E, Bentley L, Udachina A, Delespaul P, et al. Experience sampling research in individuals with mental illness: reflections and guidance. Acta Psychiatr Scand 2011 Jan;123(1):12-20. [CrossRef] [Medline]
    48. Dejonckheere E, Erbas Y. Designing an experience sampling study. In: The Open Handbook of Experience Sampling Methodology: A Step-by-step Guide to Designing, Conducting, and Analyzing ESM Studies. -: Independently published; 2021.
    49. Videos. PETRA.   URL: https://www.petrapsy.nl/en/videos/ [accessed 2021-11-03]
    50. Eisele E, Kasanova Z, Houben M. Questionnaire designevaluation. In: The Open Handbook of Experience Sampling Methodology: A Step-by-step Guide to Designing, Conducting, and Analyzing ESM Studies. -: ‎ Independently published; 2021.
    51. Eisele G, Vachon H, Lafit G, Kuppens P, Houben M, Myin-Germeys I, et al. The effects of sampling frequency and questionnaire length on perceived burden, compliance, and careless responding in experience sampling data in a student population. Assessment 2022 Mar;29(2):136-151. [CrossRef] [Medline]
    52. Riese H, von Klipstein L, Schoevers RA, van der Veen DC, Servaas MN. Personalized ESM monitoring and feedback to support psychological treatment for depression: a pragmatic randomized controlled trial (Therap-i). BMC Psychiatry 2021 Mar 10;21(1):143 [FREE Full text] [CrossRef] [Medline]
    53. Reininghaus U, Klippel A, Steinhart H, Vaessen T, van Nierop M, Viechtbauer W, et al. Efficacy of Acceptance and Commitment Therapy in Daily Life (ACT-DL) in early psychosis: study protocol for a multi-centre randomized controlled trial. Trials 2019 Dec 26;20(1):769 [FREE Full text] [CrossRef] [Medline]
    54. van der Krieke L, Blaauw FJ, Emerencia AC, Schenk HM, Slaets JP, Bos EH, et al. Temporal dynamics of health and well-being: a crowdsourcing approach to momentary assessments and automated generation of personalized feedback. Psychosom Med 2017;79(2):213-223. [CrossRef]
    55. Krieke LV, Jeronimus BF, Blaauw FJ, Wanders RB, Emerencia AC, Schenk HM, et al. HowNutsAreTheDutch (HoeGekIsNL): a crowdsourcing study of mental symptoms and strengths. Int J Methods Psychiatr Res 2016 Jun;25(2):123-144 [FREE Full text] [CrossRef] [Medline]
    56. Schoevers RA, van Borkulo CD, Lamers F, Servaas M, Bastiaansen JA, Beekman AT, et al. Affect fluctuations examined with ecological momentary assessment in patients with current or remitted depression and anxiety disorders. Psychol Med 2020 Apr 01;51(11):1906-1915. [CrossRef]
    57. Reininghaus U, Kempton MJ, Valmaggia L, Craig TK, Garety P, Onyejiaka A, et al. Stress sensitivity, aberrant salience, and threat anticipation in early psychosis: an experience sampling study. Schizophr Bull 2016 May 01;42(3):712-722 [FREE Full text] [CrossRef] [Medline]
    58. Knapen S. Rhythm & Blues: Chronobiology in the Pathophysiology and Treatment of Mood Disorders. [Groningen]: Rijksuniversiteit Groningen; 2019.
    59. Tyler E, Jones S, Black N, Carter L, Barrowclough C. The relationship between bipolar disorder and cannabis use in daily life: an experience sampling study. PLoS One 2015 Mar 4;10(3):e0118916 [FREE Full text] [CrossRef] [Medline]
    60. Myin-Germeys I, Peeters F, Havermans R, Nicolson N, DeVries MW, Delespaul P, et al. Emotional reactivity to daily life stress in psychosis and affective disorder: an experience sampling study. Acta Psychiatr Scand 2003 Mar;107(2):124-131. [CrossRef] [Medline]
    61. Barrantes-Vidal N, Chun CA, Myin-Germeys I, Kwapil TR. Psychometric schizotypy predicts psychotic-like, paranoid, and negative symptoms in daily life. J Abnorm Psychol 2013 Nov;122(4):1077-1087. [CrossRef] [Medline]
    62. Schreuder MJ, Groen RN, Wigman JT, Hartman CA, Wichers M. Measuring psychopathology as it unfolds in daily life: addressing key assumptions of intensive longitudinal methods in the TRAILS TRANS-ID study. BMC Psychiatry 2020 Jul 06;20(1):351 [FREE Full text] [CrossRef] [Medline]
    63. Kroeze R, van der Veen DC, Servaas MN, Bastiaansen JA, Oude Voshaar RC, Borsboom D, et al. Personalized feedback on symptom dynamics of psychopathology: a proof-of-principle study. J Pers Oriented Res 2017 Nov 03;3(1):1-10 [FREE Full text] [CrossRef] [Medline]
    64. Bos F. Ecological Momentary Assessment as A Clinical Tool in Psychiatry Promises, Pitfalls, and Possibilities. Groningen, The Netherlands: University of Groningen; 2021.
    65. Zijlema WL, Stolk RP, Löwe B, Rief W, BioSHaRE, White PD, et al. How to assess common somatic symptoms in large-scale studies: a systematic review of questionnaires. J Psychosom Res 2013 Jun;74(6):459-468. [CrossRef] [Medline]
    66. Wichers M, Peeters F, Rutten BP, Jacobs N, Derom C, Thiery E, et al. A time-lagged momentary assessment study on daily life physical activity and affect. Health Psychol 2012 Mar;31(2):135-144. [CrossRef] [Medline]
    67. Laura Bringmann's intensive longitudinal data lab. LaBLab.   URL: https://www.laurabringmannlab.com/ [accessed 2021-12-20]
    68. MYIN-GERMEYS I, KRABBENDAM L, DELESPAUL PA, VAN OS J. Do life events have their effect on psychosis by influencing the emotional reactivity to daily life stress? Psychol Med 2003 Feb 14;33(2):327-333. [CrossRef]
    69. Booij SH, Wichers M, de Jonge P, Sytema S, van Os J, Wunderink L, et al. Study protocol for a prospective cohort study examining the predictive potential of dynamic symptom networks for the onset and progression of psychosis: the Mapping Individual Routes of Risk and Resilience (Mirorr) study. BMJ Open 2018 Jan 21;8(1):e019059 [FREE Full text] [CrossRef] [Medline]
    70. Leverich G, Post R. The NIMH life chart manual for recurrent affective illness: The LCM. NIMH Monograph. 1993.   URL: http:/www.bipolarnews.org/pdfs/Clinician%20Prospective%20Manual.pdf [accessed 2022-06-30]
    71. Schiepek G, Stöger-Schmidinge B, Kronberger H, Aichhorn W, Kratzer L, Heinz P, et al. The Therapy Process Questionnaire - Factor analysis and psychometric properties of a multidimensional self-rating scale for high-frequency monitoring of psychotherapeutic processes. Clin Psychol Psychother 2019 Sep 10;26(5):586-602 [FREE Full text] [CrossRef] [Medline]
    72. Carney C, Buysse D, Ancoli-Israel S, Edinger J, Krystal A, Lichstein K, et al. The consensus sleep diary: standardizing prospective sleep self-monitoring. Sleep 2012 Mar 01;35(2):287-302 [FREE Full text] [CrossRef] [Medline]
    73. Wichers M, Riese H, Hodges TM, Snippe E, Bos FM. A narrative review of network studies in depression: what different methodological approaches tell us about depression. Front Psychiatry 2021 Oct 28;12:719490 [FREE Full text] [CrossRef] [Medline]
    74. Bastiaansen JA, Kunkels YK, Blaauw FJ, Boker SM, Ceulemans E, Chen M, et al. Time to get personal? The impact of researchers choices on the selection of treatment targets using the experience sampling methodology. J Psychosom Res 2020 Aug 05;137:110211 [FREE Full text] [CrossRef] [Medline]
    75. Lutz W, Rubel JA, Schwartz B, Schilling V, Deisenhofer A. Towards integrating personalized feedback research into clinical practice: development of the Trier Treatment Navigator (TTN). Behav Res Ther 2019 Sep;120:103438. [CrossRef] [Medline]
    76. Lutz W, Schwartz B, Hofmann SG, Fisher AJ, Husen K, Rubel JA. Using network analysis for the prediction of treatment dropout in patients with mood and anxiety disorders: a methodological proof-of-concept study. Sci Rep 2018 May 18;8(1):7819 [FREE Full text] [CrossRef] [Medline]
    77. Rubel J, Fisher A, Husen K, Lutz W. Translating person-specific network models into personalized treatments: development and demonstration of the dynamic assessment treatment algorithm for individual networks (DATA-IN). Psychother Psychosom 2018 Apr 20;87(4):249-251. [CrossRef] [Medline]
    78. Frumkin MR, Piccirillo ML, Beck ED, Grossman JT, Rodebaugh TL. Feasibility and utility of idiographic models in the clinic: a pilot study. Psychother Res 2021 Apr 24;31(4):520-534 [FREE Full text] [CrossRef] [Medline]
    79. Ellison WD. An initial study of practicing psychologists' views of the utility of ecological momentary assessment for difficult psychotherapy cases. Adm Policy Ment Health 2021 Jul 12;48(4):597-607. [CrossRef] [Medline]
    80. Myin-Germeys I. Psychiatry. In: Handbook of Research Methods for Studying Daily Life. New York, NY United States: Guilford Press; 2012:636.
    81. Myin-Germeys I, Kasanova Z, Vaessen T, Vachon H, Kirtley O, Viechtbauer W, et al. Experience sampling methodology in mental health research: new insights and technical developments. World Psychiatry 2018 Jun 24;17(2):123-132 [FREE Full text] [CrossRef] [Medline]
    82. Burns MN, Begale M, Duffecy J, Gergle D, Karr CJ, Giangrande E, et al. Harnessing context sensing to develop a mobile intervention for depression. J Med Internet Res 2011 Aug 12;13(3):e55 [FREE Full text] [CrossRef] [Medline]
    83. Depp CA, Ceglowski J, Wang VC, Yaghouti F, Mausbach BT, Thompson WK, et al. Augmenting psychoeducation with a mobile intervention for bipolar disorder: a randomized controlled trial. J Affect Disord 2015 Mar 15;174:23-30 [FREE Full text] [CrossRef] [Medline]
    84. Saunders K, Bilderbeck A, Panchal P, Atkinson L, Geddes J, Goodwin G. Experiences of remote mood and activity monitoring in bipolar disorder: a qualitative study. Eur Psychiatry 2017 Mar 27;41(1):115-121 [FREE Full text] [CrossRef] [Medline]
    85. Fisher AJ, Bosley HG, Fernandez KC, Reeves JW, Soyster PD, Diamond AE, et al. Open trial of a personalized modular treatment for mood and anxiety. Behav Res Ther 2019 May;116:69-79. [CrossRef] [Medline]
    86. Vaessen T, Steinhart H, Batink T, Klippel A, Van Nierop M, Reininghaus U, et al. ACT in daily life in early psychosis: an ecological momentary intervention approach. Psychosis 2019 Mar 19;11(2):93-104. [CrossRef]
    87. Bell IH, Rossell SL, Farhall J, Hayward M, Lim MH, Fielding-Smith SF, et al. Pilot randomised controlled trial of a brief coping-focused intervention for hearing voices blended with smartphone-based ecological momentary assessment and intervention (SAVVy): feasibility, acceptability and preliminary clinical outcomes. Schizophr Res 2020 Feb;216:479-487. [CrossRef] [Medline]
    88. van Roekel E, Keijsers L, Chung JM. A review of current ambulatory assessment studies in adolescent samples and practical recommendations. J Res Adolesc 2019 Sep 17;29(3):560-577 [FREE Full text] [CrossRef] [Medline]
    89. Heron K, Everhart R, McHale S, Smyth J. Using mobile-technology-based Ecological Momentary Assessment (EMA) methods with youth: a systematic review and recommendations. J Pediatr Psychol 2017 Nov 01;42(10):1087-1107. [CrossRef] [Medline]
    90. Cain AE, Depp CA, Jeste DV. Ecological momentary assessment in aging research: a critical review. J Psychiatr Res 2009 Jul;43(11):987-996 [FREE Full text] [CrossRef] [Medline]
    91. Vahabzadeh M, Epstein D, Mezghanni M, Lin J, Preston K. An electronic diary software for ecological momentary assessment (EMA) in clinical trials. In: Proceedings of the 17th IEEE Symposium on Computer-Based Medical Systems. 2004 Presented at: 17th IEEE Symposium on Computer-Based Medical Systems; Jun 25, 2004; Bethesda, MD, USA. [CrossRef]
    92. Activity monitoring. Movisens.   URL: https://www.movisens.com/en/ [accessed 2021-11-11]
    93. Ilumivu homepage. Ilumivu.   URL: https://ilumivu.com/ [accessed 2021-11-11]
    94. Klein A, Clucas J, Krishnakumar A, Ghosh SS, Van Auken W, Thonet B, et al. Remote digital psychiatry for mobile mental health assessment and therapy: mindlogger platform development study. J Med Internet Res 2021 Nov 11;23(11):e22369 [FREE Full text] [CrossRef] [Medline]
    95. m-Path homepage. m-Path.   URL: https://m-path.io/landing/ [accessed 2021-12-10]
    96. Verhagen SJ, Berben JA, Leue C, Marsman A, Delespaul PA, van Os J, et al. Demonstrating the reliability of transdiagnostic mHealth Routine Outcome Monitoring in mental health services using experience sampling technology. PLoS One 2017 Oct 12;12(10):e0186294 [FREE Full text] [CrossRef] [Medline]
    97. Daniëls NE, Hochstenbach LM, van Bokhoven MA, Beurskens AJ, Delespaul PA. Implementing experience sampling technology for functional analysis in family medicine - a design thinking approach. Front Psychol 2019 Dec 11;10:2782 [FREE Full text] [CrossRef] [Medline]
    98. Palmier-Claus JE, Rogers A, Ainsworth J, Machin M, Barrowclough C, Laverty L, et al. Integrating mobile-phone based assessment for psychosis into people's everyday lives and clinical care: a qualitative study. BMC Psychiatry 2013 Jan 23;13:34 [FREE Full text] [CrossRef] [Medline]
    99. Hetrick SE, Goodall J, Yuen HP, Davey CG, Parker AG, Robinson J, et al. Comprehensive online self-monitoring to support clinicians manage risk of suicide in youth depression. Crisis 2017 May;38(3):147-157. [CrossRef] [Medline]
    100. Torous J, Staples P, Shanahan M, Lin C, Peck P, Keshavan M, et al. Utilizing a personal smartphone custom app to assess the patient health questionnaire-9 (PHQ-9) depressive symptoms in patients with major depressive disorder. JMIR Ment Health 2015 Mar 24;2(1):e8 [FREE Full text] [CrossRef] [Medline]
    101. Drake RE, Cimpean D, Torrey WC. Shared decision making in mental health: prospects for personalized medicine. Dialogues Clin Neuroscience 2022 Apr 01;11(4):455-463. [CrossRef]
    102. Hamann J, Leucht S, Kissling W. Shared decision making in psychiatry. Acta Psychiatr Scand 2003 Jun;107(6):403-409. [CrossRef] [Medline]
    103. Joosten EA, DeFuentes-Merillas L, de Weert GH, Sensky T, van der Staak CP, de Jong CA. Systematic review of the effects of shared decision-making on patient satisfaction, treatment adherence and health status. Psychother Psychosom 2008;77(4):219-226. [CrossRef] [Medline]
    104. Hetrick SE, Robinson J, Burge E, Blandon R, Mobilio B, Rice SM, et al. Youth codesign of a mobile phone app to facilitate self-monitoring and management of mood symptoms in young people with major depression, suicidal ideation, and self-harm. JMIR Ment Health 2018 Jan 23;5(1):e9 [FREE Full text] [CrossRef] [Medline]
    105. Rubeis G. iHealth: the ethics of artificial intelligence and big data in mental healthcare. Internet Interv 2022 Apr;28:100518 [FREE Full text] [CrossRef] [Medline]
    106. van Genugten CR, Schuurmans J, Lamers F, Riese H, Penninx BW, Schoevers RA, et al. Experienced burden of and adherence to smartphone-based ecological momentary assessment in persons with affective disorders. J Clin Med 2020 Jan 23;9(2):322 [FREE Full text] [CrossRef] [Medline]
    107. Drake G, Csipke E, Wykes T. Assessing your mood online: acceptability and use of Moodscope. Psychol Med 2012 Nov 13;43(7):1455-1464. [CrossRef]
    108. Gibbons C, Porter I, Gonçalves-Bradley DC, Stoilov S, Ricci-Cabello I, Tsangaris E, et al. Routine provision of feedback from patient-reported outcome measurements to healthcare providers and patients in clinical practice. Cochrane Database Syst Rev 2021 Oct 12;10:CD011589. [CrossRef] [Medline]
    109. Witteman C, Bolks L, Hutschemaekers G. Development of the illness perception questionnaire mental health. J Ment Health 2011 Apr 20;20(2):115-125. [CrossRef] [Medline]
    110. Monger B, Hardie SM, Ion R, Cumming J, Henderson N. The Individual Recovery Outcomes Counter: preliminary validation of a personal recovery measure. Psychiatrist 2018 Jan 02;37(7):221-227. [CrossRef]
    111. Munder T, Wilmers F, Leonhart R, Linster HW, Barth J. Working Alliance Inventory-Short Revised (WAI-SR): psychometric properties in outpatients and inpatients. Clin Psychol Psychother 2010;17(3):231-239. [CrossRef] [Medline]
    112. Giota K, Kleftaras G. Mental health apps: innovations, risks and ethical considerations. E-Health telecommun syst Netw 2014;03(03):19-23. [CrossRef]
    113. The Experience Sampling Method (ESM) item repository. OSF HOME.   URL: https://osf.io/kg376/ [accessed 2022-06-30]
    114. Kirtley OJ, Lafit G, Achterhof R, Hiekkaranta AP, Myin-Germeys I. Making the black box transparent: a template and tutorial for registration of studies using experience-sampling methods. Advances Method Pract Psychol Sci 2021 Mar 02;4(1):251524592092468. [CrossRef]
    115. Bringmann LF, van der Veen DC, Wichers M, Riese H, Stulp G. ESMvis: a tool for visualizing individual Experience Sampling Method (ESM) data. Qual Life Res 2021 Nov 22;30(11):3179-3188 [FREE Full text] [CrossRef] [Medline]
    116. Epskamp S, van Borkulo CD, van der Veen DC, Servaas MN, Isvoranu A, Riese H, et al. Personalized network modeling in psychopathology: the importance of contemporaneous and temporal connections. Clin Psychol Sci 2018 May 19;6(3):416-427 [FREE Full text] [CrossRef] [Medline]
    117. von Klipstein L, Riese H, van der Veen DC, Servaas MN, Schoevers RA. Using person-specific networks in psychotherapy: challenges, limitations, and how we could use them anyway. BMC Med 2020 Nov 23;18(1):345 [FREE Full text] [CrossRef] [Medline]
    118. Smit A, Snippe E, Wichers M. Increasing restlessness signals impending increase in depressive symptoms more than 2 months before it happens in individual patients. Psychother Psychosom 2019 Jun 28;88(4):249-251 [FREE Full text] [CrossRef] [Medline]
    119. Wichers M, Smit AC, Snippe E. Early warning signals based on momentary affect dynamics can expose nearby transitions in depression: a confirmatory single-subject time-series study. J Pers Oriented Res 2020 Sep 10;6(1):1-15 [FREE Full text] [CrossRef] [Medline]
    120. Perez Arribas I, Goodwin GM, Geddes JR, Lyons T, Saunders KE. A signature-based machine learning model for distinguishing bipolar disorder and borderline personality disorder. Transl Psychiatry 2018 Dec 13;8(1):274 [FREE Full text] [CrossRef] [Medline]

    ‎
    CeHRes: Center for eHealth Research
    EHR: electronic health record
    EMA: ecological momentary assessment
    GDPR: General Data Protection Regulation
    PETRA: Personalized Treatment by Real-time Assessment
    RGOc: Rob Giel Research Center
    UX: user experience

    Edited by J Torous; submitted 14.01.22; peer-reviewed by A Klein, M Richter, C Skirrow, V Wiemker; comments to author 24.02.22; revised version received 11.04.22; accepted 06.05.22; published 09.08.22

    Copyright

    ©Fionneke M Bos, Lino von Klipstein, Ando C Emerencia, Erwin Veermans, Tom Verhage, Evelien Snippe, Bennard Doornbos, Grietje Hadders-Prins, Marieke Wichers, Harriëtte Riese. Originally published in JMIR Mental Health (https://mental.jmir.org), 09.08.2022.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Mental Health, is properly cited. The complete bibliographic information, a link to the original publication on https://mental.jmir.org/, as well as this copyright and license information must be included.