This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Mental Health, is properly cited. The complete bibliographic information, a link to the original publication on https://mental.jmir.org/, as well as this copyright and license information must be included.
Smartphone self-monitoring of mood, symptoms, and contextual factors through ecological momentary assessment (EMA) provides insights into the daily lives of people undergoing psychiatric treatment. Therefore, EMA has the potential to improve their care. To integrate EMA into treatment, a clinical tool that helps clients and clinicians create personalized EMA diaries and interpret the gathered data is needed.
This study aimed to develop a web-based application for personalized EMA in specialized psychiatric care in close collaboration with all stakeholders (ie, clients, clinicians, researchers, and software developers).
The participants were 52 clients with mood, anxiety, and psychotic disorders and 45 clinicians (psychiatrists, psychologists, and psychiatric nurses). We engaged them in interviews, focus groups, and usability sessions to determine the requirements for an EMA web application and repeatedly obtained feedback on iteratively improved high-fidelity EMA web application prototypes. We used human-centered design principles to determine important requirements for the web application and designed high-fidelity prototypes that were continuously re-evaluated and adapted.
The iterative development process resulted in Personalized Treatment by Real-time Assessment (PETRA), which is a scientifically grounded web application for the integration of personalized EMA in Dutch clinical care. PETRA includes a decision aid to support clients and clinicians with constructing personalized EMA diaries, an EMA diary item repository, an SMS text message–based diary delivery system, and a feedback module for visualizing the gathered EMA data. PETRA is integrated into electronic health record systems to ensure ease of use and sustainable integration in clinical care and adheres to privacy regulations.
PETRA was built to fulfill the needs of clients and clinicians for a user-friendly and personalized EMA tool embedded in routine psychiatric care. PETRA is unique in this codevelopment process, its extensive but user-friendly personalization options, its integration into electronic health record systems, its transdiagnostic focus, and its strong scientific foundation in the design of EMA diaries and feedback. The clinical effectiveness of integrating personalized diaries via PETRA into care requires further research. As such, PETRA paves the way for a systematic investigation of the utility of personalized EMA for routine mental health care.
Ecological momentary assessment (EMA; also referred to as the experience sampling method or ambulatory monitoring) [
In many medical disciplines, self-monitoring of, for example, daily blood pressure or glucose levels, is already a common practice and is informative for diagnosis and treatment planning [
Self-monitoring through EMA is suggested to improve clients’ self-management by increasing their insights into their well-being. By frequently reflecting on one’s symptoms and learning what types of activities or situations positively or negatively influence well-being, clients may become more in control [
The first empirical investigations into the clinical effectiveness of EMA are promising. In qualitative research, clients and clinicians describe the beneficial effects of EMA on client self-management, therapeutic alliance, and treatment effectiveness [
Given that clients, clinicians, and researchers align in their positive evaluation of the clinical utility of EMA, the field needs a flexible, user-friendly, and evidence-based application for the use of personalized EMA diaries in mental health care. More specifically, this entails the development of a digital infrastructure that enables clients and clinicians to intuitively construct personalized EMA diaries, deliver them to clients’ smartphones, and dynamically visualize the gathered EMA data.
In this paper, we describe the development of Personalized Treatment by Real-time Assessment (PETRA), a web-based application that enables the use of personalized EMA diaries in Dutch mental health care. PETRA was built into electronic health record (EHR) systems to ensure sustainability and easy access. The web application comprises a decision aid to help clients and clinicians construct personalized EMA diaries, an EMA diary item repository, a diary delivery system, and a feedback module to visualize the gathered EMA data. PETRA was developed in close collaboration with clients and clinicians as they will be the primary users. Involving them early in the development process is crucial to ensure the uptake of PETRA.
PETRA was developed using the road map of the Center for eHealth Research (CeHRes) of the University of Twente, the Netherlands [
During each of these phases, stakeholders were actively involved to ensure their perspectives are integrated into PETRA. The CeHRes road map follows human-centered design principles, attempting to understand the needs of users and actively involving them in designing solutions to meet these needs [
Overview of the developmental phases of PETRA (Personalized Treatment by Real-time Assessment). EMA: ecological momentary assessment.
The PETRA web application was developed from 2016 to 2022 at the Department of Psychiatry, University Medical Center Groningen (the Netherlands), facilitated by iLab, a collaborative initiative to implement scientific innovations into clinical practice [
Participants comprised a convenience sample of clients and clinicians recruited from RGOc facilities, who took part in interviews, focus groups, and usability sessions. All clients with a (self-reported) history of mental health problems (regardless of type) were eligible to participate in the study. Similarly, all clinicians working in an RGOc facility were eligible. Clinicians could be psychiatrists, psychologists, or psychiatric nurses. Participant characteristics for the first 2 phases can be found in the studies by Bos et al [
The University Medical Center Groningen Medical Ethics Committee exempted this research from a full review (reference number 201900401). Clients signed a written informed consent form to participate in the focus groups, interviews, and usability sessions.
Phases 1 and 2 aimed to identify perceived problems in the status quo of mental health treatment, perceived advantages and challenges of personalized electronic diaries, and core requirements for the EMA web application to be developed. In-depth qualitative interviews and focus groups were conducted with 40 clients and 27 clinicians between June 2016 and March 2018 on the perceived utility of EMA for psychiatric care and perceived important requirements for a clinical EMA tool. Focus groups with clients or clinicians were conducted in groups of 3 to 7 participants until data saturation was reached (ie, no new themes emerged). The focus groups and interviews were part of 2 qualitative studies that broadly focused on identifying applications of EMA in diverse clinical contexts and implementation requirements. For more details, we refer the reader to previous studies [
On the basis of the recommendations in phases 1 and 2, we designed prototypes for the web-based personalized EMA tool PETRA. These were continuously tested and redesigned in usability sessions with clients and clinicians held from May 2019 to August 2021. Owing to the COVID-19 pandemic, all usability sessions after March 2020 were conducted via video calls (for interview guides, see Table S1 in
On the basis of phases 1 and 2, clinicians were considered the primary users of the decision aid and feedback module: they would need to integrate PETRA into their working routines and introduce it to clients. Therefore, most of the usability sessions were held with clinicians. A total of 18 clinicians participated in ≥1 of the 31 usability sessions (5/18, 28% were men). Approximately 77% (24/31) of the sessions were held individually, and 23% (7/31) of the sessions took place in groups of 2 to 3 clinicians. Most clinicians were psychologists (10/18, 56%), followed by psychiatrists (4/18, 22%) and psychiatric nurses (4/18, 22%). Of the 18 clinicians, 7 (39%) participated once, 8 (44%) participated twice, and 3 (17%) participated thrice. During each session, clinicians were asked to describe their thoughts and expectations regarding EMA and their requirements for a diary web application. Subsequently, the clinicians were walked through a digital prototype of the PETRA web application. They were encouraged to describe what they saw on the screen and provide their initial thoughts on each page (the think-aloud method). When the clinicians offered suggestions for improvement, follow-up questions were asked until an in-depth understanding of the proposed adaptation was reached. The clinicians were also invited to explain how they would use PETRA in treatment and to suggest new features or feedback types.
A total of 12 clients participated in ≥1 of the 4 usability sessions, which were similar to those with the clinicians. Clients were currently or had been previously in treatment for mood, psychotic, or anxiety disorders (8/12, 67% were men). Of the 12 clients, 9 (75%) participated in 1 usability session, and 3 (25%) participated in 2 sessions. The sessions took place in groups of 3 to 7 clients. One of the sessions specifically focused on designing low-fidelity (paper-and-pencil) prototypes for EMA diary feedback. The other usability sessions were used to obtain feedback on the digital high-fidelity prototypes of the PETRA web application. Feedback was sought on the workflow and its intended use in treatment.
The interviews and focus groups were transcribed and analyzed thematically using the Qualitative Analysis Guide of Leuven [
After each usability session, wireframes were created with the proposed changes to the PETRA web application (see
The first wireframe that was designed of the first page of the decision aid, on which participants select the goal of the EMA diary. The usability sessions indicated a need for an introduction and a more detailed progress menu. More detailed wireframes can be found here [
The ninth wireframe that was designed of the first page of the decision aid, on which participants select the goal of the EMA diary. As suggested in usability sessions, it incorporates an introduction and a more detailed progress menu. More detailed wireframes can be found here [
The final (visual) design of the first page of the decision aid, on which participants select the goal of the EMA diary. More detailed wireframes can be found here [
The first 2 phases of the CeHRes road map resulted in an in-depth overview of clients’ and clinicians’ perceptions of perceived problems in the status quo of mental health treatment, perceived advantages and challenges of personalized electronic diaries, and core requirements for the EMA web application to be developed (
Results of phases 1 and 2 of the Center for eHealth Research road map in the development of the PETRA (Personalized Treatment by Real-time Assessment) application.
Theme | Description | |
|
||
|
Reliability or efficiency of current monitoring instruments |
Often paper-and-pencil based and deemed less reliable and efficient Often focus too much on symptoms and fail to take important contextual factors and strengths into account Usually only administered 1 to 2 times per month; missing relevant information about clients’ daily lives throughout the day Usually not person-specific enough to be directly relevant to clients |
|
Insights |
Limited insight into overall progress and treatment outcomes Limited insight into the effects of treatment or lifestyle adaptations Limited insight into the frequency and severity of symptoms and when they occur Limited insight into triggers and relapse signals Recall of well-being in between sessions is biased by current mood or otherwise difficult to recall for clients (recall bias) |
|
Therapeutic alliance |
Knowledge imbalance between client and clinician, weakening the therapeutic alliance Current monitoring instruments are often not discussed in treatment, weakening the therapeutic alliance Limited contact between client and clinician in between sessions Client does not believe in the current treatment approach as the effects are not clearly visible |
|
||
|
Reliability or efficiency |
More reliable and efficient as assessments take place multiple times per day via smartphone and are less easily forgotten or ignored |
|
Insight |
Offers insights into progress, treatment effects, and the flow of symptoms throughout daily life, thereby increasing client self-management Offers insights into contextual factors and personal strengths |
|
Therapeutic alliance |
Client and clinicians share the same information and can, therefore, collaborate more easily Relevant diary feedback can more easily be integrated in treatment, strengthening the therapeutic alliance and clients’ trust in their clinician |
|
||
|
Diary construction |
What kind of clinical questions can be answered with electronic diaries? How to formulate or select relevant and valid diary items? How to determine the number of assessments per day, balancing client burden and the number needed to answer a clinical question? How to determine the necessary diary schedule (eg, time-contingent or event-contingent) for a clinical question? How to make sure the diary maximizes its advantages and minimizes disadvantages? |
|
Diary feedback |
How to automatically analyze and visualize the diary data without the need for statistical knowledge of the clinician? How to interpret the diary feedback in a clinical context? |
All clients and clinicians stressed the necessity for extensive personalization of the diary content, schedule, and duration. They suggested that clients and clinicians should be able to compile different EMA items based on the clients’ current care needs in a flexible manner. This means that EMA diaries should be transdiagnostic and adaptable in diverse stages of treatment. The alternative, where diaries are constructed based on client diagnosis, was viewed as limiting, more burdensome, and not in line with the real-world situation where many clients present with symptoms of multiple diagnoses. Furthermore, EMA diaries should not only focus on symptoms but also on personal strengths and (social) contextual factors. This need for personalization has also been highlighted in research on monitoring in general [
Both clients and clinicians emphasized that the use of EMA in treatment should be easy, time efficient, and fit with the existing workflow of clinicians. This means that constructing personalized EMA diaries and interpreting the resulting feedback should be intuitive and cost clients and clinicians limited time, effort, and resources. This corresponds with research demonstrating that eHealth is most useful if it matches client and clinician expectations [
Clients and clinicians indicated that they found it difficult to assess the effectiveness of eHealth applications, which corresponds to eHealth research [
The final core requirement clients and clinicians mentioned was that the gathered diary data should be safely stored, protecting the privacy of clients, according to data protection laws (eg, the General Data Protection Regulation [GDPR] law of the European Union). Given the far-reaching personalization of the diaries, the data should be treated accordingly.
The core requirements for a web application for personalized EMA diaries were translated into the current design of PETRA. PETRA was developed as a web-based application, built into EHR systems via RoQua. As per the suggestion of clinicians, this means that clients and clinicians do not need additional log-in data, and clients do not need to install an app on their phones, allowing easy access for both. PETRA was developed for adult clients diagnosed with mood, anxiety, or psychotic disorders receiving treatment in specialized mental health care. Proficiency in the Dutch language and client possession of a smartphone with internet access are required to use PETRA.
PETRA comprises four main parts: (1) a decision aid, (2) an item repository of diary items, (3) an SMS text message–based diary delivery system, and (4) a feedback module. An overview of the various parts of PETRA is shown in
Overview of PETRA.
A decision aid was developed to assist clients and clinicians in constructing personalized EMA diaries. This decision aid is intended to be used collaboratively by clients and clinicians. The following specifications can be personalized via the decision aid: (1) the content (or diary items), (2) the number of assessments per day, (3) the diary schedule (timing of the assessments), and (4) the length of the EMA monitoring period. These specifications depend on the intended goal of the diary, anticipated burden for the client, and the client’s main symptom profile.
PETRA’s decision aid was developed based on the results of phases 1 and 2 and guidelines for designing EMA studies [
In addition to the goal, clients and clinicians select the symptom profile they wish to focus on. We developed a list of prevalent symptom profiles in specialist mental health care (eg, depression, anxiety, and psychosis). Each symptom profile was linked to potentially relevant EMA items. For example, relevant constructs for clients in depression treatment could be mood, anhedonia, worry, self-esteem, and resilience.
The decision aid first prompts clients and clinicians to decide on the clinical goal, symptom profile, and preferred diary schedule (see
The EMA item repository was developed as clients and clinicians indicated a preference for a
The EMA item repository was developed by the scientific team based on extensive experience with the items used in EMA studies and subsequent evaluations and suggestions by clients and clinicians. First, we determined potentially relevant constructs to be assessed; that is, constructs pertaining to symptoms, strengths, and contextual factors. For each construct, we formulated 4 to 10 diary items.
This resulted in four item sets based on the EMA assessment schedule: (1) at semirandom moments (128 items), (2) at fixed moments (126 items), (3) once daily in the evening (62 items), and (4) once daily in the morning (18 items). EMA assessments can occur either at fixed time points (eg, at noon) or at random points in predefined intervals (eg, somewhere between 10:30 AM and noon), termed fixed or semirandom EMA designs, respectively. Fixed designs are believed to be less burdensome for clients, whereas semirandom designs supposedly provide a more representative overview of daily experiences [
Semirandom and fixed diary schedules are mutually exclusive. Once-daily items are often an informative addition to these more frequent assessment schedules. Therefore, in the decision aid, the constructs of the once-per-day schedules are integrated within the construct structure of the semirandom and fixed schedules. This ensures that clients and clinicians can select items based on the construct. For example, the construct
Once clients and clinicians have constructed the EMA diary, clients receive SMS text messages on their personal smartphones. The messages are sent according to the chosen diary schedules and contain a link to the EMA diary items, which are presented and filled in via the browser. Clients receive reminders if the diary has not been completed within 20 minutes, and the link is disabled after 30 minutes (semirandom and fixed schedules) or 3 hours (once-daily schedules).
The gathered EMA data for each client are visualized in the PETRA feedback module, which clients and their clinicians can discuss during a regular treatment session (see
Clients and clinicians indicated that the understanding of EMA data is greatly facilitated by qualitative descriptions of the context provided by clients. Therefore, clients are actively encouraged to provide such descriptions. In the feedback module, these qualitative evaluations can be linked to high and low scores to provide contextual information on moments the client experiences high or low symptoms. Furthermore, all entered text is summarized in word clouds to intuitively summarize the main themes that emerged from the qualitative data and provide context for when these themes emerged. The PETRA feedback is descriptive as statistical analysis of diary data was considered too complex, unclear, and vulnerable to misinterpretation [
An interactive graph on the variation in mood and symptoms in the feedback module. All continuous ecological momentary assessment (EMA) diary items can be selected from the menu on the right side of the screen. Clicking on any of the assessment points will activate a sliding pane depicted in Figure 7. For a video, see here [
Sliding panel in the feedback module that provides relevant contextual information for a specific moment in the time series and surrounding moments. Moments are selected by clicking on assessment points in the graph depicted in Figure 6. For a video, see here [
A word cloud of all freely entered text in the feedback module. Clicking on a word provides more context on when this word was used. For a video, see here [
In this paper, we outline the development of PETRA, a web-based application for personalized EMA in Dutch psychiatric care. Interviews, focus groups, and usability sessions with clients and clinicians demonstrated that they expected an added value of integrating personalized EMA into treatment. Crucial requirements entailed a personalized and user-friendly diary tool built on scientific foundations, which adheres to privacy regulations. The subsequent thorough cocreation design process ensured that PETRA was considered intuitive, user-friendly, and useful for clients and clinicians.
By involving clients and clinicians from the start of the development, PETRA can be considered a significant advancement of current EMA (web-based) applications that are mostly targeted at researchers. By systematically integrating the perspectives of clients and clinicians with those of researchers and software developers, the PETRA web application has promise as a tool for assisting both clients and clinicians in the personalized treatment of mental health problems.
In this project, we attempted to include a diverse group of clients and clinicians with differing levels of interest in using personalized EMA for treatment. However, this was a relatively small sample, and participants were selected based on their interest in EMA. Therefore, it cannot be ruled out that we overestimated the eventual uptake in clinical practice. As EMA requires significant time and resources from both clients and clinicians, it may not appeal to everyone [
Furthermore, there are some limitations that pertain specifically to the PETRA tool. First, PETRA is only accessible to clients via their clinicians via their EHRs. This can be a psychiatrist, psychologist, or psychiatric nurse affiliated with a RoQua mental health care facility. We are currently developing client access to PETRA directly via their personal health record system to enable clients to interpret feedback independently of their clinician. This is in line with clients’ desire to have access to their own data [
Thus far, EMA software development has mainly focused on researchers [
Qualitative work demonstrates that clients and clinicians consider personalized EMA a useful add-on tool for diagnosis and treatment [
Several ethical considerations arise when developing tools for highly personalized self-monitoring. First, the (sometimes qualitative) data are sensitive, and the privacy of the client should be well protected [
Finally, research suggests that clinician training and support via a helpdesk are important requirements for the implementation of EMA in clinical settings [
Although self-monitoring is already a prominent feature in several treatment modalities and protocols [
The second step concerns further scientific validation of the EMA items [
PETRA was developed to meet the demands of clients and clinicians for a personalized and user-friendly EMA tool embedded in routine psychiatric care. By collaboratively constructing EMA diaries and interpreting the resulting dynamic feedback, PETRA offers clients and clinicians a new tool to illuminate daily life processes that worsen or alleviate mental health problems. This approach may have beneficial effects on client self-management and the therapeutic alliance and, thus, has the potential to significantly improve clinical care. Our findings demonstrate the importance of a multidisciplinary approach to the development of personalized EMA tools, including clients, clinicians, researchers, and software developers, to ensure that their needs are sufficiently addressed in the design of the tool. PETRA is unique in its codevelopment process, extensive but user-friendly personalization options, integration into EHR systems, transdiagnostic focus, and strong scientific foundation in the design of EMA diaries and feedback. As such, PETRA paves the way for a systematic investigation of the effect of personalized EMA on specialized mental health care.
Supplementary Materials.
Center for eHealth Research
electronic health record
ecological momentary assessment
General Data Protection Regulation
Personalized Treatment by Real-time Assessment
Rob Giel Research Center
user experience
The authors thank all clients and clinicians for their invaluable insights. The authors also thank the RoQua team, in particular, Henk van der Veen and Inge ten Vaarwerk, for integrating the PETRA (Personalized Treatment by Real-time Assessment) web application within RoQua. The authors gratefully acknowledge Meike Bak for assisting with client recruitment and our research assistants Jannie Brouwer and Roan Carlier. The authors thank Judith Rosmalen for her insightful inputs on relevant ecological momentary assessment items and for her help in setting up the project. Finally, the authors thank Anoek Houben for designing the manual and the animation videos for the PETRA web application.
This study was financially supported by Innovation Fund “Stichting De Friesland” (grant DS81 to HR and MW), the charitable foundation “Stichting tot Steun VCVGZ” (grant 239 to HR and Dr Jojanneke Bastiaansen), the European Research Council under the European Union’s Horizon 2020 research and innovation program (ERC-CoG-2015; grant 681466 to MW), and the Rob Giel Research Center. The funding agencies had no role in the interpretation of the results and the drafting of this manuscript.
FMB, HR, and LvK conceptualized the study, with assistance from the scientific advisors MW, ES, client representative GHP, and clinician representative BD.
HR and MW contributed to funding acquisition. FMB contributed to the data acquisition, formal analysis, project coordination, and writing of the original paper. ACE, TV, EV, and FMB contributed to software development. HR contributed to study supervision. FMB, HR, LvK, ACE, TV, EV, ES, BD, and GHP reviewed and edited the manuscript.
None declared.