Published on 01.10.18 in Vol 5, No 4 (2018): Oct-Dec
Preprints (earlier versions) of this paper are available at http://preprints.jmir.org/preprint/10157, first published Feb 16, 2018.
A Smartphone App to Foster Power in the Everyday Management of Living With Schizophrenia: Qualitative Analysis of Young Adults’ Perspectives
Background: Literature indicates that using smartphone technology is a feasible way of empowering young adults recently diagnosed with schizophrenia to manage everyday living with their illness. The perspective of young adults on this matter, however, is unexplored.
Objective: This study aimed at exploring how young adults recently diagnosed with schizophrenia used and perceived a smartphone app (MindFrame) as a tool to foster power in the everyday management of living with their illness.
Methods: Using participatory design thinking and methods, MindFrame was iteratively developed. MindFrame consists of a smartphone app that allows young adults to access resources to aid their self-management. The app is affiliated with a website to support collaboration with their health care providers (HCPs). From January to December 2016, community-dwelling young adults with a recent diagnosis of schizophrenia were invited to use MindFrame as part of their care. They customized the resources while assessing their health on a daily basis. Then, they were invited to evaluate the use and provide their perspective on the app. The evaluation was qualitative, and data were generated from in-depth interviews. Data were analyzed using a hermeneutical approach.
Results: A total of 98 individuals were eligible for the study (mean age 24.8, range 18-36). Of these, 27 used MindFrame and 13 participated in the evaluation. The analysis showed that to the young adults, MindFrame served to foster power in their everyday management of living with schizophrenia. When MindFrame was used with the HCPs consistently for more than a month, it could provide them with the power to keep up their medication, to keep a step ahead of their illness, and to get appropriate help based on their needs. This empowered them to stay on track with their illness, thus in control of it. It was also reported that MindFrame could fuel the fear of restraint and illness exacerbation, thereby disempowering some from feeling certain and secure.
Conclusions: The findings demonstrate that young adults diagnosed with schizophrenia are amenable to use a smartphone app to monitor their health, manage their medication, and stay alert of the early signs of illness exacerbation. This may empower them to stay on track with their illness, thus in control of it. This indicates the potential of smartphone-based care being capable of aiding this specific population to more confidently manage their new life situation. The potentially disempowering aspect of MindFrame accentuates a need for further research to understand the best uptake and the limitations of smartphone-based schizophrenia care of young adults.
JMIR Ment Health 2018;5(4):e10157
- mental health;
- mobile app;
- participatory design;
- patient empowerment;
- patient involvement;
- patient participation;
- young adults
It is well established that self-management knowledge and skills are the cornerstones of preventing exacerbations and relapse of psychotic illness [- ]. However, many young adults recently diagnosed with schizophrenia skip their clinical visits [ , ], leaving them with only little knowledge and skill power to manage everyday living with the illness efficiently. This causes a serious threat to their current and future health and quality of life [ , ]. This proves the need to find new and innovative approaches to build competencies to empower them to manage the illness in the context of their daily lives. An approach could be smartphone-based care. The pervasive nature of the smartphone and smartphone apps allows to monitor health and for customized information and self-management tools to be disseminated in real time and in real-life settings [ , - ], where and when it is needed [ - ].
Smartphone Apps for Schizophrenia Care
Smartphone apps have been developed for schizophrenia care [- ], yet only limited attention has been paid to mobile health (mHealth) apps to provide illness management support to individuals with schizophrenia outside the confines of the mental health clinic [ ]. A review of smartphone apps for schizophrenia identified only 1 app providing this kind of support [ , ]. This app offered prescheduled and on-demand resources to facilitate symptom management, mood regulation, medication adherence, social functioning, and improved sleep. Evaluation of the app in 33 individuals with schizophrenia or schizoaffective disorder showed that the participants were willing and capable of using the app independently in their own environment [ ].
Although sparse, the existing literature indicates that a smartphone app is a promising way to empower young adults recently diagnosed with schizophrenia to manage everyday living with their illness. The viewpoint of this matter from the perspective of those living with the illness as part of their daily lives, however, is unknown. As interests in smartphone apps in schizophrenia care grow [, , , , , ], this seems increasingly important to explore.
Qualitative research is a systematic inquiry seeking to explore, and eventually understand, the experiences of a particular group of people [, ]. A qualitative inquiry may provide insider perspectives to aid the understanding of the viability of apps to make those recently diagnosed with schizophrenia more capable and confident in managing their lives. Using a qualitative inquiry, the objective of this study was, therefore, to explore how young adults recently diagnosed with schizophrenia used and perceived a smartphone app (MindFrame) as a tool to foster power in the everyday management of living with their illness.
Using participatory design thinking [- ] and methods [ - ], MindFrame was iteratively developed to run on the Monsenso mHealth platform powered by Monsenso ApS. The platform has been technically and clinically validated in various clinical evaluation studies and randomized clinical trials (RCTs) [ ]. First, interviews were conducted with young adults recently diagnosed with schizophrenia to explore their perspective of needs to be supported in the everyday management of living with the illness and to generate ideas of using the smartphone to accommodate the needs [ ]. Then, young adults recently diagnosed with schizophrenia, health care providers (HCPs), a researcher, and software designers collaboratively designed resources to accommodate the needs [ ]. shows MindFrame, which consists of a smartphone app that allows young adults diagnosed with schizophrenia to access resources to aid their self-management. The app is affiliated with a website to support collaboration with their HCPs. A comprehensive description of the resources in MindFrame, including its aims, capabilities, and intended use, is provided in .
|Resource||Aim of resource||Capabilities of resource||Intended use of resource|
|Self-assessment||Monitor health||Data input to report the mental health state, for example, mood, activity, sleep, stress, medication, alcohol, hallucinations, hash, isolation, exercise, hygiene, paranoia, self-harm, sensitivity, and drugs. A note function allows explaining the assessment scores.|
|Visualization||Psychoeducation||Data display of the reported mental health state.|
|Early warnings signs||Awareness on changes in health||Display of early signs of exacerbation of illness and suggestions of how to tackle changes to stay well.|
|Triggers and alerts||Notifications of changes in health||Data survey to notify signs of exacerbation of illness and to provide feedback on actions to take to stay well.|
|Action plan||Strategies to stay in good health||Display of 3 levels of relapse prevention strategies: (1) stay well, (2) what can help, and (3) get help.|
|Medication overview||Medication management||Reminders and tracking of medication adherence.|
|Settings||Customization of resources||Customization of reminders and change of pin code. Access to user guide and a film introducing MindFrame.|
The study design was qualitative and constituted the third phase of a participatory design process. The phases of the overall study are available elsewhere . MindFrame was tested as an intervention during the period of January 1 to December 31, 2016. Subsequently, the intervention was evaluated.
The setting of the research was OPUS. OPUS is a bio, psycho, and social course of intensive outpatient care in Denmark available to young adults, aged 18 to 36 years, for the first 2 years following diagnosis . The course of care is publicly funded. Effects of the OPUS program have been extensively researched and documented [ - ].
MindFrame was implemented as an add-on tool to regular OPUS care in 1 OPUS clinic in Denmark. The criteria for participation in the intervention were the ability to read Danish and willingness to download and use the smartphone app.
The HCPs provided the young adults the invitations to use MindFrame. The invitation informed that (1) MindFrame had been developed in close collaboration with individuals with schizophrenia and HCPs from OPUS as a collaborative tool to support the everyday management of living with the illness, (2) they could use the app for free and for an unlimited period during the intervention period, (3) it was voluntary to use the app, (4) they could terminate use of the app at all times, (5) early termination of the app would not influence their course of care, and (6) they would be invited to share their views on the usefulness and impact of MindFrame at the end of the intervention period. Invitations to use MindFrame were provided throughout the intervention period. Thus, the length of the intervention and the time when the intervention was applied in the course of care differed from person to person.
When a young adult consented to use MindFrame, they were registered on the MindFrame website, and the smartphone app was downloaded from Google Play or App Store. An install guide was provided for this purpose. A secretary at OPUS made the registrations and handled any install problems. The registration procedure automatically generated an email that was sent to the young adult´s private inbox with a secure log-in code (see Ethics section). The log-in code was used to open the app. Individuals who did not own their own smartphone were offered one to use during the intervention period.
The HCPs in OPUS were responsible for teaching and guiding the young adults in using and customizing the resources in MindFrame. Therefore, HCPs received training ahead of the intervention period. The first author and a MindFrame software designer conducted the training. The training was group-based and held as a 2-hour hands-on session, where the app and the website were carefully explained and then put into their hands to play around. The HCPs who were unable to partake in the group training were offered a one-on-one session by the first author. After the training session, the HCPs were provided a hard copy of a user guide describing each resource in MindFrame in depth, customization of the MindFrame resources, and how to receive first-level support. The first author was available for questions and supervision throughout the intervention period.
Following the intervention period, MindFrame was evaluated qualitatively. The evaluation process used for this study was inspired by interpretative hermeneutics. As such, it strove to bring out and manifest what is normally hidden in human experiences and human relations . Data were collected through telephone interviews, which have shown to be productive in qualitative research [ ].
All the young adults who had used MindFrame at some point during the intervention period were invited to participate in the evaluation. Thus, the recruitment strategy for the interviews was pragmatic and convenient . The only criterion for participation in the evaluation was willingness to share experiences of MindFrame use by virtue of knowledge. The HCPs in OPUS distributed the invitation, and the first author phoned those consented to be contacted explaining more about the purpose of the evaluation and their rights as study participants. The young adults were encouraged to ask questions and were given time to make a decision on participation. All made their decision immediately and provided written consent. Characteristics of the evaluation sample are outlined in .
The interviews lasted between 35 and 66 min. They were conducted in Danish and recorded using the TapeACall app from Epic Enterprises. To guide and direct the interviews, a semistructured thematic interview guide  regarding personal power, knowledge power, and skills power [ ] was used. To encourage the participants to speak freely about their views on how MindFrame contributed in the management of their lives with the illness, interview questions were open-ended. However, at the end of each interview, 15 close-ended questions were posed to work around the concept of empowerment and to prompt more direct answers. Examples of the close-ended questions are “When I use MindFrame I feel more in control” and “When I use MindFrame I become more uncertain of what is right and wrong.” Answering could by default be “yes,” “no,” or “I don´t know,” yet most answered in sentences. Given the questions had not been touched upon in the first part of the interview, the participants were invited to unfold their answers.
An interpretative hermeneutical approach, grounded in the work of the German philosopher Hans-Georg Gadamer, guided the data analysis. A hermeneutic interpretative approach goes beyond mere descriptions to look for meaning embedded in common life practices. These meanings are not always apparent to the participants but can be gleaned from the narratives produced by them .
In Gadamer’s perspective, interpretation of meaning is not a stepwise approach. He emphasizes the canon principle that meaning comes from the hermeneutical circle of iteratively moving between part and the whole of the text . Consequently, Gadamer does not provide a method for analyzing text, for example, interview transcripts, audio recordings, observations, and notes [ ]. Nevertheless, he states that to obtain understanding, methodological direction through a systematic approach is needed [ ].
To provide structure in the process of analysis, 4 tasks grounded in the hermeneutical circle served as a guide. The tasks that were derived from Gadamer’s work and proposed by Fleming et al  were as follows: (1) finding fundamental meaning of the text as a whole, (2) exploring parts for meaning, (3) comparing the meaning of the whole with the parts, (4) and identifying passages representative of the interpreted meaning.
Guided by hermeneutical thinking, the analysis began with listening to the tapes multiple times and obtaining a fundamental meaning of the interviews from an empowerment perspective. Then, the fundamental meaning was split into smaller parts that were explored by listening to smaller sections and individual sentences. Using the analytical question “what is said in relation to power,” sections and individual sentences were selected. To obtain meaning from the sections and sentences, they were deconstructed through interpretation, and the interpretations were constantly compared and contrasted with the meaning of the whole. According to Gadamer, there is no understanding without the activity of questioning . Hence, explorative questions were constantly posed to the text in the process of interpretation. To ensure a rigorous analysis, questioning continued until an inner unity, which was free from logical contradictions, had been reached. At this point, categories of synthesized meaning were constructed.
|Gender, n (%)|
|Age in years, mean (range)||24.8 (18-36)|
|Education in years, n (%)|
|Low: ≤9||4 (31)|
|Middle: 10-12||6 (46)|
|High: ≥13||3 (23)|
|Employment status, n (%)|
|Living conditions, n (%)|
|Living alone||8 (61)|
|Living with spouse or partner||4 (31)|
|Living with family||1 (8)|
|Has children, n (%)|
|Support worker, n (%)|
|Medication for mental health issues, n (%)|
In accordance with the Danish law, a formal ethics approval of the study was not required. Authorization by the Danish Data Protection Agency (Datatilsynet) was obtained (2008-58-0028).
The study was consistent with the Declaration of Helsinki , meaning that the participants were fully informed about the purpose of the research. The informed consent was obtained verbally and in writing before the enrollment, and information about the right to withdraw from the study was provided. The participants were carefully explained that any withdrawal from the study would not influence their course of care.
MindFrame was established under the standard security approval and procedures of the information and technology department in the specific region in Denmark where it was applied.
Use of MindFrame
As evidenced in, a total of 98 individuals were eligible to use MindFrame during the intervention period and 27 used it. One of the individuals was excluded from using it as a result of not being able to speak Danish and 50 refused to use it. In 20 cases, individuals were not invited by their HCPs to use MindFrame. On being asked why, the HCPs owed the opt-out decision for exclusions to concerns that these individuals were too ill to use and engage with the app. Out of the 27 young adults who used the app, 13 participated in the evaluation.
The participants in the evaluation described MindFrame as easy and intuitive to use. In accordance with needs and preferences, the period of use of MindFrame differed among the participants. Some participants terminated use within 1 month (n=5), others terminated use within 2 to 3 months (n=4), and others used MindFrame for 6 to 12 months, terminating their use when the intervention period stopped (n=4). Reasons given for self-initiated termination of MindFrame included boredom, lack of motivation and energy, fatigue, and problems quantifying their mental health.
Perceived Use of MindFrame
On the basis of the participants’ descriptions of use, 2 main and very different categories were generated about the usefulness and impact of MindFrame. When MindFrame was used with HCPs consistently for more than a month, it could provide the participants with the power to keep up their medication, to keep a step ahead of their illness, and to get appropriate help based on their needs. This empowered them to stay on track with the illness, thus in control of it. Furthermore, MindFrame could fuel the fear of restraint and illness exacerbation, thereby disempowering some from feeling certain and secure. This was observed when MindFrame was applied early in the course of care when the participants barely knew their HCP.
Five subcategories led to the 2 main categories. These are outlined inand presented in the following section.
MindFrame Can Provide the Participants With the Power to Keep Up Their Medication
A total of 9 participants received psychotropic drugs for their mental illness during the study. They explained how their memory had been disabled by the illness, yet emphasized how MindFrame had helped them take the medication more regularly. As health tracking covered whether the medication had been taken, not taken, or taken with changes, the self-assessment procedure worked as a daily medication reminder, making it easier to comply with the medication regime. This was a comforting way of staying in control of the medication:
Every day I was reminded to take my medication through the app. That worked really, really well. When I was reminded about it I asked myself, “have you remembered to take your medication today.” If not, I ran out to take it straight away.
Some participants explained how they forgot to take the medication deliberately although they knew by heart that they needed it to stay well. One participant who had used MindFrame for 9 months explained how the self-assessment scores had helped her discover that irregular consumption of medication impacted her mental health state. Insight into this pattern of behavior helped her to make the decision to resume her medication regime:
Sometimes the scores made me realize that I needed to take my medication. It is easier to make decisions on [...] resuming taking the pills when I can see that my symptoms are progressing when I don´t take them.
|Words on tapes (quotations)||Immediate answers: what is said in relation to power?||Decontextualization through interpretation: empowering aspects of MindFrame||Categories||Result and major categories|
|When I first started in OPUS I had problems with compliance. MindFrame helped me to remember to take my pills.||Helped me to remember to take my pills.||Power to maintain medication||MindFrame can provide the participants with the power to keep up their medication||MindFrame can empower the participants to stay on track with the illness|
|Triggers and alerts made me conscious about signs I had to pay attention to, and act upon to stay well [...].I believe that has helped me to stay on track.||Made me conscious; pay attention; act [...] to stay well; stay on track.||Power to act timely to stay on the track of health||MindFrame can provide the participants with the power to keep a step ahead of their illness||MindFrame can empower the participants to stay on track with the illness|
|MindFrame helped me to get anti-depressants pretty fast, when I needed it. I don´t believe that would have happened if my psychiatrist had not had the chance to look at my scores.||Helped me [...] fast; when I needed it.||Ability for health care providers (HCPs) to be more responsive to needs||MindFrame can assist the participants with the power to get appropriate help based on their needs||MindFrame can empower the participants to stay on track with the illness|
|If my scores were really bad, then, could they [HCPs] use my scores to put me under restraint? I was really uncertain of that in the beginning.||Could they put me under restraint?; really uncertain.||Lack of power to feel secure||MindFrame can increase participant fears and worries of restraint||MindFrame can also fuel the fear of restraint and illness exacerbation, thereby disempowering some participants from feeling certain and secure|
|Triggers and alerts gave attention to any early signs of change. What is this? Am I getting worse? That made me worried at some point.||What is this?; Am I getting worse?; made me worried.||Lack of power to feel certain||MindFrame can increase uncertainties in the participants about their mental health state||MindFrame can also fuel the fear of restraint and illness exacerbation, thereby disempowering some participants from feeling certain and secure|
Thus, MindFrame seemed to provide the participants receiving psychotropic drugs the power to keep up with their medication so as to stay well. This was the case when the self-assessment procedure was used passively as a reminder to take medication or when the self-assessment scores were used actively to make the decision that medication should be resumed to stay on track.
MindFrame Can Provide the Participants With the Power to Keep a Step Ahead of Their Illness
The participants stressed how they had to react quickly to early signs of exacerbation of illness to prevent symptoms from progressing into full psychosis. The participants who had set up threshold values for triggers emphasized how MindFrame was a powerful resource to this end. They explained how their scores had prompted a trigger, which alerted them to be aware of the early signs of change, causing them to act upon these signs to stay in good health:
The trigger and alert function was really smart. It showed when things changed, and made one aware to do something in order to stay well.
Awareness was brought to mind automatically through the visualization feature in MindFrame even when threshold values had not been set. Most of the participants made self-assessments on a daily basis for a period of time during the intervention period and emphasized how the display of their scores helped them to see when they should behave differently to stay well. This encouraged the belief that the illness would remain within their control:
It is so comforting that I know that the system shows me if the illness is getting worse. Then I know when I should act to prevent it from getting out of control.
To stay in control of the illness, it was not enough to know when action should be taken. Knowing which action should be taken and how to stay on track were equally important. A few of the participants had customized their action plan with their HCPs, and they explained how the plan of action had provided them with strategies to stay well, saying, “The action plan tells me what to do to stay well.” Other few participants had used the action plan without customization, which some found useful.
Thus, MindFrame seemed to provide the participants with the power to keep a step ahead of their illness rather than at the rare end of it by making them aware of when to act and how. This was the case when self-assessments had been conducted for more than a month and especially the case when the self-assessments were used with triggers and a customized action plan.
MindFrame Can Assist the Participants With the Power to Get Appropriate Help Based on Their Needs
All the participants described how cognitive difficulties challenged them when trying to remember how their health had been over time. In this respect, they stressed how MindFrame had empowered their memory to keep track of their state and progress:
It [MindFrame] helps me a great deal when remembering how I was last week or a couple of weeks ago. I cannot find back to how things were, MindFrame has helped me to keep track of this: how it was.
The ability to keep track of their mental health state and progress was strongly emphasized by the participants, as it provided a solid basis for discussing their health and needs of care with the HCPs. To this end, most underlined how mental health tracking assisted their HCPs to ask more direct questions about the fluctuations in their mental health and relations between the mental health state and their behaviors and actions:
It has been easier for [name of HCP] to ask questions since she could see my scores: “I can see that you have had a bad day what happened that day?” She knew how my week had been and could ask more direct questions.
Ultimately, this contextualized dialogue enabled the HCPs to be more responsive to the needs of the participants, which empowered them to receive the help they needed when they needed it:
I was in a period where my thoughts were getting darker and darker and [name of HCP] said to me: “I can see from your scores that your mood and sleep is not good at the moment. I don´t think your antidepressants help you enough.” She was right. Then the dose was increased, and after some time I started to get better.
As such, MindFrame seemed to provide the participants with the power to get appropriate help based on their needs. This was the case when they reported their mental health state and the scores were used by the HCPs as a basis for assessing and adjusting care to their needs.
Several of the participants emphasized how they wanted their HCPs to take even more advantage of using their scores in their course of care. They explained how looking at the scores with the HCPs and getting expert help to add meaning to the score enabled them to better understand the causations of fluctuations in the mental health state and allowed the effectiveness of behavior change to be evaluated. They believed that learning generated from their own data could equip them to more confidently and independently navigate the everyday management of the illness in the long run.
MindFrame Can Increase Participant Fears and Worries of Restraint
As evidenced in the previous section, it seems that MindFrame could provide the participants with the power to stay on track of their illness. However, it also seemed that MindFrame could increase fears and worries in some of the participants, thereby disempowering them from feeling certain and secure. This was observed in 3 participants who had just been enrolled in OPUS and had only known their HCP for a short period. Shared for these participants were concerns of using MindFrame even before beginning its use. They stressed how they were worried that their HCPs could survey their mental health state on a day-to-day basis or keep them under surveillance in the time between consultations. They feared that surveillance could lead to situations where they were unwillingly put under restraint and committed to hospital:
My biggest concern about starting using MindFrame was that my nurse would observe my condition every day. Then, would there be consequences? Could she use my scores to admit me to the hospital?
The fear of surveillance seems to fade with use. Two of the 3 participants stressed how concerns and fears had become less dominant over time as they had become more familiar with MindFrame and certain about the fact that their HCPs were only interested in their scores to provide the best possible care:
At first I was a bit worried that [name of HCP] could see all my scores, but when I found out that she was only interested in my scores to help me my worries disappeared.
In 1 of the 3 participants, fears and concerns of restraint remained. Consequently, this participant did not report his true state of mind on his bad days. Rather, he touched up the scores making his mental health seem better than it was. This participant stopped using MindFrame within 1 month. The rest of the sample did not address fears and worries of restraint in relation to their mental health state being observed by their HCPs. Rather, they talked about surveillance of their mental health as a way of careful watching, helping them to get timely and appropriate help based on their needs.
Thus, for some participants, MindFrame seemed to increase fears and worries of restraint, which prevented them from feeling confident and safe. Worries and fears seemed to fade with use of the app but remained with 1 participant who embellished his data to stay in control.
MindFrame Can Increase Uncertainties in the Participants About Their Mental Health State
MindFrame seemed to provide the participants with the power to keep a step ahead of their illness, thus staying on track. Being a step ahead of the illness, however, was not always perceived positively. Two participants who had conducted self-assessments continuously for several months addressed this. Both participants felt that the notifications felt comforting and allowed them to act timely; however, occasionally it was stressful to be alerted about all the changes in their mental health state, as it left them wondering if their condition was worsening:
Being notified of all the changes sometimes made me anxious. It made me wonder if the illness was maybe about to get out of control.
The 2 participants explained that doubt and hesitation about their mental health state was something they dealt with on a daily basis, thus it was not something new. However, they stressed how the notifications in some ways increased their uncertainty. They experienced this when there were incongruences between their perception of their mental health state and the state communicated by MindFrame. When their personal interpretation of the information gained from their senses did not match the notifications from MindFrame, they were left in doubt of what to think and whether or not to act:
When the notifications tell me to take care and I feel fine, it makes me question myself even more. Is it ok now, or should I do something?
Thus, for some participants, MindFrame seemed to increase uncertainties regarding their mental health state and thereby disempowered them from feeling self-confident and on track with their illness. It was only observed in 2 participants, and they stressed that their uncertainty often disappeared when the notifications were shared and discussed with their HCPs.
This study explored how young adults recently diagnosed with schizophrenia used and perceived the smartphone app MindFrame as a tool to foster power in the everyday management of living with their illness. Findings from the interviews showed that when MindFrame was used continuously for more than a month and with the HCPs, the participants were provided with the power to keep up their medication, to keep a step ahead of their illness, and to get appropriate help based on their needs. This empowered them to stay on track with the illness, thus in control of it.
The findings showed that prolonged and continuous self-assessments were main components responsible for the efficiency of MindFrame. When data were collected consistently over a period of time, a picture of the mental health state of the participant was generated, and this picture worked as a tool to inform decisions about medication and as a tool to alert timely actions to stay in good health. In addition, prolonged and continuous self-assessments worked as a tool to inform the HCPs about the mental health state of the participant, which enabled them to deliver timely care more responsive to their needs. The findings highlight that as a tool to foster power in the everyday management of living with schizophrenia, MindFrame is mostly viable in young adults with schizophrenia who are willing, able, and capable of assessing their health over the course of time. Tenacious use of smartphone apps in the care for persons with schizophrenia may be difficult to obtain [- ], which was also evident in our study where 5 out of 13 participants terminated use of MindFrame within the first month. This was true, although the resources in MindFrame were closely aligned with the needs and preferences of the intended user group, which is suggested to foster engagement [ , - ]. This shows that MindFrame—despite being codesigned—was neither applicable nor appealing to all. The fact that only approximately 35% (27/77) of those invited to use the app accepted to use it further underlines this and indicates that MindFrame may not generalize to the broader population of young adults recently diagnosed with schizophrenia. Further research is needed to establish this.
The findings showed that collaborative use of MindFrame was another main component of its efficacy. When the self-assessment scores of the participants were shared with their HCPs, the HCPs were enabled to deliver care more responsive to their needs, which empowered them to stay on track. The participants stressed how they wanted their HCPs to take even more advantage of using their scores in their course of care. They believed that learning generated from their own data could equip them with the knowledge and skills to more confidently and independently navigate the everyday management of the illness in the long run. In line with previous research, the findings indicate that HCPs are responsive to integrating smartphone technology into young adult schizophrenia care , yet, that HCPs uptake could be better [ ]. Successful implementation and dissemination of smartphone apps as part of schizophrenia care for young adult population will rely on provider uptake as well as client use [ ]. Future research will need to address how to increase provider uptake and evaluate the impact of provider engagement on the ability to navigate the everyday management both in the short and long run.
As evidenced, the findings suggest that MindFrame can be used as a tool to foster power in the everyday management of living with schizophrenia. However, we identified 2 key aspects of use to take into account.
First, we identified that MindFrame could increase fears and worries of restraint, thereby disempowering some participants from feeling certain and secure. The fears and worries were related to data sharing when participants did not know their HCP very well. Ben-Zeev et al  investigated passive monitoring through sensors in a smartphone app. Using a sample of 11 inpatients and 9 outpatients with schizophrenia, for 1- or 2-week periods, respectively, they observed that approximately 20% of the sample felt upset by monitoring. This substantiates that worries related to health monitoring are rather common in individuals with schizophrenia even when data are generated automatically. We found that the feeling of uncertainty blurred when the participants got more familiar with the monitoring aspect of the app and their HCPs. This suggests that certainty may develop with use over time. However, we found that 1 participant embellished his data to stay in control, which accentuates that this might not always be the case. This advocates that health monitoring may have its limitations and highlights the paramount importance of carefully assessing the most appropriate time in the course of care to introduce and use an app for empowering purposes. Future research will need to look closer into the characteristics of those feeling upset from monitoring to fully understand its limitations.
Second, we identified that MindFrame could increase uncertainties about participants’ own mental health state, thereby disempowering some from feeling certain and secure. The uncertainty was related to notifications of exacerbations of illness and arose when the app indicated worsening, but the participant was fine. The findings show that being notified may lead to an emotional response of disturbance when the notification does not correspond to the participant´s sense of health. The same was identified in individuals with severe and very severe chronic obstructive pulmonary disease. Hunicke et al found that disturbance arose when individuals felt better or worse than what the technology indicated . The former is in line with our findings and highlights how monitoring may increase uncertainty even in individuals who have been living and managing their illness for a long time. The finding highlights the paramount importance of using an app as part of a collaborative partnership with the HCP to increase certainty. HCPs have clinical knowledge and insights of importance that would help young adults diagnosed with schizophrenia set up the right threshold values to notify changes in their mental health state and to adjust the values as the illness stabilizes or exacerbates. Given the young adult is left alone to do this, it is likely that the amount of false-negative or false-positive notifications may increase. Ben-Zeev et al stress that in the future, evidence-based mHealth apps will be downloaded directly onto the smartphone and used by individuals with little or no contact to mental health care facilities [ ]. Our findings suggest that in the case of young adults recently diagnosed with schizophrenia, this may leave some worried and uncertain.
Schermer has sketched 2 possible future scenarios of the use of smartphone technologies in mental health care. One scenario is the Big Brother scenario, where monitoring technology will reproduce the old paternalistic paradigm of patient-HCP interaction in which compliance and monitoring are the aims. The other scenario is that it will create a new situation that centers on shared decision making and self-management, adding to the autonomy of the service user . Our findings suggest for the latter scenario to be feasible.
A number of key limitations must be acknowledged. The recruitment strategy restricted 20 individuals from choosing for themselves whether or not to engage in the intervention. Opt-out decisions where HCPs set up their own criteria for excluding individuals with mental health issues from participation in interventions appear rather common [- ]. In our study, this may have contributed to an evaluation sample nonrepresentative of the population and the impression that MindFrame may not generalize to the broader population of young adults with a recent diagnosis of schizophrenia.
The evaluation sample was small, and most of the participants had positive attitudes toward MindFrame. The poor retention of study participants may have overvalued the positive effects of the technology. A replication of the study with a larger sample size and maximum variation sampling in the interviews could help clarify this. Contrary to convenience sampling, maximum variation sampling and extreme case sampling allow the researcher to purposefully select participants to learn from the most extreme and unusual cases .
The evaluation sample was one of convenience and consisted of 9 women and 4 men. Research has established that first episode schizophrenia incident rates are approximately 2 times higher in men than in women [, ]. This suggests that our findings are gender biased and potentially in favor of women. A replicative study with a sample more representative of the population would be interesting to see whether these study findings are gender consistent. This might not be the case, as previous research has provided findings that male gender is a specific predictor of nonadherence to mHealth interventions [ ].
The sample was interviewed post intervention. For the participants who had terminated using MindFrame after a short period, the evaluation was conducted several months after they had stopped using it. Given the cognitive deficits addressed in the analysis and broadly in the scientific literature [, ], it is possible that our study design has contributed to recall bias, which may have prevented some complexities from unfolding. The research process, however, does not indicate this. When interview questions were posed, the participants easily shared their views and experiences.
Our findings demonstrate that young adults recently diagnosed with schizophrenia are amenable of using a smartphone app as part of their everyday life to monitor their health, to manage medication, and to stay alert of early signs of exacerbation of illness. Given the app is used consistently for more than a month and in close collaboration with HCPs, it may empower them to keep the illness within their control.
The findings encourage the application of smartphone-based care to aid this population to better help themselves in the time following the diagnosis. The disempowering aspect of MindFrame accentuates that a smartphone app should be used in a reflected manner at the right time in the course of care and with the right amount of support. Further research is required to understand the best uptake and limitations of smartphone-based young adult schizophrenia care.
This study was funded by a grant from the Tryg Foundation, Denmark, for which the authors want to express their gratitude. The funding source had no involvement in the study. The authors want to express their heartfelt thanks to the HCPs from OPUS for recruiting participants for the study and the young adults who took time to participate in the intervention and in the evaluation. Without their commitment, this study would never have become a reality. Last, but not least, the authors want to thank Julie Walsh, Research Facilitator at Parkwood Institute Research (London, Ontario, Canada), for taking time to proofread and edit this paper.
Conflicts of Interest
- van der Krieke L, Wunderink L, Emerencia AC, de Jonge P, Sytema S. E-mental health self-management for psychotic disorders: state of the art and future perspectives. Psychiatr Serv 2014 Jan 1;65(1):33-49. [CrossRef] [Medline]
- Lee H, Schepp KG. Lessons learned from research with adolescents with schizophrenia and their families. Arch Psychiatr Nurs 2013 Aug;27(4):198-203. [CrossRef] [Medline]
- Zou H, Li Z, Nolan M, Arthur D, Wang H, Hu L. Self-management education interventions for persons with schizophrenia: a meta-analysis. Int J Ment Health Nurs 2013 Jun;22(3):256-271. [CrossRef] [Medline]
- Xia J, Merinder LB, Belgamwar MR. Psychoeducation for schizophrenia. Cochrane Database Syst Rev 2011 Jun 15(6):CD002831 [FREE Full text] [CrossRef] [Medline]
- Stowkowy J, Addington D, Liu L, Hollowell B, Addington J. Predictors of disengagement from treatment in an early psychosis program. Schizophr Res 2012 Apr;136(1-3):7-12. [CrossRef] [Medline]
- O'Brien A, Fahmy R, Singh SP. Disengagement from mental health services. A literature review. Soc Psychiatry Psychiatr Epidemiol 2009 Jul;44(7):558-568. [CrossRef] [Medline]
- Nordentoft M, Melau M, Iversen T, Petersen L, Jeppesen P, Thorup A, et al. From research to practice: how OPUS treatment was accepted and implemented throughout Denmark. Early Interv Psychiatry 2015 Apr;9(2):156-162. [CrossRef] [Medline]
- Jørgensen P, Nordentoft M, Abel MB, Gouliaev G, Jeppesen P, Kassow P. Early detection and assertive community treatment of young psychotics: the Opus Study and design of the trial. Soc Psychiatry Psychiatr Epidemiol 2000 Aug 11;35(7):283-287. [CrossRef] [Medline]
- Kay M, Santos J, Takane M. mHealth: new horizons for health through mobile technologies: based on the findings of the second global survey on eHealth. Geneva: World Health Organization; 2011. URL: http://www.who.int/ehealth/mhealth_summit.pdf [accessed 2018-09-17] [WebCite Cache]
- Donker T, Petrie K, Proudfoot J, Clarke J, Birch MR, Christensen H. Smartphones for smarter delivery of mental health programs: a systematic review. J Med Internet Res 2013 Nov 15;15(11):e247 [FREE Full text] [CrossRef] [Medline]
- Sandoval LR, Torous J, Keshavan MS. Smartphones for smarter care? Self-management in schizophrenia. Am J Psychiatry 2017 Dec 1;174(8):725-728. [CrossRef] [Medline]
- Depp CA, Moore RC, Perivoliotis D, Granholm E. Technology to assess and support self-management in serious mental illness. Dialogues Clin Neurosci 2016 Jun;18(2):171-183 [FREE Full text] [Medline]
- Fiordelli M, Diviani N, Schulz PJ. Mapping mHealth research: a decade of evolution. J Med Internet Res 2013;15(5):e95 [FREE Full text] [CrossRef] [Medline]
- Ben-Zeev D, Wang R, Abdullah S, Brian R, Scherer EA, Mistler LA, et al. Mobile behavioral sensing for outpatients and inpatients with schizophrenia. Psychiatr Serv 2016 May 1;67(5):558-561. [CrossRef] [Medline]
- Klasnja P, Pratt W. Healthcare in the pocket: mapping the space of mobile-phone health interventions. J Biomed Inform 2012 Feb;45(1):184-198 [FREE Full text] [CrossRef] [Medline]
- Ben-Zeev D. Technology in mental health: creating new knowledge and inventing the future of services. Psychiatr Serv 2017 Feb 1;68(2):107-108. [CrossRef] [Medline]
- Ainsworth J, Palmier-Claus JE, Machin M, Barrowclough C, Dunn G, Rogers A, et al. A comparison of two delivery modalities of a mobile phone-based assessment for serious mental illness: native smartphone application vs text-messaging only implementations. J Med Internet Res 2013;15(4):e60 [FREE Full text] [CrossRef] [Medline]
- Palmier-Claus JE, Ainsworth J, Machin M, Barrowclough C, Dunn G, Barkus E, et al. The feasibility and validity of ambulatory self-report of psychotic symptoms using a smartphone software application. BMC Psychiatry 2012;12:172 [FREE Full text] [CrossRef] [Medline]
- Palmier-Claus JE, Taylor PJ, Ainsworth J, Machin M, Dunn G, Lewis SW. The temporal association between self-injurious thoughts and psychotic symptoms: a mobile phone assessment study. Suicide Life Threat Behav 2014 Feb;44(1):101-110. [CrossRef] [Medline]
- Niendam TA, Tully LM, Iosif A, Kumar D, Nye KE, Denton JC, et al. Enhancing early psychosis treatment using smartphone technology: a longitudinal feasibility and validity study. J Psychiatr Res 2018 Jan;96:239-246. [CrossRef] [Medline]
- Macias C, Panch T, Hicks YM, Scolnick JS, Weene DL, Öngür D, et al. Using smartphone apps to promote psychiatric and physical well-being. Psychiatr Q 2015 Dec;86(4):505-519. [CrossRef] [Medline]
- Aschbrenner KA, Naslund JA, Barre LK, Mueser KT, Kinney A, Bartels SJ. Peer health coaching for overweight and obese individuals with serious mental illness: intervention development and initial feasibility study. Transl Behav Med 2015 Sep;5(3):277-284 [FREE Full text] [CrossRef] [Medline]
- Naslund JA, Aschbrenner KA, Barre LK, Bartels SJ. Feasibility of popular m-health technologies for activity tracking among individuals with serious mental illness. Telemed J E Health 2015 Mar;21(3):213-216. [CrossRef] [Medline]
- Schlosser D, Campellone T, Kim D, Truong B, Vergani S, Ward C, et al. Feasibility of PRIME: a cognitive neuroscience-informed mobile app intervention to enhance motivated behavior and improve quality of life in recent onset schizophrenia. JMIR Res Protoc 2016 Apr 28;5(2):e77 [FREE Full text] [CrossRef] [Medline]
- Kumar D, Tully LM, Iosif AM, Zakskorn LN, Nye KE, Zia A, et al. A mobile health platform for clinical monitoring in early psychosis: implementation in community-based outpatient early psychosis care. JMIR Ment Health 2018 Feb 27;5(1):e15 [FREE Full text] [CrossRef] [Medline]
- Firth J, Torous J. Smartphone apps for schizophrenia: a systematic review. JMIR Mhealth Uhealth 2015;3(4):e102 [FREE Full text] [CrossRef] [Medline]
- Ben-Zeev D, Kaiser SM, Brenner CJ, Begale M, Duffecy J, Mohr DC. Development and usability testing of FOCUS: a smartphone system for self-management of schizophrenia. Psychiatr Rehabil J 2013 Dec;36(4):289-296 [FREE Full text] [CrossRef] [Medline]
- Ben-Zeev D, Brenner CJ, Begale M, Duffecy J, Mohr DC, Mueser KT. Feasibility, acceptability, and preliminary efficacy of a smartphone intervention for schizophrenia. Schizophr Bull 2014 Nov;40(6):1244-1253 [FREE Full text] [CrossRef] [Medline]
- Wang R, Aung MS, Abdullah S, Brian R, Campbell AT, Choudhury T, et al. CrossCheck: Towards passive sensing and detection of mental health changes in people with schizophrenia. In: Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing. New York, NY, USA: ACM; 2016 Presented at: UbiComp ’16; September 12-16, 2016; Heidelberg, Germany p. 886-879 URL: https://precisionconference.com/~ubicomp16a/papers/553/paper553.pdf?userRef=QQ8K7OIJ8F11QNJ4XW6JKR9RJOZBEGD1%5Cnpapers2://publication/uuid/B3E8E7A5-9087-4410-AA39-A5F3540F3EA9
- Polit DF, Beck CT. Nursing Research: Generating and Assessing Evidence for Nursing Practice - 9th Edition. Philadelphia: Lippincott Williams & Wilkins; 2011.
- Teherani A, Martimianakis T, Stenfors-Hayes T, Wadhwa A, Varpio L. Choosing a qualitative research approach. J Grad Med Educ 2015 Dec;7(4):669-670 [FREE Full text] [CrossRef] [Medline]
- Spinuzzi C. The methodology of participatory design. Tech Commun 2005 May;52(2):163-174 [FREE Full text]
- Sanders EB. From user-centered to participatory design approaches. In: Frascara J, editor. Design and the Social Sciences. London: Taylor & Francis, Inc; Apr 2002:1-7.
- Simonsen J, Robertson TJ. Participatory design: an introduction. In: Routledge Handbook of Participatory Design. Hoboken: Taylor and Francis; 2013:1-17.
- Kensing F, Greenbaum J. Heritage: having a say. In: Simonsen J, Robertson T, editors. Routledge International Handbook of Participatory Design. Hoboken: Taylor and Francis; 2013:21-36.
- Brandt E, Binder T, Sanders EB. Tools and techniques. Ways to engage telling, making and enacting. In: Simonsen J, Robertson T, editors. Routledge International Handbook of Participatory Design, Edition 1. Hoboken: Routledge International Handbooks; 2013:145-181.
- Sanders EB, Stappers PJ. Probes, toolkits and prototypes: three approaches to making in codesigning. CoDesign Int J CoCreation Des Arts 2013 Mar 6;10(1):5-14. [CrossRef]
- Kensing F. Methods and Practices in Participatory Design. Copenhagen: ITU Press; 2003.
- Healthcare DENMARK. Monsenso URL: https://www.healthcaredenmark.dk/profiles/monsenso.aspx
- Terp M, Bjørnes CD, Jørgensen R, Mainz J, Laursen BS. Collaborating with young adults diagnosed with schizophrenia: a participatory design study to shape the healthcare system. Open J Nurs 2017;7(7):743-758. [CrossRef]
- Terp M, Laursen BS, Jørgensen R, Mainz J, Bjørnes CD. A room for design: through participatory design young adults with schizophrenia become strong collaborators. Int J Ment Health Nurs 2016 Dec;25(6):496-506. [CrossRef] [Medline]
- Melau M, Jeppesen P, Thorup A, Bertelsen M, Petersen L, Gluud C, et al. The effect of five years versus two years of specialised assertive intervention for first episode psychosis - OPUS II: study protocol for a randomized controlled trial. Trials 2011 Mar 10;12:72 [FREE Full text] [CrossRef] [Medline]
- Secher RG, Hjorthøj CR, Austin SF, Thorup A, Jeppesen P, Mors O, et al. Ten-year follow-up of the OPUS specialized early intervention trial for patients with a first episode of psychosis. Schizophr Bull 2015 May;41(3):617-626 [FREE Full text] [CrossRef] [Medline]
- Jeppesen P, Petersen L, Thorup A, Abel MB, Oehlenschlaeger J, Christensen T, et al. Integrated treatment of first-episode psychosis: effect of treatment on family burden: OPUS trial. Br J Psychiatry Suppl 2005 Aug;48:s85-s90. [CrossRef] [Medline]
- Lopez KA, Willis DG. Descriptive versus interpretive phenomenology: their contributions to nursing knowledge. Qual Health Res 2004 May;14(5):726-735. [CrossRef] [Medline]
- Sturges JE, Hanrahan KJ. Comparing telephone and face-to-face qualitative interviewing: a researach note. Qual Res 2002;4(1):107-118. [CrossRef]
- Kvale S, Brinkmann S. InterViews: learning the craft of qualitative research interviewing 2nd edition. Los Angeles: Sage Publications; 2009.
- Cordova D, Bauermeister JA, Fessler K, Delva J, Nelson A, Nurenberg R, Youth Leadership Council. A community-engaged approach to developing an mHealth HIV/STI and drug abuse preventive intervention for primary care: a qualitative study. JMIR Mhealth Uhealth 2015 Dec 18;3(4):e106 [FREE Full text] [CrossRef] [Medline]
- Gadamer HG. Truth and Method Revised Edition. London: Bloomsbury Academic; 2013.
- Fleming V, Gaidys U, Robb Y. Hermeneutic research in nursing: developing a Gadamerian-based research method. Nurs Inq 2003 Jun;10(2):113-120. [Medline]
- WMA. World Medical Association Declaration of Helsinki. JAMA [Internet] 2013 Nov 27;310(20):2191-2194. [CrossRef]
- Palmier-Claus JE, Rogers A, Ainsworth J, Machin M, Barrowclough C, Laverty L, et al. Integrating mobile-phone based assessment for psychosis into people's everyday lives and clinical care: a qualitative study. BMC Psychiatry 2013;13:34 [FREE Full text] [CrossRef] [Medline]
- Killikelly C, He Z, Reeder C, Wykes T. Improving adherence to web-based and mobile technologies for people with psychosis: systematic review of new potential predictors of adherence. JMIR Mhealth Uhealth 2017 Jul 20;5(7):e94 [FREE Full text] [CrossRef] [Medline]
- Gleeson JF, Lederman R, Wadley G, Bendall S, McGorry PD, Alvarez-Jimenez M. Safety and privacy outcomes from a moderated online social therapy for young people with first-episode psychosis. Psychiatr Serv 2014 Apr 1;65(4):546-550. [CrossRef] [Medline]
- Ben-Zeev D, Scherer EA, Gottlieb JD, Rotondi AJ, Brunette MF, Achtyes ED, et al. mHealth for schizophrenia: patient engagement with a mobile phone intervention following hospital discharge. JMIR Ment Health 2016 Jul 27;3(3):e34 [FREE Full text] [CrossRef] [Medline]
- Orlowski SK, Lawn S, Venning A, Winsall M, Jones GM, Wyld K, et al. Participatory research as one piece of the puzzle: a systematic review of consumer involvement in design of technology-based youth mental health and well-being interventions. JMIR Hum Factors 2015 Jul 9;2(2):e12 [FREE Full text] [CrossRef] [Medline]
- Goodwin J, Cummins J, Behan L, O'Brien SM. Development of a mental health smartphone app: perspectives of mental health service users. J Ment Health 2016 Oct;25(5):434-440. [CrossRef] [Medline]
- Torous J, Roux S. Patient-driven innovation for mobile mental health technology: case report of symptom tracking in schizophrenia. JMIR Ment Health 2017 Jul 6;4(3):e27 [FREE Full text] [CrossRef] [Medline]
- Španiel F, Hrdlička J, Novák T, Kožený J, Höschl C, Mohr P, et al. Effectiveness of the information technology-aided program of relapse prevention in schizophrenia (ITAREPS): a randomized, controlled, double-blind study. J Psychiatr Pract 2012 Jul;18(4):269-280. [CrossRef] [Medline]
- Huniche L, Dinesen B, Nielsen C, Grann O, Toft E. Patients' use of self-monitored readings for managing everyday life with COPD: a qualitative study. Telemed J E Health 2013 May;19(5):396-402. [CrossRef] [Medline]
- Schermer M. Telecare and self-management: opportunity to change the paradigm? J Med Ethics 2009 Nov;35(11):688-691. [CrossRef] [Medline]
- Pinfold V, Cotney J, Hamilton S, Weeks C, Corker E, Evans-Lacko S, et al. Improving recruitment to healthcare research studies: clinician judgements explored for opting mental health service users out of the time to change viewpoint survey. J Ment Health 2017 Jul 12:1-7. [CrossRef] [Medline]
- Jørgensen R, Munk-Jørgensen P, Lysaker PH, Buck KD, Hansson L, Zoffmann V. Overcoming recruitment barriers revealed high readiness to participate and low dropout rate among people with schizophrenia in a randomized controlled trial testing the effect of a Guided Self-Determination intervention. BMC Psychiatry 2014 Feb 3;14:28 [FREE Full text] [CrossRef] [Medline]
- Mason VL, Shaw A, Wiles NJ, Mulligan J, Peters TJ, Sharp D, et al. GPs' experiences of primary care mental health research: a qualitative study of the barriers to recruitment. Fam Pract 2007 Oct;24(5):518-525. [CrossRef] [Medline]
- Lal S, Dell'Elce J, Tucci N, Fuhrer R, Tamblyn R, Malla A. Preferences of young adults with first-episode psychosis for receiving specialized mental health services using technology: a survey study. JMIR Ment Health 2015;2(2):e18 [FREE Full text] [CrossRef] [Medline]
- Amminger GP, Harris MG, Conus P, Lambert M, Elkins KS, Yuen HP, et al. Treated incidence of first-episode psychosis in the catchment area of EPPIC between 1997 and 2000. Acta Psychiatr Scand 2006 Nov;114(5):337-345. [CrossRef] [Medline]
- Bowie CR, Harvey PD. Cognitive deficits and functional outcome in schizophrenia. Neuropsychiatr Dis Treat 2006 Dec;2(4):531-536 [FREE Full text] [Medline]
- Harvey PD. Cognitive aspects of schizophrenia. Wiley Interdiscip Rev Cogn Sci 2013 Nov;4(6):599-608. [CrossRef] [Medline]
|HCPs: health care providers|
|mHealth: mobile health|
Edited by G Eysenbach; submitted 16.02.18; peer-reviewed by S Berrouiguet, D Fulford, K Mishina, Y Perry; comments to author 15.03.18; revised version received 08.08.18; accepted 04.09.18; published 01.10.18
©Malene Terp, Rikke Jørgensen, Birgitte Schantz Laursen, Jan Mainz, Charlotte D Bjørnes. Originally published in JMIR Mental Health (http://mental.jmir.org), 01.10.2018.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Mental Health, is properly cited. The complete bibliographic information, a link to the original publication on http://mental.jmir.org/, as well as this copyright and license information must be included.