This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Mental Health, is properly cited. The complete bibliographic information, a link to the original publication on http://mental.jmir.org/, as well as this copyright and license information must be included.
Quality of life (QoL) is a prominent outcome measure in mental health. However, conventional methods for QoL assessment rely heavily on language‐based communication and therefore may not be optimal for all individuals with severe mental health problems. In addition, QoL assessment is usually based on a fixed number of life domains. This approach conflicts with the notion that QoL is influenced by individual values and preferences. A digital assessment app facilitates both the accessibility and personalization of QoL assessment and may, therefore, help to further advance QoL assessment among individuals with severe mental health problems.
This study focused on the development of an innovative, visual, and personalized QoL assessment app for people with severe mental health problems: the QoL-ME.
This study targeted 3 groups of individuals with severe mental health problems: (1) people with psychiatric problems, (2) people treated in forensic psychiatry, and (3) people who are homeless. A group of 59 participants contributed to the 6 iterations of the cocreative development of the QoL-ME. In the brainstorming stage, consisting of the first iteration, participants’ previous experiences with questionnaires and mobile apps were explored. Participants gave their feedback on initial designs and wireframes in the second to fourth iterations that made up the design stage. In the usability stage that comprised the final 2 iterations, the usability of the QoL-ME was evaluated.
In the brainstorming stage, participants stressed the importance of privacy and data security and of receiving feedback when answering questionnaires. Participants in the design stage indicated a preference for paging over scrolling, linear navigation, a clean and minimalist layout, the use of touchscreen functionality in various modes of interaction, and the use of visual analog scales. The usability evaluation in the usability stage revealed good to excellent usability.
The cocreative development of the QoL-ME resulted in an app that corresponds to the preferences of participants and has strong usability. Further research is needed to evaluate the psychometric quality of the QoL-ME and to investigate its usefulness in practice.
Quality of life (QoL) has risen to prominence as an outcome in mental health care. Still, many authors agree that there is further room for improvement in the field of QoL assessment, especially regarding the instruments used to assess QoL [
Over the last few years, many digital electronic health (eHealth) technologies for use in mental health care have been developed [
Previous studies reveal that websites and apps that are well designed for the general public may not be appropriate for people with severe mental health problems [
This research covers the cocreative development of a QoL assessment app that does not rely solely on language-based communication, facilitates personalization, and is useful for both patients and clinicians: the QoL-ME. The aforementioned design recommendations will be taken into account, but the development of the QoL-ME will primarily be based on the input of end users, which continues to be the standard in design in general [
This study targeted 3 groups of individuals with severe mental health problems: (1) people with psychiatric problems, (2) people treated in forensic psychiatry, and (3) people who are homeless. Homeless individuals were included in this study because of the high prevalence of severe mental health problems in this group [
Participants were recruited with the help of 6 societal institutions that collaborated in a consortium to facilitate this research project, including a mental health institution; a hospital for forensic psychiatry; a multimodal day treatment center for multiproblem young adults; a day center for people who are homeless; and 2 research institutions focusing on lifestyle, homelessness and addiction.
The QoL-ME was cocreatively developed in an iterative development process in which the 3 aforementioned groups of people with severe mental health problems played an essential and indispensable role. The process consisted of 6 iterations divided over 3 stages: (1) brainstorming stage, (2) design stage, and (3) usability stage. Theoretically, the development process fits in the
Every iteration involved 3 separate user test sessions, and the total number of test sessions was 18. A new group of participants was recruited in every test session. The 3 target groups were involved in every single iteration. In addition, the age distribution of participants was roughly the same in every iteration. Between 2 and 5 individual participants contributed in every test session. The feedback, tips, and insights of end users gathered during test sessions were of vital importance and were fed back to the professional designers who took care of the technical side of the development. In between iterations, the researchers and professional designers discussed the feedback gathered during the previous iteration. If the end users’ opinions and preferences contradicted each other, an attempt at a synthesis was made during this discussion. If necessary and possible, 2 rivaling preferences were tested in the next iteration. In all stages of the development, the input and opinions of end users were instrumental and were used to expand and refine the initial designs and early versions of the app.
The brainstorming stage involved iteration 1. In this stage, participants were invited to share their past experiences with apps, share ideas regarding the improvement of QoL assessment, and comment on basic initial designs. The topic list that was used during the brainstorming stage is available in
The design stage covered iterations 2, 3, and 4. Initially, paper sketches (wireframes) were used to test alternative navigational structures, various possible page-layouts, and possible forms of interaction for the app. In the remainder of the design stage, digitalized versions of these wireframes were gradually refined, expanded, and made functional. Finally, the first prototype was developed.
Schematic overview of the development of the QoL-ME, involving 3 stages and 6 iterations.
In iterations 5 and 6, which together formed the usability stage, the prototype was subjected to usability testing. Participants were invited to complete a single task: to fill out the QoL-ME. To test if participants were able to use the prototype independently, no explanation regarding the QoL-ME was provided. The usability of the prototype was systematically assessed using a modified Dutch version of the System Usability Scale (SUS) [
In 7 of the 12 test sessions, participants consented to audio recordings. In the other 5 test sessions, the researchers took extensive notes. The researchers made an elaborate summary of every test session of the first 4 iterations, based on either the recordings or the notes. The summaries included all the participants’ insights, ideas, and feedback and were discussed together with the designers. On the basis of these discussions, the designers elaborated, adjusted, and polished the QoL-ME.
The English version of the SUS was developed by Brooke [
At the start of every test session, the researcher explained the goal of the research project and how participants were invited to contribute. Next, participants gave their informed consent and were asked if they consented to the audio recording of the test session. To prevent acquiescence bias, the researcher emphasized that they did not create the designs or prototypes themselves. In addition, the researcher stressed that there were no right or wrong answers but that participants’ opinions, ideas, and insights counted. In the brainstorming stage, participants were asked a number of questions, after which they were invited to comment on a number of basic initial designs. In the design stage, participants were invited to comment on the layout of the wireframes and to test various forms of interaction and navigation. In the usability stage, participants were invited to use the QoL-ME and to fill out the SUS afterward. At the end of a session, participants were asked if they had any additional feedback, tips, or questions. Moreover, the researchers explained that participants’ feedback was used to refine the designs, and participants received a 10 Euro gift voucher.
All designs and prototypes were tested using an Apple iPad Air 2, which had a 9.7-inch touchscreen display. The researcher provided this iPad.
Ethical approval was obtained from the Ethics Committee of the Tilburg School of Behavioural and Social Sciences at Tilburg University (EC-2015.44). Informed consent was obtained from each participant. All procedures performed in this study involving human participants were in accordance with the ethical standards of the institutional and national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards.
The results of the development of the QoL-ME app are difficult to interpret without additional knowledge of the structure and content of the QoL-ME. To enable an adequate understanding of the results of this study, the conceptual framework underlying the QoL-ME is described in this section.
The QoL-ME consists of 2 main components: a core version and additional modules. The core version comprises a fixed set of universal QoL domains, and every respondent is required to answer questions on these domains. Research indicates that having meaning in life is especially important for people who are homeless [
The additional modules serve to ensure the personalization of the QoL-ME. Every module corresponds with a domain of QoL, and users are free to select any combination of the 8 modules. The following 8 domains of QoL are included: (1) Support and Attention, (2) Social Contacts, (3) Happiness and Love, (4) Relaxation and Harmony, (5) Leisure, (6) Lifestyle, (7) Finances, and (8) Health and Living. These domains were identified in a visual concept mapping study of the QoL of people with severe mental health problems [
This structure, involving both a core version and additional modules, makes for a flexible QoL assessment app. The core version is useful in contexts where group-level data are of interest, such as comparisons of the QoL of different client populations. The additional modules are especially suitable for use in individual care planning.
A total of 59 participants contributed to the development of the QoL-ME. Their mean age was 40.8 years (SD 15), and over 80% were male (see
Participants in the brainstorming stage reported using apps primarily for communication and maintaining social relations. In addition, 4 younger participants treated in forensic psychiatry reported using apps for services such as internet banking and Web-based shopping. The single most important factor for why participants used certain apps over others was their confidence in the trustworthiness of the apps. The majority of participants indicated having privacy concerns when using apps, but this did not seem to deter them from using apps frequently.
All participants had prior experience with questionnaires, primarily in the context of professional care or research. Participants reported several annoyances regarding their previous experiences with questionnaires, 2 of which were relevant for the development of the QoL-ME: (1) lack of feedback and (2) lack of transparency regarding data use. These considerations were fed back into the development of the QoL-ME. In practice, a feedback module that provided users with insight into their scores was implemented, and special consideration was given to the issue of data ownership, leading to the decision that users retain the ownership of their data.
The participants in the brainstorming stage had a number of ideas regarding the QoL-ME. Some participants indicated a preference for the personalization of the app’s appearance by selecting a personal background or by changing the colors of the app. In addition, participants pointed out that not every patient has their own device and therefore advocated a multiplatform app. Furthermore, a combination of visual- and language-based communication was proposed, and some participants even indicated a preference for audio. Whenever possible, these ideas were incorporated into the initial designs of the QoL-ME that were tested in the subsequent iterations.
As displayed in
First, participants commented on the functionality of the QoL-ME. Specifically, these comments were related to different forms of interaction, operating the app, and receiving feedback. Several mechanisms for selecting the additional modules of the QoL-ME were tested.
Basic demographic characteristics of participants per iteration of the development of the QoL-ME.
Stage of development and iteration number | Participants (n) | Male participants, n (%) | Age, mean (SD) | ||||
Iteration 1 | 10 | 7 (70) | 34.2 (12.8) | ||||
Iteration 2 | 8 | 7 (88) | 32.8 (13.6) | ||||
Iteration 3 | 8 | 7 (88) | 38.9 (12.8) | ||||
Iteration 4 | 9 | 8 (89) | 41 (14.9) | ||||
Total | 25 | 22 (88) | 37.7 (14.3) | ||||
Iteration 5 | 9 | 6 (67) | 42 (17.5) | ||||
Iteration 6 | 15 | 13(87) | 49.7 (11.3) | ||||
Total | 24 | 19 (79) | 46.8 (14.4) | ||||
Total entire development | 59 | 48 (81) | 40.8 (15) |
Overview of the feedback obtained during the 3 iterations of the design stage of the development of the QoL-ME.
Category and subcategory | Feedback | |
Interaction: selecting additional modules | Swiping icons preferred in domain selection | |
Interaction: items additional modules | Visual analog scale preferred to answer questions of additional modules | |
Input | Most participants had no difficulty with the touchscreen, but some did: enable alternative options such as a keyboard and mouse | |
Feedback | Simple visualization of results, avoiding graphs | |
Linear structure | Inevitable choices in hierarchical structure were confusing: preference for linear structure | |
Confirming choices | Confirmation of choices (next and previous) was appreciated | |
Size and position of buttons | Large buttons with fixed sizes (bottom left and bottom right of screen) | |
Creating user profiles | Too much effort and no added value | |
Personalization of background and colors | No added value | |
Layout | Calm and professional layout was evaluated positively | |
Font size | Large font was advised | |
Contrasts | Sufficient contrast between text and background |
Four mechanisms for selecting additional modules. In the top-left mechanism, users rate the importance of every domain individually. In the top right corner, the same mechanism is displayed for every domain at the same time. In the bottom left panel, every domain is rated by giving it one to three stars. In the bottom right panel, the icons on the left and right have to be swiped or dragged to one of the two circles.
Participants indicated a strong preference for the option in which icons had to be swiped (see the bottom right panel in
Second, initial versions of the QoL-ME allowed participants to select the order in which they wanted to progress through the app. Participants had the opportunity to choose 1 of the 4 menu items (see
Most of the participants in the design stage were unsure which of the 4 options to select and preferred a linear navigational structure, which was adopted in later versions of the QoL-ME. The QoL-ME requires participants to navigate the app explicitly by selecting buttons at the bottom of the screen (see
Third, possibilities for the personalization of the QoL-ME were explored. Versions of the QoL-ME that were tested in this stage allowed participants to create a user profile (see
Fourth, throughout the design stage, participants had a fondness for the calm and clean layout of the QoL-ME (see
The average SUS score was 76.8 (SD 14.9; median=76.3), and scores ranged between 35 and 97.5. According to the classification reported by Bangor and Kortum [
Three possible mechanisms for interaction in the items of the additional modules.
Earlier versions of the QoL-ME required users to select 1 of 4 menu options.
Users are required to navigate explicitly by selecting 1 of the 2 buttons at the bottom left and bottom right of the screen.
Early versions of the QoL-ME included the possibility to create a user profile.
Earlier versions of the QoL-ME allowed users to customize the colors of the QoL-ME.
After filling out the SUS, participants were invited to share any additional feedback. Most of the participants in the usability stage did not have any additional feedback and found the QoL-ME to be easy to use, as reflected by their SUS scores. Some participants wanted more explanation on how to select the content of the additional modules of the QoL-ME. Others had difficulty in placing the VAS exactly at the halfway point. These minor remarks were taken into consideration, and some slight modifications to the prototype were made, resulting in the QoL-ME that is described in the following section.
The following section contains a brief walkthrough of the QoL-ME. An accompanying video is provided in
An affirmative answer will lead the user to the core version for people who are homeless. Alternatively, users are invited to fill out the core version for people with psychiatric problems. Having selected the appropriate core version, users arrive at an explanation of the core version. Examples of 2 items of the core version are presented in
Having completed the core version, users are asked to indicate which domains of the additional modules are important to them. A screenshot of the mechanism used to select add-on modules is available in
Next, users answer questions corresponding to their selection of additional modules.
Examples of 2 items of the QoL-ME’s core version.
Four screenshots depicting the additional modules of the QoL-ME. The top left panel displays the mechanism for selecting additional modules. Respondents are invited to drag 8 icons, corresponding to the 8 modules, to either a circle that says important or a circle that says not important. The top right panel shows how respondents are asked to confirm their choice of additional modules. The 2 remaining panels provide examples of items of the additional modules.
Results section of the QoL-ME. The top panel displays how the results of the core version are displayed, whereas the bottom panel demonstrates the results of the additional modules.
This study pertains to the cocreative development of the QoL-ME: an innovative, personalized, and visual QoL assessment app. A diverse group of people with severe mental health problems contributed to every iteration of the development. The feedback regarding the design and functionality of the QoL-ME that was provided by participants played an essential and central role in the development. The usability evaluation using the SUS revealed good to excellent usability of the QoL-ME.
The feedback gathered during the development of the QoL-ME can be split into 3 categories: (1) feedback that corresponds to previous design recommendations [
Second, some feedback deviated from the design guidelines found in the literature. Of the main recommendations of the FEDM 1 covers the navigational structure of a digital technology. The FEDM advocates a shallow hierarchical structure, whereas participants in this study strongly preferred a linear structure, as it removed the need for making navigational choices. Furthermore, the FEDM promotes scrolling down a page for additional content over paging: having to go to another page for additional content. However, in this study, participants indicated a clear preference for paging over scrolling. The fact that the FEDM primarily targets websites, whereas the QoL-ME is an app, may explain this deviation. General guidelines that target smartphone apps specifically do recommend minimizing navigational choices and advise against scrolling [
Third, 2 preferences indicated by participants are specific to the functionalities of the QoL-ME and are, therefore, unrelated to existing design recommendations. First, participants preferred the use of VAS scales over the Likert scale to answer the items of the additional modules. This finding confirms earlier research [
Usability evaluations of the QoL-ME using the SUS reveal good to excellent usability. The average SUS score of 76.8 obtained in this study is similar to SUS scores gathered in usability evaluations of comparable apps. Kooistra et al [
The diversity of the study population is an important strength. Participants from diverse age groups and care backgrounds shared their insights regarding the QoL-ME. This diverse sample ensures that the QoL-ME appeals to a large and diverse group of potential users and may enhance the generalizability of the results to people with severe mental health problems. The strong emphasis on collaboration with people with severe mental health problems can be seen as another strength [
Apart from these strengths, several limitations ought to be taken into account when interpreting the results of this study. First, the sample was not selected randomly but by a combination of convenience sampling and stratified sampling. This sampling strategy may negatively affect the generalizability of the results. At the same time, the aforementioned diversity of the sample indicates that the negative consequences of the sampling strategy are minimal. Second, the results may be biased by a selection effect. It is likely that clients who were interested in this study had at least some affinity and experience with digital technology and apps. If this is the case, potential issues in the design of the QoL-ME may not have been uncovered. However, a number of participants described themselves as digital illiterates and some even indicated never having used apps or touchscreen devices before. This anecdotal evidence appears to indicate that no major selection effect occurred. Nevertheless, participants’ previous experience with digital technologies was not investigated systematically, and therefore, no firm conclusion can be drawn. Third, the group of participants who evaluated the usability of the QoL-ME using the SUS was rather small. However, a study by Tullis and Stentson [
The cocreative development of the QoL-ME resulted in an innovative, personalized, and visual app for QoL assessment. Overall, participants had little difficulty in operating the QoL-ME and were positive regarding its usability. Participants indicated a preference for paging over scrolling, linear navigation, a clean and minimalist layout, and the use of touchscreen functionality in various modes of interaction. Further research is needed to evaluate both the validity and reliability of the QoL-ME. In addition, it is important to investigate the usefulness of the QoL-ME for both clients and care professionals in practice. Moreover, for the QoL-ME to be of use in clinical practice, its sensitivity to change in QoL ought to be examined.
Topic list used during the brainstorming stage of the development of the QoL-ME.
Short video introducing the QoL-ME. Users may choose to watch the video directly after logging into the QoL-ME.
Video walkthrough of the QoL-ME.
electronic health
Flat Explicit Design Model
Lancashire Quality of Life Profile
quality of life
System Usability Scale
visual analog scale
This work is part of the research program
None declared.