This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Mental Health, is properly cited. The complete bibliographic information, a link to the original publication on https://mental.jmir.org/, as well as this copyright and license information must be included.
Engagement with mental health smartphone apps is an understudied but critical construct to understand in the pursuit of improved efficacy.
This study aimed to examine engagement as a multidimensional construct for a novel app called
We examined app use in a pilot study (n=31) and identified 5 patterns of behavioral engagement: consistently low, drop-off, adherent, high diary, and superuser.
We present a series of cases (5/31, 16%) from this trial to illustrate the patterns of behavioral engagement and cognitive and affective engagement for each case. With rich participant-level data, we emphasize the diverse engagement patterns and the necessity of studying engagement as a heterogeneous and multifaceted construct.
Our thorough idiographic exploration of engagement with
Over the past 2 decades, the number of available mental health smartphone apps has grown to well over 10,000 [
A critical challenge to realizing the potential of mental health apps is attrition; app use has been found to decline significantly after the first 2 weeks [
Although there is an implicit assumption of a meaningful relationship between app use and benefit, the relationship between app use and clinical outcomes is complex. Greater app use has not consistently been associated with better clinical outcomes (eg, Lin et al [
Although there are many definitions of engagement, most concur on its multifaceted and dynamic nature [
In this study, we aimed to operationalize the model of engagement by Nahum-Shani et al [
In a small pilot study,
This study aimed to (1) present an operationalization of engagement with
This study included 31 participants who were randomly assigned to
Full sample demographics (N=31).
Characteristics | Values | ||
Age (years), mean (SD) | 29.2 (10) | ||
|
|||
|
Nonbinary transmasculine | 1 (3) | |
|
Woman | 19 (61) | |
|
Man | 11 (36) | |
|
|||
|
Queer | 1 (3) | |
|
Bisexual | 3 (10) | |
|
Gay or lesbian | 2 (7) | |
|
Heterosexual | 25 (81) | |
|
|||
|
Do not know | 1 (3) | |
|
Asian and White | 3 (10) | |
|
Asian | 2 (7) | |
|
Hispanic or Latinx | 2 (7) | |
|
Non-Hispanic White | 24 (77) |
Exclusion criteria included current mania, psychosis, or severe clinical acuity, as judged by clinic staff, which would impair the understanding of consent and research procedures. Forgeard et al [
This study was approved by the Mass General Brigham Institutional Review Board (2018P000252).
Features of
Feature or strategy | Empirical support | What does this look like in |
Human support | [ |
App use was guided during acute care as support staff checked in with participants daily or less frequently if preferred. Postdischarge support was continued through weekly email check-ins. |
Customization and notifications | [ |
Participants were prompted to schedule 3 exercise sessions per week in the month after discharge and were then sent push notifications at the scheduled times. Exercise scheduling was customizable such that participants could schedule and change exercise session timing, promoting participants’ sense of control and feasibility to use in the context of the participant’s busy life. |
Personalization | [ |
Increased relevancy of HabitWorks by only offering it to those who demonstrated at least a minimal level of interpretation bias. Participants completed personalization checklists assessing demographic characteristics and worry domains (eg, social situations, panic symptoms, and relationships). The app algorithm then selects relevant word-sentence pairs (see the study by Beard et al [ |
Novelty | [ |
HabitWorks presented variations of the interpretation bias exercise in format and length through the “level up” and bonus functions. When participants reached 90% accuracy, they progressed to the next out of 10 levels, which featured increasingly positive interpretations and introduced novel word-sentence pairings [ The app presented 17 randomized encouraging GIFs, such as a celebrity giving a thumbs up, at the end of each exercise session. |
Mood and tracking features | [ |
Participants completed mood surveys prompted by the app weekly and self-initiated surveys as desired. HabitWorks included progress graphs of mood check-in data, as well as exercise performance. The exercise graphs depicted changes in reaction time and interpretation accuracy over time. |
Habitdiary | [ |
The Habitdiary asked participants to reflect on their week and record instances in which they found themselves jumping to negative conclusions or noticed changes in their thinking or behavior. Participants were prompted to complete entries once weekly during check-ins and could also initiate additional entries as desired. |
Feedback | [ |
HabitWorks provided feedback during the exercise to participants immediately following each trial based on the accuracy of their responses (ie, “Correct!” Or “Try Again!”), as well as at the end of each exercise on overall reaction time, accuracy, and percentage improvement (see the study by Beard et al [ HabitWorks provided PHQ-9a and GAD-7b scores. |
Privacy and data security | [ |
Users required a unique passcode to access HabitWorks. HabitWorks enabled touch ID to access the app and ensured thorough understanding of participant rights, data collected, data storage techniques, and data uses by going over consent documentation and storing this document within the app. |
aPHQ-9: Patient Health Questionnaire-9.
bGAD-7: Generalized Anxiety Disorder-7.
The interpretation bias exercises were based on the WSAP [
Supplemental screenshots of a
Participants were asked to use the app daily during acute care, with support from bachelor’s degree–level research staff as desired. This report focuses on engagement during the month following discharge, during which participants were asked to complete exercises 3 times per week independently, as well as a weekly in-app check-in that included a mood check-in (ie, depression and anxiety scores) and the habit test. During this postdischarge period, participants continued to be supported via weekly email check-ins from the staff. Participants were asked to complete assessments after treatment (1 week after discharge) and after 1 month (1 month after discharge). Participants were compensated US $100 for completing the study assessments but were not compensated for their app use.
Measures were administered via the
Operationalization of engagement in HabitWorks based on the visual model used by Nahum-Shani [
We calculated the number of exercises completed per week, number of Habitdiary entries completed, and number of self-initiated mood surveys.
Adherence to the protocol was defined as the completion of the suggested 12 exercises and 4 weekly check-ins during the 1-month postacute phase of the study.
After the first session of
Participants were asked to complete free-response diary entries weekly during the 1-month postdischarge phase and were able to initiate additional entries as desired from the dashboard of the app (
Supplemental screenshot of the
The participants progressed through a series of 10 levels in the
Participants were asked to provide feedback on the
We administered a self-reported measure of satisfaction [
Several items (ie, “What did you think about the
The expectancy items assessed how participants
App use data were passively collected within the app and stored on a secure REDCap server. Upon study completion, data were exported and aggregated by participants for the following variables: type of use, date, and content related to use (eg, accuracy score for exercises, mood symptom score, and Habitdiary content). Use during the month after discharge was focused on, as many factors (ie, insurance, clinical acuity, and logistics) affected the length of stay in acute care, making comparisons of use during acute care challenging. We calculated the following summary variables for the month after discharge: number of exercises completed per week, number of weekly check-ins completed (of 4), number of Habitdiary entries completed, and number of user-initiated mood surveys completed.
After a thorough visual inspection of the data, the first (RR), second (EB), and last (CB) authors discussed and came to a consensus to identify 5 patterns of engagement in the month after discharge. The 3 authors then independently categorized participants into one of the 5 use patterns: consistently low (0-2 exercises per week; 5/31, 16%), adherent (9-15 exercises during month; 14/31, 45%), drop-off (adherent initially, then dropout; 2/31, 6%), high diary (adherent plus >2 diaries per week; 3/31, 10%), and superuser (>16 exercises during month, 7/31, 23%). We then selected the cases that represented each engagement pattern.
Summary of participant engagement.
Facet and indicator | Participant A |
Participant B |
Participant C |
Participant D |
Participant E |
|
|
||||||
|
Exercises during 1 month after discharge (suggested 12), n (%) | 4 (33) | 13 (108) | 13 (108) | 10 (83) | 60 (500) |
|
Number of Habitdiaries | 4 | 4 | 6 | 11 | 17 |
|
Weekly check-ins (suggested 4), n (%) | 3 (75) | 3 (75) | 2 (50) | 4 (100) | 4 (100) |
|
Number of user-initiated surveys 1 month after discharge | 3 | 3 | 7 | 1 | 22 |
|
||||||
|
1 (not at all) to 9 (completely) | 7 | 6 | 5 | 9 | 7 |
|
Credibility: useful—1 (not at all) to 9 (completely) | 6 | 7 | 3 | 5 | 5 |
|
Level completion by 1 month (out of 10 levels), n (%) | 4 (40) | 8 (80) | 10 (100) | 1 (10) | 10 (100) |
|
Habitdiary content | Relationship functioning, eating behaviors and symptoms, and interpersonal conflict | Dating, current treatment, general mental health status, and awareness of symptom improvement | Symptom improvement and current treatment, social functioning, work, and COVID-19–related worries | Free-response record (ie, monitored with timings): sleep, food, symptoms, and medication | Worries about the future, romantic relationships, family, and health |
|
||||||
|
Expected improvement (%) | 80 | 30 | 10 | 70 | 30 |
|
Exit questionnaire: 1 (completely disagree) to 7 (completely agree), mean (SD) | 6.6 (0.55) | 6 (0.71) | N/Aa | 5.6 (0.55) | 6.6 (0.55) |
aN/A: not applicable.
Participant A was a college student with a primary diagnosis of bipolar disorder. Participant A maintained a low level of activity in the app throughout the month after discharge and completed the 1-month follow-up assessment.
During the month after discharge, participant A completed 75% (3/4) of the weekly check-ins, as well as 3 self-initiated mood check-ins. Exercise completion during the month after discharge was low (ie, 4), reflecting low and sporadic use: participant A completed 1 exercise in week 1, a total of 2 exercises in week 2, no exercise in week 3, and 1 exercise in week 4.
At baseline, participant A’s cognitive engagement, assessed by credibility ratings (out of 9=“completely”), was good (treatment logicality=7 and usefulness of treatment=6). Participant A completed 4 Habitdiary entries that covered several themes such as relational functioning and interpretations (ie, family, social, and romantic relationships), eating-related symptoms, and interpersonal conflict. Level completion was low; they reached level 4 (out of 10) by the 1-month time point. At the 1-month assessment, participant A indicated that they enjoyed the weekly mood check-ins and that these were “eye opening” with regard to their symptoms.
At baseline, affective engagement measured by expectancy was high (80%). At the 1-month follow-up, affective engagement reflected by the exit questionnaire ratings (out of 7=“completely agree”) was excellent (satisfaction=6, helpfulness=7, and user-friendliness=7). At the 1-month assessment, participant A indicated that they liked the notifications and the ability to schedule and reschedule exercise sessions at their convenience.
Participant A was considered “Consistently low” as they did not reach an adherent level of use on a weekly basis, or cumulatively, throughout the month following discharge. Despite low use, participant A demonstrated moderate cognitive engagement and strong affective engagement. Therefore, we speculate that other factors may have affected their behavioral engagement. Notably, participant A’s month after discharge coincided with the onset of the COVID-19 pandemic. Participant A’s lack of activity in week 3 seemed to coincide with an increase in suicidality, for which they received a risk evaluation from the senior author. Their qualitative data revealed other life factors that increased their stress level during their transition out of acute care (ie, moving out of their parents’ home during the onset of the COVID-19 pandemic and conflict with family), which may have contributed to their low use.
Participant B had a primary diagnosis of major depression, was living alone, and was preparing to apply to college. Participant B completed all follow-up assessments.
During the month after discharge, participant B completed 75% (3/4) of the weekly check-ins, as well as 3 self-initiated mood check-ins. Participant B was categorized as adherent as they completed 13 of the 12 suggested exercises.
At baseline, cognitive engagement, assessed by credibility ratings on a scale out of 9 (“completely”), was good (treatment logicality=6 and usefulness of treatment=7). During the 1-month postdischarge period, participant B completed 4 Habitdiary entries that covered several themes such as dating, current treatment, general mental health status, and awareness of improvement of symptoms. Level completion was good; they completed level 8 by the 1-month time point. At the 1-month assessment, participant B mentioned “[HabitWorks] allowed me to have more control over negative automatic thoughts.”
At baseline, affective engagement measured by expectancy was low, with the expected symptom improvement rated at 30%. In the daily sessions, participant B consistently reported finding the app easy to use. At the 1-month follow-up, affective engagement reflected by the exit questionnaire ratings (out of 7=“completely agree”) was excellent (satisfaction=6, helpfulness=6, and user-friendliness=7). In the qualitative interview, participant B said that they found the app easy to use and feasible to fit into the structure of the day.
Participant B was considered “Adherent” as they met the suggested exercise completion benchmarks. Despite their low expectancy early in treatment, they demonstrated strong behavioral, cognitive, and affective engagement throughout the month. They did not exhibit high initiation to use app features outside of the prompted occasions.
Participant C was a teacher and had a primary diagnosis of major depression. Participant C adhered to the study protocol through week 3 of the postdischarge month. Drop-off during week 4 coincided with the transition from remote to in-person learning at their school, and participant C subsequently did not complete the 1-month follow-up assessment.
During the month after discharge, participant C completed 50% (2/4) of the weekly check-ins, as well as 7 self-initiated mood check-ins, all before a drop-off in week 4. Exercise completion during the month after discharge (ie, 13) was adherent but reflected a drop-off in use; participant C completed 5 exercises in weeks 1 and 2, a total of 3 exercises in week 3, and no exercises in week 4.
At baseline, participant C’s cognitive engagement, assessed by credibility ratings (out of 9=“completely”), was low to moderate (treatment logicality=5 and usefulness of treatment=3). Participant C commented on having trouble with the WSAP and ambiguous situations related to work. Participant C completed 6 Habitdiary entries that covered several themes such as symptom improvement and current treatment, social functioning, work, and COVID-19–related worries (ie, getting COVID-19 at work and wearing a mask). Level completion was excellent; they reached level 10 by the end of week 3.
At baseline, affective engagement measured by expectancy was low, with expected symptom improvement rated at 10%. As participant C did not complete the 1-month follow-up, the exit questionnaire ratings and qualitative interviews could not be used to indicate the level of affective engagement.
Participant C was considered “Drop-off” as they initially exceeded the suggested number of exercises and then suddenly dropped off in use and did not complete the follow-up assessment. Although participant C was active, they used all app features (ie, diary, mood surveys, and exercises) and showed good cognitive engagement. Participant C’s drop-off coincided with the transition from remote to in-person school during the COVID-19 pandemic, and they had previously voiced concerns about this transition because of the fear of contracting COVID-19.
Participant D had a primary diagnosis of panic disorder. Participant D was excited to participate and “contribute to science” and was attuned to the app, frequently reporting perceived glitches or malfunctions to study staff. Participant D stated that they wanted to be completely adherent and completed all study assessments.
During the month after discharge, participant D completed 100% (4/4) of the weekly check-ins, as well as 1 self-initiated mood check-in. Exercise completion during the postdischarge month was generally adherent, although slightly less than suggested (ie, 10): a total of 5 exercises in week 1, a total of 2 exercises in weeks 2 and 3, and 1 exercise in week 4.
At baseline, cognitive engagement, assessed by credibility ratings (out of 9=“completely”), was very good (treatment logicality=9 and usefulness of treatment=5). Participant D completed 11 Habitdiary entries and seemed to primarily use this feature as a tool for monitoring sleep, food, symptoms, and medication changes. Level completion was very low, remaining at level 1 by the end of the month after discharge. Despite not improving in exercise accuracy, participant D reported that it was “cool that [the app made me] notice my negative automatic thoughts” and that it was “eye-opening” in that it created greater awareness of interpretive style in daily life.
At baseline, affective engagement measured by expectancy was good, with expected symptom improvement rated at 70%. At the 1-month follow-up, affective engagement reflected by the exit questionnaire ratings (out of 7=“completely agree”) was good (satisfaction=6, helpfulness=5, and user-friendliness=6). In the qualitative interview, participant D reported that they liked the checklists to personalize stimuli and subsequently found all presented stimuli relatable.
Participant D was considered “High diary” as they clearly developed a preference for the Habitdiary feature. Indeed, although participant D completed 10 exercises during the postdischarge month, they seemed to use
Participant E had a primary diagnosis of major depression. Participant E was extremely interested in participating mentioning past positive experiences with mental health apps and an interest in continuing to use apps to address mental health concerns. Participant E was active throughout the study and completed all the study assessments.
During the month after discharge, participant E completed 100% (4/4) of the weekly check-ins, as well as 22 self-initiated mood check-ins. Exercise completion during the postdischarge month was extremely high (ie, 60 total, 15 exercises per week).
At baseline, cognitive engagement, assessed by credibility ratings (out of 9=“completely”), was moderate to good (treatment logicality=7 and usefulness of treatment=5). Participant E completed 17 Habitdiary entries, using this feature as intended to track negative automatic thoughts, as well as negative interpretations of events occurring in daily life. Themes present in the diary entries included worries about the future, romantic relationships, family, and health. Level completion was high, reaching level 10 by the end of the month after discharge. During the follow-up assessment, participant E reported that they found the situations personally relevant and noticed that handling some real-life situations was more challenging after they stopped using the app.
At baseline, affective engagement measured by expectancy was low, with the expected symptom improvement rated at 30%. At the 1-month follow-up, affective engagement reflected by the exit questionnaire ratings (out of 7=“completely agree”) was excellent (satisfaction=7, helpfulness=6, and user-friendliness=7). Throughout the study, participant E reported that the exercises were fun and enjoyable. In the 1-month qualitative interview, participant E reported that they enjoyed both the routineness (ie, consistent daily and weekly elements) and the “game component” of the app. They also mentioned sometimes struggling to quantify symptoms over the past 24 hours during weekly check-ins and sometimes found the app stimuli redundant.
Participant E was considered a “Super user” as they far exceeded benchmarks for exercise completion during the month after discharge. They also completed an extremely high number of Habitdiaries and user-initiated mood surveys during this period.
We examined patterns of behavioral engagement with a new mental health app designed to facilitate a healthier interpretive style as well as cognitive therapy skills practice following discharge from short-term psychiatric care. First, we operationalized engagement using a model that captures its multifaceted and dynamic nature and presented 5 cases reflecting the engagement patterns present in the sample. The data revealed heterogeneity across participants in behavioral use patterns, as well as variability within participants in their behavioral, cognitive, and affective engagement.
We identified 5 patterns of engagement in our sample: consistently low, adherent, drop-off, high diary, and superuser. Most of the participants (22/31, 71%) were categorized as adherent or superuser. This finding differs from the typical pattern of quick disengagement with mental health apps. Indeed, only 16% (5/31) participants were categorized as consistently low in use. This may be because of the framing of the app as an augmentation and extension of care, motivation and excitement to use the app in our sample, and the engagement enhancement strategies used in
High behavioral engagement may have been because of the use of bachelor’s degree–level staff for human support throughout the protocol [
The evidence supporting the usefulness of human support brings to the forefront the therapeutic alliance within app research, a well-documented, robust predictor of treatment outcome in traditional mental health care [
Indicators of cognitive engagement varied across the 5 cases. Although cognitive engagement assessed by initial credibility ratings ranged from average to good, level completion varied dramatically across the cases. Level progression in
However, in addition to level completion, cognitive engagement with
Qualitative data from all 5 cases added further nuance to our understanding of cognitive engagement, indicating that these participants found that the app helped them become aware of and assert control over their negative automatic thoughts, notice their interpretive style in their daily life, and better handle daily life situations. Participants’ use of CBT language (ie, negative automatic thoughts) in their feedback may illustrate a useful integration between the app and their CBT-based partial hospital treatment.
Affective engagement, measured by expectancy for treatment to improve symptoms, was quite low for participants B, C, and E. However, at the 1-month assessment, all participants rated
Although early affective engagement (ie, expectancy of app benefits) was low for some participants and high for others, these early ratings did not correspond in the expected direction with behavioral engagement throughout the 1 month. The typical relationship between expectancy and treatment engagement is such that lower expectancy is associated with lower engagement in treatment [
Participant C (“Drop-off”) illustrates the connection between cognitive engagement and behavioral engagement and the difficulty of relying on just one or the other to determine meaningful use. Although participant C’s use of the app suddenly dropped off after week 3 (ie, behavioral: shorter duration of use), they had already completed the prescribed number of exercises (ie, behavioral: adherent number of exercises) and had achieved the highest level possible in the app (ie, cognitive: interpretation bias accuracy). Their level completion indicates that they reached a “healthy” interpretation level (ie, 90% accuracy) at each level. Considering their behavioral and cognitive engagement together, we can surmise that they effectively used
This discussion aligns with previous research suggesting that behavioral engagement alone does not necessitate better outcomes [
Our study had some limitations. First, the current case series included some indicators of engagement that were chosen post hoc and were specific to the
This case series of
Supplemental measures.
cognitive behavioral therapy
Credibility and Expectancy Questionnaire
randomized controlled trial
Research Electronic Data Capture
Word Sentence Association Paradigm
This research was supported by a grant from the National Institute of Mental Health (R34MH113600) awarded to the senior author and was registered on ClinicalTrials.gov (identifier NCT03509181).
None declared.