We developed a survey with input from VHA operational leadership and mental health providers, including some involved in facilitating VHA’s MBC initiative. The survey was designed to address the potential impacts of the MHC app on the facets of MBC that comprise VHA’s Collect, Share, Act model of MBC, as well as other important constructs highlighted by stakeholders. Survey topics included frequency of use, preferences, barriers, and satisfaction with using the MHC app. A veteran member of our evaluation team reviewed survey question items for clarity and readability. We initiated the first survey mailing to 2690 veterans in June 2021. We initiated a follow-up mailing to nonresponders in July 2021 to facilitate response. Included in the survey mailings were a cover letter and a postage-paid reply envelope. We gave veterans the option to complete surveys by phone as preferred. Survey respondents were subsequently mailed a US $10 CVS (CVS Pharmacy, Inc) gift card in appreciation of their time.

Consistent with data integration through “building” in mixed methods [23], we recruited a subset of survey respondents to participate in follow-up semistructured telephone interviews intended to elaborate on survey responses. We purposefully stratified the interview sample to reflect variation in the following survey responses: app use and experiences with the app, including number of unique log-ins, app satisfaction, types of assessments completed in the app, age, and gender. We developed a semistructured interview guide with feedback from VHA operational leadership and VHA mental health providers and questions that broadly mapped to the domains of the well-established Non-Adoption, Abandonment, and Challenges to the Scaleup, Spread, and Sustainability (NASSS) of Health and Care Technologies Framework [24].

Interviews assessed veteran experiences with completing assessments in the app and using the app’s other features, as well as the perceived impacts of app use on a veteran’s care. Specifically, we asked veterans to describe (1) the condition for which they were using the MHC app, and the mental health care that they receive; (2) their previous experiences with MBC and how using MHC compares (eg, Can you please tell me about if and how you completed VA mental health care assessments before you began using MHC?); (3) their current experiences with R-MBC (eg, Are you still currently using the MHC app to complete assessments assigned to you by your VA mental health providers?); (4) their experience learning to use and using the MHC app (eg, What challenges have you encountered using MHC? What do you think is valuable about MHC?); (5) their perceived impact of MHC use on their mental health care, self-management, and outcomes (eg, How has completing assessments through MHC impacted your experiences with your VA mental health treatment?); and (6) their suggestions for improving MHC/R-MBC (eg, What changes could be made to make using the MHC app easier for veterans?).

Two evaluation team members with qualitative expertise (JP and FRB) conducted all interviews by telephone between August 2021 and September 2021, which coincided with the COVID-19 pandemic. Interviews lasted an average of 42 minutes (range 17-76 minutes). Each interview was audio recorded with the veteran’s permission and then professionally transcribed verbatim. Veterans who participated in an interview were mailed a US $20 CVS gift card in appreciation of their time.

We characterized responses to the survey using descriptive statistics (frequencies and proportions) and compared survey responses between veterans who indicated on their survey that they used the app for ≥3 months with veterans who used the app for ˂3 months using the chi-square test statistic. Only veterans who indicated that they received and completed an assessment in MHC and also responded to the survey question regarding the duration of MHC use were included in the analyses. Stata (version 17; StataCorp LLC) was used for analysis of the survey data.

We analyzed interview data using thematic analysis facilitated by NVivo (version 12.0; QSR International). Two qualitatively trained investigators (RTH and JP) initially drafted a codebook using a deductively driven approach informed by topics in the interview guide. They jointly coded the first 25% of transcripts (5/20 interviews), iteratively refining the codes and definitions in an inductive manner. They then finalized the codebook and independently coded all the remaining transcripts. Any cross-analyst discrepancies in coding decisions were resolved through discussions in weekly meetings. We reviewed and synthesized findings in thematic reports, noting the frequency and emphasis of veteran comments, and selected exemplary quotes. These synthesized reports were then reviewed and discussed with the larger evaluation team for interpretation and integration with the survey findings.

We used a number of established methods [25] that strengthen the rigor and credibility of our findings, including using a sequential-explanatory design that compared quantitative and qualitative data, triangulating interpretations through discussions among a team of qualitatively trained doctoral-level analysts, and incorporating both confirmatory and contrasting cases of emergent themes.

Findings from the integrated quantitative and qualitative data sets supported one another and highlighted 4 key themes [23]. Specifically, (1) the use of the MHC app had positive impacts on care processes for veterans, including increased engagement in mental health care and self-management and (2) enhanced veteran-provider communication and rapport. However, (3) veterans also reported a variety of challenges using MHC. Last, (4) interviewees offered suggestions for improving the app’s future functionality and adoption.

Veterans reported completing a variety of assessments in the MHC app: PTSD (209/267, 78%), depression (191/267, 72%), and anxiety (180/267, 67%) were most common; others included assessments related to sleeping habits (125/267, 47%), quality of life (102/267, 38%), alcohol use (75/267, 28%), and physical health (68/267, 26%). Approximately two-thirds (169/268) of survey respondents indicated that they had been using MHC for ≥3 months. Veterans using the app for ≥3 months rated their overall satisfaction with the app similarly to those who used MHC for <3 months. However, a greater proportion of veterans using the app for ≥3 months endorsed several benefits of using the app (Table 2). A greater proportion of veterans who used MHC for ≥3 months also reported agreement that MHC helped to improve their VHA mental health care when compared with those who used it for <3 months (101/163, 62% versus 44/98, 45% agreed or strongly agreed, respectively; P=.007).

The MHC app offers veterans the ability to view their assessment data in graphs over time. While only 39% (n=104) of survey respondents reported viewing graphs of their assessment results, those who did described how the visualizations helped them gauge personal progress.

For some, the visualizations of their assessment data were an encouraging factor to continue therapy.

Seeing their assessment scores also enhanced some veterans’ self-awareness. As one veteran said, “it lets you know whether you’re in the ‘green level’ or whether you need to seek more help.” Others remarked:

A greater proportion of veterans who had used MHC for ≥3 months (versus <3 months) agreed that using the app helped them better communicate (112/164, 68% versus 51/98, 52% agreed/strongly agreed; P=.009) and improve rapport (95/164, 58% versus 42/98, 43% agreed/strongly agreed; P=.02) with their VHA providers (Table 3).

Consistent with the survey findings, in interviews, veterans reported that using MHC facilitated communication with their providers, particularly initiating or directing therapy conversations. One veteran reported that MHC allowed their provider to “open the book on me.” Others said:

Other veterans explained that sharing assessment scores from MHC improved communication with their providers because it helped them focus and structure their therapy time. As one veteran stated, the assessment scores help his provider “work more on what she knows I’m in need of and what we’re supposed to be going into.” Others said:

Importantly, discussing assessment findings also facilitated rapport and trust.

While veterans described how the use of the MHC app had positive impacts on their self-management and engagement in care, as well as their interactions with their providers, both the survey and interview findings indicated that MHC posed several challenges for veterans.

Survey findings indicated that veterans who used MHC for <3 months reported almost all challenges at a similar frequency as veterans who used the app for ≥3 months (Table 4). However, a greater proportion of veterans who had used the MHC app for <3 months reported that not being able to answer assessment questions as well as they could when they were at a VHA facility was a somewhat serious or serious challenge (32/98, 33% versus 30/166, 18%; P=.007). Despite the challenges reported, 78% (n=207) of veterans expressed that they were comfortable using the app, and over 41% (n=108) described not having trouble using the MHC app. When veterans encountered trouble using MHC (N=154), the most frequently reported actions they took were securely messaging (84/154, 55%) or calling their VHA providers (57/154, 37%), and to a lesser extent, calling the VHA Help Desk for technical support (29/154, 19%).

Veterans described similar challenges within the follow-up semistructured interviews. Notably, many of the challenges they discussed were not unique to the MHC app; rather, they pertained to uncertainty about the purpose of MBC; emotional burden related to completing assessments; and challenges related to PRO assessments more generally.

One veteran, confused about the impact of his scores on his overall mental health care, noted his uncertainty about the purpose of measurement: “I wasn’t sure what the results or effect was gonna be.”

Some veterans also reported emotional burdens related to completing assessments. For example, completing assessments could be emotionally triggering.

Emotional burden was also experienced when a veteran felt anxiety related to the assessments: feeling “pressure” about not scoring well or “failing the program” if they did not score well and the implications this might have for their provider:

Last, some veterans experienced challenges with PRO assessments themselves. First, a veteran found assessing their emotional status “... hard to quantify.... There’s no definition of what severe means... ” (Veteran 0041)

Assessments could also be challenging given competing demands on a veteran’s attention and time:

Another veteran pointed out how completing the same type of assessment over time was “tedious...just because it’s the same questions every time.” (Veteran 0087)

Veterans found inadequate instructions about how to use the app to be a further challenge in completing assessments.

In addition to sharing comments about the challenges they faced, interviews also provided an ideal opportunity for veterans to share their thoughts about how to overcome some of the aforementioned challenges and how to improve the functionality of the MHC app. Technical suggestions included reducing the number of steps to log into MHC to just a username and password, and including a reminder function (eg, an alarm) or a link to one’s electronic calendar to support the completion of assessments and, by extension, sustained engagement. Other suggestions pertained to the assessments themselves. veterans described the potential utility of defining scale terms (eg, does “frequently” mean 3 times/week or 5 times/week?), reordering assessment items so they do not feel redundant, and reducing the time frames over which veterans are asked to reflect upon and recall the symptoms they are experiencing. Veterans also wanted providers to talk to them about how assessments “fit into the big picture” of their mental health care. Finally, some veterans offered their thoughts about ways to support the broader implementation of the MHC app, including marketing campaigns within and beyond clinical settings, providing individual training opportunities and further technical support, and ensuring awareness of the MHC app and its consistent use in practice among VHA mental health providers.