Formative evaluation (FE) is a process of evaluating and modifying the content and design of training as it is developed [21]. As part of the FE, interviews were conducted with VA staff to identify learner needs, tailor training objectives, and adapt content from existing training materials. For example, FE helped the project team to determine how to modify the training content to accommodate the needs of staff from different disciplines. This process was ongoing, meaning that the training was modified over time, as new information was collected from participants. This information was obtained formally (via surveys) and informally (via verbal participant feedback), as well as through interviews with Facilitators and members of the project team who delivered the training and served as the primary point of contact with participating sites.

Analyses focused on descriptive statistics detailing the composition and characteristics of the key groups and primary outcomes. Inferential statistics, including 2-sided t tests and general linear models, were used to analyze continuous outcomes. Chi-square tests were used to compare independent groups, and Wilcoxon signed-rank tests were used to compare paired groups when analyses included nominal and ordinal outcomes. In addition to assessing the statistical significance of the results, we provided effect size estimates. For t tests, standardized mean differences (Cohen d) are presented using the conventional interpretative guidelines of 0.20=small, 0.50=medium, and 0.80=large effects [29]. Effect sizes for the general linear models are summarized using partial η² with interpretative guidelines for 0.01=small, 0.09=medium, and 0.25=large effects [29]. All the available data were used for each analysis. Because multiple statistical tests were conducted, family-wise error was controlled using the Holm step-down method, which yielded a criterion of .001 for unadjusted P values [30]. We report unadjusted P values, and those equal to or smaller than this criterion (P=.001) were considered statistically significant. All analyses were conducted using SPSS (version 24.0; IBM Corp).

Missing data rates varied from less than 1% for many of the background variables (eg, service lines and disciplines), approximately 25% (about 278/1110) for basic skills and behavioral intention items, and approximately 30% (about 333/1110) for sociodemographic items (eg, age, gender, and education), which appeared at the end of the survey. Of the 1110 participants enrolled in the project, 862 (77.66%) started the pretraining survey and 1023 (92.16%) started the posttraining survey. As most statistical procedures used (ie, chi-square analyses and Wilcoxon rank sum tests) do not have easily accessible routines to pool the results required to use modern multiple imputation procedures, sensitivity analyses were conducted using 20 multiply imputed data sets, in which (1) the analyses described were replicated using the 20 imputed sets, and (2) analyses using similar but more complex statistical procedures answering similar questions (eg, binary or ordinal regression) were explored. The patterns of the results remained unchanged across the analyses. Thus, initial analyses are provided.

Ethics approval was obtained from the Stanford Institutional Review Board (Protocol #60207), and this project was determined not to qualify as human subjects research.

The sample consisted of 1110 participants and included 42.42% (467/1101) staff members who reported that they provided mental health services or psychotherapy. In addition, 53.18% (569/1070) of participants reported conducting safety planning interventions for suicide prevention. Tables 1 and 2 present the staff work characteristics and sociodemographic information of MHSL staff (743/1109, 67%) and AOSL staff (366/1109, 33%, one participant’s service line could not be identified). A total of 39 participants reported serving both on the mental health and another service line and were counted in the MHSL group. The 2 staff types were similar in gender, disability status, years of experience working with veterans, and average number of hours per week spent interacting with veterans. The 2 groups differed on two background characteristics. First, MHSL staff members were slightly younger (mean 44.2, SD 10.16) compared with AOSL staff members (mean 46.0, SD 10.9), t₇₄₃=2.11, P=.04, Cohen d=0.17. Second, MHSL staff reported higher educational attainment compared with AOSL staff, χ²₅=73.41, P<.001.

The 1110 participants represented a range of service lines with 67% (743/1109) from MHSL, 14.92% (165/1106) from primary care, 14.29% (158/1106) from Medical Specialty service lines, and 7.96% (88/1106) from other service lines. Across service lines, 68.17% (754/1106) of the participants worked in outpatient programs, 20.80% (230/1106) reported working across 2 or more program types, 9.13% (101/1106) worked in inpatient or residential programs, and 1.90% (21/1106) reported working in research or administrative roles. The participants represented 34 different disciplines (Table 3).

Table 4 summarizes the participants’ ownership, attitudes, and skills using VA MH apps broken down by staff type at baseline. Both groups reported a high degree of personal smartphone ownership (828/862, 96.06%). The vast majority (753/830, 90.72%) were either very comfortable or quite comfortable using apps, with most spending at least 1 hour per day using them (719/844, 85.19%). Most participants reported being confident in their skills to show veterans how to download apps (655/841, 77.88%) and showing veterans how to use VA MH apps (593/839, 70.68%) at baseline.

The 2 staff types differed in 3 ways related to technology. First, MHSL staff (204/602, 33.89%) were more likely than AOSL staff (51/260 19.62%) to have government-issued smartphones, χ²₁=17.75, P<.001. Second, MHSL staff had higher mean percentage correct scores on the knowledge check at pretraining (mean 82.1% correct, SD 16%) compared with the AOSL staff, who averaged 76.5% correct (SD 15.22%), t₈₂₉=−4.74, P<.001, Cohen d=−0.36. Finally, MHSL staff (550/574, 95.82%) were more likely to report that they definitely or probably could think of at least one veteran who they planned to use apps with compared with AOSL staff (215/252, 85.32%), χ²₂=29.55, P<.001.

The program exceeded the goals set for participant recruitment. Across the 19 sites, 1110 staff attended the training program with an average of 58.4 participants per site (SD 36.49, median [IQR] 51). This was 247% of the minimum goal of 25 participants from 18 locations. All sites met the goal of recruiting both MHSL and AOSL staff members to participate in the program.

Table 5 summarizes the data from the posttraining survey, which included measures of training satisfaction. There were no significant differences based on staff type for training satisfaction, χ²₃=7.93, P=.047 or for willingness to recommend training to others, χ²₃=7.71, P=.052. The vast majority (961/1024, 93.85%) of participants were satisfied with the training and 92.44% (941/1018) would recommend the training to others.

A training program was implemented to facilitate VA staff’s use of VA MH apps with veterans seen in mental health and non–mental health settings. The number of participants (N=1110) exceeded the project’s recruitment target of 25 per site, suggesting a higher-than-anticipated staff interest in this training. Participants represented 34 disciplines, and 33% (366/1109) of participants worked in settings other than mental health, reflecting the relevance of VA MH apps in diverse settings across the health care system. Given that most VA patients do not use mental health services, with only 22% of those diagnosed with a mental illness receiving psychotherapy and over half dropping out by the second session [31], the level of participation by staff outside mental health settings is promising for veterans who might otherwise not be reached. Furthermore, the training was well received: over 90% of staff members reported being satisfied with the training and said that they would recommend it to others.

In addition to the successful recruitment and engagement of participants, measures of the effectiveness of training indicated significant increases in (1) key knowledge about VA MH apps, (2) confidence in the ability to use VA MH apps with veterans, and (3) behavioral intentions to use VA MH apps with veterans. AOSL and MHSL staff knowledge was high at pretraining, particularly for MHSL staff, but increased significantly from pre- to posttraining for both groups. In terms of skills, about 96% of participants reported being comfortable showing veterans how to both download (980/1018, 96.27%) and use (979/1023, 95.70%) VA MH apps after the training. Finally, although behavioral intentions to use VA MH apps with veterans at pretraining were high, they also increased following training. Nearly all participants endorsed intentions to recommend apps to veterans and could think of at least one specific veteran to whom they could recommend an app.

To summarize, both AOSL and MHSL staff made significant gains in their knowledge and intention to use VA MH apps. Both before and directly after the training, AOSL staff had less knowledge of VA MH apps and were less likely than their MHSL counterparts to be able to think of a Veteran to whom they could introduce a VA MH app. However, the differences between staff types were small, and both groups indicated an increased likelihood of using MH apps with veterans after training.

This quality improvement project relied on voluntary participation of VA employees at facilities that attained leadership support for the implementation of VA MH apps. The extent to which the positive outcomes were observed could be attributed to this being a voluntary sample of early adopters. Furthermore, because we focused on VA Medical Centers, it is unclear how these findings might generalize to other health care settings. Research is needed to understand the potential impacts of introducing VA MH apps outside mental health contexts. For example, future studies could further explore whether VA MH apps help individuals self-refer to mental health services.

Next, there were concerns about ceiling effects at baseline for some key metrics (staff members’ knowledge and confidence in their skills to use VA apps). However, significantly more staff members met the goal criteria after the training. In addition, we did not deliver the same exact training across sites, as we modified our training based on feedback from each site. We also adapted to unanticipated circumstances by switching from an in-person to a live, web-based training model owing to COVID-19 restrictions. The effects of these two factors on outcomes are unclear, but they are consistent with quality improvement work.

Finally, the self-reported survey data presented in this paper were collected before and immediately after the training and did not directly address how attitudes and beliefs were translated into action. Participants reported intentions to act, which, according to the Theory of Planned Behavior [28], can inform actual behavior. This limitation will also be addressed in a subsequent manuscript that will discuss posttraining implementation efforts and, specifically, participants’ adoption of VA MH apps.

To our knowledge, this is the first VA MH app training program to demonstrate that lay health care staff are willing and able to be trained to introduce patient-facing mental health apps in a variety of VA settings. Armstrong et al [17] trained DoD and VA behavioral health providers on a similar suite of mental health applications. Although the authors used different measures of satisfaction, skill acquisition, and behavioral intentions, they similarly showed that participants rated the program well, had skill improvements, and intended to adopt practices following training. The results were promising: most of the 760 clinicians trained over 3 years demonstrated knowledge gains and reported immediately following the training that they believed the training was helpful and intended to apply what they learned in their clinical work. The current project adds to the extant literature by including nonmental health staff and training more staff in a shorter timeframe (1 year vs 3 years) while also increasing staff’s knowledge and behavioral intentions. Importantly, this project coincided with a global pandemic and a concurrent increase in telehealth [32], which may have generated increased interest in digital health technology, such as mental health apps, that can be used remotely.

The VA MH apps training far exceeded the initial goals for staff recruitment and training in all three metrics: program reach, participant satisfaction, and training effectiveness. Over 33% (366/1109) of the participants came from AOSLs, which suggests the utility of VA MH apps across the health care system. Veterans’ access to mental health resources is not limited to traditional referrals to mental health providers. The pathway to getting the needed help may be self-guided or initiated by a peer, chaplain, or other VA employee. By training an occupationally diverse cadre of VA staff to be capable and willing to share mental health resources with veterans, we can potentially reach a wide range of veterans who can benefit from these tools.