Evaluating a Smartphone App (MeT4VeT) to Support the Mental Health of UK Armed Forces Veterans: Feasibility Randomized Controlled Trial

Introduction

Most of the 14,000 individuals who leave the UK Armed Forces (UKAF) every year [1] transition to civilian life successfully; however, some experience difficulties with their mental health [2]. Approximately 6% of UK veterans are likely to have posttraumatic stress disorder (PTSD), 22% report symptoms of common mental disorders such as anxiety or depression, and 10% report alcohol misuse [2]. More than half of the veterans experiencing mental health difficulties do not seek professional help [3], with studies consistently demonstrating that veterans often prefer to manage their mental health problems alone [4]. Veterans with mental health difficulties are prone to negative life outcomes, such as lower employment rates [5,6], emphasizing the need to encourage mental health support in this population.

Previous research investigating barriers to help-seeking in male veterans [7] identified 3 core barriers: veterans struggled to define the problems they were experiencing as mental health difficulties; veterans did not recognize their need for help until their difficulty reached a crisis point; and once help was sought, veterans found it challenging to navigate the care pathway and make sense of the number of services available to them. Given the poor uptake of formal mental health support in this population and the drive to manage mental health independently, an app that enables veterans to actively manage their symptoms and encourages them to access formal care if self-help has been ineffective could be of substantial benefit.

The Mental Health Toolkit for Veterans Project (MeT4VeT) was initiated to develop a smartphone app to help transitioning UK veterans by identifying mental health difficulties, providing a degree of self-help, and assisting them in recognizing when more formal support is required [8]. The app was developed by King’s Centre for Military Health Research, King’s College London according to the UK Medical Research Council (MRC) complex intervention framework [9,10]. This included a plan for the app development process, the involvement of veterans and stakeholders from services providing veteran support, and a scoping review of published research evidence for existing toolkits (eg, apps and websites). A smartphone app was the preferred format because of its geographical reach, 24/7 availability, lack of requirement for face-to-face contact, and ability to be downloaded at no cost [11]. The app was designed according to the MRC complex intervention framework by drawing on existing theories of cognitive behavioral therapy (CBT), which have been used as the foundation for many self-help tools and have demonstrated effectiveness in treating common mental disorders [12].

The MeT4VeT app consisted of five main sections: (1) People, enabling participants to identify symptoms of mental health difficulties through veterans’ stories; (2) Tasks, encouraging participants to set tasks to help them achieve goals in different areas of their life, for example, work, family and physical health; (3) Tools, providing participants with resources to independently manage their mental health; (4) Tracking, allowing participants to monitor their mental health and see their progression through the app; (5) Notifications, daily notifications to encourage engagement with the app. The elements of the app represent the intersection between core components of CBT, behavior change constructs, and potential means to overcome each of the barriers outlined in earlier research [7]. Core principles of CBT include the self-monitoring of thoughts, feelings, and behaviors (incorporated within the People section), behavioral activation and goal setting (included within the Tasks section), the use of techniques such as mindfulness and cognitive reframing (contained within the Tools section), and feedback on behavior (included in the Tracking section) [12].

This paper outlines a feasibility trial conducted in line with the MRC complex intervention framework [9,10]. The framework outlines the 4 phases in the development of an intervention and the trial focused on the second phase: “feasibility: assessing feasibility and acceptability of intervention and evaluation design to make decisions about progression to the next stage of evaluation” [10].

In line with this framework, a set of objectives were created to cover the following three core areas:

1. Assess the ability to recruit and retain participants throughout the trial period (28 days);
2. Evaluate technical app delivery and measurement processes, including the acceptability of 4 outcome measures: General Health Questionnaire (GHQ-12), PTSD Checklist-Civilian Version (PCL-C), Warwick Edinburgh Mental Well-Being Scale (WEMWBS), and World Health Organization Quality of Life Assessment (WHOQOL-BREF);
3. Examine the acceptability of the intervention, including engagement with and usability of the intervention.

Methods

Design

The trial had a parallel design. Participants were randomly assigned on a 1:1 ratio using block randomization to either an intervention group, who received the full smartphone app, or a control group, who received a noninteractive app that contained only signposting information. Data were collected at baseline and follow-up periods of 1 month (28 days) and 3 months (84 days) postbaseline. These time periods were chosen to align with other self-help intervention trials targeting a change in GHQ-12 scores and upon which our initial power calculation was based [13]. The time points enabled an assessment of any change at the end of the trial period (28 days) and again at 3 months to ascertain whether the change was continued. All participants were asked to use the app for a minimum of 28 days and app notifications were sent to the intervention group to encourage engagement.

Participants

Participants were eligible for the study if they (1) are male, (2) own a smartphone, (3) served at least 2 years in the UKAF, (4) left the UKAF within the last 2 years, (5) are not currently undertaking formal mental health treatment (eg, mental health therapy conducted by a medical professional), and (6) indicated a degree of mental health distress assessed via a score of 2 or more on the GHQ-12 [14], similar to previous research [15]. Participants received compensation of a GBP £10 (US $12.85) Amazon voucher for completing each questionnaire (GBP £30 total, US $38.56), and participants in the intervention group who completed an interview received an additional GBP £10 (US $12.85) Amazon voucher.

Recruitment

Participant recruitment was attempted at 3 UK military medical centers between October 2020 and February 2021. Personnel undergo a medical check-up when they leave the military as part of the discharge process, which includes a mental health screening. Three participating medical centers were asked to identify eligible participants at these check-ups and invite them to participate in the study. However, this approach became unfeasible due to the COVID-19 pandemic. The recruitment strategy was then expanded to include the use of social media and advertising in veteran third-sector organizations between February and November 2021. The social media platforms Facebook (Meta Platforms Inc), Instagram (Meta Platforms Inc), and Twitter (Twitter Inc) were used to post both paid and free promotional advertisements, which included a link to the participant information sheet and consent form on the questionnaire platform Qualtrics (Qualtrics International Inc).

App Design

Full details regarding the app development and its components have been reported previously [8]. The intervention group received the MeT4VeT app with full functionality (Figure 1 and Table 1). This included a daily schedule of notifications for the first 7 days related to the main app sections, followed by targeted notifications depending on participants’ usage of the app. For example, participants who completed at least 2 tasks in a week would receive the notification: “You’re making great progress with the ‘Tasks’ section. Are there any other behaviors you’d like to work on?” compared with participants who completed less than 2 tasks in a week: “It’s been a while since you have set some tasks, maybe add some more or change the ones you have set?” The control group received an app containing only signposting information about a range of statutory and third-sector organizations that focused on veterans’ mental health. Both apps were available on Android (Google) through the Google Play Store (Google) and on iOS (Apple Inc) via the TestFlight (Apple Inc) app.

‎

Randomization

Once recruited into the study, participants were assigned a unique proxy identifier. This was linked to a randomization sequence generated in Stata (StataCorp) statistical software, which assigned participants on a 1:1 ratio to either the intervention or control group. All members of the research team were blind to group allocations except for authors SP, BC, and DL. SP and BC managed participant recruitment, conducted primary analyses, and had access to the raw study data. DL developed the MeT4VeT app and had access to the raw study data. Instructions for downloading the relevant app were sent to the participants via email. The apps could only be activated using a QR code or token code, which was sent to the participants by the research team upon successful registration to the trial.

Trial Objectives

Overview

A set of objectives were developed a priori to assess areas of uncertainty before progression to a randomized controlled trial (RCT). These were based on the CONSORT (Consolidated Standards of Reporting Trials) guidelines for conducting a feasibility trial [17] (Table 2).

Recruitment and Retention

Recruitment rates throughout the trial were presented according to the CONSORT flow diagram [17], with a breakdown of reasons provided for participant dropout. Demographic information was captured from all participants to assess their similarity to those who had left the UKAF in the last 2 years, drawn from UK Ministry of Defence (MOD) statistics [18]. Information was captured regarding age, military service branch, military rank, and regular or reservist status (Table 3).

Trial Procedures and Outcome Measures

Trial procedures were assessed by examining the technical delivery of the app and a review of the feedback option within the Settings section where participants were asked to note any technical issues as they occurred. The number of completed in-app surveys and the amount of time required to complete the surveys were also assessed.

Four outcome measures were trialed and completed by the participants in the Questionnaires section of the app. The GHQ-12 is a measure of common mental disorders such as anxiety and depression [14]. It is a 12-item self-report questionnaire measured on a 4-point scale. An example item is “Have you recently been feeling unhappy and depressed?” The GHQ-12 has demonstrated good psychometric properties (eg, Cronbach α=.91) [19].

To evaluate symptoms of PTSD, the PCL-C was used [20]. It is a 17-item self-report questionnaire, measured on a 5-point scale from (1) not at all to (5) extremely. An example item is “Over the past month how much have you been bothered by repeated, disturbing memories, thoughts or images of a stressful experience?” The PCL-C has been shown to have good psychometric properties (eg, Cronbach α=.96) [21].

The WEMWBS measures mental well-being within the previous 2 weeks and is a 14-item self-report questionnaire [22]. It is measured on a 5-point scale from (1) none of the time to (5) all of the time and an example item is “I’ve been thinking clearly.” The WEMWBS has been shown to have good internal consistency (Cronbach α=.89) and test-retest reliability (Cronbach α=.83) [22].

Quality of life was assessed using the WHOQOL-BREF, a 26-item self-report questionnaire [23]. It is measured on a 5-point scale and an example item is “How satisfied are you with yourself?” The WHOQOL-BREF has been shown to have good psychometric properties (eg, Cronbach α>.80 on 3 domains) [24].

Acceptability and Usability

App acceptability was assessed through engagement with the app, obtained from the app usage data. Engagement is defined as usage of the intervention including the temporal patterns (frequency and duration) and depth (specific intervention content) [25]. Engagement has been linked to behavior change through specific mechanisms of action such as the content and delivery of the intervention [25]. App usage data were collected from Google Analytics via Firebase and included information such as the number of times the app was opened, the duration of each app use, and the number of times specific sections of the app were used. Participants’ devices automatically reported the data to Google Analytics, although participants could choose to disable app tracking from their own devices outside of the MeT4VeT app. This type of automatic tracking of usage is a commonly used measure of engagement for digital behavior change interventions [25].

Usability data were obtained from the mHealth App Usability Questionnaire (MAUQ) [26], completed within the Questionnaires section of the app. The MAUQ is a 21-item self-report questionnaire measured on a 7-point scale ranging from strongly disagree (1) to strongly agree (7). Similar to a previous veteran app study [27], 5 questions were removed (resulting in a 16-item questionnaire), as the questions related to communicating with health care services through the app, which was not applicable in our study. Participant responses were aggregated into overall usability and 3 other domains: ease of use and satisfaction, interface and functionality, and usefulness [26]. The mean total score and the mean of each 3 domains were scored out of 7, with a score of 4 indicating a neutral response and higher scores indicating greater app usability. The MAUQ has been shown to have good psychometric properties (eg, Cronbach α>.80) for all 3 domains [26].

Data Analysis

Data were analyzed using Stata v17 (StataCorp). For the outcome measures trialed, independent sample t tests and chi-square tests (for categorical data) were used to assess changes in outcome measures between the groups at 3 time points (baseline, baseline + 1 month, and baseline + 3 months) (Table S1 in Multimedia Appendix 1). App usage data were reported using medians and IQR. The MAUQ was reported using means and SD. A power analysis calculation was performed using the GHQ-12 with a threshold score of 3, as the common thresholds used in previous research are 2 or 3 [15]. Statistical significance is reported using P values with statistical significance being determined as P<.05.

Ethics Approval

The feasibility trial was granted full ethical approval by the UK Ministry of Defence Research Ethics Committee (1074/MODREC/20) and the Research Governance Office at King’s College London (DPRF-19/20-16015).

Results

Overview

Three main areas of the feasibility trial were assessed (recruitment and retention; trial procedures and outcome measures; and acceptability and usability) to provide data on areas of uncertainty prior to progression to an RCT (Table 4).

Recruitment and Retention

Following recruitment, 791 people completed the participant information sheet and 261 did not meet the eligibility criteria (Figure 2). There were multiple reasons why interested participants were not eligible to participate in the study (Table 5). The most common reasons were undergoing mental health treatment and not having left the UKAF in the last 2 years. A total of 530 participants completed the consent form and 377 of those either declined or could not be contacted for recruitment into the study. Overall, 153 participants were recruited into the study, but 99 did not download the app and 4 did not complete the baseline measures. The final sample consisted of 50 participants randomly assigned to the intervention group (n=24) and control group (n=26). After 1 month, 10 participants were lost to follow-up (n=6 intervention group, n=4 control group) and after 3 months, another 6 were lost to follow-up (n=3 in both groups).

‎

At baseline, there were no significant differences between the groups for any participant characteristic (Table 6). The mean age of the participants was 41.7 years (95% CI 38.8-44.7). Participants were predominantly White (47/50, 94%) and married, cohabiting, or in a long-term relationship (41/50, 82%). Two-thirds of participants had served in the Army (32/50, 64%), followed by the Royal Navy or Royal Air Force (18/50, 36%). Participants were predominately noncommissioned officers (40/50, 80%) and regular serving personnel (45/50, 90%). Of the reported reasons for leaving the military, there were several responses in the “other” category (25/50, 50%) such as health problems and dissatisfaction with their military career. Over half of all participants used an Android device (27/50, 54%).

Participant characteristics were compared to the UK MOD personnel statistics [18] for male UKAF regulars leaving service from March 2020 to March 2022 (n=62,740). Statistics including reservists were not available by gender. The findings indicated an approximate mapping for rank: officer, 20% (11% MOD); other ranks, 80% (89% MOD); and service branch (Army, 64% [61% MOD], Royal Navy or Royal Air Force, 36% [39% MOD], Royal Navy or Royal Marines and Royal Air Force). Comparison of the age of participants highlights a far younger age demographic compared to the population included in this study: 36% (75% MOD) were younger than 40 years, 38% (17% MOD) were 40 to 49 years, and 26% (8% MOD) were 50 years and older.

A power calculation was performed to determine the sample size needed for a future RCT based on the difference in GHQ-12 scores at baseline and follow-up (3 months). To detect a difference in the GHQ-12 means of 3 points between groups at the 3-month follow-up, with 80% power and a significance level of 5%, each group would need to have 39 participants (n=78 in total). To allow for an attrition rate of 32% at the 3-month follow-up (16/50) based on this feasibility trial, 115 participants would need to be recruited. To account for a response rate of 29% (153/530) based on this feasibility trial, recruiting 397 participants is recommended for a future RCT.

Trial Procedures and Outcome Measures

Technical delivery of the app was successful, with only minor resolvable technical issues identified by either the research team or the participants. Three participants (total N=50) provided technical feedback within the app; 1 needed support to clear notifications and 2 participants from the control group questioned whether there was additional content. The protocol was successful in collecting all measures outlined and the participants completed 124 surveys within the app, totaling 19 hours (median 21 minutes per participant).

At baseline, there were no significant differences between the groups on any outcome measures (Table S1 in Multimedia Appendix 1). For the intervention group, symptoms of mental health distress (GHQ-12) decreased significantly (P<.001) from preintervention (mean 7.0, SD 3.6) to the 3-month follow-up (mean 2.7, SD 3.6) with a mean difference of 4.3 (95% CI 1.9-6.7) (Table S4 in Multimedia Appendix 1). Symptoms of PTSD (PCL-C) decreased significantly (P=.03) from preintervention (mean 44.2, SD 15.0) to the 3-month follow-up (mean 33.2, SD 13.9) with a mean difference of 11.0 (95% CI 1.3-20.7). Well-being (WEMWBS) improved significantly (P=.04) from preintervention (mean 38.3, SD 9.2) to the 3-month follow-up (mean 44.9, SD 9.4) with a mean difference of −6.6 (95% CI −12.8 to −0.5). For the WHOQOL-BREF subdomain of psychological, quality of life increased significantly (P=.01) from preintervention (mean 11.1, SD 2.6) to the 3-month follow-up (mean 13.3, SD 2.4) with a mean difference of −2.2 (95% CI −3.9 to −0.5).

For the control group, there were no significant differences in any of the outcome measures from preintervention to the 3-month follow-up (Table S5 in Multimedia Appendix 1). Analyses of the outcome measures across all time points are available in Multimedia Appendix 1.

Acceptability

Over the trial period (28 days), the intervention group initialized the app a median of 8.5 times (IQR 3-18) and over a median period of 2.5 weeks (IQR 1-3.5) (Table 7). The median session duration was 20.7 seconds (IQR 17.2-34.3). The control group initialized the noninteractive app a median of 2 times (IQR 1-6) and over a median period of 1 week (IQR 1-1). The median session duration was 29 seconds (IQR 20-39.2).

The People section of the app was accessed most frequently by the intervention group with an average of 14 (IQR 4-45) visits per participant, followed by the Dashboard section with an average of 11 (IQR 6-21) visits per participant (Table 8). Other than the Questionnaires section, participants spent the most time on the People section with an average visit length of 2.8 minutes (median 169.3 seconds, IQR 54.2-430.4). Out of the 4 main sections of the app (People, Tasks, Tools, and Tracking), the Tracking section was accessed by the least number of participants (n=19, 79%), the Tasks section was used the least number of times (median 8.5, IQR 4-43) and the Tools section was used for the least amount of time (median 49.5, IQR 6.7-166.8).

Usability

Participants in the intervention group rated the overall usability (mean 5.3, SD 1.5) on the MAUQ and each of the subdomains of ease of use and satisfaction (mean 5.5, SD 1.5), interface and functionality (mean 4.9, SD 1.6), and usefulness (mean 5.3, SD 1.8) of the app with positive (above neutral) responses. The control group had higher scores compared to the intervention group for overall usability (mean 5.6, SD 1.2), ease of use and satisfaction (mean 5.9, SD 1.1), interface and functionality (mean 5.5, SD 1.4), and lower scores for usefulness (mean 4.8, SD 1.6).

Discussion

Principal Findings

The aims of the trial were to collect data to determine the feasibility of running an RCT by assessing (1) recruitment and retention, (2) trial procedures and outcome measures, and (3) the acceptability of the MeT4VeT app.

Recruitment and Retention

The COVID-19 pandemic prevented effective recruitment through military bases and social media was used as an alternative recruitment strategy. This presented some challenges in obtaining valid responses and retaining participants, which likely accounts for the high dropout rate. These challenges are common in research using social media as a recruitment tool [28,29]. If a future RCT uses social media recruitment, further work will be required to reduce dropout rates or increase the provision of resources to support this degree of dropout. Recruitment may have also been impacted by stringent restrictions from Apple for publishing apps on their app store and another app (TestFlight) was required to be installed before MeT4VeT could be downloaded, which may have led to reduced engagement. Once participants were engaged in the trial, the retention rate was 68% (34/50) from baseline measures to follow-up, which is acceptable compared to other trials assessing interventions within a military veteran population [30,31]. A trial exploring a mental health monitoring app within this population indicated a retention rate of 81% from baseline to follow-up [30] and a trial evaluating an alcohol use reduction intervention reported a retention rate of 22% to follow-up [31].

The sample in this study is comparable to UK MOD statistics [18] in terms of service branch and rank division across the UKAF. However, our sample represented a substantially older demographic when compared to those leaving the UKAF between 2020 and 2022 [18]. To improve comparability, we suggest recruiting from a wider age range to include younger veterans and females.

Examination of the eligibility criteria showed that a large number of potential participants were not eligible as they had not left the UKAF within the last 2 years. Future studies could relax this criterion to enable veterans to enter the trial at any stage of transition. An even greater number of potential participants were not eligible to participate because they were currently undertaking mental health treatment, which could indicate a need for the development of an app to enable self-management to complement therapy within this population. The presence of a group of potential participants who did not meet the GHQ-12 cut-off but were interested in an app to manage their mental health suggests a potentially broader scope of interest to maintain positive mental health or prevent mental health distress.

Trial Procedures and Outcome Measures

The results of the trial indicate it would be feasible to conduct an RCT using the same technical delivery of the app and the same set of measurement processes. Four outcome measures were deemed acceptable for data collection, with the intervention indicating a potential improvement in mental health from preintervention to the 3-month follow-up, contrasting with no significant improvement in the GHQ-12 scores for the control group over the same time period. Due to the small sample size, it is difficult to draw any firm conclusions from these results, but they provide a promising signal for the MeT4VeT app and the use of the GHQ-12 as an outcome measure to be explored in a future RCT with longer follow-up periods and a larger sample size. A provisional sample size for a later RCT of 397 was calculated.

Acceptability and Usability

The MAUQ scores indicated positive responses to the app, with scores ranging from 4.9 to 5.5, all of which were above a neutral response of 4. However, the scores were below those of a similar app designed to reduce veterans’ alcohol consumption where scores of 5.7 to 5.9 were reported across the MAUQ domains [31]. Before a future RCT, work should be conducted using qualitative feedback on the app to refine its usability. Qualitative data were collected from this trial and will be reported in a future publication.

The app usage data indicated that the intervention group engaged with the MeT4VeT app more often (initializations and sessions) and for a longer duration than the control group (weeks active). When focusing on the 4 core sections of the app (People, Tasks, Tools, and Tracking), the People section was accessed most frequently and for the longest duration and it required active participation. The remaining sections were accessed less frequently by participants. This may have been because they could be accessed via other resources (eg, Tools), whereas the People section provided a novel resource (videos with veterans) not readily available elsewhere. A future iteration of the app may be best focused as a gateway into existing self-help tools, with continued development of the People, Tasks, and Tracking sections and replacement of the Tools section with links to existing resources. This aligns with the original intention of the app, to create a way to help people at the beginning of their journey to identify symptoms and track them over time to recognize when difficulties occur and when self-help, or more formal support, may be beneficial. The median duration of app usage in the intervention group was 2.5 weeks, which falls below the 4-week trial period. Before an RCT is initiated, further research is required to explore the reasons behind early disengagement with the app and potentially refine the notification schedule to encourage app usage throughout the full trial period.

Strengths

The strengths of this study include its rigorous design, with random assignment to groups and the inclusion of a control group. The MeT4VeT app development used a strong conceptual framework informed by the military mental health literature and previous research [7]. Furthermore, feedback on the app was provided throughout the app development process by stakeholders working on veterans’ mental health at 4 stakeholder events, through interviews with veterans, and veteran involvement and engagement groups.

Limitations

The limitations of the study include the relatively small sample size, partly due to the COVID-19 pandemic, which impacted recruitment, and partly due to the loss of potential participants lacking interest in trial initiation. The generalizability of the results is affected by the limitations of the sample, which included only male veterans and those that had left the UKAF within the last 2 years. The development of the app was based on data drawn from research on male veterans [7], so the same barriers could not be assumed to be synonymous for the female veteran population. Future research should explore whether female veterans experience the same barriers and evaluate the use of the app in this population. The feasibility study tentatively demonstrated that MeT4VeT may be useful for those within 2 years of leaving the UKAF and whose departure from the UKAF was over a longer time period; further research is warranted to explore this.

Conclusions

The results of the trial suggest that although the study protocol was both feasible and acceptable, the recruitment process requires further refinement to provide an appropriate number of participants for a future RCT. The outcome measures used within the trial were acceptable for use in a future RCT to robustly examine the effectiveness of the app. The findings indicate that the MeT4VeT app shows some promise as a useful, accessible way for veterans to monitor and manage their mental health.