The target population for the survey was United States (US) military veterans of the Operation Enduring Freedom-Operation Iraqi Freedom (OEF-OIF) service era (September 2001-present), also referred to as Iraq and Afghanistan era veterans. To be eligible, individuals needed to be age 18 or older, and on active duty in the US Armed Forces after September 2001 but not presently. We excluded individuals who completed surveys in less than five minutes, had more than one survey response, or incorrectly answered a military-related “insider knowledge” question to reduce the chance of online survey misrepresentation [18,21]. Survey completion was defined as those respondents who reached the end of the online survey and were not excluded based on the above quality control measures.

Facebook offers myriad options related to placement and targeting of ads; the same parameters were used for all ads. Ads were run simultaneously, to identical audiences, with the same ad budget, and for the same duration of time. Ads ran for a total of 45 days between January 13, 2017, and March 18, 2017, except for one ad that was briefly deactivated by Facebook for technical reasons. Ads were exclusively placed in the News Feed on computers and not mobile phones as the survey was developed for computer administration. Ads were optimized per Facebook’s algorithm for clicks, meaning that ads were automatically shown to users whom Facebook anticipated would click at the highest rates, in a targeting process adjusted by actual clicks during the campaign.

Study ads were broadly targeted at Facebook users in the US of any age or gender who had at least one of a variety of veteran-related characteristics (eg, interest in “United States Armed Forces” or “Supporting Our Veterans” as determined by their Facebook profile). Text above the ad image indicated that veterans who served between 2001 and 2017 were needed for an “online health survey.”

We designed a total of 15 ad variations in a 3x5 factorial design, with 3 images (ie, a person taking a survey on a tablet device; a veteran with his family; and soldiers marching) varied against five headlines. Ad images are illustrated in Figure 1. Headlines were informed by empirical research in psychology and survey methodology as well as established principles in behavioral economics known to help nudge behavior [22].

Specific approaches that can motivate research participation include: targeting feelings of altruism or a “warm-glow” effect [23], harnessing “psychological capital”, which is closely correlated with a sense of empowerment [24,25], or using a statement of what others in similar situations do, also known as descriptive social norms [26]. Providing financial incentives [27] or encouraging the sharing of content with social network members could also increase engagement [28]. Based on these ideas, we crafted and used the following 5 ad headlines:

Ads were hosted by Oregon Health & Science University (OHSU) and linked to an online survey. To calculate survey participation and other outcomes by an ad, we constructed separate uniform resource locators (URLs) to the online survey for each ad. Prospective participants initially completed an online consent and eligibility screener. As an incentive for survey completion, we informed potential participants of an optional sweepstakes or lottery, in which two randomly selected survey participants who provided contact information would receive an iPad. Eligible, consented participants then completed the full online survey. Before and after survey, we provided all participants with a series of online, phone, and text messaging mental health treatment resources, including options for crisis situations, non-urgent treatment referral, and—as we would not be aware of the particular location of respondents—ways to locate local support and treatment resources. The institutional review board of OHSU approved all study procedures.

Our primary outcome was survey completion, which represents the highest level of engagement with a Facebook ad [20]. As additional outcomes, we included measures of ad engagement that are automatically tabulated by Facebook for advertisers:

The online survey included a series of self-report questions to obtain background information about the sample including sociodemographic characteristics, military history, social media use, and interest in social media-based interventions. Using survey items from the National Survey of Veterans [29], we assessed the period of service (ie, “Have you ever served on active duty in the US Armed Forces? Active duty includes serving in the US Armed Forces as well as activation from the Reserves or National Guard.” and “ When did you serve on active duty in the US. Armed Forces? Mark all service eras that apply. ”), branch of service (ie, “In which branch or branches did you serve on active duty?”) and deployment history (ie, “Did you deploy in support of Operation Enduring Freedom (OEF) or Operation Iraqi Freedom (OIF) or Operation New Dawn (OND)”) Frequency of Facebook use was assessed by adapting previously validated survey items used by Pew Research [30]. We used two additional items from the National Survey of Veterans to determine VA health service use (ie, “Have you ever been enrolled in VA health care?” and “In the past 12 months, did you use any VA health care services?”) [29]. Participants who responded “Don’t know” to health service use questions were classified as not non-users.

To screen for mental health problems, we employed reliable and valid self-report tools. For PTSD, we used the Primary Care PTSD Screen for DSM-5 (PC-PTSD), a five-item scale assessing past-month symptoms of a lifetime traumatic event. A score of three or higher on the PC-PTSD indicates a positive screen [31]. For alcohol misuse, we used the Alcohol Use Disorders Identification Test Alcohol Consumption Questions (AUDIT-C), a three-item scale on frequency and intensity of drinking. An AUDIT-C score of four or higher for men, or three or higher for women, indicates a positive screen for problematic drinking [32]. For major depression, we used the Patient Health Questionnaire-2 (PHQ-2), a two-item scale on anhedonia and depressed mood in the previous two weeks. A score of two or higher on the PHQ-2 indicates a positive screen [33]. For suicidality, we used the Depressive Symptom Inventory Suicidality Subscale (DSI-SS), a four-item scale on suicidal ideation within the past two weeks [34]. A score of two or higher on the DSI-SS indicates a positive screen in a population-based sample [35].

Demographic variables were compared by ad text and image for participants in the analytic sample using Pearson’s chi-square test, or ANOVA for age. All outcomes were modeled as negative binomial counts. The study design parameters of image, headline, and the interaction were included as independent factors. The model for clicks and CTR included an offset for the number of impressions; the model for reactions included an offset for the number of clicks.

Over the 45 days of the advertising campaign (Figure 2), the Facebook ads produced 827,918 impressions, 9,527 clicks (CTR=9,527/827,918, 1.20%), and 1,787 reactions. There were 1,329 complete responses to the eligibility screener, of which 711 met eligibility criteria, and 605 completions of the online survey (ie, 605/711, 85% response rate). A total of 18 responses were excluded from the analysis based on quality control measures. Ten took less than 5 minutes, 2 claimed nonexistent pay grades, and 6 were duplicate responses. This left a final sample of 587 (ie, roughly 13 new participants each day). Total ad expenditure was US $11,427, yielding an average cost per analyzed survey of US $19.47.

Characteristics of survey participants are described in Table 1. Their mean age was 40 years. A total of 81% (474/587) were male, and 81% (477/587) were white and non-Hispanic. In addition to serving during the Iraq and Afghanistan era, many also served during prior eras, particularly the Gulf War era (213/587, 36%). The majority (426/587, 73%) had been deployed in support of OEF-OIF. In this sample, the majority (326/587, 56%) were in the Army, as compared with approximately 36% among active-duty military personnel [36].

With regards to VA enrollment, 33% of participants (193/587) had not enrolled in VA health services and 55% (322/587) had not used VA care in the prior year. Positive screens for current mental health problems were common: PTSD (266/585, 45%), problematic drinking (243/584, 42%), and major depression (164/586, 28%). Twenty-three percent (132/585) indicated current suicidality. Of the participants not enrolled in VA health services, 21% (40/193) reported current suicidality.

Gender of respondents varied by ad text, (χ²₄ (N=583)=10.7, P=.03), with sharing and empowerment messages having a higher proportion of women. Age varied by text (F(2, 585)=11.84, P<.01) and image (F(2, 585)=10.09, P<.01), with the soldiers marching image and incentive text attracting the youngest respondents and the survey-taking image and altruism and social norms headlines attracting the oldest. Service era varied correspondingly, with the soldiers marching image (χ²₂ (N=585)=9.90, P<.01) and incentive text (χ²₄, N=585)=22.03, P<.01) attracting a higher proportion of respondents who had only served during the OEF-OIF era. Race, ethnicity, education, military branch, and deployment to Iraq or Afghanistan did not vary significantly by ad text or image.

There was a main effect for ad image across impressions, CTR, and reactions, but not survey participation. In terms of both impressions and click-through rates, the soldiers marching image performed better than the survey-taking and family images (P<.001 for all comparisons). In addition, the soldiers marching image generated significantly more reactions than the survey-taking (P=.001) and family (P<.001) images. However, there were no significant differences by ad image in terms of survey participation.

There was also a main effect for ad headline on ad engagement outcomes. Specifically, the sharing headline was associated with more impressions than the incentive (P=.045) and empowerment (P=.004) headlines; more reactions than the altruism (P=.004) and empowerment (P=.014) headlines; and higher survey participation than the social norms headline (P<.001). In addition, the social norms headline was associated with higher click-through rates than the incentive (P<.001), altruism (P<.001), and empowerment P=.001) headlines.

Two of the 15 ad versions generated nearly half (285/587, 49%) of the participants (Table 2). These were the ads containing the image of soldiers marching with either the incentive or sharing headline. These two ad versions had significantly higher impressions (P<.001) and CTR (P<.001) than the other 13. Consequently, these two ads were most cost effective, at US $4.88 and US $5.90 per participant, respectively. Results were similar when examining individuals who completed the online eligibility screener, regardless of whether they were eligible or completed the full survey.

Among veterans who completed the survey, the probability of suicidal ideation ranged from an estimated 15%-50% across the 15 ad variants, and the probability of not being enrolled in VA health care ranged from an estimated 18%-50% (Table 3). Recruitment of veterans with suicidal ideation was significantly higher for ads with the survey-taking image, as compared to the soldiers marching (P=.007) image. There were no statistically significant differences in recruitment of non-enrolled veterans by ad image or headline.

Our study demonstrated that Facebook ads are a potentially powerful tool to recruit research subjects. With the support of a single, half-time research assistant, we engaged veterans in enrollment in our online survey at a rapid clip (ie, nearly 100 participants per week). Though our click-through rate was similar to prior studies, our response rate was very high, which may have reflected ease of participation in this online survey. Average cost per participant was less than US $20, and our best-performing ads were dramatically cheaper, approximately US $5-6 per survey participant, a figure that compares very favorably with most prior studies [37,38]. Facebook ads were further disseminated through social sharing, as is illustrated by the “likes”, comments, and sharing of ads that we observed. This is a significant “externality” from a cost efficiency standpoint.

The feasibility of reaching and engaging younger veterans in research through this approach has important public health and clinical implications. We reached not only a relatively broad target population (ie, recent military veterans) but also were effective in engaging subpopulations that can be hard-to-reach and are of heightened interest. Being able to rapidly reach veterans who are experiencing current suicidal ideation and unengaged in VA health care—as we did—is a major challenge for the VA, health policy-makers and other stakeholders interested in improving veteran mental health outcomes.

It is worth emphasizing the high rate of detection of potentially serious psychiatric problems in this sample; we found high rates of screening positive for active PTSD (266/585, 45%), problematic drinking (243/584, 42%), major depression (164/586, 28%) and serious suicidal ideation (132/585, 23%). Facebook ads, together with other digital media advertising strategies that can support help-seeking (eg, Google AdWords) [39], may comprise critical tools in the design of effective campaigns for mental health treatment engagement. The major—and more imposing—next challenge is how to move individuals from endorsing their distress (online) to engaging in treatment in a health care or other therapeutic setting.

A critical novel component of this study was the use of an experiment, or as close to a “true experiment” as is possible in the Facebook advertising environment, to determine what ad features are most likely to result in engagement with the ad. One of the most intriguing novel findings here was that a headline encouraging users to share the ad resulted in better ad engagement. “Sharing” is a request that is uniquely suited to the social media milieu, and also appeals to a military ethos of helping peers. Results also varied depending on the level of engagement being measured and target population. What works as a “hook” regarding the generating clicks may not translate into more active participation, as was similarly found in a study reaching concerned partners of heavy drinking service members and veterans [19]. We found that for a more modest level of engagement (eg, impressions and clicks) with a broad spectrum of recent military veterans, using an image of soldiers or headlines containing a social norms message may be more effective. In contrast, if the goal is more proactive engagement (ie, survey completion) by individuals with active suicidality, an image of a person taking a survey may work better. One reason careful development and selection of image and text may be necessary to optimize ads for individuals with suicidal ideation could be related to cognitive differences in these individuals [40]. There is, for instance, an emerging set of empirical studies showing attentional biases toward certain words [41,42].

Our results should be considered in the context of several limitations. As participation in this study only involved a one-time online survey, it is not clear if the same strategies would be effective for treatment engagement [39], or engagement in research requiring a higher burden (eg, intervention or longitudinal cohort study). Also, it is possible that individuals perceived our ads in ways different than hypothesized (eg, the “survey-taking” image could have been perceived as that of “computer technology”). Military culture may also impact response to advertisements. For example, our military-related ad images favored the army, which may have contributed a higher representation of them in the sample. Future research focused on testing the effectiveness of online ads should consider a qualitative component to gain more insight into differential ad performance. If future studies can confirm and further identify ad features that result in more response and engagement by veterans with suicidal ideation, there is significant potential for targeted interventions or campaigns to enhance outreach, health messaging, help-seeking, or other behaviors.

Taken together, our study demonstrates that Facebook ads are an effective medium for rapidly identifying, reaching, and recruiting recent military veterans, and can particularly help in reaching individuals who screen positive for current mental health problems. These results provide a foundation to inform efforts to engage veterans disconnected from the health care system or at risk for suicidal ideation.