Personalization Strategies for Increasing Engagement With Digital Mental Health Resources: Sequential Multiple Assignment Randomized Trial

Introduction

In 2023, 53.9% of the 58.7 million United States adults living with a mental illness received mental health treatment in the past year [1]. Self-guided mental health websites may be especially helpful for those who face structural barriers (eg, time, geographic constraints, and cost) and individual barriers (eg, stigma, low perceived need, and lack of readiness) to treatment [1-3]. Mental health websites bypass many of these obstacles by maintaining user privacy and offering accessible and free resources; they may also be preferred by some over face-to-face care [4]. For instance, the Mental Health America (MHA) [5] website annually attracts approximately 10 million visitors and provides free screening tools and resources, such as psychoeducational articles, therapist directories, and self-help tools. These resources are designed to support individuals at any stage of their mental health journey, ranging from those exploring mental health topics for the first time to those actively seeking treatment.

Helping individuals find and use quality web-based mental health resources has positive implications for future treatment decisions and stabilization [6,7]. However, visitors to mental health websites, including MHA’s, typically engage for only seconds to minutes [8]. One qualitative study of young adults who completed screening tools on MHA found that most participants did not have plans for immediate action after completing a screening tool [9]. Given the underutilization of web-based mental health resources and their potential benefits, there is a critical need to identify strategies that encourage visitors to access resources that can help improve their mental health.

Personalization may be an effective strategy for promoting engagement with mental health websites [10,11]. Research shows that tailored information sometimes influences health behaviors more effectively than nontailored information [12,13]. For example, one meta-analysis found that tailoring was associated with greater efficacy of web-based interventions in health-promoting behavior change compared with control [14]. However, whether personalization improves engagement with a real-world self-guided mental health website remains unclear.

Tailoring in this study was based on factors associated with engagement identified by previous research. In preparation for this study, our team conducted unpublished micro-randomized trials (MRTs) testing design elements that might optimize engagement from April 2022 to September 2023. These MRTs and previous research indicated different usage patterns by age group (eg, older adults tend to access resources related to treatment, younger adults are more likely to use mental health websites than older adults) [6]. In addition, our MRTs indicated that lesbian, gay, bisexual, transgender, and queer (LGBTQ+) individuals tended to have lower engagement rates than visitors who did not identify as LGBTQ+, despite evidence that mental health websites are vital sources of information for LGBTQ+ individuals [15]. One literature review found that tailoring content to LGBTQ+ identity was important for LGBTQ+ people who used digital mental health interventions [16].

Tailoring was also informed by the Health Action Process Approach (HAPA), a stage-based theoretical framework that proposes that health behaviors are adopted and maintained through a process of developing behavioral intentions (ie, the motivation phase) followed by planning to action (ie, the volition phase) [17,18]. Mechanisms that influence an individual’s progression through these stages include, for example, outcome expectancies, perceived risks, and perceived need [17-19]. Research indicates that perceived need may be associated with the formation of behavioral intentions to engage with eHealth interventions [19] and mental health websites [20]. The aim of tailoring to perceived need in the current study was to facilitate engagement by encouraging individuals with low perceived need to transition from preintention to intention and to provide positive reinforcement to those with a perceived need to take action to improve their mental health. In addition, the purpose of tailoring to visitors’ intended next steps on the website was to provide resources relevant to each individual’s stage of change. For instance, individuals who wanted to learn more about depression were provided with links to psychoeducational articles, and those who wanted to find a treatment provider received links to therapist directories and articles related to seeking treatment.

The goal of this study was to determine whether engagement with a mental health website could be improved by tailoring content to age, LGBTQ+ status, perceived need, and intended next steps via a sequential multiple assignment randomized trial (SMART) [21]. Before this study, visitors who completed the Patient Health Questionnaire-9 (PHQ-9) [22] were shown an optional demographics survey before the screening results page. However, acquiring information to inform tailoring necessitates the addition of tailoring questions, and previous research shows that longer surveys are associated with higher dropout [23]. For this study, we used the Next Steps survey, which included 2 tailoring questions before the items on the demographics survey, to assess whether the addition of tailoring questions hindered engagement. Submitting the demographics or Next Steps survey took visitors to the screening results page, which showed nontailored or tailored messages and featured resources (ie, prominently displayed links to resources on MHA). Additional persistent general resources were displayed underneath featured resources on the screening results page to all participants independent of condition. This study addressed the following aims:

1. Does adding tailoring questions negatively impact engagement?
2. Does personalization to demographics improve engagement?
3. Does tailoring to perceived need, intended next steps, or a combination of both improve engagement?
4. Among the strategies examined in this study, which is the optimal tailoring strategy?

These findings will inform the implementation of tailoring on the MHA website, offer novel insights on how to optimally deliver information to people seeking web-based mental health information, and provide empirical evidence on the effectiveness of tailoring on a large, real-world self-guided mental health website.

Methods

Overview

A session was defined as activity on the MHA website from a unique IP address bounded by 30 minutes of inactivity. Visitors with ≥30 minutes of inactivity were considered to have disengaged, and only the first session of each unique IP address was used for analyses. Data were collected from 258,383 sessions from December 1, 2023 to February 6, 2024 and from April 1, 2024 to April 30, 2024 on MHA. Data from February 7, 2024 to March 31, 2024 were excluded due to minor programming errors that were subsequently resolved by MHA.

Participants

Participants were recruited on MHA through voluntary response sampling. Inclusion criteria were residing in the United States and viewing a postscreening survey after completing the PHQ-9 (n=169,647). See Multimedia Appendix 1 for MHA’s data cleaning procedures.

Procedure

Overview

This study used a SMART design. Participants could progress through 3 web pages displaying: (1) the PHQ-9, (2) a postscreening survey, and (3) their screening results. Participants began on the PHQ-9 web page (see Figure 1), which did not vary by condition. Participants were randomized via simple randomization to 1 of 2 Level 1 conditions (ie, demographics survey or Next Steps survey) upon clicking the PHQ-9 web page, on which a link to the appropriate postscreening survey was generated. Upon completing the PHQ-9 and clicking the “Next” button, participants were taken to the postscreening survey web page, which displayed either the demographics survey or Next Steps survey (see Figure 2 for a sample demographics survey web page).

Postscreening survey items were optional. Participants who clicked the “View Results” button at the bottom of the postscreening survey progressed to the screening results page and were randomized to 1 of 5 Level 2 conditions. Participants who submitted the demographics survey were randomized to 1 of 2 possible conditions: (1) nontailored or (2) tailored resources to demographics (TR-D). Participants who submitted the Next Steps survey were randomized to 1 of 4 conditions: (1) nontailored, (2) tailored message to perceived need (TM-PN), (3) tailored resources to intended next steps (TR-INS), or (4) tailored message to perceived need + tailored resources to intended next steps (TM-PN+TR-INS).

The following descriptions of Level 2 conditions provide examples of messages and resources shown in each condition. For a complete list of messages and featured resources, see Multimedia Appendix 2.

‎

‎

Nontailored Condition

Participants from either Level 1 condition could be randomized to the nontailored condition on the screening results page. The nontailored condition included nontailored messages and nontailored featured resources. A nontailored message showed visitors a brief description of their PHQ-9 results, which was shown across all conditions. For example, participants with a PHQ-9 score from 20‐27 were shown, “Based on your responses, you may have symptoms of severe depression. This result is not a diagnosis. A doctor or therapist can help you get a diagnosis and/or treatment.” Nontailored featured resources comprised 4 featured resources (ie, links to resources on MHA that were prominently displayed in large, colorful bubbles across all conditions), shown in random order, that were selected by MHA’s algorithm as the most popular resources accessed by visitors who complete the PHQ-9.

Tailored Resources to Demographics (TR-D) Condition

Participants who completed the demographics survey could be randomized to the TR-D condition, which showed participants a nontailored message (ie, brief description of PHQ-9 results) and featured resources tailored to age and LGBTQ+ status. A pool of 3 to 4 possible resources for each age group was chosen by an algorithm that identified the most popular resources within each age group. Participants who provided their age and did not self-identify as LGBTQ+ were shown all possible resources for their age group. For example, a participant between 25‐45 years old who did not self-identify as LGBTQ+ was shown the following links in random order: “I can’t do anything right—depression,” “I feel numb,” “I can’t get out of bed,” and “Does depression go away on its own?”

Tailored resources to LGBTQ+ status comprised resources on MHA that cater to the LGBTQ+ population. Participants who self-identified as LGBTQ+ and did not provide age were shown all 4 possible LGBTQ+ resources: “I’m bullied because I’m LGBTQ+/queer,” “Q-chat space,” “Prepare for difficult conversations,” and “How do I find LGBTQ-friendly therapy?” LGBTQ+ participants who also provided age were shown 2 age-tailored resources and 2 LGBTQ+ resources; the 2 LGBTQ+ resources were selected from a pool of 4 possible resources for participants under 18 years or 3 possible resources for those 18 years and older. See Figure 3 for an example of a screening results page for a participant randomized to the TR-D condition who was between 45‐54 years old and self-identified as LGBTQ+.

‎

Tailored Message to Perceived Need (TM-PN) Condition

The TM-PN condition was 1 of 4 conditions available to participants who completed the Next Steps survey. Tailored messages were based on participants’ responses to the perceived need item, followed by text based on their PHQ-9 results. The TM-PN condition displayed nontailored featured resources. See Figure 4 for an example of a screening results page for a participant randomized to the TM-PN condition who endorsed “Yes” to perceived need.

‎

Tailored Resources to Intended Next Steps (TR-INS) Condition

Participants who completed the Next Steps survey could also be randomized to the TR-INS condition, which included a nontailored message and 4 featured resources tailored to participants’ responses to the intended next steps item. For example, a participant who selected “Understand what depression is like (reading articles)” was shown “What is depression really like?,” “Am I depressed or just sad?,” “Are there types of depression?,” and “Am I broken?” in random order.

Tailored Message to Perceived Need + Tailored Resources to Intended Next Steps (TM-PN+TR-INS) Condition

The TM-PN+TR-INS condition was available to participants who completed the Next Steps survey. The TM-PN+TR-INS condition included a tailored message to perceived need and 4 featured resources tailored to participants’ intended next steps. See Figure 5 for an example of a screening results page for a participant randomized to the TM-PN+TR-INS condition who endorsed “No” to perceived need and “Find a treatment provider near you” as their intended next steps.

‎

Persistent General Resources

Persistent general resources included links to resources related to depression. Persistent general resources were shown underneath featured resources on the screening results page to all participants, independent of condition (see Figure 6).

‎

Programming Errors

Two minor programming and execution errors occurred. All conditions were intended to receive 4 resources, but only 3 were shown to participants in the 45+ years age group in the TR-D condition, unless they also indicated being LGBTQ+. However, this only affected a small portion of the sample. People who reported that they lived in Minnesota (2.1% of the 77% of participants who provided their state of residence) were shown a Minnesota-specific resource in place of one of the featured resources.

Measures

Demographics Survey

Participants were invited to provide their age range, gender, race and ethnicity, and household income. They were also asked to select applicable descriptors among veteran or active-duty military, caregiver of someone living with emotional or physical illness, LGBTQ+, student, trauma survivor, new or expecting parent, and healthcare worker.

Next Steps Survey

The Next Steps survey included 2 tailoring questions shown before the demographics survey items. The first question asked about participants’ intended next steps: “What is the main thing you want to do after taking this mental health test? (Select one).” Response options included, but were not limited to, “Take another mental health test,” “Understand what depression is like (reading articles),” and “Understand and manage self-harm or suicidal thoughts.” The second tailoring item assessed perceived need (“Do you feel like you need to do something to improve your mental health?”) with options of “Yes,” “No,” and “I don’t know.”

Engagement

Our primary engagement outcome was disengagement, operationalized as leaving the website or having ≥30 minutes of inactivity (0=engaged, 1=disengaged). We also measured engagement on the screening results page by whether participants clicked a featured or persistent general resource (0=clicked a persistent general resource, 1=clicked a featured resource).

Analyses

Impact of Adding Tailoring Questions on Engagement

Data were analyzed in R Version 4.3.3 (R Core Team) [24], and all research questions were examined via logistic regressions. We assessed whether Level 1 condition (ie, demographics survey or Next Steps survey) was a significant predictor of disengaging from the postscreening survey to examine the costs, if any, of adding tailoring questions. For the remaining analyses, we used 2-part models to first examine the odds of disengaging from the screening results page by Level 2 condition. We then examined the odds of clicking a featured versus a persistent general resource among participants who chose to engage on the screening results page. Statistical significance was not impacted after corrections for multiple comparisons using the Benjamini-Hochberg procedure with a false discovery rate of 0.05 [25]. We report odds ratios (OR) with 95% CIs and P values.

Impact of Tailoring on Engagement

Among participants who submitted the demographics survey (n=38,490), we examined whether the odds of disengaging (versus engaging) and clicking a featured resource (versus clicking a persistent general resource) on the screening results page significantly differed for participants in the TR-D condition compared with the nontailored condition. Among participants who submitted the Next Steps survey (n=34,204), we tested whether engagement significantly differed for participants in the TM-PN, TR-INS, and TM-PN+TR-INS conditions compared with the nontailored condition.

Identifying the Optimal Tailoring Strategy in This Study

We aimed to identify the optimal tailoring strategy among those tested in this study by comparing engagement across all possible conditions on the screening results page, with TR-D set as the reference group. Conditions included nontailored (including participants from both the demographics survey and Next Steps survey conditions), TR-D, TM-PN, TR-INS, and TM-PN+TR-INS.

Ethical Considerations

This study was approved by the University of Washington Institutional Review Board (STUDY00010958). The Institutional Review Board and the study’s external Data Safety and Monitoring Board concluded that there were no foreseeable risks associated with participation in the study, and further application for ethical approval was not necessary. MHA visitors engaged with the website naturalistically, and the website’s terms and conditions include provisions around the use of user data for research. Thus, the Institutional Review Board waived informed consent, and participants were not compensated for their involvement in the study. Data were protected under the University of Washington Institutional Review Board and were deidentified via random identifiers before data transfers between MHA and the University of Washington in accordance with a Data Security Protocol. The deidentified data were stored on a secure encrypted institutional server with access restricted to approved research personnel only.

Results

Sample Characteristics

Among participants who clicked the PHQ-9 (n=258,383), 131,275/258,383 (50.8%) were randomized to the demographics survey, and 127,108/258,383 (49.2%) were randomized to the Next Steps survey. See Figure 7 for the study flow diagram. Among participants who viewed a postscreening survey (n=169,647), 87,712/169,647 (51.7%) participants were allocated to the demographics survey condition, and 81,935/169,647 (48.3%) were allocated to the Next Steps survey condition. Most participants who submitted a postscreening survey (n=72,694) had moderately severe or severe depression (mean 15.9, SD 6.1). Among participants who also provided demographic data, most were White (n=37,311/69,385, 53.8%); Hispanic or Latino (n=10,218/69,385, 14.7%); Asian (n=6647/69,385, 9.6%); Black or African American (n=6532/69,385, 9.4%); women (n=43,517/71,963, 60.5%); and between 18‐24 years old (n=15,604/71,529, 21.8%; see Table 1). Among participants who responded to the intended next steps tailoring question (n=33,002), the most common responses were “I don’t want to do anything” (n=12,034, 36.5%) followed by “Take another mental health test” (n=4740, 14.4%). Among participants who responded to the perceived need item (n=33,352), 60.9% (n=20,325) of participants endorsed “Yes” to perceived need (see Table 2). Among participants randomized to a Level 2 condition (n=72,694), disengagement rates on the screening results page ranged from 44.3% to 47.5% by condition. Among those who engaged on the screening results page (n=39,137), rates of clicking a featured resource ranged from 26% to 42.2%, and rates of clicking a persistent general resource ranged from 57.8% to 74% by condition.

‎

Small Cost to Engagement by Adding Tailoring Questions

Table 3 displays logistic regression model estimates. Compared with the demographics survey condition, 625 more participants in the Next Steps survey condition disengaged from the postscreening survey web page. Disengagement rates were 25% for participants in the demographics survey condition and 27.5% for participants in the Next Steps survey condition. Participants who viewed the Next Steps survey had a 14% increase in the odds of disengaging from the postscreening survey (odds ratio [OR] 1.14, 95% CI 1.11‐1.16; P<.001) compared with participants who viewed the demographics survey.

TR-D Reduced Disengagement and Increased Clicks on Featured Resources

Among participants who viewed the screening results page after submitting the demographics survey, those who received resources tailored to demographics had a 10% decrease in the odds of disengaging (OR 0.90, 95% CI 0.87‐0.94; P<.001) compared with participants in the nontailored condition. Participants in the TR-D condition had a 90% increase in the odds of clicking a featured resource versus a persistent general resource (OR 1.90, 95% CI 1.79‐2.01; P<.001) compared with participants in the nontailored condition.

TR-INS and TM-PN+TR-INS Increased Clicks on Featured Resources

Among participants who viewed the screening results page after submitting the Next Steps survey, there were no significant differences in the odds of disengaging for participants in the TM-PN (P=.33), TR-INS (P=.51), and TM-PN+TR-INS (P=.52) conditions compared with the nontailored condition. However, there were statistically significant differences in the odds of clicking a featured resource versus a persistent general resource among those who engaged. Compared with the nontailored condition, those in the TR-INS condition had a 25% increase in the odds of clicking a featured resource (OR 1.25, 95% CI 1.14‐1.37; P<.001), and those in the TM-PN+TR-INS condition had a 34% increase in the odds of clicking a featured resource (OR 1.34, 95% CI 1.23‐1.47; P<.001). There was no significant difference in the odds of clicking a featured resource between the nontailored and TM-PN conditions (P=.11).

Tailoring to Demographics Was the Optimal Tailoring Strategy in This Study

Tailoring resources to demographics yielded the best engagement outcomes compared with the remaining Level 2 conditions. Compared with participants shown resources tailored to demographics, those in the remaining conditions had 8%-13% greater odds of disengaging (P≤.004) and, among participants who engaged, had 30%-52% fewer odds of clicking a featured resource versus a persistent general resource on the screening results page (P<.001).

Discussion

Principal Findings

This study examined whether personalizing content increased engagement with a self-guided mental health website. There was a small but statistically significant cost to engagement from adding tailoring questions assessing perceived need and intended next steps, and tailoring resources to demographics was the most effective strategy used in this study. Appealing to perceived need alone via tailored messages, as chosen in this study based on HAPA theory, did not improve engagement. We speculate that this may be the case because although perceived need may be associated with intent to take action, intent is not necessarily associated with taking health actions (ie, the intention-behavior gap) [26]. Individuals with a perceived need to act may not be ready to do so immediately after viewing their screening results, and it is possible that increased cognitive load due to the additional text of tailored messages discouraged further engagement [27]. However, we also observed that while tailored resources to intended next steps increased the odds of clicking a featured resource by 25%, combining a tailored message to perceived need with tailored resources to intended next steps further increased those odds to 34%. It is possible that there was an interaction effect, such that tailored messages to perceived need may have facilitated engagement when individuals were also provided with resources relevant to their needs. Moreover, the HAPA has been used to describe behaviors that change over periods of days to weeks [18]. Whether the HAPA or other frameworks of health behavior change apply to long-term engagement with mental health websites remains unknown.

Design Implications

Our findings have important generalizable design implications for implementing tailoring on websites to enhance engagement:

1. Defining engagement: we used indicators for disengagement and clicks on web-based resources to assess the impact of tailoring on early engagement. It is important that the operationalization of engagement aligns with the context of each study. For example, research questions related to long-term engagement may be better addressed by engagement outcomes such as total number of clicks, length of time spent on each web page, and whether visitors return to the website.
2. Selecting tailoring variables: tailoring variables may include, but are not limited to, any number of demographics, screening results, behavioral characteristics, and theoretical constructs of behavior change. Our preliminary MRTs identified variables that may be related to engagement specific to the MHA website. However, it is likely that engagement with websites that offer different types of resources or that are targeted toward specific populations is associated with variables not used for tailoring in this study. We recommend conducting pilot studies to identify potential tailoring variables related to engagement unique to each website. Researchers are also advised to consider the production costs of tailoring (eg, whether existing materials can be used or if new materials and design elements will need to be developed).
3. Acceptability of tailoring: researchers must also consider the acceptability of tailoring to demographics without suggesting that those characteristics are causal factors of visitors’ mental health problems [28]. For example, while tailoring to demographics was an effective strategy in this study, tailoring solely to LGBTQ+ status may have an inadvertent stigmatizing effect by implying to visitors that their depressive symptoms stem from their LGBTQ+ identity. Alternatively, researchers may ask participants about what they believe contributes to their mental health issues and tailor content accordingly.
4. Balancing costs and benefits: it is critical to identify the appropriate threshold for the number of tailoring questions to add to a website such that the costs of tailoring (ie, in this study, disengagement due to the addition of tailoring questions) do not outweigh the benefits. However, what constitutes costs and benefits depends on the goals of each website. For example, in some contexts, research questions may be better answered by retaining visitors who are willing to invest time and energy into answering many tailoring questions.
5. Randomization procedures: in the current study, participants were allocated to a Level 1 condition through simple randomization on the PHQ-9 web page, on which a link to the appropriate postscreening survey was generated. In addition, MHA does not retain screening data from visitors who do not submit a postscreening survey. Allocations to Level 1 conditions among participants who viewed a postscreening survey were slightly imbalanced (51.7% versus 48.3%) due to when randomization was triggered. To ensure balanced allocation to experimental conditions, future studies may use a 1:1 allocation among participants who meet inclusion criteria while taking existing data saving procedures into account.

Strengths and Limitations

Our study was a large-scale randomized trial in an ecologically meaningful setting with a help-seeking population. Additional strengths include the use of objective measures of engagement and our design implications for implementing tailoring on websites. The generalizability of our findings may be limited to self-guided mental health websites, as opposed to other digital mental health formats (eg, apps and interventions).

One limitation of the study was that content could only be tailored for participants who responded to tailoring questions. By necessity, we could not assess the impact of tailoring for individuals who did not provide tailoring data. In addition, both featured resources and persistent general resources may have varied in appeal regardless of visitors’ personal characteristics and preferences. Nevertheless, our approach used MHA’s existing resources to align with the research questions as closely as possible in a real-world setting.

Another limitation was the use of IP addresses to identify individual website visitors. This method may not reliably distinguish between responses from multiple individuals accessing the website from within the same household or from visitors connected to a shared public network. However, MHA’s internal data cleaning procedures exclude entries from IP addresses of known public spaces (eg, libraries and schools; see Multimedia Appendix 1). Our analyses retained only the first entry from each unique IP address, which may have excluded data from additional visitors accessing the website from the same IP address. Nonetheless, our approach was a practical means of tracking website activity at a large scale, minimizing duplicate entries, and preemptively excluding responses from potential bots before analyses.

Future Directions

This study focused on identifying strategies to enhance immediate engagement outcomes and was not designed to assess health behavior change over the long term. Tailoring to perceived need based on HAPA theory was effective in increasing the odds of clicking featured resources only when paired with tailored resources to participants’ intended next steps on the website. It is possible, however, that these decision processes are not well-suited for HAPA theory and that other guiding frameworks are needed for future research in this area. For example, other theoretical frameworks that have been used to tailor web-based interventions (eg, the theory of planned behavior [29], the transtheoretical model [30], and social cognitive theory [31]) may further guide research toward establishing optimal tailoring strategies on mental health websites. Future research may also test tailored messages that incorporate stronger calls to action intended to shift ambivalence (eg, via motivational interviewing language [32]). Given our evaluation of a restricted set of tailoring strategies, our findings provide an indication toward an optimal tailoring strategy for mental health websites. It is possible that other strategies not examined in this study, such as mode tailoring [33] or psychographic targeting [11], may also increase engagement in similar contexts. In addition, personal contexts and barriers can be assessed through open-ended questions to better understand contextual factors that may impact engagement with mental health websites, and long-term engagement patterns may be explored by identifying optimal times for re-engaging visitors when they are ready to access mental health resources.

Conclusion

Mental health websites have the potential to serve millions of people with mental health concerns, particularly those who face barriers to treatment, yet their resources are underutilized. This study addresses the critical need for empirical research on effective strategies for connecting mental health website visitors with digital mental health resources. Our findings provide evidence that personalizing resources to demographic characteristics can enhance engagement with mental health websites: in this study, visitors who were shown resources tailored to demographics had 10% fewer odds of disengaging and, among those who engaged, 90% greater odds of clicking a featured versus a persistent general resource. We provide design implications for future research, which may explore alternative frameworks of behavior change, test different tailoring variables and strategies, and evaluate engagement over repeated visits. Researchers are encouraged to continue to address these gaps to improve engagement with mental health websites and better support individuals seeking web-based mental health information.