StayWell is a fully remote trial and has various designs: (1) a pre-post comparison, in which we assessed depression and anxiety symptoms for all patients before and after the intervention and (2) a randomized controlled trial with two groups: Uniform Random (UR) messaging and a Reinforcement Learning (RL) messaging. Owing to a coding error in the algorithm, all participants received messages randomly (UR condition). Therefore, we altered the study’s main aims by focusing on the pre-post effects of participating in StayWell on depression and anxiety symptoms. The institutional review board of the University of California (UC) Berkeley reviewed and approved all study procedures.

Participants were enrolled in the StayWell trial using the HealthySMS platform—an automated text messaging platform developed by the authors [20,24]. HealthySMS has been successfully used with various low-income English- and Spanish-speaking populations to send automated text messages and manage participant responses using a secure, Health Insurance Portability and Accountability Act–compliant platform [20]. Participants received 2 messages daily for 60 days: 1 skills-based message and 1 message inquiring about their mood. The skills-based messages included tips on how to deal with worry and stress brought on by the COVID-19 pandemic. Half of these messages were based on BA strategies, and half were based on skill-based strategies. We developed the messages to highlight evidence-based practices used in depression and anxiety interventions that promote behavior change. Templates for these messages were developed from previous work conducted by authors SMD and AA [16]. The study team edited the original messages to fit the COVID-19 context and improve readability. The team also translated and culturally adapted messages in Spanish to expand the reach of the intervention. Messages were sent daily at a random time between 9 AM and 6 PM. Three hours following the delivery of the skills-based messages, participants were sent a message inquiring about their mood on a scale of 1-9, with 9 being the best mood.

Participants received messages uniform randomly (ie, a microrandomized trial design [16]), where every day during the study treatment allocation was characterized by a full factorial design with two factors: skills-based messages (M) and the time frame (T) when the message was sent. M has 2 levels (BA and skill-based), and T has 3 levels (9 AM-12 PM, 12 PM-3 PM, and 3 PM-6 PM). Participants received 1 daily skills-based message and 1 daily mood message that did not vary.

Adult Spanish and English speakers aged 18 years and older who had a mobile phone were recruited via web-based media advertisements on Facebook, Craigslist, and university websites (UC Berkeley and UC San Francisco) to participate. Data were collected through a web-based Qualtrics survey and HealthySMS. Participants were excluded if they used a web-based text messaging app or were outside of the United States. Web-based text messaging apps are more prone to scams on the internet and facilitate the creation of multiple phone numbers for 1 participant. The study lasted from April to December 2020. The CONSORT (Consolidated Standards for Reporting Trials) flow diagram for data collection is shown in Figure 1.

Our primary outcomes, including the 8-item Patient Health Questionnaire 8 [26] and the 2-item Generalized Anxiety Disorder 2 [27] (GAD-2) scale, were collected through a Qualtrics survey at pre- and postintervention.

Secondarily, we were interested in assessing engagement in the intervention by measuring response rates to mood rating messages and calculating how many participants stopped the text messaging.

We will conduct a pre-post comparison among all participants. The depression score measured using the 8-item Patient Health Questionnaire (PHQ-8) and the anxiety score measured using the 2-item Generalized Anxiety Disorder (GAD-2) scale will be improved over the 60-day study.

The sample size calculation was performed in a previous protocol paper [23], which includes 2 aims. This paper only considers the primary aim. At a medium standardized effect size (ie, Cohen d=0.5), a sample size of 64 participants is required to detect an improvement of either the depression or anxiety score from baseline to 60-day at 80% power and 5% level of significance. The sample size of this study was 193 participants, which is based on the secondary aim. The secondary outcome focuses on the proximal effect of daily improvement on the mood rating. However, the analysis will be presented in a separate manuscript.

A total of 303 people entered the study and received text messages. Of these, 12 were recruited via ClinicalTrials.gov, 75 via Craigslist, 184 via Facebook, and 32 by texting the StayWell phone number. We show the distribution of baseline characteristics in Table 1. Most baseline respondents were female (76.0%) and spoke English (88.4%). While almost half of the respondents identified as White or Caucasian (47.9%), the sample was relatively diverse with 20.5% Latinx, 13.2% Asian or Pacific Islander, 11.5% multiethnic, and 6.6% Black or African American participants. Of the baseline participants, 193 also completed an exit questionnaire and were included in the main analysis.

^bGAD-2: 2-item Generalized Anxiety Disorder.

Table 2 displays the raw scores and the distributions of change in depression (PHQ-8) and anxiety (GAD-2) scores at 60 days from baseline for respondents who completed the baseline and exit surveys, respectively. The data in Table 2 indicate that average PHQ-8 and GAD-2 scores decreased significantly from baseline to the end of the study, suggesting improvements in depression and anxiety symptoms. There was a reduction in the mean PHQ-8 and GAD-2 scores of 18.5% and 17.2%, respectively, at 60 days compared to the baseline scores. The normality assumption of each score change is valid.

To evaluate the generalizability of our data in terms of anxiety and depression prevalence and symptoms in our baseline samples, we compared the clinical parameters between participants who only responded to the baseline survey versus those who responded to both the baseline and 60-day surveys (Tables 3 and 4). Likely major depressive disorder and likely GAD were assessed using cutoff scores of ≥10 on the PHQ-8 and ≥3 on the GAD-2, respectively. There was no significant difference (for all, P>.05) in clinical parameters between people who only responded to the baseline survey and those who responded to the baseline and 60-day assessment. This suggests that the mental health burden was similar between our study sample and individuals who did not complete the 60-day survey.

To determine whether the changes in PHQ-8 and GAD-2 scores remain constant after accounting for participants’ engagement in the intervention, we used a simple linear regression model (Table 5) with participants’ fraction of mood messages answered (ie, response rates) as the main predictor. The outcome was the change in GAD-2 and PHQ-8 scores at 60 days from the baseline; thus, a negative coefficient indicates a greater improvement (larger decrease in scores) in anxiety and depressive symptoms than the reference group. We found that even when accounting for engagement in StayWell, the average improvements in both PHQ-8 and GAD-2 scores remain significant. The average improvements in PHQ-8 and GAD-2 scores controlling for engagement are –2.7 points (P=.001) and –0.78 points (P=.01), respectively.

To assess the influence of other factors (COVID-19 infection rates, self-rated health, and other demographic variables) on GAD-2 and PHQ-8 score improvements, we conducted a post hoc exploratory analysis (Table 6). This analysis also adjusts for response rates. Eight individuals lacked a valid zip code and were excluded from the analysis. Compared to females, the change in PHQ-8 score at 60 days from baseline was 2.4 points larger (P=.01) among males, adjusting for all other covariates; this suggests that males experienced relatively lesser improvement in depression symptoms. Having very good self-rated health was associated with less improvement in anxiety symptoms (increase of 0.83 in the GAD-2 score, P=.04) at 60 days from baseline compared to those with poor health and adjusting for covariates.

As a sensitivity analysis, we conducted 2 separate 1-way analysis of variance to assess the differences in the average change in PHQ-8 scores between genders and explore the association between self-rated health and the average change in GAD-2 scores. Our results show that gender had a significant effect on the change in PHQ-8 scores at 60 days from baseline (F_2,190=4.106; P=.02). This suggests that there are true differences in the average improvement in PHQ-8 scores among male- and female-identifying participants. However, the mean change in GAD-2 scores did not differ between self-rated health categories (F_3,189=2.954; P=.37). Thus, we cannot conclude that there are differences in the average improvement in GAD-2 scores by self-rated health.

The results in Table 2 show that both the depression and anxiety scores decreased over the 60-day period. Table 5 shows that the changes in both scores remained significant, adjusting for the response rate, as determined using a simple linear regression model (PHQ-8, P=.01 and GAD-2, P=.50). We also found that participants’ response rates are not significant predictors of improved PHQ-8 and GAD-2 scores (Table 5 and Table 6). In Table 6, we adjusted for the response rate and other demographic and clinical variables using a multivariable linear regression model, and we observed that the change in depression scores remained significant, but the change in anxiety score was not significant. However, we had previously modeled the change in PHQ-8 and GAD-2 scores by only adjusting for demographic and clinical variables, and the change in GAD-2 scores was also not significant (Multimedia Appendix 1). Therefore, the nonsignificant results for the change in GAD-2 scores are not necessarily attributed to the response rates.

Participants answered the mood text messages on average 60.0% of the time (ranging from 0% to 100%). Furthermore, 21 people did not respond to any mood-related messages, and 13 participants opted out of text messaging by texting “STOP” to our system. The 303 baseline participants were in the study for a range of 2 to 72 days (mean 59 days). Seventy participants went beyond the 60-day time frame owing to a system glitch; these participants were in the study for an average number of 63 days.

Participants who received the StayWell text messaging program showed improved depression and anxiety symptoms at completion of the program (60 days) on average. These results are similar to those of previous studies utilizing text messaging as a public mental health intervention to counteract the deleterious emotional and mental health effects of the COVID-19 pandemic in Canada [12-15]. Additionally, engagement with our texting study (2 messages per day) was relatively high. Response rates averaged to 60% in the daily mood check-in, and only 4% of participants opted out of the text messages during the study. This study supports the use of text messaging as a broad-based tool for improving mental health, especially in the context of a global pandemic when in-person behavioral health visits are inaccessible.

It was important to assess whether other factors (local COVID-19 infection rates, self-rated health, and other demographic variables) influenced the positive outcomes; however, we found that improvements in GAD-2 scores were not related to other measured variables. We found a greater improvement in PHQ-8 scores for female-identifying participants, but improvements held despite local infection rates or other demographic factors. These findings suggest that women may experience greater benefits in their depression symptoms from participating in the StayWell program. Nonetheless, it is particularly notable that weekly local infection rates were not related to changes in outcomes. For example, it is possible that decreases in symptoms could be influenced by reductions in local infection rates and accompanying lowered concerns of infection or reduction in policies such as stay-at-home orders. However, improvements in PHQ-8 scores persisted after accounting for changes in weekly infection rates and other covariates, suggesting that the intervention improved symptoms beyond the influence of any change in the severity of the COVID-19 pandemic in participants’ area of residence.

Over a third (36.3%) of participants did not complete the final assessment, and we took steps to assess how noncompletion impacted our results. First, we assessed whether reductions in depression and anxiety symptoms remained after controlling for participants’ response rates, and our findings suggest that the intervention may be beneficial overall even if participants do not respond. It is possible that reading messages can be beneficial even if participants do not respond to subsequent mood ratings. On average, participants who completed the study had a significantly higher response rate to mood rating messages (70%, 42/60 responses) than those who dropped out (35%, 21/60 responses) (P<.001). Despite no significant differences in baseline mood ratings, PHQ-8, or GAD-2 scores between completers and noncompleters, those who completed the final assessment had increased daily mood ratings (0.0037; P<.001), whereas those who did not complete the final assessment reported decreases in mood ratings (difference of 0.0041 from 0.0078 to 0.0037; P<.001). Therefore, it is possible that people with worsening symptoms respond less often or that less engagement is related with worse outcomes. It is possible that the study outcomes are biased on the basis of the mood rating post hoc analyses (attrition bias); however, this is in contrast with significant differences in outcomes remaining after controlling for the response rate. Ideally, we would be able to assess any differences in primary outcomes (PHQ-8 and GAD-2 scores), but we do not have those data for participants who dropped out of the study. This provides a mixed picture but suggests that it is more likely that participants experiencing worsening symptoms drop out of the study more often.

As stated in the trial protocol, an additional aim of this study was to test whether a reinforcement learning algorithm could improve personalization and outcomes beyond randomly selecting messages within different categories (behavioral activation and coping skills). Considering technical difficulties, we could not randomize participants into the distinct conditions, and all participants received the same intervention (random message condition). To prevent technical errors, future studies using RL should incorporate pilot work before commencing the trial to check for errors; however, we were unable to do so because we needed a quick roll-out during the pandemic. Further studies are needed to compare the effectiveness of personalizing messages using a reinforcement learning algorithm to that of a random message condition. Other studies in progress may be better suited to answer questions related to the impact of reinforcement learning models’ utility for improving personalization [23].

This study had a racially/ethnically diverse sample in the United States, which included 11.3% of Spanish speakers. On average, participants entered the study with mild/moderate symptoms of depression, which improved to mild symptoms at the end of the intervention period. While reductions in PHQ-8 and GAD-2 were not large, the low-intensity intervention approach is highly scalable and can accommodate a large number of people. This study also shows that participants are open to receiving 2 messages per day, including one mood response, while maintaining a high response rate (60%). Lastly, the longitudinal data collected provides opportunities for further inquiry into daily mood ratings and response rates.

The main limitation of this study is the lack of a control group with no or an inactive intervention. Given the impacts of the COVID-19 pandemic on mental health, the study team felt that it was unethical to withhold mental health support at this time. In addition, we were interested in assessing whether the intervention might be improved by applying a reinforcement learning algorithm to personalize messages, but technical errors prevented us from examining this. Lastly, a significant portion of the sample did not complete the final assessment, which may result in attrition bias, and further exploration of the engagement-outcome relationship is merited.

Our study provides support for low-intensity text messaging interventions to improve mental health at the population level. Our data show the feasibility of sending 2 messages a day and asking for daily mood responses. Texting and other mobile interventions could serve as ways to identify individuals in need of more intensive intervention on the basis of reported daily mood ratings. Future studies should continue to assess the impacts of text messaging and related mobile health interventions for mental health and continue to assess methods including machine learning to improve personalization.

Participation in a CBT-based text messaging program focused on COVID-19 cognitive flexibility, and behavioral activation and acceptance were related to improved depression and anxiety symptoms. In addition to improvements in outcomes, this study reported high levels of engagement during a 60-day intervention that sent 2 messages per day. Text messaging interventions could serve as an important public health tool to disseminate strategies for managing mental health.