Original Paper
Abstract
Background: Culturally relevant interventions are needed to help American Indian and Alaska Native (AI/AN) teenagers and young adults navigate common risky situations involving family and friends, including drug and alcohol misuse, dating violence, and suicidality. Leveraging We R Native, a multimedia health resource for Native teenagers and young adults, staff of the Northwest Portland Area Indian Health Board designed the BRAVE intervention for Native youth. The program is delivered via SMS text messaging and includes role model videos, mental wellness strategies, links to culturally relevant resources, and social support from family and friends.
Objective: We aim to conduct a randomized controlled trial of the BRAVE intervention among AI/AN teenagers and young adults (aged 15-24 years) to assess its impact on their physical, mental, and spiritual health; their resilience and self-esteem; and their coping and help-seeking skills.
Methods: From October to December 2019, we recruited 2334 AI/AN teenagers and young adults nationwide via social media channels and SMS text messages and enrolled 1044 participants. AI/AN teenagers and young adults enrolled in the study received either BRAVE SMS text messages, designed to improve mental health, help-seeking skills, and cultural resilience, or 8 weeks of science, technology, engineering, and math (STEM) SMS text messages, designed to elevate and reaffirm Native voices in STEM and medicine and then received the BRAVE SMS text messages. The impacts of the BRAVE intervention were tested using linear mixed-effect models and linear regressions.
Results: A total of 833 AI/AN teenagers and young adults were included in the analysis. Individuals in the BRAVE and STEM arms showed significant positive trends over the course of the study for all outcomes, except cultural identity and help-seeking behavior. Mean scores were significantly different for health (P<.001), resilience (P<.001), negative coping (P=.03), positive coping (P<.001), self-efficacy (P=.02), and self-esteem (P<.001). Changes in help-seeking self-efficacy were significant in those exhibiting risky behaviors at baseline to exit (P=.01). Those who reported positive coping scores at baseline also reported better health on average; however, no difference was found in risky drug and alcohol use (P<.001). The number of participants who used SMS text messages to help themselves increased from 69.1% (427/618) at 3 months to 76% (381/501; P<.001) at 8 months. Similarly, the number of participants who used SMS text messages to help friends or family members increased from 22.4% (138/616) at 3 months to 54.6% (272/498) at 8 months.
Conclusions: This is the first national randomized controlled trial of a mobile health intervention among AI/AN teenagers and young adults to test the efficacy of a mental wellness intervention in relation to STEM career messages. This study provides new insights for supporting the next generation of AI/AN changemakers.
Trial Registration: ClinicalTrials.gov NCT04979481; https://clinicaltrials.gov/ct2/show/NCT04979481
doi:10.2196/26158
Keywords
Introduction
Background
There are 1.6 million youth aged ≤18 years in the United States who self-identify as American Indian and Alaska Native (AI/AN) [
]. With 573 federally recognized tribes throughout the United States, AI/AN populations are culturally diverse with distinct languages, cultural practices, ceremonies, traditions, and histories. Despite their immense cultural resilience and pride, AI/AN teenagers and young adults are disproportionally affected by stress, depression, and suicide [ ]. As a population, AI/AN individuals experience substantial behavioral health needs that are compounded by limited access to mental health professionals and services [ ]. Differences in risk behavior and health outcomes among AI/AN teenagers and young adults have been linked to poor economic and social conditions and historical trauma stemming from colonization [ ]. Combined with chronic underfunding of the Indian Health Service, these conditions mean that many AI/AN teenagers and young adults do not have sufficient access to needed health services while living in medically underserved communities [ , ].Mobile Health Interventions
Mobile health (mHealth) technologies, including mobile devices, tablets, PDAs, and computers, are increasingly being used to address unmet health needs because they are cost-effective, developmentally responsive, and have the potential to reach large groups of AI/AN people living across the United States [
]. A recent analysis conducted by MarketCast found that AI/AN teenagers and young adults are open to social media about their mental health and coping strategies. From January to March 2020, participants mentioned mental health, stress, anxiety, and depression more than 1500 times on public social media channels [ ]. A recent article by Around Him et al [ ] underscored the relevance of social media outlets such as Twitter, Facebook, and Instagram. These platforms offer unique opportunities to address health disparities by building social connections, cultural advocacy, and supportive peer networks [ ]. In recent surveys assessing the accessibility and acceptability of technologies to promote health, 93% of AI/AN youth reported having regular access to a smartphone and accessing the internet from their phones daily and 38% reported spending an average of 3-4 hours on social media per day [ ].SMS text messaging interventions are another example of how mHealth technologies are being used to fill the gaps in existing health delivery systems to improve health behavior. Mobile phones are nearly ubiquitous across age groups, socioeconomic classes, and language preferences. A meta-analysis of SMS text messaging to improve diverse health outcomes showed that interventions with tailored and personalized messages have greater effects than interventions without unique messages. However, questions remain about the best strategies for designing message content, the impact of generic versus more tailored SMS text messages, the optimal message timing and dose, and whether unidirectional or bidirectional messaging is important [
- ]. This study is the first to leverage SMS text messaging to promote mental health among AI/AN teenagers and young adults.Designing Interventions for AI/AN Teenagers and Young Adults
Culturally tailored and user-centered interventions are critically needed to increase the degree by which health messages are perceived as personally relevant by AI/AN youth, thus inspiring and supporting behavior change [
, , ]. Although mHealth interventions (delivered via SMS text messaging and social media) have been used to improve multiple health outcomes [ ], the extent to which mHealth can promote mental health for AI/AN teenagers and young adults is not yet known.From 2015 to 2018, the Northwest Portland Area Indian Health Board (NPAIHB) carried out formative research involving youth as co-designers to create a behavioral intervention addressing alcohol misuse, intimate partner violence, and suicidality among AI/AN teenagers and young adults living across the United States. In 2015, we conducted key informant interviews with 10 Native young men to better understand their perceptions on alcohol misuse and the context of violence in their communities [
]. In 2016, we designed a series of theoretically informed, culturally relevant SMS text messages and a role model script to demonstrate and reinforce the skills described in the SMS text messages. In 2017, we pilot-tested the messages with Native youth and topical experts to assess the tone, content, and frequency of planned SMS text messages and video episodes.Grounded in the principles of inclusion, equity, belonging, and diversity, the NPAIHB confirmed that the intervention could reflect the rich cultural diversity of tribes and urban youth living throughout the United States, and multiple phases of youth-driven research informed the design of the BRAVE intervention, the first mHealth intervention delivered via SMS text messages that uses culturally relevant images, narrative role model videos, and help-seeking resources [
].This study has two aims: (1) to test the BRAVE intervention and (2) to test secondary associations, including self-efficacy, self-esteem, resilience, coping strategies, substance use, and cultural identity.
Research Partners
Overview
The BRAVE intervention was designed and evaluated over 5 years through a series of community-based participatory research activities led by the NPAIHB (
). The first phase was carried out in collaboration with the Harvard School of Public Health and focused on better understanding alcohol misuse and the context of violence among Native males between the ages of 18 and 24 years using key informant interviews.The second phase was carried out in collaboration with the NPAIHB’s Tribal Health: Reaching out InVolves Everyone project and We R Native. The NPAIHB is a regional, tribal nonprofit organization that represents 43 federally recognized tribes in Washington, Oregon, and Idaho. The Northwest Tribal Epidemiology Center is housed under NPAIHB and provides support through research, surveillance, and public health capacity building in partnership with the Northwest tribes. During the second phase of the study, the team designed and pilot-tested the intervention with AI/AN young men with a history of alcohol use and violence and 8 topical experts in alcohol prevention, alcohol treatment, violence prevention, health communication, and adolescent health.
During the efficacy phase of the study, the NPAIHB partnered with the mHealth Impact Lab at the Colorado School of Public Health. The lab works to facilitate the rapid and rigorous development, implementation, and evaluation of mobile and digital technology for health promotion and disease prevention to address inequalities in health outcomes. During this phase of the project, the NPAIHB recruited study participants and delivered SMS text messages, and mHealth led the design of data collection tools, data collection, and analysis. The partnership was supported by the Technology & Adolescent Mental Wellness (TAM) program, run by the Social Media and Adolescent Health Research Team and housed within the Department of Pediatrics at the University of Wisconsin-Madison.
We R Native
To reach AI/AN teenagers and young adults with culturally relevant health messages, the NPAIHB built We R Native—a holistic, multimedia health resource that reaches over 5000 viewers per day across its messaging channels. We R Native was designed using a youth-centered approach, in which youth were actively involved in selecting, writing, and designing their content. The service includes a website, an Ask Auntie Q&A service, an SMS text messaging service (text NATIVE to 97779), a YouTube channel, and social media accounts (ie, Facebook, Instagram, and Twitter). Given the widespread use of mobile phones by Native youth, mHealth interventions provide a promising tool to reach AI/AN youth who live across vast geographies in urban and rural communities.
We R Native regularly engages users to ensure that its tools and messages are relevant, timely, and culturally appropriate. In 2019, the team interviewed 13 AI/AN teenagers and young adults who regularly viewed We R Native channels (website, SMS text message, or social media). Participants shared ways in which We R Native messages had improved their mental health, cultural connectedness, sense of self-worth, and access to health resources, both for themselves and their loved ones. When asked to reflect on their own mental health concerns, participants mentioned grief and depression as the most common topics, followed by stress and mental wellness skills (eg, coping mechanisms).
These interviews also informed the final design of the BRAVE intervention, highlighting common coping strategies (ie, alcohol and drug misuse), preferred wellness strategies (ie, goal setting and self-care), help-seeking skills (ie, reaching out for help for themselves or a friend), and related protective factors, including cultural resilience, identity, and cultural pride [
]. Participants and key informants recommended expanding the inclusion criteria for BRAVE to females and expanding the age group to 15-24 years. Research is needed to test the efficacy of the BRAVE intervention using the expanded inclusion criteria.Study Design and Objectives
To evaluate its efficacy, we conducted a randomized controlled trial (RCT) of the BRAVE intervention among AI/AN teenagers and young adults (15-24 years old) living across the United States and assessed its impact on their physical, mental, and spiritual health; their resilience and self-esteem; and their coping and help-seeking skills. Participants were surveyed at four time points (
presents the primary and secondary hypotheses).The use of RCTs in AI/AN communities is often considered an inappropriate study design, given its culture of inclusion that values shared benefits [
]. The NPAIHB used an attention control design to ensure that all participants could receive the same benefits from the study. An alternative study design offers superior methodological rigor compared with a no-intervention control arm [ ]. An equivalent control arm was imperative to respect and honor the time and participation of the research participants. We selected science, technology, engineering, and math (STEM)–related SMS text messages because we felt that they would not interfere with the help-seeking skills demonstrated in the BRAVE arm and would be of equal value to participants and because we had access to STEM role model videos that could closely mirror the content and frequency in the study arm.All data collection methods were approved by the Portland Area Indian Health Service Institutional Review Board in Portland, OR (principal investigator: SCR, protocol no: 1384639). A waiver for parental consent was requested and approved by the Institutional Review Board. In addition, a consent form was included on the cover page of the pre- and postsurveys. All instruments and data collection methods were reviewed and approved by the Portland Area Indian Health Service Institutional Review Board before data collection.
Methods
Participant Eligibility and Recruitment
The efficacy study included self-identified AI/AN teenagers and young adults aged 15-24 years. All participants were required to have a cell phone with SMS text messaging capabilities. Eligibility was described via SMS text messages and confirmed using a presurvey.
From September to December 2019, the NPAIHB recruited AI/AN teenagers and young adults via We R Native social media channels (ie, Facebook, SMS text message, and Instagram) [
]. Additional recruitment took place through listservs associated with tribes, Indian health and Indian education organizations, and human service organizations that serve AI/AN young adults (Indian Health Service, Methamphetamine Suicide Prevention Initiative, Healthy Native Youth, etc). Interested teenagers and young adults were asked to text the keyword BRAVE to 97779, which triggered a series of eligibility and consent SMS text messages. Over 2330 AI/AN teenagers and young adults texted BRAVE to learn more about the study.Enrollment and Randomization
To enroll in the study, participants were required to complete a presurvey. Those who met the eligibility criteria were randomized into the study (n=1044) using a 1:1 allocation ratio. mHealth used Excel (Microsoft) software to randomize participants into each study arm. There were a total of eight waves of enrollment from October to December 2019. Data were collected using web-based questionnaires in Qualtrics (SAP) at baseline and at 3, 5, and 8 months. In appreciation of their time, participants received a US $10 Amazon gift code for each survey they completed, up to US $40 per person.
Intervention and Control Messages
AI/AN teenagers and young adults randomized into the intervention arm were immediately exposed to the BRAVE campaign, containing 8 weeks of SMS text messages with three messages per week. Participants in the control arm received messages designed to elevate and reaffirm Native voices in STEM and medicine. The STEM messages were delivered three times per week for 8 weeks. Both arms were designed to include a similar number of messages per week, with a combination of information, role model videos, images, and opportunities for reflection and engagement with the campaign (ie, reply for more information, provide a Q&A response, and click links to access resources). A paper by Stephens et al [
] described the theory that informed the design of campaign messages, as well as the sequence and content of the messages in greater detail.Data Collection
Data collection began on September 1, 2019. All four pre- and postsurveys were delivered using Qualtrics, an online data collection platform [
]. Questions collected information about standard demographic information, resilience measures, self-esteem, positive and negative coping skills, help-seeking self-efficacy, mental health, and cultural pride (see and for the complete BRAVE survey). Data from the surveys were collected, stored, and maintained by mHealth Impact Lab.The retention was high across the two study arms. For BRAVE enrollees, only 41 participants opted out during the intervention, and only 25 opted out at crossover. For STEM enrollees, only 45 participants opted out during the control phase, and only 18 opted out at crossover. The washout period between the study arms was 1 week. In total, 86 participants opted out of the study during the first arm, and 43 opted out after the crossover, resulting in an 87% retention rate, as shown in
.All participants were required to complete the baseline survey at the time of enrollment (presurvey). The presurvey was used to assess eligibility, randomize participants, and gauge baseline measures. Enrollees were asked to complete the same survey after the first set of messages (first postsurvey at 3 months). At the end of the intervention period, participants crossed over to receive the second set of messages and were asked to complete the survey a third time (second postsurvey at 5 months). The study team discontinued communication and asked participants to complete the final survey 90 days later (third postsurvey at 8 months), as shown in
.Survey dimension (number of items) | Sample question or statement and validated tool | Analysis | Answer choices |
Health (3) | Rate your physical health, YRBSa [ | ]Changes in perceived physical, mental, and spiritual health | Excellent, very good, good, fair |
Coping negative (2) | I’ve been using alcohol to make myself feel better, Youth Coping Responses Inventory [ | ]Changes in problem drinking or drug use | Strongly agree, agree, disagree, strongly disagree, don’t know or not sure, don’t want to answer |
Coping positive (5) | I’ve been taking action to try to make the unpleasant situation in my life better, Youth Coping Responses Inventory [ | ]Proportion of respondents who report high scores of coping skills | Strongly agree, agree, disagree, strongly disagree, don’t know or not sure, don’t want to answer |
Resilience (9) | I try to finish what I start, Child Youth Resilience Measure [ | ]Proportion of respondents who report high scores of resilience | Strongly agree, agree, disagree, strongly disagree, don’t know or not sure, don’t want to answer |
Self-efficacy (3) | I am able to reach out for help when I need it, Self-Efficacy Scale [ | ]Proportion of respondents who report high scores of their own ability to succeed in different situations and tasks | Very confident, confident, somewhat confident, not at all confident, don’t know or not sure, don’t want to answer |
Help-seeking skills (3) | I offer help or give advice to my friends who are struggling, Counseling Helping Questionnaire [ | ]Proportion of respondents who report high scores of help-seeking | Often, sometimes, rarely, never, don’t know or not sure, don’t want to answer |
Self-esteem (4) | As a whole, I am satisfied with myself, Rosenberg Self-Esteem Scale [ | ]Proportion of respondents who report high scores of self-satisfaction, having good qualities, pride, self-worth, and self-respect | Strongly agree, agree, disagree, strongly disagree, don’t know or not sure, don’t want to answer |
Cultural resilience and identity (6) | I enjoy my community’s traditions, Child Youth Resilience Measure [ | ]Proportion of respondents who report high scores of connection, identity, and cultural pride | Strongly agree, agree, disagree, strongly disagree, don’t know or not sure, don’t want to answer |
aYRBS: Youth Risk Behavior Survey.
BRAVE data collection timeline and activities.
0-2 Weeks
- Administration of survey 1 (baseline or presurvey) to determine eligibility, enrollment, and assignment to study arms
2 Months
- Sending out first set of messages (BRAVE intervention or science, technology, engineering, and math [STEM] control)
1 Week
- Administration of survey 2 (1st postsurvey)
1 Week
- Washout period
2 Months
- Crossover, second set of messages (BRAVE or STEM)
1 Week
- Administration of survey 3 (2nd postsurvey) and discontinued communication for 90 days
3 Months
- Administration of survey 4 (3rd postsurvey)
Survey Measures
Survey measures were influenced by the Healing of the Canoe Survey (Tribal Health: Reaching out InVolves Everyone project) [
] and taken from validated survey tools, including the Youth Risk Behavior Surveillance Survey [ ], the Youth Coping Responses Inventory [ ], the Child and Youth Resilience [ ], the Bandura’s Self-Efficacy Beliefs of Adolescents Scale [ ], the Counseling and Help Seeking Questionnaire [ ], and the Rosenberg Self-Esteem Scale [ ]. The primary outcomes assessed were changes in AI/AN teenager and young adult intentions, behavior, and self-efficacy related to mental health, alcohol and drug misuse assessed by negative coping scales, and help-seeking skills. Outcomes were assessed using standardized scales and Likert-type ratings, wherein answers were converted to numeric scales, and averages and SDs were calculated for each outcome. For example, the health measure is an average score generated from three questions assessing physical, mental, and spiritual health. The secondary outcomes of the study included self-efficacy, self-esteem, resilience, coping strategies, substance use, and cultural identity. outlines the survey constructs, the number of items for each construct, and examples for each question in the questionnaire (see for a complete list of questions).Data Analysis
The study data used in this analysis were collected by mHealth team members and managed using Qualtrics survey software [
] hosted at the University of Colorado Anschutz Medical Campus. All statistical analyses were performed using SAS statistical software (version 9.4) [ ] and R (version 3.6) [ ].The study team explored the carryover effects between the BRAVE and STEM arms throughout the study period. Significant differences between the BRAVE and STEM arms were compared at baseline, crossover, and follow-up. Two analyses were conducted to explore the differences among primary outcome measures. First, a two-tailed paired t test was used to assess the treatment effect, comparing BRAVE with STEM arms using the difference in outcome scores for each measure. Second, a linear mixed-effect model (using a random effect for the subject) was used to test for changes over time. Treatment effects and changes in outcomes were considered significant at P<.05. The secondary outcome measures were analyzed using a series of statistical tests. Multiple linear regressions were used to assess the differences in the baseline and follow-up survey scores. Two models were used to assess the associations at baseline and changes in scores from baseline to study completion. A linear mixed-effect model (with random effects for subjects at 3, 5, and 8 months) was used to assess changes over time.
Results
Demographics
Participants who completed the presurvey were aged 15-24 years and identified as AI/AN were included in this analysis (n=833;
). Most participants were female (552/833, 66.3%). Three-fourth were straight or heterosexual (628/833, 75.4%). Most planned to go to college (440/833, 52.8%), although some were undecided (88/833, 10.6%). Participants most frequently identified as AI/AN only, followed by two and three races. Only 11.3% (94/833) of the participants identified as Hispanic. Study participants represented nearly every state in the United States. Most were from western states including Oregon (n=213), Arizona (n=188), California (n=169), and Washington (n=138) as shown in .Characteristic | Participant, n (%) | |
Gender | ||
Female | 552 (66.2) | |
Gender fluid | 9 (1.1) | |
Male | 241 (28.9) | |
Transgender female to male | 2 (0.2) | |
Transgender male to female | 1 (0.1) | |
Two spirit | 14 (2) | |
Other (please specify) | 6 (0.7) | |
Prefer not to answer | 8 (1) | |
Sexual orientationa | ||
Asexual | 6 (0.7) | |
Bisexual | 78 (9.4) | |
Indigiqueer | 7 (0.8) | |
Lesbian, gay, or homosexual | 42 (5.1) | |
Pansexual | 24 (2.9) | |
Queer | 7 (0.8) | |
Straight or heterosexual | 628 (75.8) | |
Two spirit | 12 (1.5) | |
Otherb (please specify) | 11 (1.3) | |
Prefer not to answer | 14 (1.7) | |
Race | ||
Identify as AI/ANc only | 657 (78.9) | |
Identify as 2 races | 157 (18.9) | |
Identify as s3 | 19 (2.3) | |
Hispanicd | ||
Yes | 94 (11.3) | |
No | 714 (85.8) | |
Prefer not to answer | 24 (2.9) |
aDoes not include four missing responses.
bOther responses were as follows: demisexual (n=2), questioning (n=2), straight (n=4), poly-panromantic asexual two spirit (n=1), no sexuality (n=1), and demisexual and biromantic (n=1).
cAI/AN: American Indian and Alaska Native.
dDoes not include one missing response.
Primary Outcomes
To test the study hypotheses and explore the primary and secondary outcomes of interest, the research team compared subjects with themselves and explored differences from pre- to postsurvey throughout the study period. Tests for differences between the two study arms did not show a significant difference within subjects, and statistical analyses revealed no carryover effects (
).Participants in the BRAVE arm did not report better outcomes than those in the STEM arm. There were no significant differences in outcome measures at follow-up based on individual and aggregate survey measures. However, individuals in both study arms showed significant positive trends over the course of the study for all primary outcomes except cultural identity and help-seeking behavior.
Longitudinal Effects
A review of 3-, 5-, and 8-month mean scores when controlling for treatment order showed a positive and upward trend for nearly all constructs measured (
). Mean scores were significantly different for health (P<.001), resilience (P<.001), negative coping (P=.03), positive coping (P<.001), self-efficacy (P=.02), and self-esteem (P<.001; ).Measure | Score, mean (SD) | P value | ||||
Baseline | 3 months | 5 months | 8 months | |||
Health | 3.87 (0.88) | 3.95 (0.87) | 4.08 (0.85) | 4.12 (0.83) | <.001a | |
Resilience | 5.13 (0.56) | 5.18 (0.52) | 5.25 (0.51) | 5.23 (0.49) | <.001 | |
Negative coping | 2.88 (1.05) | 2.89 (1.02) | 2.92 (1.08) | 2.92 (0.95) | .03 | |
Positive coping | 5.04 (0.68) | 5.13 (0.61) | 5.17 (0.59) | 5.22 (0.53) | <.001 | |
Self-efficacy | 5.15 (0.61) | 5.24 (0.59) | 5.21 (0.6) | 5.22 (0.55) | .02 | |
Self-esteem | 5.07 (0.66) | 5.17 (0.62) | 5.18 (0.6) | 5.19 (0.6) | <.001 | |
Cultural resilience and identity | 5.4 (0.55) | 5.38 (0.51) | 5.4 (0.46) | 5.4 (0.49) | .84 | |
Help-seeking skills | 4.02 (0.89) | 4.03 (0.85) | 4.02 (0.81) | 3.98 (0.8) | .06 |
aItalicized values indicate statistical significance (P<.05).
Health and Identity, Cultural Pride, and Resilience
Participants who reported better health at baseline reported stronger cultural identity, cultural resilience, identity, and cultural pride. Results from the linear regression (with the health score as the outcome and the cultural identity score as the covariate) showed an association between health and cultural identity (P<.001) at baseline. At baseline, health measures were positively associated with cultural identity. On average, those who reported better health also reported greater cultural resilience, identity, and cultural pride (P<.001). The R2 value was 0.488, indicating that 48% of the variation in health scores was explained by cultural identity. Those who reported a positive change in health from baseline to 8 months were also more likely to report a positive change in cultural identity (
).Secondary outcome measures and covariatesa | β | P value | Effect size | ||||
Change in help-seeking self-efficacy | |||||||
Intercept | .452 | <.001 | 0.021 | ||||
Under 21 | −.162 | .18 | —b | ||||
Gender male | −.154 | .09 | — | ||||
Risky baseline | −.280 | .01 | — | ||||
Health and cultural identity at baseline | |||||||
Intercept | 1.23 | — | 0.092 | ||||
Identity baseline | .488 | <.001 | — | ||||
Change in health and cultural identity | |||||||
Intercept | .144 | <.001 | 0.027 | ||||
Identity baseline difference | .241 | <.001 | — | ||||
Coping strategies, health, and substance use at baseline | |||||||
Intercept | 0 | <.001 | 0.153 | ||||
Risky baseline | −.004 | .94 | — | ||||
Health baseline | .301 | <.001 | — | ||||
Change in coping strategies, health, and substance | |||||||
Intercept | .066 | .02 | 0.028 | ||||
Risky baseline | .114 | .10 | — | ||||
Health baseline | .123 | .001 | — |
aEach outcome is presented as a separate linear model. Effect sizes are measured by R2.
bThe effect size is only reported for the intercept and compares the secondary outcome measures and covariates (eg, risky baseline and health baseline).
Positive Coping Strategies
Participants who reported positive coping scores also reported better health on average, with no difference in drug and alcohol misuse (P<.001). The R2 value was 0.301, indicating that 30% of the variation in coping was explained by better health. Those who reported a positive change in coping strategies from baseline to 8 months were also more likely to report a positive change in health during the study (P=.001). The R2 value was 0.123, indicating that 12% of the variation in health was explained by positive coping strategies (
).Help-Seeking Skills and Behavior
The changes observed in help-seeking self-efficacy were significant for those who reported risky behavior at baseline to 8 months (P=.01). The mean change from baseline in help-seeking behavior was −0.281 units lower for those exhibiting risky behavior at baseline. The R2 value was 0.021, indicating that 2% of the variation in help-seeking self-efficacy was explained by the model containing only participants who reported risky behaviors at baseline. No differences were found among men or young adults aged ≥21 years (
).User Engagement and Interaction
Participants reported and demonstrated a high level of interaction and engagement with the SMS text messages of the intervention. They reported reading the majority of the messages sent; this activity rate remained high throughout the study, with more than 91.2% (458/502) reporting all or most SMS text messages read at 8 months. The number of participants who shared SMS text messages with family and friends increased from 3 months (206/574, 35.8%) to 8 months (277/502, 55.2%; P<.001;
).Questions and responsesa | Participants, n (%) | Trend | P value | ||||||||||
3 months (n=647) | 5 months (n=561) | 8 months (n=514) | |||||||||||
How many of the study’s SMS text messages did you read? | Decreasing | .01b | |||||||||||
Don’t want to answer | 1 (0.1) | 2 (0.3) | 4 (0.8) | ||||||||||
None | 2 (0.3) | 1 (0.2) | 3 (0.6) | ||||||||||
Some | 35 (5.6) | 43 (7.9) | 33 (6.6) | ||||||||||
Don’t know or not sure | 6 (1) | 4 (0.7) | 4 (0.8) | ||||||||||
Most | 148 (24) | 133 (24.5) | 106 (21.1) | ||||||||||
All | 426 (68.9) | 360 (66.3) | 352 (70.1) | ||||||||||
How many of the study’s videos did you watch? | Decreasing | .60 | |||||||||||
Don’t want to answer | 3 (0.5) | 2 (0.4) | 2 (0.4) | ||||||||||
None | 57 (9.2) | 38 (7) | 30 (5.9) | ||||||||||
Some | 139 (22.5) | 110 (20.2) | 94 (18.7) | ||||||||||
Don’t know or not sure | 21 (3.4) | 19 (3.5) | 10 (1.9) | ||||||||||
Most | 143 (23.2) | 163 (30) | 224 (44.6) | ||||||||||
All | 254 (41.2) | 211 (38.9) | 142 (28.3) | ||||||||||
How many of the messages did you share with your friends or family? | Increasing | <.001 | |||||||||||
Don’t want to answer | 4 (0.7) | 5 (0.9) | 4 (0.8) | ||||||||||
None | 122 (19.8) | 74 (13.6) | 59 (12) | ||||||||||
Some | 254 (41.2) | 204 (37.6) | 144 (28.7) | ||||||||||
Don’t know or not sure | 30 (4.9) | 18 (3.3) | 18 (3.6) | ||||||||||
Most | 137 (22.2) | 168 (31) | 210 (41.8) | ||||||||||
All | 69 (1.2) | 74 (13.6) | 67 (13.4) |
aResults of the linear mixed-effect model with a random effort to subject (one for each question) based on 3-, 5-, and 8-month timepoints.
bItalicized values indicate statistical significance (P<.05).
Changes in Help-Seeking Skills and Behavior
A comparison of 3-, 5-, and 8-month data showed significant improvements in help-seeking skills among BRAVE participants. Results from the linear mixed-effect model (with random effects for subjects) showed a significant increase. The number of participants who used the SMS text messages to help themselves increased from 69.1% (427/618) at 3 months to 76% (381/501) at 8 months (P<.001). Similarly, the number of participants who used the SMS text messages to help a friend or family member increased from 22.4% (138/616) at 3 months to 54.6% (272/498) at 8 months (P<.001;
). At the end of the survey, participants were also asked to provide open-ended feedback on ways to improve the program for others. The team plans to conduct a qualitative analysis of user feedback to inform future improvements.Questions and responsesa | Participants, n (%) | Trend | P value | ||||||||||
3 months (n=647) | 5 months (n=561) | 8 months (n=514) | |||||||||||
How likely are you to use any of the resources you learned about if you, a family member, or friend need them? | Decreasing | .008b | |||||||||||
Don’t want to answer | 3 (0.5) | 2 (0.4) | 3 (0.6) | ||||||||||
Very unlikely | 2 (0.3) | 1 (0.2) | 0 (0) | ||||||||||
Unlikely | 18 (3) | 16 (3) | 14 (3) | ||||||||||
Don’t know or not sure | 33 (5) | 23 (4.2) | 17 (3) | ||||||||||
Likely | 338 (54.8) | 352 (64.8) | 330 (65.7) | ||||||||||
Very likely | 223 (36.1) | 149 (27.4) | 138 (27.5) | ||||||||||
Did you use any of the messages to help yourself? | Increasing | <.001 | |||||||||||
No | 89 (14) | 44 (8) | 39 (8) | ||||||||||
Did not need them | 102 (16.5) | 65 (12) | 81 (16) | ||||||||||
Yes | 427 (69.1) | 433 (79.9) | 381 (76.1) | ||||||||||
Did you use any of the messages to help a friend or family member? | Increasing | <.001 | |||||||||||
No | 363 (58.9) | 196 (36.1) | 119 (23.9) | ||||||||||
Did not need them | 115 (18.7) | 122 (22.5) | 107 (21.5) | ||||||||||
Yes | 138 (22.4) | 225 (41.4) | 272 (54.6) | ||||||||||
Did the messages encourage you to participate in any new wellness activities? | Increasing | .07 | |||||||||||
No | 119 (19.3) | 88 (16.3) | 79 (15.8) | ||||||||||
Yes | 498 (80.7) | 453 (83.7) | 421 (84.2) | ||||||||||
Is there anything else we can do to improve this program for others? | Decreasing | <.001 | |||||||||||
No | 537 (87.6) | 492 (90.7) | 472 (94.8) | ||||||||||
Yes | 76 (12.4) | 50 (9.2) | 26 (5.2) |
aResults of the linear mixed-effect model with a random effort to subject (one for each question) based on 3-, 5-, and 8-month timepoints.
bItalicized values indicate statistical significance (P<.05).
Discussion
Principal Findings
This is the first national RCT of an mHealth intervention among AI/AN teenagers and young adults to rigorously test the efficacy of a mental wellness intervention in relation to STEM career messages, conducted in a 100% virtual format—recruiting, consenting, and surveying participants—via SMS text messaging (
).Although our results did not fully support the proposed hypotheses, there were no measurable treatment effects between the two study arms. The findings indicate that both culturally relevant mHealth interventions improved health outcomes, an important programmatic finding. For both groups, most survey measures improved over time, including notable improvements in mental health, resilience, coping skills, and better self-esteem, despite the study conducted in the midst of the global pandemic. There were also significant reductions in reported alcohol and drug misuse by participants over the course of the study. Both study arms showed positive improvements in composite scores and trends, arguably the most promising outcome of all, demonstrating that culturally tailored SMS text messages can improve health and wellness outcomes for Native teenagers and young adults.
The lack of significant treatment effects may be partially explained by the lack of representation of Native youth and young adult experience in popular media, making the control arm similarly novel and inspiring to study participants. These findings have direct implications for future research. When designing RCTs for AI/AN groups, it is essential to consider what constitutes a control versus intervention arm. The study team believed that STEM messages would not interfere with the modeling of BRAVE-related skills. This RCT suggests, however, that the STEM role model videos had a similar effect on the study participants.
The need for culturally relevant STEM interventions is also critical for Native youth and young adults [
]. Native students are chronically underrepresented in training programs for STEM professions. Expanding the diversity of the STEM and medicine workforce is needed to meet the unique needs and worldviews of AI/AN communities. Messaging campaigns, such as those offered in the control arm, may inspire Native youth and young adults to join the STEM workforce pathway.Health and Identity, Cultural Pride, and Resilience
At baseline, physical, mental, and spiritual health measures were positively associated with cultural identity. This suggests that resilience, identity, and cultural pride may be associated with better health among AI/AN teenagers and young adults. These findings are consistent with a previous study that reported that AI/AN identity may be protective against a myriad of risk behaviors, including alcohol and drug misuse [
]. This research adds to a growing body of evidence that recognizes the foundational importance of building cultural pride and positive self-worth in adolescent health programs for AI/AN youth, further demonstrating that health and identity are inextricably intertwined.Positive Coping Strategies
Notably, participants who reported better coping strategies at baseline (ie, having healthy outlets to use when stressed, taking active steps to improve mental health, or using positive self-talk to overcome unpleasant feelings) also reported better health on average but did not have any difference in risky behavior compared with their peers. In addition, participants who reported a positive change in coping strategies over the course of the study were more likely to report a positive change in physical, mental, and spiritual health during that period. These findings reinforce a previous study on AI/AN populations, which suggested that having healthy coping strategies may directly impact health [
]. Health advocates can build on these findings by encouraging students to find and practice healthy coping strategies that reflect their unique skills and preferences.Alcohol and Drug Misuse May Compromise Help Seeking
Although we hypothesized that the BRAVE intervention might be more effective to the original audience of AI/AN males aged 21-24 years involved in formative research, it was not correct; we found that changes in help-seeking self-efficacy were not significantly different from younger AI/AN males aged 18-20 years. This was a welcome surprise, reinforcing our decision to expand the eligibility criteria for the study.
There was, however, one significant difference among males at baseline: males who reported greater alcohol and drug misuse also reported lower help-seeking behavior. Previous research in other populations suggests that help-seeking interventions are most effective when an individual is symptomatic and felt the need to ask for help [
]. Future research should determine the points or conditions in which help-seeking interventions, such as BRAVE, are the most beneficial for reducing alcohol and drug misuse.Help-Seeking Skills and Behavior
Finally, self-reported help-seeking skills (ie, ability to recognize when a friend is struggling, and willingness to contact a helpline, seek counseling, or treatment if needed, for themselves or a friend) were moderate to high among study participants. Healthy participants with high help-seeking characteristics who entered an intervention showed less change in these attributes over time.
Similarly, self-reported data showed a high level of interaction and engagement throughout the study; however, this engagement did not translate to a significant intervention effect on the primary outcomes of interest. Future work could explore the efficacy of the BRAVE intervention among AI/AN teenagers and young adults who have been identified as high-risk due to their involvement with juvenile justice systems, foster care, treatment, or behavioral health programs.
Strengths and Limitations
The BRAVE study has several strengths and limitations. A major strength of the study was the RCT design, where random allocation allowed for a clean comparison between the intervention and control groups. The biggest strength of BRAVE was its development with input from AI/AN teenagers and young adults, tribal health educators, and topical experts. The focus of BRAVE on culturally relevant images, language, and the use of role model videos demonstrated respect for diversity and value. BRAVE created health communication messaging that AI/AN youth and young adults could identify with.
There are several limitations that should be noted. First, all data were self-reported. The sample was recruited largely through We R Native, which is a multimedia health and wellness resource. Participant familiarity with We R Native may have influenced their responses and participation. The initial goal was to enroll 125 participants from the 12 Indian Health Service geographic regions (n=1500) to explore the efficacy of BRAVE across regional differences. However, only 1030 participants, which is an insufficient sample size to explore regional differences, enrolled in BRAVE. Of these, only 833 were included in the analysis to improve data quality. The washout period of 1 week between the intervention and control arms may not have been sufficient to observe an effect in the treatment and control conditions. Finally, most participants reported high or favorable scores on the survey measures at baseline. This may have contributed to the lack of treatment effects between the two arms.
Conclusions
This study builds upon the extensive digital research activities already carried out by the NPAIHB and the mHealth Impact Lab and fills a critical need for evidence-based interventions that reflect the unique needs and worldview of AI/AN teenagers and young adults. This research provides new information on the development of mHealth interventions for AI/AN youth using evidence-based strategies and furthers our understanding of the connections between health and cultural resilience, identity, and cultural pride. This work demonstrates the feasibility of recruiting a national sample of AI/AN teenagers and young adults online, which is largely due to the trusted nature of the We R Native platform. It also underscores the acceptability of SMS text messaging as a vehicle to promote and support mental health and wellness, addressing a priority concern among AI/AN teenagers and young adults.
Researchers, practitioners, policy makers, and AI/AN-serving organizations may use this example to improve the relevance, efficacy, and use of other mHealth interventions to reach high-risk, underserved populations. BRAVE lessons can be easily integrated into the flow of services provided by clinics, schools, treatment centers, and other community-based programs and can be tailored to the needs and time constraints of any setting [
]. Future research may address the recommendations offered by study participants to improve campaigns for future users and assess the uptake of the service outside the study setting. Native youth and young adults can now text BRAVE to 97779 or STEM to 97779 to receive the SMS text message sequence on their own.Taken together, these results demonstrate the importance of culturally relevant health resources to support AI/AN teenagers and young adults as they navigate common risky situations involving family and friends, including alcohol and drug misuse, dating violence, and suicidality. User-designed mHealth interventions are critical to reach and engage at-risk populations and nurture lifelong decision-making skills. In the words of one participant, demonstrating the power and promise of BRAVE, “It encouraged me to better my health not only for myself but for the future generations.”
Acknowledgments
This study was funded by a grant from the TAM program at the University of Wisconsin-Madison. The content is solely the responsibility of the authors and does not necessarily represent the official views of the university or the TAM program. The BRAVE intervention was developed, in part, with staff funded by grant H79SM082106 from Substance Abuse and Mental Health Service Administration. The views, opinions, and content of this publication are those of the authors and contributors and do not necessarily reflect the views, opinions, or policies of the Centers for Medicare and Medicaid Services, Substance Abuse and Mental Health Service Administration, or Health and Human Services and should not be construed as such.
Editorial Notice
This randomized study was only retrospectively registered, explained by authors with "(1) It was not a process or requirement; (2) It was funded with private/foundation funds and not federal/NIH." The editor granted an exception from ICMJE rules mandating prospective registration of randomized trials, because the risk of bias appears low and the study was considered formative, guiding the development of the application. However, readers are advised to carefully assess the validity of any potential explicit or implicit claims related to primary outcomes or effectiveness, as retrospective registration does not prevent authors from changing their outcome measures retrospectively.
Conflicts of Interest
Authors SCR, DS, RP, CB, TGD, CM, DLB, MT, CC, MS, and PS are employees of Northwest Portland Area Indian Health Board, which developed the BRAVE intervention.
Healthy Native Youth BRAVE Website and Curriculum.
PDF File (Adobe PDF File), 412 KB
BRAVE study hypotheses.
DOCX File , 14 KB
BRAVE survey questions.
DOCX File , 17 KB
Treatment effect results and mean differences.
DOCX File , 22 KB
Graphical abstract of the BRAVE study.
PDF File (Adobe PDF File), 142 KB
CONSORT-EHEALTH checklist (V 1.6.1).
PDF File (Adobe PDF File), 716 KBReferences
- American Indian and Alaska Native Alone or in Combination With One or More Races, 2013-2018 American Community Survey 5-year Estimates (B02010). United States Census Bureau. 2013. URL: https://www.census.gov/newsroom/press-releases/2019/acs-5-year.html [accessed 2020-11-20]
- National Academies of Sciences, Engineering, and Medicine. Advancing Health Equity for Native American Youth. Washington, USA: The National Academies Press; 2016.
- Gone JP, Trimble JE. American Indian and Alaska Native mental health: diverse perspectives on enduring disparities. Annu Rev Clin Psychol 2012;8:131-160. [CrossRef] [Medline]
- Strickland CJ, Walsh E, Cooper M. Healing fractured families: parents' and elders' perspectives on the impact of colonization and youth suicide prevention in a pacific northwest American Indian tribe. J Transcult Nurs 2006 Jan;17(1):5-12. [CrossRef] [Medline]
- Kelley A, Small C. Documenting resiliency of American Indian youth: preliminary results from native PRIDE’s intergenerational connections project. J Indig Stud 2020;8:5. [CrossRef]
- Rushing SC, Stephens D. Use of media technologies by Native American teens and young adults in the Pacific Northwest: exploring their utility for designing culturally appropriate technology-based health interventions. J Prim Prev 2011 Aug;32(3-4):135-145. [CrossRef] [Medline]
- Around Him D, Weilin L, Gross E, Warren J, DeMand A, Garcia-Baza I, et al. Twitter Analysis Can Help Practitioners, Policymakers, and Researchers Better Understand Topics Relevant to American Indian/Alaska Native Youth. Child Trends. URL: https://www.childtrends.org/publications/twitter-analysis-practitioners-policymakers-researchers-understand-topics-american-indian-alaska-native-youth [accessed 2020-11-20]
- Around HD, Weilin L, Gross E, Warren J, DeMand A, Garcia-Baza I, et al. Twitter Analysis Can Help Practitioners, Policymakers, and Researchers Better Understand Topics Relevant to American Indian/Alaska Native Youth. Child Trends. 2020. URL: https://www.childtrends.org/publications/twitter-analysis-practitioners-policymakers-researchers-understand-topics-american-indian-alaska-native-youth [accessed 2020-11-20]
- Reed N, Craig RS. Youth Health Tech Surveys. Healthy Native Youth. 2020. URL: https://www.healthynativeyouth.org/wp-content/uploads/2021/02/YHTS-No-Labels-FINAL.pdf [accessed 2021-02-25]
- Gurol-Urganci I, de Jongh T, Vodopivec-Jamsek V, Atun R, Car J. Mobile phone messaging reminders for attendance at healthcare appointments. Cochrane Database Syst Rev 2013 Dec 5(12):CD007458 [FREE Full text] [CrossRef] [Medline]
- de Jongh T, Gurol-Urganci I, Vodopivec-Jamsek V, Car J, Atun R. Mobile phone messaging for facilitating self-management of long-term illnesses. Cochrane Database Syst Rev 2012 Dec 12;12:CD007459 [FREE Full text] [CrossRef] [Medline]
- Vodopivec-Jamsek V, de Jongh T, Gurol-Urganci I, Atun R, Car J. Mobile phone messaging for preventive health care. Cochrane Database Syst Rev 2012 Dec 12;12:CD007457 [FREE Full text] [CrossRef] [Medline]
- Head KJ, Noar SM, Iannarino NT, Harrington NG. Efficacy of text messaging-based interventions for health promotion: a meta-analysis. Soc Sci Med 2013 Nov;97:41-48. [CrossRef] [Medline]
- Craig Rushing S, Gaston A, Kaufman CE, Markham C, Jessen C, Gorman G, et al. Using Technology to Promote Health and Wellbeing among American Indian and Alaska Native Teens and Young Adults. In: Dyson LE, Grant S, Hendriks M, editors. Indigenous People and Mobile Technologies. New York, NY: Routledge; 2015:163-178.
- Schnall R, Rojas M, Bakken S, Brown W, Carballo-Dieguez A, Carry M, et al. A user-centered model for designing consumer mobile health (mHealth) applications (apps). J Biomed Inform 2016 Apr;60:243-251 [FREE Full text] [CrossRef] [Medline]
- Kaufman CE, Schwinn TM, Black K, Keane EM, Big Crow CK. The Promise of Technology to Advance Rigorous Evaluation of Adolescent Pregnancy Prevention Programs in American Indian and Alaska Native Tribal Communities. Am J Public Health 2016 Sep;106(S1):S18-S20. [CrossRef] [Medline]
- Craig Rushing S, Kelley A, Hafner S, Stephens D, Singer M, Bingham D, et al. The BRAVE Study: Formative Research to Design a Multimedia Intervention for American Indian and Alaska Native Young Adults. Am Indian Alsk Native Ment Health Res 2021;28(1):71-102. [CrossRef] [Medline]
- Fisher P, Ball T. Tribal participatory research: mechanisms of a collaborative model. Am J Community Psychol 2003 Dec;32(3-4):207-216. [CrossRef] [Medline]
- Freedland KE, Mohr DC, Davidson KW, Schwartz JE. Usual and unusual care: existing practice control groups in randomized controlled trials of behavioral interventions. Psychosom Med 2011 May;73(4):323-335 [FREE Full text] [CrossRef] [Medline]
- Stephens D, Peterson R, Singer M, Johnson J, Rushing S, Kelley A. Recruiting and Engaging American Indian and Alaska Native Teens and Young Adults in a SMS Help-Seeking Intervention: Lessons Learned from the BRAVE Study. Int J Environ Res Public Health 2020 Dec 16;17(24):12 [FREE Full text] [CrossRef] [Medline]
- Qualtrics. 2020. Qualtrics. URL: http://www.qualtrics.com/ [accessed 2020-11-20]
- Grunbaum J, Kann L, Kinchen S, Williams B, Ross J, Lowry R, et al. Youth risk behavior surveillance--United States, 2001. J Sch Health 2002 Oct;72(8):313-328. [CrossRef] [Medline]
- Hernandez B, Vigna J, Kelley ML. The Youth Coping Responses Inventory: development and initial validation. J Clin Psychol 2010 Sep;66(9):1008-1025. [CrossRef] [Medline]
- Ungar M, Liebenberg L. Assessing Resilience Across Cultures Using Mixed Methods: Construction of the Child and Youth Resilience Measure. Journal of Mixed Methods Research 2011 Apr;5(2):126-149. [CrossRef]
- Bandura A. Guide for constructing self-efficacy scales. In: Self-efficacy beliefs of adolescents. Charlotte: Information Age Publishing; 2006:307-337.
- Dauphinais P, LaFromboise T, Rowe W. Perceived problems and sources of help for American Indian Students. Counselor Education and Supervision 1980;20:37-44.
- Rosenberg M. Society the adolescent self-image. In: Princeton University Press. Princeton: Princeton University Press; 1965:1-326.
- Donovan D, Thomas L, Sigo R, Price L, Lonczak H, Lawrence N, et al. Healing of the canoe: preliminary results of a culturally tailored intervention to prevent substance abuse and promote tribal identity for Native youth in two Pacific Northwest tribes. In: Am Indian Alsk Native Ment Health Res. Healing of the Canoe: Preliminary results of a culturally grounded intervention to prevent substance abuse and promote tribal identity for Native youth in two Pacific Northwest tribe. Am Indian Alsk Native Ment Health Res (Online); 2015:42-76.
- SAS. URL: https://www.sas.com/en_us/home.html [accessed 2020-12-25]
- The R Project for Statistical Computing. URL: https://www.r-project.org/ [accessed 2020-12-23]
- Stevens S, Andrade R, Page M. Motivating Young Native American Students to Pursue STEM Learning Through a Culturally Relevant Science Program. J Sci Educ Technol 2016 Jun 6;25(6):947-960. [CrossRef]
- Brown RA, Dickerson DL, D'Amico EJ. Cultural Identity Among Urban American Indian/Alaska Native Youth: Implications for Alcohol and Drug Use. Prev Sci 2016 Oct;17(7):852-861 [FREE Full text] [CrossRef] [Medline]
- Elmquist DL, McLaughlin CL. Social Media Use Among Adolescents Coping with Mental Health. Contemp School Psychol 2017 Nov 16;22(4):503-511. [CrossRef]
- Gulliver A, Griffiths KM, Christensen H, Brewer JL. A systematic review of help-seeking interventions for depression, anxiety and general psychological distress. BMC Psychiatry 2012 Jul 16;12:81 [FREE Full text] [CrossRef] [Medline]
- BRAVE Users Guide. Healthy Native Youth. URL: https://www.healthynativeyouth.org/curricula/brave/ [accessed 2021-02-24]
Abbreviations
AI/AN: American Indian and Alaska Native |
mHealth: mobile health |
NPAIHB: Northwest Portland Area Indian Health Board |
RCT: randomized controlled trial |
STEM: science, technology, engineering, and math |
TAM: Technology & Adolescent Mental Wellness |
Edited by A Jolliff, M Moreno; submitted 30.11.20; peer-reviewed by E Rink, B Kerr; comments to author 29.01.21; revised version received 25.02.21; accepted 16.03.21; published 15.09.21
Copyright©Stephanie Craig Rushing, Allyson Kelley, Sheana Bull, David Stephens, Julia Wrobel, Joshva Silvasstar, Roger Peterson, Corey Begay, Thomas Ghost Dog, Celena McCray, Danica Love Brown, Morgan Thomas, Colbie Caughlan, Michelle Singer, Paige Smith, Kanku Sumbundu. Originally published in JMIR Mental Health (https://mental.jmir.org), 15.09.2021.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Mental Health, is properly cited. The complete bibliographic information, a link to the original publication on https://mental.jmir.org/, as well as this copyright and license information must be included.