This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Mental Health, is properly cited. The complete bibliographic information, a link to the original publication on http://mental.jmir.org/, as well as this copyright and license information must be included.
Supporting Our Valued Adolescents (SOVA), a social media website for adolescents, was designed to increase mental health literacy and address negative health beliefs toward depression and/or anxiety diagnosis and treatment. This stakeholder-informed site underwent iterative user testing to evolve into its current version with daily blog posts, round-the-clock site moderation, and Web-based peer interaction to create an online support community.
The aim of this study was to evaluate the technological feasibility (at least 100 users on the site, logging in 12 to 18 times in the first 6 weeks) and acceptability of the SOVA site determined by the System Usability Scale (SUS).
Adolescents and young adults (aged 14-26 years) with a self-reported history of depressive and/or anxiety symptoms were recruited to access the research website (sova.pitt.edu). Participants were screened out if they reported active suicidality or a prior suicide attempt. Baseline survey measures included demographics, symptomatology using the Patient Health Questionnaire-9 modified for adolescents (PHQ-9A) and Screen for Child Anxiety Related Disorders (SCARED-C), and mental health treatment history. The 6-week follow-up measures taken in addition to the symptomatology, included feasibility (total number of log-ins), usability, and acceptability of SOVA using SUS.
Most of the 96 participants identified as female (75% [72/96]) and white (67% [64/96]). Most participants (73% [70/96]) reported having taken prior professional psychological help. The average PHQ-9A score was 11.8 (SD 5.5), and for SCARED-C, 85% (80/94) of the participants reported a score consistent with being susceptible to a diagnosed anxiety disorder. There were 46% (41/90) of eligible users who ever logged in. Out of the total users who ever logged in, the mean of total log-ins over the entire study was 4.1 (SD 6.9). Median number of users rated the user-friendliness of the site as “good.” The average SUS score was 71.2% (SD 18.7), or a “C-grade,” which correlated to an acceptable range. The participants reported to have liked the “easy-to-understand format” and “positive, helpful atmosphere,” but they also reported a desire for greater social interaction. Iterative recruitment resulted in incremental improvements to the site.
The SOVA site met feasibility goals of recruiting almost 100 users and establishing acceptable usability. Subsequent interventions are planned to increase site engagement and to evaluate efficacy in increasing uptake of primary care–recommended depression and/or anxiety treatment.
Suicide is the second most common cause of death in adolescents and young adults in the United States [
To seize this potential, we designed a social media website for adolescents called SOVA (Supporting Our Valued Adolescents) and a separate companion website for parents called wiseSOVA (not discussed in this paper). SOVA aims to (1) challenge negative health beliefs and increase depression and anxiety knowledge through daily blog posts enhanced with peer commentary, (2) promote social support through Web-based peer interactions, and (3) encourage parent-adolescent mental health offline communication through same-day blog posts with questions for discussion. SOVA’s goal is to increase the perceived need for services in both, adolescents referred for treatment and their parents, ultimately leading to increasing the use of adolescent mental health services.
From the inception of the SOVA sites, we knew we would need to use multiple strategies to buffer against the lack of engagement which affects many ehealth interventions [
The SOVA site's process of iterative development and design is described in detail elsewhere [
Supporting Our Valued Adolescents (SOVA) key intervention components.
Adolescents and young adults (AYA) aged 14 to 26 years were recruited in-person from clinical settings by behavioral health clinicians and from online websites (eg, Craigslist, University of Pittsburgh Research Registry). Young adults up to the age of 26 years were included as the online community we sought to form that could benefit from peer involvement from young adults who had had positive experiences in the mental health care system and exhibited resilience [
After creating a username, password, and agreeing with a set of common-ground rules emphasizing anonymity on the main website, participants were automatically redirected to a Web-based survey (Qualtrics, Provo, UT). Individuals aged 14 to 26 years were included if they self-reported a history of depression and/or anxiety symptoms, had Internet and email access, could read and write in English, and had completed the 6th grade. We obtained a waiver of parental permission because of anticipated difficulty with recruitment due to the study being online, and because minors aged 14 years and older can seek mental health services without parental permission in Pennsylvania, United States. Due to the unknown safety profile of the intervention, we excluded participants with active suicidality, defined by thoughts with intent to act on these thoughts, or a history of a suicide attempt. Participants screening in were asked for their contact information and for 1 supportive adult; this was described as a requirement of the study for safety reasons. Those screening in would be redirected to complete a baseline 52-item Web-based survey. After 6 weeks, they received a follow-up 63-item Web-based survey by email. Passive data were also collected regarding the number of log-ins and text from comments in response to blog posts. Participants received compensation in the form of a prepaid debit card on the completion of the first and 6-week surveys.
At baseline, participants were asked their age, gender, and race.
Depression symptoms were measured using the Patient Health Questionnaire-9 Item (PHQ-9) modified for adolescent use and using the cut-off score of 11 for detecting major depression [
Anxiety symptoms were measured using the 5-item version of the Screen for Child Anxiety Related Emotional Disorders (5-item SCARED-C) [
Mental health treatment history was ascertained by asking AYA whether they had ever received treatment from a professional psychologist or counselor and/or taken a medication such as an antidepressant [
The Positive Youth Development 17-item Very Short Form (PYD-VSF) is an abbreviated version of the full PYD, which has been used to measure positive attributes in AYA based on the Lerner and Lerner Five Cs Model of PYD [
User log-in data over the initial 6 weeks of site use and afterwards (some individuals continued to use the site after 6 weeks) was collected, as well as the frequency of viewing specific blog post categories, such as Education (twice as many posts as other categories), Social Media, Positivity, and Resources was also reported. Aggregate data of site use was also collected by views of unique Internet protocol (IP) addresses, filtering out the IP addresses of the study team. A data visualization module was created to view daily, weekly, and monthly log-ins, unique IP addresses, and blog post article comments on the same display over time and allowed notation of events that may affect use (eg, opening site articles to public).
Website usability and acceptability were measured using the modified SUS [
Descriptive analyses were used for summary statistics of all measures listed above. The primary outcome for feasibility was mean number of log-ins over a 6-week period, and for usability and acceptability, the SUS mean score. Paired
Screening for eligibility took place with 226 individuals, of whom 130 were ineligible, mostly because of suicidality (N=121; see
Participant flowchart.
The baseline sample (
There were 46% (41/90) of participants who ever logged in. As 61 participants completed the 6-week survey on usability, as many as 13 may have viewed content without logging in once it was public or only viewed the content in their notification email, but this cannot be determined from the data collected because of anonymity of 6-week data collection and inability to match it to usernames. Out of those who ever logged in (users), the mean total log-ins over the initial 6 weeks were 1.9 (SD 2.3) and over the total study were 4.1 (SD 6.9; see
Data visualization showed a sharp increase in site views by unique IP addresses in June 2016 after the site blog posts were made public and a problem with users not getting notification emails was resolved (
Demographics and baseline measures of study population (N=96).
Variable | Value | |
Age in years, median (range) | 23 (14-26) | |
Female | 72 (75) | |
Female to male transgender | 0 (0) | |
Male | 21 (22) | |
Male to female transgender | 0 (0) | |
Not sure | 0 (0) | |
Other | 3 (3) | |
White | 64 (67) | |
Black | 14 (15) | |
Asian/Pacific Islander | 12 (12) | |
Hispanic | 7 (7) | |
North American Native | 1 (1) | |
Other | 0 (0) | |
Don’t want to answer | 2 (2) | |
Depressive symptoms: PHQ-9 Scoreb, mean (SD)c | 11.8 (5.5) | |
Feeling sad most days in the past year, n (%) | 67 (70) | |
Not difficult at all | 9 (9) | |
Somewhat difficult | 55 (58) | |
Very difficult | 23 (24) | |
Extremely difficult | 8 (8) | |
None (1-4) | 8 (8) | |
Mild (5-9) | 30 (32) | |
Moderate (10-14) | 27 (28) | |
Moderately severe (15-19) | 21 (22) | |
Severe (20-27) | 9 (9) | |
SCARED-Cd score consistent with anxiety (≥3), n (%)e | 80 (85) | |
Professional psychologist or counselor | 70 (73) | |
Medication like antidepressants | 55 (57) | |
54.4 (7.5) | ||
Competence | 7.6 (2.0) | |
Confidence | 8.0 (2.5) | |
Character | 13.6 (2.6) | |
Caring | 13.0 (2.2) | |
Connection | 12.2 (3.2) |
aPercentage may equal greater than 100 due to participants answering more than one racial category.
bPHQ-9: Patient Health Questionnaire-9 modified for adolescents.
cN=95 due to exclusion of those who did not answer all PHQ-9 questions.
dSCARED-C: 5-item Screen for Child Anxiety Related Emotional Disorders.
eN=94 due to those who did not answer all SCARED-C questions.
fN=92 due to those who did not answer questions for all PYD categories.
Feasibility and usability of Supporting Our Valued Adolescents (SOVA).
Outcomes | Value | |||
All users | 0.9 (1.8) | |||
Users who ever logged in | 1.9 (2.3) | |||
All users | 1.8 (5.0) | |||
Users who ever logged in | 4.1 (6.9) | |||
All users | 2.3 (10.1) | |||
Users who ever logged in | 5.1 (14.7) | |||
All users | 0.9 (3.7) | |||
Users who ever logged in | 2.0 (5.3) | |||
All users | 1.2 (4.8) | |||
Users who ever logged in | 2.8 (6.9) | |||
All users | 0.8 (3.4) | |||
Users who ever logged in | 1.8 (4.9) | |||
System Usability Scale scorec, mean (SD) | 71.2 (18.7) | |||
Worst imaginable | 0 (0) | |||
Awful | 1 (2) | |||
Poor | 3 (5) | |||
OK | 11 (18) | |||
Good | 19 (31) | |||
Excellent | 21 (34) | |||
Best imaginable | 6 (10) |
aData available for 90 accounts as 6 users requested to be withdrawn.
bDifferences were not statistically significant (
cN=61 due to loss to follow-up; 70 users completed follow-up but not all completed each measure.
Data visualization.
The median number of users thought the user-friendliness of the site was “good.” Over a third of users (34% [21/61]) thought the user-friendliness of the site was excellent (
Only 4 users participated in poststudy interviews. One interviewee commented the site was, “better than I expected it to be” and later clarified:
I didn’t expect [the site] to be that very in-depth. I thought it was very straightforward, but it was very in-depth and very fun to be on. The blog posts on positive quotes and stories, I’d say they felt very uplifting actually. I really enjoyed it. It felt like I wasn’t reading something in a psychological book. It felt like on a personal level.
On comments to an article on what to share and what not to share on social media, users remarked that although they don’t feel comfortable posting on other websites, they do feel comfortable on sova.pitt.edu due to its anonymity (
In open-ended questions, remarks on what to change mostly centered around increasing interactivity of the site and including less structured ways for users to communicate, such as on a discussion forum. In poststudy interviews, users remarked an app would improve usability due to logging in:
Definitely now with smartphones like if there’s an app it’s so much better than having to log onto the Internet.
When asked about a potential future direction of including peer users who compose their own blog posts, a user remarked:
I think that’s a great idea. I think that’s what makes it more likeable you know when people have their own input so they can share their own stories. They write their own journals per se about their experiences. That’s a good idea.
User commenting on anonymity.
Due to loss to follow-up and missing data, only 57 participants completed enough of the 6-week survey to calculate scoring scales for depression, anxiety, and positive youth development.
There was a statistically significant
Overall, there was a statistically significant positive change in mean total PYD-SF from baseline (54.0, SD 7.6 ) to 6-week (57.6, SD 8.0),
Participants commenting on blog posts shared personal stories and support (
Change in depression, anxious symptoms and obtaining treatment at follow-up, N=56. N differs from original group due to loss to follow-up and incomplete data, for example, not completing full scale.
Outcome | Baseline | 6-week | Test parameter | Difference | ||
PHQ-9a score, mean (SD) | 11.6 (5.1) | 10.2 (6.1) | Paired |
|||
SCARED-Cb score, score consistent with anxiety (≥3), n (%) | 48 (86) | 49 (88) | McNemar test | N=56 | ||
Professional psychologist/counselor | 43 (66) | 45 (69) | McNemar test | N=57 | ||
Medication such as antidepressant | 32 (56) | 34 (52) | McNemar test | N=57 |
aPHQ-9: Patient Health Questionnaire-9 modified for adolescents.
bSCARED-C: 5-item Screen for Child Anxiety Related Emotional Disorders.
Changes in Positive Youth Development-Short Form score (N=57). N differs from original group due to loss to follow-up.
Outcome | Baseline | 6-week | Test parameter | Difference | ||
54.0 (7.6) | 57.6 (8.0) | Paired |
||||
Competence | 7.6 (1.9) | 8.2 (2.1) | Paired |
|||
Confidence | 7.9 (2.5) | 8.6 (2.3) | Paired |
|||
Character | 13.4 (2.6) | 14.0 (3.1) | Paired |
|||
Caring | 12.9 (2.2) | 13.2 (2.2) | Paired |
|||
Connection | 12.2 (3.2) | 13.6 (3.1) | Paired |
aPYD-SF: Positive Youth Development-Short Form.
User commenting on providing support.
The aim of this study was to determine the usability of a social media website designed to challenge negative health beliefs and increase depression and anxiety knowledge in adolescents and young adults through daily blog posts enhanced with peer commentary from an online community. We found that maintaining the site was technologically feasible as we experienced very few major errors, aside from finding that notification emails were not being sent because of incorrect settings. We were able to moderate the site and examine all new content in a timely manner. Additionally, there were no safety concerns identified. Feasibility goals were not fully achieved. While we were able to recruit about 100 AYA to the study, which implied interest about the site, only about half of the users ever logged in. We expected that all users would log in and the mean number of log-ins to the site would be 12 to 18 over the 6-week study, but the actual mean of 2 log-ins over the first 6 weeks was much lower than the mean value we expected. We reached our usability goals and found that the median number of users found that SOVA sites had “good” user-friendliness and the SUS scale acceptability mean was 71.2, only slightly lower than our goal of 73. We also found users in this study experienced a slight decrease in depression symptoms and increase in competence, confidence, and connection.
We think we were able to achieve good usability results mainly because of the stepwise stakeholder-informed approach we took, which employed human computer interaction techniques and uncovered user preferences such as a desire for anonymity and functionality to allow them to share their experiences with others [
A review of the literature finds few comparable studies in adolescent and young adult mental health which were tested for usability in the same stage of development or having similar intervention components or goals. The most comparable is a recent study evaluating the feasibility and acceptability of a minimal viability version of ProjectTECH, a Web-based skill-building intervention for adolescents at risk for depression including peer support, in 4 groups of 8 to 12 individuals over an 8-week period [
Web-based interventions to facilitate mental health help-seeking in young adults are feasible [
A recent review of 19 Internet-based cognitive behavioral therapy programs for adolescent depression found that some techniques may increase user engagement. For example, real-time guidance, surface credibility or a competent “look and feel” of the site, including video, animation, and interactive exercises, tailoring, and self-monitoring components [
This study was not designed to test effectiveness, but the direction of slightly decreasing depression symptoms is encouraging. Overall, the AYA participating in this study had high levels of caring and altruistic intentions. Contributing to Web-based interventions which have a goal of sharing experiences in a safe and positive environment may offer opportunities for these AYA to increase aspects of positive development, especially competence, confidence, and connection. Recruiting new users to the site who have high levels of the caring characteristic and have a desire to share what they have learned about being mentally healthy may be a method to increase site engagement as well.
There are several limitations of this study. Due to finding that anonymity was important to users in our previous design study [
In conclusion, we found that using a stakeholder-informed user design process [
Adolescents and young adults
Internet protocol
Patient Health Questionnaire-9
Patient Health Questionnaire-9 modified for adolescents
Positive Youth Development-Very Short Form
Screen for Child Anxiety Related Emotional Disorders, 5-item version
Supporting Our Valued Adolescents
System Usability Scale
AR was supported by an institutional career development award during this study (AHRQ PCOR K12 HS 22989-1) and is currently on a second career development award (NIMH 1K23MH111922-01A1). This research was also supported by the National Institutes of Health through Grant Number UL1TR000005AR and, in part, by Children’s Hospital of Pittsburgh of the UPMC Health System.
The authors thank Yves Patrick Tagasi Fotso, Siddharth Coontoor, Prashanth Kumar Ravindragopal, and Dr Michael Spring for technical assistance with measuring usability and website development. The authors thank Dr Bruce Rollman and Dr Bea Belnap and their research team with technical and study design assistance. The authors thank Alexandra DeMand, Laura Liuyi Lin, and Shannon Donofy and also former social work students and graduates Kara Mackinson Peters, Amie Ditomasso, Katrina Keane, Kaitlin Glover, Danielle Washington, and Anna Rastatter for assistance in blog writing and site moderation. The authors thank and acknowledge the SOVA community members and stakeholders for informing this study and making it possible.
AR, TG, BS, and EM contributed to study design. AR and TG collected the data. AR, TG, CL, and JW contributed to the analysis. All authors contributed to interpretation of the data. AR wrote the manuscript with guidance from the other authors. All authors approved the final version of the manuscript.
None declared.