Background: COVID-19 and its public health response are having a profound effect on people’s mental health. To provide support during these times, Canada’s largest mental health and addiction teaching hospital (Centre for Addiction and Mental Health [CAMH]) launched the Mental Health and COVID-19 Pandemic website on March 18, 2020. This website was designed to be a nonstigmatizing psychoeducational resource for people experiencing mild to moderate distress due to COVID-19 and the public health response to the pandemic.
Objective: The aim of this study was to examine the reach, usability, and user perceptions of the CAMH Mental Health and COVID-19 Pandemic website.
Methods: This study used a mixed methods sequential explanatory design approach, which consisted of the following 2 distinct phases: (1) quantitative data collection and analysis and (2) qualitative semistructured interviews. In phase 1, we analyzed Google Analytics data to understand how many people visited the website and which were the most visited pages. We conducted a survey to identify users’ sociodemographic backgrounds, and assess the usability of the website using the System Usability Scale and users’ subjective stress levels using the Perceived Stress Scale (PSS-10). For phase 2, we conducted semistructured interviews to explore user experiences; user motivation, engagement, satisfaction, and perception of the stress reduction strategies; reflections of the website’s functionality, ease of use, navigation, and design; and recommendations for improvement.
Results: Google Analytics results showed 146,978 unique users from June 2020 to March 2021. Most users were from Canada (130,066, 88.5%). Between February 20, 2021, and June 4, 2021, 152 users completed the survey. Most users identified as white, female, and having at least a college degree. Based on the PSS-10 scores, most participants were experiencing moderate to high stress when they visited the website. Users rated the usability of the website as acceptable. Ten users completed in-depth interviews between May 2021 and June 2021. Positive feedback related to the content was that the website was a trustworthy source of mental health information with helpful evidence-based stress reduction strategies. Areas for improvement included the text heavy design of the website, wider dissemination/marketing, and greater accessibility of the website to meet the needs of diverse populations.
Conclusions: Adding stress reduction resources to a website from a well-respected institution may be a practical method to increase awareness and access to evidence-based stress reduction resources during times of crisis, where there is severe disruption to usual health care contacts. Efforts to ensure that these resources are more widely accessed, especially by diverse populations, are needed.
The COVID-19 pandemic and its public health response are having a profound effect on all aspects of society, including mental health . Introductions of public health measures, such as quarantine, travel restrictions, and physical closure of places of learning and businesses, were part of the response to reduce the spread of COVID-19 [ - ]. Quarantine, social isolation, and uncertainty surrounding the pandemic are known to negatively impact mental health [ ]. For example, one study of individuals quarantined during the 2003 SARS pandemic in Toronto, Ontario, found that 31% of the sample developed posttraumatic stress disorder symptoms, and 29% demonstrated depressive symptom severity similar to that of a patient with clinically diagnosed depression [ ].
In response to the disruption of in-person care during the current COVID-19 pandemic, health systems rapidly adopted virtual care options . Where possible, care was delivered virtually by telephone or through video-conferencing platforms and web applications, which was especially true and possible with mental health conditions [ ]. Not everyone needed or wanted in-person clinical services, but rather wanted accessible trustworthy psychoeducation to manage the uncertainty-related stress and anxiety associated with the disease itself and the impacts on their lives [ , ]. This led to a concern that in the absence of credible Canadian sources, most would turn to uncurated information through social media [ - ]. In Canada, the capacity of mental health and addiction services in the province of Ontario was also of concern [ ]. Even before the pandemic, Ontario’s infrastructure for mental health and addiction services was overburdened and underfunded [ ]. Thus, expecting this system to respond to people with mild to moderate distress was not deemed to be appropriate.
Given the growing evidence suggesting that positive outcomes can be achieved through the use of digital mental health technologies [- ], on March 18, 2020 (the same month the World Health Organization declared COVID-19 a pandemic), the Centre for Addiction and Mental Health (CAMH), Toronto, Canada [ ] launched a COVID-19 resource page on the CAMH website [ ]. The website was designed to be a digital, plain language, nonstigmatizing psychoeducational resource for Canadians experiencing mild to moderate distress due to COVID-19. Stress was assumed to be universal due to the uncertainty associated with an evolving pandemic. The response would affect those with and without pre-existing mental illness to varying degrees, with the majority of the population experiencing mild to moderate distress, but in the absence of resources, this could overwhelm the health care system that was focusing on containing COVID-19. The website provides practical problem-solving tips based on cognitive behavioral therapy in both English and French. Specifically, it contains (1) a moderated discussion board, (2) written content on managing mental health during COVID-19, (3) self-assessments for anxiety using the Generalized Anxiety Disorder-7 (GAD-7) scale and perceived stress and resilience using the Perceived Stress Scale (PSS), and (4) a series of answers to frequently asked questions (FAQs) by the lay public on specific topics including intimate partner violence and grief. Those with diagnosable anxiety and depression might have more severe stress responses, which in turn hampers engagement with digital mental health interventions [ ]. The lead developers (PS and RD) felt that these individuals should be directed to seek clinical services either in-person or virtually because they might need more intensive services.
The rapid development of digital strategies to minimize the negative impact of COVID-19 on public mental health has outpaced the evaluation of digital health interventions [- ], creating a gap in our understanding of the utility and quality of the interventions [ ]. In this study, we report on the reach, usability, and user perceptions of the CAMH COVID-19 website.
This study used a mixed methods sequential explanatory design approach, which consisted of the following 2 distinct phases: (1) quantitative data collection and analysis and (2) qualitative semistructured interviews. To report this study, we adhered to the guidelines for Good Reporting of A Mixed Methods Study (GRAMMS) .
Recruitment and Data Collection
Phase 1: Reach (Google Analytics) and Usability (Self-administered Survey)
In phase 1, Google Analytics data were used to determine the number of unique users, the countries and cities associated with the users, what pages they visited and in what order, and the channels (eg, Twitter and Facebook) that drove the most traffic. We collected Google Analytics data from June 2020 to March 2021.
In order to obtain additional descriptive information on who uses CAMH’s COVID-19 website tools, we placed an advertisement for this study on the website and sent a tweet letting people know. Tweets were also sent from the organizational accounts, @CAMHnews and @PSQuitSmoking, asking users/readers if they were interested in participating in the study. After consenting, participants completed a survey via Research Electronic Data Capture (REDCap), which included questions about sociodemographics; the PSS-10, a 10-item tool scored on a 5-point Likert scale ranging from 0 (never) to 4 (very often), which assesses subjective stress levels ; and the System Usability Scale (SUS) for assessing perceived usability of applications and services [ , ]. The SUS is a 10-item questionnaire containing response options ranging from strongly agree to strongly disagree. The decision to use the SUS tool for measuring usability was based on its validity to effectively distinguish between usable and nonusable systems [ ]. The SUS can be easily administered to participants and has been shown to yield reliable results, even with small sample sizes [ ]. Once the participants completed the survey, they were redirected to a new page via REDCap to provide their contact details to enter into a draw for a gift card. This was optional, and participants were able to choose not to enter the draw. Participants were also asked about their interest in participating in an interview and about consent for future communication (both optional), and contact information (email and phone number) was collected from those who were interested. Participants were informed that information collected from the survey might be used by CAMH for future research, in which case, the data were deidentified and only aggregated data were reported. We collected survey data from February 20, 2021, to June 4, 2021.
Phase 2: User Perceptions (Semistructured Interviews)
To explore and describe participant perceptions of the CAMH COVID-19 website, semistructured interviews were conducted from May 25, 2021, to June 29, 2021. Of the 152 participants who completed the CAMH COVID-19 website evaluation survey, 53 indicated that they were interested in being interviewed. All 53 participants were eligible to be interviewed as they visited the CAMH COVID-19 website and were over the age of 18 years. They were contacted by email/telephone, and a consent discussion was scheduled with individuals who agreed to proceed with an interview. Once participants consented and signed the informed consent using REDCap, they were booked for a 30-minute semistructured interview over the phone or a video-conferencing system. All interviews were audio recorded and transcribed verbatim. Ten interviews were conducted, and the data were assessed as adequate to permit deep case-oriented analysis that resulted in an in-depth understanding of users’ experiences with the website . Several attempts were made to reach and recruit a heterogeneous sample of users for interviews. However, those who consented to participate were primarily college educated, white, and female. All participants who consented to an interview were from Ontario; thus, our sample was not representative of the overall population of Canada.
After updates were made, qualitative semistructured interviews were then used to further elaborate on the usability of the website. Usability was defined as the degree to which a program can be used easily, efficiently, and with satisfaction . Specifically, the interview topic guide explored participants’ experiences with the website, including their motivation, engagement, satisfaction, or dissatisfaction, and their perceived experiences on whether the website helped to reduce their stress. It also explored participants’ reflections on the CAMH COVID-19 website’s functionality and ease of use/navigation, the overall look/design, and how the intervention could be improved.
Timeline of Project Activities
provides a timeline of project activities. The website was launched on March 18, 2020. Content was added over the year including making the website available offline through a downloadable app available on the App Store for Apple devices and Google Play Store for Android devices. Topics were added, based on the evolution of the pandemic and perceived needs to be addressed, by the planning committee (eg, parenting, grief, trauma and loss, parent support, and back to school). On April 23, 2021, based on preliminary feedback from this study and a review of the website by a trained behavioral psychologist and professional designer, the website was updated to improve navigation, eliminate duplication, and include additional content (eg, support for caregivers), based on what was going on at the time.
Phase 1: Google Analytics and Self-administered Survey
Google Analytics data collected from June 2020 to March 2021 were exported to Microsoft Excel, which we used to conduct descriptive statistics. We calculated means, standard deviations, and medians where applicable.
Quantitative data from the survey were analyzed using SPSS (version 25; IBM Corp). The scoring guidelines provided by the US Department of Health and Human Services were used to inform the analysis of the SUS questionnaire in the survey . A SUS score between 0 and 100 was generated for each participant by converting the score for each question into a new number, summing the converted scores, and multiplying the score by 2.5. Descriptive statistics were used to report sociodemographic characteristics, usability, and engagement. The PSS-10 was analyzed by first reversing scores for 4 of the positive items on the scale and then summing across all 10 items to receive a score between 0 and 40 for each participant.
Phase 2: Semistructured Interviews
Qualitative data analysis for the study involved an iterative team-based process. Transcribed interviews were entered into NVivo 12 (QSR International) for qualitative data management and analysis. Two research staff read interview transcripts multiple times to achieve immersion prior to code development. The author AG crafted the initial codebook, and then, AG and AS jointly coded the first 3 transcripts to refine the codebook and defined the codes through consensus. Two other transcripts were coded independently by AG and AS, and the calculated intercoder reliability was assessed using the kappa coefficient in NVivo 12. There was 93% agreement on 96% of codes . The remaining transcripts were independently coded either by AG or AS. Data were analyzed using thematic analysis [ ]. A deductive approach was used to identify the coding scheme for the transcripts that allowed for the development of codes corresponding to the components of usability and stress management.
We merged both the quantitative and qualitative analysis results to provide a more comprehensive overview of participants’ perceptions of the CAMH COVID-19 website.
This study, including the recruitment and consent process, received ethics approval from the CAMH Quality Project Ethics Review Board (QPER #: 2020_34) .
Phase 1: Google Analytics and Self-administered Survey
From June 2020 to March 2021, there were 146,978 unique users (unduplicated/counted only once) of the website. Of the 146,978 users from all over the world, most were from Canada (130,066, 88.5%), and of these, 30,964 (23.8%) were from Toronto and 6538 (5.0%) were from Ottawa. The homepage was the most visited page of the website, which had 91,866 unique page views during the 10 months, followed by the page on “Coping with Stress and Anxiety,” which had 29,889 unique page views during the 10 months [, ]. Organic search traffic (ie, entry through a search engine) accounted for the highest proportion of user visits to the website. From June 2020 to March 2021, 108,332 users (73.7% of all unique visitors) visited the site through organic search and 36,630 users (24.9% of all unique visitors) visited the site by typing the URL directly into their address bar.
Characteristics of Survey and Interview Participants
Survey responses are shown in. We have divided participant responses into 3 columns. The first column includes participants who responded between February 20, 2021, and April 23, 2021 (before the website was updated); the second includes participants who responded between April 24, 2021, and June 4, 2021 (after the website was updated); and the third includes participants we interviewed.
|Variable||Survey participants (February 20-April 23, 2021) (N=101a)||Survey participants (April 24-June 4, 2021) (N=51b)||Interview participants (May-June 2021) (N=10)|
|Race/ethnicity, n (%)|
|White||76 (75.3)||36 (70.6)||5 (50.0)|
|Latin American||5 (5.0)||1 (2.0)||1 (10.0)|
|Mixed (eg, African and White)||2 (2.0)||2 (3.9)||1 (10.0)|
|East/South/South East Asian||6 (5.9)||6 (11.8)||2 (20.0)|
|First Nations/Indigenous||0 (0.0)||1 (2.0)||0 (0.0)|
|Black||3 (3.0)||1 (2.0)||0 (0.0)|
|Indian–Caribbean||1 (1.0)||0 (0.0)||1 (10.0)|
|Did not answer||8 (7.9)||4 (7.8)||0 (0.0)|
|Age (years), mean (SD)||44 (18)||45 (18)||45 (15)|
|Gender, n (%)|
|Female||86 (85.1)||39 (76.5)||10 (100.0)|
|Male||10 (9.9)||7 (13.7)||0 (0.0)|
|Nonbinary||0 (0.0)||4 (7.8)||0 (0.0)|
|Unknown/did not know/did not answer||5 (5.0)||1 (2.0)||0 (0.0)|
|Education, n (%)|
|Graduate degree||16 (15.8)||17 (33.3)||7 (70.0)|
|University degree||33 (32.7)||9 (17.6)||1 (10.0)|
|College diploma||24 (23.8)||12 (23.5)||1 (10.0)|
|Trade/vocational school||6 (5.9)||5 (9.8)||1 (10.0)|
|High school||18 (17.8)||7 (13.7)||0 (0.0)|
|Elementary school||3 (3.0)||0 (0.0)||0 (0.0)|
|Did not answer||1 (1.0)||1 (2.0)||0 (0.0)|
|Income (CAD $)c, n (%)|
|$0-$29,999||19 (18.8)||7 (13.7)||2 (20.0)|
|$30,000-$59,999||19 (18.8)||12 (23.5)||0 (0.0)|
|$60,000-$89,999||24 (13.9)||5 (9.8)||2 (20.0)|
|$90,000-$119,999||12 (11.9)||7 (13.7)||3 (30.0)|
|$120,000-$149,999||5 (5.0)||2 (3.9)||0 (0.0)|
|$150,000 or more||15 (14.9)||9 (17.6)||2 (20.0)|
|Do not know/prefer not to disclose||17 (16.8)||9 (17.6)||1 (10.0)|
|Marital status, n (%)|
|Married/in a relationship||47 (46.5)||27 (54.0)||7 (70.0)|
|Divorced/separated||14 (13.9)||2 (4.0)||1 (10.0)|
|Single/never married||37 (36.6)||17 (34.0)||1 (10.0)|
|Widowed||2 (2.0)||2 (4.0)||1 (10.0)|
|Do not know/did not answer||1 (1.0)||2 (4.0)||0 (0.0)|
|Level of perceived stress (Perceived Stress Scale score range), n (%)|
|Low stress (0-13)||8 (9.0)||6 (12.5)||0 (0.0)|
|Moderate stress (14-26)||44 (49.4)||25 (52.1)||6 (60.0)|
|High perceived stress (27-40)||37 (41.6)||17 (35.4)||4 (40.0)|
|System Usability Scale, n (%)|
|Above average (greater than 68)||42 (48.3)||28 (58.3)||7 (70.0)|
|Below average (lower than 68)||45 (51.7)||20 (41.7)||3 (30.0)|
aFor the variables age, level of perceived stress, and System Usability Scale finding, the N values were 99, 89, and 87, respectively.
bFor the variables age, marital status, level of perceived stress, and System Usability Scale finding, the N values were 47, 50, 48, and 48, respectively.
cA currency exchange rate of CAD $1=US $0.77 is applicable.
In total, 152 participants completed the survey. Most participants identified as white and female, and had at least a college degree. Based on the PSS-10 scores, most participants were experiencing moderate to high stress when they visited the website.
Survey data from February 20, 2021, to April 23, 2021 (before the website was updated) indicated that participants reported a mean SUS score of 66.44 (SD 16.72; IQR 10-97.5; N=87). According to the SUS, this score was below average for a website. Specific SUS items that received a low score included the website being well-integrated (41/87, 47% agreed or strongly agreed) and the likelihood of using the website frequently (34/87, 39% agreed or strongly agreed).
After changes to the website were implemented, survey data from April 24, 2021, to June 4, 2021, indicated that the participants reported a mean SUS score of 70 (SD 17.02; IQR 22.5-100; N=48). According to the SUS, this score was above average for a website. Improved scores were reported for the website being well-integrated (31/48, 65% agreed or strongly agreed) and the likelihood of using the website frequently (26/48, 55% agreed or strongly agreed).
Phase 2: Interview Results
Background on Participants Interviewed
Interview participants (n=10) were experiencing moderate to high levels of stress at the time they visited the website. Four participants had a history of mental illness or had a family member with a mental illness who received services from CAMH. Of 2 participants, whose family members received services from CAMH, one was an essential worker and the other a retired health care worker looking for mental health resources. The 2 participants with a diagnosed mental illness had recently contracted and recovered from COVID-19. Both stated that they were frequent users of the general CAMH website and visited the CAMH COVID-19 page for COVID-19–related information. Five other participants were researchers or professionals in a health-related field working with individuals who required mental health services. One participant was a stay-at-home parent with young children.
Functionality, Ease of Use, and Design
Most participants who were interviewed reported having visited the CAMH COVID-19 website once or twice, and having gone there to look for resources on how to deal with stress (either for themselves or for others). All participants (n=10) shared that their first impression of the website was positive, and that there were many helpful resources. Generally, participants found the website visually appealing, and liked the diversity in the images.
I liked the overall design, sort of the esthetic of the website. So, I found all of the photos that are in there to be quite diverse, in terms of the way that people look and the sort of different activities that they’re doing. They all seem very relatable….
Several participants (n=5) reported that the information was helpful, but it was not well organized. Two participants specifically mentioned feeling overwhelmed by the amount of information on the website.
Well, my immediate impressions were that it was really overwhelming. … There’s so many options…. Great…I mean it’s great to have so many options, but because of the pandemic, I think people are already overwhelmed… There’s a lot of good stuff there.
Another common theme, mentioned by 4 participants, was that the information might not be accessible to many users who might benefit from the information on the website.
My parents are refugees, like a lot of these words would be difficult for them to understand…like stigma and prejudice…I don’t know that those words mean anything to them. Maybe making things a bit more layman’s terms. Like if you want to just, kind of just speak to the masses, versus people that are in a clinical setting. Or have a background in mental health and social work.
In addition, 3 participants mentioned that there was a need to disseminate the website to those who might need it the most.
My partner, for example, he’s dealt with mental illness and homelessness for over ten years, and he didn’t know that this website existed even.
The Website and Stress
When asked specifically about the website as a stress reduction strategy for people in their province, all participants (n=10) responded that the website had the potential to reduce stress for individuals seeking help with stress-related issues. All participants were from Ontario. Participants noted that CAMH is a trusted authority where people can access evidence-based resources without judgement.
Well, I think it’s great that, I think it’s great that you, you know, that you’re a trusted authority. And I think it’s great…I think it’s really wonderful that all…that this website has been put up. Because, you know, I mean people need, need to get stuff from a trusted place, you know. And a place where they won’t…and where they won’t be judged, which is another huge thing is judgment, right.
I think, I really liked that you can take the tests. You know the…you know like I’m familiar with the GAD7 just because I’m a Psychologist....Like, it’s just nice to see that, you know there’s evidence-based research supported measures...
Consistent with the Google Analytics data, participants mentioned that the section they visited the most was “Stress and Anxiety.” Health professionals (n=3) indicated that the self-assessment resources found on this page would be helpful to their clients.
And, I think, when I went to the website, there were like different advice kind of little columns that you can read about how to like stay active and how to decrease your stress and your burnout and whatnot. And then there’s like different little tests you can take. And so, I thought that was…it was just really nicely laid out. Yeah. For…I mean, not for myself, like I already, you know, I’m kind of like at the point where I’m kind of feeling anxious, as given my profession, but for clients and whatnot…I think it’s helpful for them...
A participant, who was single, mentioned that she found the stress resources needed and was able to employ the recommended stress and coping strategies outlined on the website. This was in direct contrast to what parents said, as noted below.
Well, it was very good. Because it tells you, you know you need to be…keep yourself busy. You need to enjoy things like, you know hobbies…. Yes. Definitely. And I think for people who, you know like myself, who lives alone. So, I read that…I read it basically to give me information as a single person what is going on.
The need for information tailored to parents was mentioned by 2 participants as follows:
I think if you had a section for parents and that’s something I think that would be helpful for sure. When you have kids it’s like, you, you are responsible for other people than yourself. And it’s just so much harder, because there is no break …so it was something that was useful, but for me in my situation, it wasn’t quite enough, just because, you know you can’t really control your children. So, like there’s only so much that, so much that’s limited to you having control over. Right. …so the coping strategies, I think, are great, if you are a single person with no other people dependent on you.
Participants with higher levels of stress or those who described their situation as more complex noted that the stress management strategies on the website were good for “normal stress,” but that these strategies were not able to address the specific mental health needs during the COVID-19 pandemic.
I think that the stress from COVID-19, is just very…it’s very unique, in the fact that like obviously none of us have really been through a pandemic like this. But I think normal coping mechanisms may not work as well, just because the nature of COVID. …you can’t get out of the pandemic. There’s no escaping it. So, I feel like the stress management strategies were very good stress management strategies for if you had normal stress.
A suggestion by 3 participants was to have a feature that allows the user to connect to a live counsellor for more personalized support, in order to get immediate help. While access to a live counsellor was not available, the website included an online peer-to-peer “Discussion Forum” where moderators were available during business hours to provide general information and encourage the use of available resources. However, only 1 of the 10 interview participants was aware of the discussion forum feature on the website.
…the only thing that you could improve, it would if you could connect somebody with a live person. I think it should lead to someone, if somebody really needs to talk to someone, I think that they should be able to…. And I…believe me, I’ve tried a lot… I think that they should be able to connect to a live person. Because people who are desperate, need humanity.
I think one of the big things that I was kind of like getting to, was just that there wasn’t…there wasn’t an option to get help right away.
The goals of the study were to collect information on the reach, usability, and user perceptions of the CAMH COVID-19 website, a web page designed to be a digital, plain-language, nonstigmatizing psychoeducational resource for people experiencing mild to moderate distress due to COVID-19.
Compared with similar websites , the CAMH COVID-19 website [ ] is a highly accessed website, with 146,978 unique individuals visiting the website in a 10-month period. The majority of the 146,978 users were from Toronto and Ottawa, which is consistent with the findings of a similar study evaluating a Canadian mental health website portal [ ] and may be reflective of the increase in the percentage of Ontario residents who indicated a poor or fair mental health status during the second wave of the pandemic [ ]. The most visited section was the home page [ ], followed by the Coping with Stress and Anxiety [ ] section, which shares different coping ideas and strategies, and includes various self-assessment tests that help users understand/determine the stress levels and anxiety they are experiencing, such as the PSS and GAD-7. Other studies have similar findings that the home page was viewed most frequently [ , ], with the second being a “self-check page, on which users could complete a depression screening tool” [ ].
Users rated the usability of the website higher after changes were made on April 23, 2021. During the first period of the survey (February 20, 2021, to April 23, 2021), the average SUS score was 66.44 (SD 16.72). The SUS increased by 3.6 points after changes were made to the website (SUS score 70, SD 17.02). Both these scores correlated with an acceptable range [, , ] and were similar to results obtained in other evaluations of mental health websites [ ].
Interview participants indicated that in the first iteration of the website, they felt overwhelmed with the volume of information and text-heavy web pages. Other studies have confirmed that text-heavy interventions can make content difficult to digest, thereby acting as a barrier to use , and website design related to page complexity, navigational simplicity, and comprehensibility should be considered to “reduce the cognitive effort required to effectively use eHealth applications” [ ].
Demographic data indicated that the majority of website users were white women with higher education aged between 20 and 59 years. This is consistent with the findings of other studies that females and adults aged 18-64 years, with high levels of socioeconomic status, are more engaged, than males and other age groups, in using health content–related websites [, - ]. The website’s reach may be a reflection of recent evidence showing that women perceived a decline in their mental health in 2020 during the second wave of the pandemic and experienced worse mental health outcomes in comparison with men [ , , , ]. Moreover, the gender differences in the website’s reach may be attributed to men’s reluctance to access mental health care because the intervention is not sufficiently sensitive to masculine identity–related factors [ ]. Given the disproportionate negative impact of COVID-19 on diverse groups [ ], there is a need to reach a wider audience. Targeted strategies need to be informed by gender sensitized services [ ] and are needed to provide mental health services to communities who have been disproportionately impacted by the COVID-19 pandemic, such as older adults, indigenous people, and other racial and ethnic minority groups [ ]. However, research shows that these communities (older adults, indigenous peoples, and other racial and ethnic minority groups) are the least likely to seek web-based health information [ , , ]. Proposed strategies to increase user engagement and technology acceptance of e-mental health among diverse populations include the implementation of a human-centered design approach that integrates designers and users in the development process from beginning to end [ ].
Interview findings showed that most participants felt positively about the website, as it was a trustworthy source of mental health information and had helpful stress reduction features. Studies have confirmed that individuals seeking mental health resources online are concerned about the trustworthiness and credibility of e-mental health, as well as the effectiveness of self-help strategies [- ], and as shown by another study, a lack of trust in a single online source can contribute to health information seekers’ stress [ ].
Several interview participants wanted to connect to a live counsellor for more personalized support and immediate help. While moderators of the website’s “Discussion Forum”  offered informational support, though not live counselling and support limited to office hours, most interview participants were unaware of this feature. This may be related to the design of the website and participants’ feedback that they found the website overwhelming and text heavy. Making the discussion forum more visible might be important, especially given that research has shown that well-designed self-guided interventions can be successful without added human support and their associated costs [ - ].
The website was designed to address the needs of individuals with low to moderate levels of stress. However, the results indicated that most of the individuals visiting the website had moderate to high levels of stress. Further research is needed to understand the characteristics of individuals with low levels of stress and their low engagement with the website. Technology-enabled care coordination, which involves the use of online technology to facilitate delivery of multidisciplinary team-based care, for patients with low to moderate levels of distress has significant potential for improving population mental health . Improvements to the website are needed to address the needs of individuals with high levels of stress, since studies have shown that individuals with more severe mental health symptoms are motivated to use digital mental health interventions, but symptom severity can also form barriers to engagement [ ]. A guided approach, which includes the provision of content that is relevant and customizable according to users’ personal preferences and offers technical assistance or training, may be suited to address the mental health of individuals with higher levels of stress [ ].
Our findings demonstrate that the CAMH COVID-19 website was useful for disseminating information about evidence-based stress reduction strategies. The website is a highly accessed website with above average usability. Users found the website to be a helpful and trustworthy source of mental health information. The recommendations suggest that the design and reach of the website need to align more closely with user needs and with greater and more equitable access to mental health care. These evaluations should be conducted with an equity-oriented approach. For example, examination of who was exposed to the website (ie, reach) allowed us to understand the need to focus on equitable reach throughout any future updates. Feedback from this evaluation will be useful for further improvements to the website, which will be beneficial to both the general public and health care professionals.
Strengths and Limitations
An important strength of this project was the mixed methods design, as it allowed us to understand the complex problem of stress during the first waves of COVID-19 in Ontario and understand what users need as part of a website intervention. Another strength of this project is that based on direct feedback from users, we were able to make significant improvements to the usability of the website. Our study has several limitations. First, we had a convenient sample and a low number of respondents compared with the number of users in both our survey and interview. Thus, they may not have been representative samples, and it is possible that those who perceived to have gained benefits from the website were more likely to participate in the surveys and interviews. In addition, the small sample of people who agreed to be interviewed (all women) also limited the ability to stratify findings by demographics or other variables of interest. Second, participants were interviewed a number of weeks after having used the website and completing the survey. As such, for some, recalling the program details was difficult. Third, there was a lack of longitudinal follow-up data on the effects of the website. Our study was restricted to the reach, usability, and user perception of the website and did not evaluate clinical efficacy. Lastly, anxiety and depression were not addressed, as the developers felt that individuals with these diagnosable illnesses needed either in-person or virtual clinical services.
Adding stress reduction resources to a website from a well-respected institution may be a practical method to increase awareness and access to evidence-based stress-reduction resources during times of crisis, where there is severe disruption to usual health care contacts. Efforts are needed to ensure that these resources are more widely accessed, especially by diverse populations. Further research is required on the health impact and role of e-mental health initiatives within the mental health services system. Given the promise and significant investment in digital mental health technologies, there is a growing need to evaluate them for low-intensity psychoeducational interventions.
We thank all the people who participated in this study. We would also like to thank Freni Jivani for her support with organizing the in-application analytics data. We thank Chantal Fougere, Mathangee Lingam, Stephanie Duench, Iqra Ashfaq, Lenka Vojtila, Huma Taj, and Victoria Kim from the Nicotine Dependence Service department for their contributions to the website’s discussion board. We thank Shirley Lin and Alan Tang (infrastructure support), Rino La Grassa and Gurpal Bubbra (app), and Anthony Gallucci (web page) from IMG for technical set up; Samantha Delenardo, Bruno Marsiaj, Paris Semanksy, and Miguel Amante from Public Affairs; and Cameron Norman (https://www.cense.ca/contact/principal/) for review and update of the website.
PS conceptualized the resource for people with mild to moderate distress during a pandemic; PS and RD designed the website; PS and NM conceptualized, designed, and supervised the evaluation with input from RD, GS, and SAK. AG and AS coordinated the study recruitment and conducted the interviews. AG and AS coded and analyzed the interviews under the supervision of NM. NM, AG, and AS drafted the manuscript. All authors participated in the critical revision of the manuscript, and read and approved the final manuscript.
Conflicts of Interest
PS reports receiving grants, salary, and/or research support from the Centre for Addiction and Mental Health, Health Canada, Ontario Ministry of Health, Canadian Institutes of Health Research (CIHR), Canadian Centre on Substance Use and Addiction, Public Health Agency of Canada (PHAC), Medical Psychiatry Alliance, Canadian Cancer Society Research Institute (CCSRI), Cancer Care Ontario, and National Institutes of Health (NIH). PS also reports receiving funding and/or honoraria from the following commercial organizations: Pfizer Inc./Canada and Patient-Centered Outcomes Research Institute. Through an open-tender process Johnson & Johnson, Novartis, and Pfizer Inc are vendors of record for providing smoking cessation pharmacotherapy, free or discounted, for research studies in which PS is the principal investigator or co-investigator.
- Xiong J, Lipsitz O, Nasri F, Lui LMW, Gill H, Phan L, et al. Impact of COVID-19 pandemic on mental health in the general population: A systematic review. J Affect Disord 2020 Dec 01;277:55-64 [FREE Full text] [CrossRef] [Medline]
- Lowe C, Keown-Gerrard J, Ng CF, Gilbert TH, Ross KM. COVID-19 pandemic mental health trajectories: Patterns from a sample of Canadians primarily recruited from Alberta and Ontario. Canadian Journal of Behavioural Science-Revue Canadienne Des Sciences Du Comportement 2022 Jan 27:1665682. [CrossRef]
- Cindrich SL, Lansing JE, Brower CS, McDowell CP, Herring MP, Meyer JD. Associations Between Change in Outside Time Pre- and Post-COVID-19 Public Health Restrictions and Mental Health: Brief Research Report. Front Public Health 2021;9:619129 [FREE Full text] [CrossRef] [Medline]
- Chum A, Nielsen A, Bellows Z, Farrell E, Durette P, Banda JM, et al. Changes in Public Response Associated With Various COVID-19 Restrictions in Ontario, Canada: Observational Infoveillance Study Using Social Media Time Series Data. J Med Internet Res 2021 Aug 25;23(8):e28716 [FREE Full text] [CrossRef] [Medline]
- Shah SMA, Mohammad D, Qureshi MFH, Abbas MZ, Aleem S. Prevalence, Psychological Responses and Associated Correlates of Depression, Anxiety and Stress in a Global Population, During the Coronavirus Disease (COVID-19) Pandemic. Community Ment Health J 2021 Jan;57(1):101-110 [FREE Full text] [CrossRef] [Medline]
- Hawryluck L, Gold WL, Robinson S, Pogorski S, Galea S, Styra R. SARS control and psychological effects of quarantine, Toronto, Canada. Emerg Infect Dis 2004 Jul;10(7):1206-1212 [FREE Full text] [CrossRef] [Medline]
- Mental Health and COVID-19: Early evidence of the pandemic’s impact: Scientific brief, 2 March 2022. World Health Organization. 2022. URL: https://www.who.int/publications/i/item/WHO-2019-nCoV-Sci_Brief-Mental_health-2022.1 [accessed 2022-08-20]
- Sorkin DH, Janio EA, Eikey EV, Schneider M, Davis K, Schueller SM, et al. Rise in Use of Digital Mental Health Tools and Technologies in the United States During the COVID-19 Pandemic: Survey Study. J Med Internet Res 2021 Apr 16;23(4):e26994 [FREE Full text] [CrossRef] [Medline]
- Mahoney AEJ, Elders A, Li I, David C, Haskelberg H, Guiney H, et al. A tale of two countries: Increased uptake of digital mental health services during the COVID-19 pandemic in Australia and New Zealand. Internet Interv 2021 Sep;25:100439 [FREE Full text] [CrossRef] [Medline]
- Tsao S, Chen H, Tisseverasinghe T, Yang Y, Li L, Butt ZA. What social media told us in the time of COVID-19: a scoping review. The Lancet Digital Health 2021 Mar;3(3):e175-e194. [CrossRef]
- Htay MNN, Parial LL, Tolabing MC, Dadaczynski K, Okan O, Leung AYM, et al. Digital health literacy, online information-seeking behaviour, and satisfaction of Covid-19 information among the university students of East and South-East Asia. PLoS One 2022;17(4):e0266276 [FREE Full text] [CrossRef] [Medline]
- Su Z, McDonnell D, Wen J, Kozak M, Abbas J, Šegalo S, et al. Mental health consequences of COVID-19 media coverage: the need for effective crisis communication practices. Global Health 2021 Jan 05;17(1):4 [FREE Full text] [CrossRef] [Medline]
- Sockalingam S, Arena A, Serhal E, Mohri L, Alloo J, Crawford A. Building Provincial Mental Health Capacity in Primary Care: An Evaluation of a Project ECHO Mental Health Program. Acad Psychiatry 2018 Aug;42(4):451-457. [CrossRef] [Medline]
- Addressing the Access Gap: Leveraging the Potential of e-Mental Health in Canada. Mental Health Commission of Canada. 2017. URL: https://mentalhealthcommission.ca/resource/addressing-the-access-gap-leveraging-the-potential-of-e-mental-health-in-canada/ [accessed 2022-08-20]
- Buchanan H, Coulson NS. Accessing dental anxiety online support groups: an exploratory qualitative study of motives and experiences. Patient Educ Couns 2007 Jun;66(3):263-269. [CrossRef] [Medline]
- Meier A, Lyons EJ, Frydman G, Forlenza M, Rimer BK. How cancer survivors provide support on cancer-related Internet mailing lists. J Med Internet Res 2007 May 14;9(2):e12 [FREE Full text] [CrossRef] [Medline]
- Eysenbach G, Powell J, Englesakis M, Rizo C, Stern A. Health related virtual communities and electronic support groups: systematic review of the effects of online peer to peer interactions. BMJ 2004 May 15;328(7449):1166 [FREE Full text] [CrossRef] [Medline]
- Klemm P, Bunnell D, Cullen M, Soneji R, Gibbons P, Holecek A. Online cancer support groups: a review of the research literature. Comput Inform Nurs 2003;21(3):136-142. [CrossRef] [Medline]
- Finn J. An exploration of helping processes in an online self-help group focusing on issues of disability. Health Soc Work 1999 Aug;24(3):220-231. [CrossRef] [Medline]
- Bäuerle A, Jahre L, Teufel M, Jansen C, Musche V, Schweda A, et al. Evaluation of the E-Mental Health Mindfulness-Based and Skills-Based "CoPE It" Intervention to Reduce Psychological Distress in Times of COVID-19: Results of a Bicentre Longitudinal Study. Front Psychiatry 2021;12:768132 [FREE Full text] [CrossRef] [Medline]
- Centre for Addiction and Mental Health. URL: https://www.camh.ca/ [accessed 2022-08-20]
- Mental Health and the COVID-19 Pandemic. Centre for Addiction and Mental Health. URL: https://www.camh.ca/en/health-info/mental-health-and-covid-19 [accessed 2022-08-20]
- Borghouts J, Eikey E, Mark G, De Leon C, Schueller SM, Schneider M, et al. Barriers to and Facilitators of User Engagement With Digital Mental Health Interventions: Systematic Review. J Med Internet Res 2021 Mar 24;23(3):e24387 [FREE Full text] [CrossRef] [Medline]
- Chang BP, Kessler RC, Pincus HA, Nock MK. Digital approaches for mental health in the age of covid-19. BMJ 2020 Jun 29;369:m2541. [CrossRef] [Medline]
- Strudwick G, Sockalingam S, Kassam I, Sequeira L, Bonato S, Youssef A, et al. Digital Interventions to Support Population Mental Health in Canada During the COVID-19 Pandemic: Rapid Review. JMIR Ment Health 2021 Mar 02;8(3):e26550 [FREE Full text] [CrossRef] [Medline]
- Balcombe L, De Leo D. Digital Mental Health Challenges and the Horizon Ahead for Solutions. JMIR Ment Health 2021 Mar 29;8(3):e26811 [FREE Full text] [CrossRef] [Medline]
- Balcombe L, De Leo D. An Integrated Blueprint for Digital Mental Health Services Amidst COVID-19. JMIR Ment Health 2020 Jul 22;7(7):e21718 [FREE Full text] [CrossRef] [Medline]
- O'Cathain A, Murphy E, Nicholl J. The quality of mixed methods studies in health services research. J Health Serv Res Policy 2008 Apr;13(2):92-98. [CrossRef] [Medline]
- Cohen S, Kamarck T, Mermelstein R. A global measure of perceived stress. J Health Soc Behav 1983 Dec;24(4):385-396. [Medline]
- System Usability Scale (SUS). Usability.Gov. URL: https://www.usability.gov/how-to-and-tools/methods/system-usability-scale.html [accessed 2022-08-20]
- Chin WSY, Kurowski A, Gore R, Chen G, Punnett L, SHIFT Research Team. Use of a Mobile App for the Process Evaluation of an Intervention in Health Care: Development and Usability Study. JMIR Form Res 2021 Oct 28;5(10):e20739 [FREE Full text] [CrossRef] [Medline]
- Sandelowski M. Sample size in qualitative research. Res Nurs Health 1995 Apr;18(2):179-183. [CrossRef] [Medline]
- ISO 9241-420:2011 Ergonomics of human-system interaction — Part 420: Selection of physical input devices. ISO. URL: https://www.iso.org/standard/52938.html [accessed 2022-08-20]
- O’Connor C, Joffe H. Intercoder Reliability in Qualitative Research: Debates and Practical Guidelines. International Journal of Qualitative Methods 2020 Jan 22;19:160940691989922. [CrossRef]
- Braun V, Clarke V. Using thematic analysis in psychology. Qualitative Research in Psychology 2006 Jan;3(2):77-101. [CrossRef]
- Research Ethics. Centre for Addiction and Mental Health. URL: https://www.camh.ca/en/science-and-research/research-ethics [accessed 2022-08-20]
- Coping with stress and anxiety. Centre for Addiction and Mental Health. URL: https://www.camh.ca/en/health-info/mental-health-and-covid-19/coping-with-stress-and-anxiety [accessed 2022-08-20]
- Dirmaier J, Liebherz S, Sänger S, Härter M, Tlach L. Psychenet.de: Development and process evaluation of an e-mental health portal. Inform Health Soc Care 2016;41(3):267-285. [CrossRef] [Medline]
- Jeong D, Cheng M, St-Jean M, Jalali A. Evaluation of eMentalHealth.ca, a Canadian Mental Health Website Portal: Mixed Methods Assessment. JMIR Ment Health 2019 Sep 06;6(9):e13639 [FREE Full text] [CrossRef] [Medline]
- Self-perceived mental health and mental health care needs during the COVID-19 pandemic. Statistics Canada. URL: https://www150.statcan.gc.ca/n1/pub/45-28-0001/2021001/article/00031-eng.htm [accessed 2022-08-20]
- Ogrodniczuk JS, Beharry J, Oliffe JL. An Evaluation of 5-Year Web Analytics for HeadsUpGuys: A Men's Depression E-Mental Health Resource. Am J Mens Health 2021 Dec 03;15(6):15579883211063322 [FREE Full text] [CrossRef] [Medline]
- King K, Schlichthorst M, Turnure J, Phelps A, Spittal MJ, Pirkis J. Evaluating the effectiveness of a website about masculinity and suicide to prompt help-seeking. Health Promot J Austr 2019 Sep;30(3):381-389. [CrossRef] [Medline]
- Bangor A, Kortum P, Miller J. Determining what individual SUS scores mean: adding an adjective rating scale. Journal of Usability Studies 2009;4(3):114-123 [FREE Full text]
- Mol M, van Schaik A, Dozeman E, Ruwaard J, Vis C, Ebert DD, Ince, et al. Dimensionality of the system usability scale among professionals using internet-based interventions for depression: a confirmatory factor analysis. BMC Psychiatry 2020 May 12;20(1):218 [FREE Full text] [CrossRef] [Medline]
- Radovic A, Gmelin T, Hua J, Long C, Stein BD, Miller E. Supporting Our Valued Adolescents (SOVA), a Social Media Website for Adolescents with Depression and/or Anxiety: Technological Feasibility, Usability, and Acceptability Study. JMIR Ment Health 2018 Feb 26;5(1):e17 [FREE Full text] [CrossRef] [Medline]
- Eccles H, Nannarone M, Lashewicz B, Attridge M, Marchand A, Aiken A, et al. Barriers to the Use of Web-Based Mental Health Programs for Preventing Depression: Qualitative Study. JMIR Form Res 2021 Jul 15;5(7):e16949 [FREE Full text] [CrossRef] [Medline]
- Rotondi AJ, Spring MR, Hanusa BH, Eack SM, Haas GL. Designing eHealth Applications to Reduce Cognitive Effort for Persons With Severe Mental Illness: Page Complexity, Navigation Simplicity, and Comprehensibility. JMIR Hum Factors 2017 Jan 05;4(1):e1 [FREE Full text] [CrossRef] [Medline]
- Baumann E, Czerwinski F, Reifegerste D. Gender-Specific Determinants and Patterns of Online Health Information Seeking: Results From a Representative German Health Survey. J Med Internet Res 2017 Apr 04;19(4):e92 [FREE Full text] [CrossRef] [Medline]
- Bidmon S, Terlutter R. Gender Differences in Searching for Health Information on the Internet and the Virtual Patient-Physician Relationship in Germany: Exploratory Results on How Men and Women Differ and Why. J Med Internet Res 2015 Jun 22;17(6):e156 [FREE Full text] [CrossRef] [Medline]
- Gallagher S, Doherty DT. Searching for health information online: characteristics of online health seekers. J Evid Based Med 2009 May;2(2):99-106. [CrossRef] [Medline]
- Escoffery C. Gender Similarities and Differences for e-Health Behaviors Among U.S. Adults. Telemed J E Health 2018 May;24(5):335-343. [CrossRef] [Medline]
- Yoon H, Jang Y, Vaughan PW, Garcia M. Older Adults' Internet Use for Health Information: Digital Divide by Race/Ethnicity and Socioeconomic Status. J Appl Gerontol 2020 Jan;39(1):105-110. [CrossRef] [Medline]
- Gandrakota N, Ali MK, Shah MK. Trends in Health Information Technology Use Among the US Population With and Without Cardiovascular Risk Factors, 2012-2018: Evidence From the National Health Interview Survey. JMIR Public Health Surveill 2021 Sep 30;7(9):e29990 [FREE Full text] [CrossRef] [Medline]
- Mitchell UA, Chebli PG, Ruggiero L, Muramatsu N. The Digital Divide in Health-Related Technology Use: The Significance of Race/Ethnicity. Gerontologist 2019 Jan 09;59(1):6-14. [CrossRef] [Medline]
- Anxiety, feelings of depression and loneliness among Canadians spikes to highest levels since spring 2020. Centre for Addiction and Mental Health. 2022. URL: https://www.camh.ca/en/camh-news-and-stories/anxiety-depression-loneliness-among-canadians-spikes-to-highest-levels [accessed 2022-08-20]
- Rice SM, Oliffe JL, Kealy D, Seidler ZE, Ogrodniczuk JS. Men's Help-Seeking for Depression: Attitudinal and Structural Barriers in Symptomatic Men. J Prim Care Community Health 2020;11:2150132720921686 [FREE Full text] [CrossRef] [Medline]
- Oliffe JL, Rice S, Kelly MT, Ogrodniczuk JS, Broom A, Robertson S, et al. A mixed-methods study of the health-related masculine values among young Canadian men. Psychology of Men & Masculinities 2019 Jul;20(3):310-323. [CrossRef]
- Jenkins EK, McAuliffe C, Hirani S, Richardson C, Thomson KC, McGuinness L, et al. A portrait of the early and differential mental health impacts of the COVID-19 pandemic in Canada: Findings from the first wave of a nationally representative cross-sectional survey. Prev Med 2021 Apr;145:106333 [FREE Full text] [CrossRef] [Medline]
- Vial S, Boudhraâ S, Dumont M. Human-Centered Design Approaches in Digital Mental Health Interventions: Exploratory Mapping Review. JMIR Ment Health 2022 Jun 07;9(6):e35591 [FREE Full text] [CrossRef] [Medline]
- Christensen H. E-Mental Health in Australia: Implications of the Internet and Related Technologies for Policy. Canberra, Australia: Commonwealth Department of Health and Ageing; 2002.
- Moock J. Support from the Internet for Individuals with Mental Disorders: Advantages and Disadvantages of e-Mental Health Service Delivery. Front Public Health 2014;2:65 [FREE Full text] [CrossRef] [Medline]
- Lattie EG, Adkins EC, Winquist N, Stiles-Shields C, Wafford QE, Graham AK. Digital Mental Health Interventions for Depression, Anxiety, and Enhancement of Psychological Well-Being Among College Students: Systematic Review. J Med Internet Res 2019 Jul 22;21(7):e12869 [FREE Full text] [CrossRef] [Medline]
- Choudhury A, Asan O, Alelyani T. Exploring the Role of the Internet, Care Quality and Communication in Shaping Mental Health: Analysis of the Health Information National Trends Survey. IEEE J Biomed Health Inform 2022 Jan;26(1):468-477. [CrossRef] [Medline]
- Coronavirus 2019 (COVID-19) Discussion Forum. Centre for Addiction and Mental Health. URL: https://covid19.camhx.ca/mod/forum/view.php?id=1 [accessed 2022-08-20]
- Renfrew ME, Morton DP, Northcote M, Morton JK, Hinze JS, Przybylko G. Participant Perceptions of Facilitators and Barriers to Adherence in a Digital Mental Health Intervention for a Nonclinical Cohort: Content Analysis. J Med Internet Res 2021 Apr 14;23(4):e25358 [FREE Full text] [CrossRef] [Medline]
- Shim M, Mahaffey B, Bleidistel M, Gonzalez A. A scoping review of human-support factors in the context of Internet-based psychological interventions (IPIs) for depression and anxiety disorders. Clin Psychol Rev 2017 Nov;57:129-140. [CrossRef] [Medline]
- Baumeister H, Reichler L, Munzinger M, Lin J. The impact of guidance on Internet-based mental health interventions — A systematic review. Internet Interventions 2014 Oct;1(4):205-215. [CrossRef]
- Heber E, Ebert DD, Lehr D, Cuijpers P, Berking M, Nobis S, et al. The Benefit of Web- and Computer-Based Interventions for Stress: A Systematic Review and Meta-Analysis. J Med Internet Res 2017 Feb 17;19(2):e32 [FREE Full text] [CrossRef] [Medline]
- Iorfino F, Occhipinti J, Skinner A, Davenport T, Rowe S, Prodan A, et al. The Impact of Technology-Enabled Care Coordination in a Complex Mental Health System: A Local System Dynamics Model. J Med Internet Res 2021 Jun 30;23(6):e25331 [FREE Full text] [CrossRef] [Medline]
|CAMH: Centre for Addiction and Mental Health|
|GAD-7: Generalized Anxiety Disorder-7|
|PSS-10: Perceived Stress Scale-10|
|REDCap: Research Electronic Data Capture|
|SUS: System Usability Scale|
Edited by J Torous; submitted 27.05.22; peer-reviewed by R Poss-Doering, L Balcombe; comments to author 01.07.22; revised version received 05.08.22; accepted 07.08.22; published 26.08.22Copyright
©Nadia Minian, Allison Gayapersad, Anika Saiva, Rosa Dragonetti, Sean A Kidd, Gillian Strudwick, Peter Selby. Originally published in JMIR Mental Health (https://mental.jmir.org), 26.08.2022.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Mental Health, is properly cited. The complete bibliographic information, a link to the original publication on https://mental.jmir.org/, as well as this copyright and license information must be included.