Maintenance Notice

Due to necessary scheduled maintenance, the JMIR Publications website will be unavailable from Wednesday, July 01, 2020 at 8:00 PM to 10:00 PM EST. We apologize in advance for any inconvenience this may cause you.

Who will be affected?


Citing this Article

Right click to copy or hit: ctrl+c (cmd+c on mac)

Published on 04.08.17 in Vol 4, No 3 (2017): Jul-Sept

This paper is in the following e-collection/theme issue:

    Original Paper

    Development and Evaluation of Digital Game-Based Training for Managers to Promote Employee Mental Health and Reduce Mental Illness Stigma at Work: Quasi-Experimental Study of Program Effectiveness

    1Department of Medical Informatics, Biometry and Epidemiology, Chair for Public Health and Health Services Research, Research Unit for Biopsychosocial Health, Ludwig-Maximilians-University Munich, Munich, Germany

    2HR EHS, Department of Psychosocial Health and Well-being, Siemens AG, Munich, Germany

    3Department of Medical Informatics, Biometry and Epidemiology-IBE, Chair for Public Health and Health Services Research, Research Unit for Biopsychosocial Health, Ludwig-Maximilians-University Munich, Munich, Germany

    4Institute and Outpatient Clinic for Occupational, Social and Environmental Medicine, WHO Collaborating Centre for Occupational Health, Clinic of Ludwig-Maximilians-University Munich, Munich, Germany

    Corresponding Author:

    Sabine Elisabeth Hanisch, MSc

    Department of Medical Informatics, Biometry and Epidemiology

    Chair for Public Health and Health Services Research, Research Unit for Biopsychosocial Health

    Ludwig-Maximilians-University Munich

    Ebene K U1

    Marchioninistr. 17

    Munich, 81377


    Phone: 49 17677504067

    Fax:49 8945361175



    Background: To counteract the negative impact of mental health problems on business, organizations are increasingly investing in mental health intervention measures. However, those services are often underused, which, to a great extent, can be attributed to fear of stigmatization. Nevertheless, so far only a few workplace interventions have specifically targeted stigma, and evidence on their effectiveness is limited.

    Objective: The objective of this study was to develop and evaluate a digital game-based training program for managers to promote employee mental health and reduce mental illness stigma at work.

    Methods: We describe the empirical development of Leadership Training in Mental Health Promotion (LMHP), a digital game-based training program for leaders. A 1-group pre-post design and a 3-month follow-up were used for training evaluation. We applied multilevel growth models to investigate change over time in the dependent variables knowledge, attitudes, self-efficacy, and intentions to promote employee mental health in 48 managers of a global enterprise in the United Kingdom. Participants were mainly male (44/48, 92%) and ranged in age from 32 to 58 (mean 46.0, SD 7.2) years.

    Results: We found a positive impact of the Web-based training program on managers’ knowledge of mental health and mental illness (P<.001), on attitudes toward people with mental health problems (P<.01), and on their self-efficacy to deal with mental health situations at work (P<.001), with the exception of intentions to promote employee mental health, which was initially high.

    Conclusions: Results provide first evidence of the effectiveness of LMHP to positively affect managers’ skills to promote employee mental health at work. Furthermore, the high rate of participation in LMHP (48/54, 89%) supports the use of digital game-based interventions to increase user engagement and user experience in mental health programs at work.

    JMIR Ment Health 2017;4(3):e31




    Due to their high prevalence (1 in 4) [1], the economic impact of mental health problems such as depression can be considerable for employers globally. In high-income countries, the trend of sick days lost due to mental health problems has been growing in recent years [2]. Resulting total work loss due to sickness absence, lost at-work productivity, and turnover are estimated to cost £26 billion a year in the United Kingdom alone [3].

    To counteract the negative impact of mental health problems on business, organizations are increasingly investing in mental health promotion, prevention, and intervention efforts [4]. For example, many organizations have implemented employee assistance programs (EAPs), which typically offer assessment, counselling, and referral services to employees with work-related or personal problems [5]. Others offer stress reduction programs such as meditation or relaxation interventions [6].

    However, there are a few drawbacks worth discussing with regard to the current practice of workplace mental health promotion. First, most interventions aiming to promote employee mental health focus on the employee level (such as in stress management) while neglecting the organizational level (working conditions) [7,8]. However, many factors that positively affect employee mental health are related to the social environment at work, such as the working culture, level of social support, and leadership style [9]. Second, with regard to leadership, few efforts have considered the special role of managers in organizations [10,11]. Because they are in close contact with employees, managers are in the best position to spot signs of deteriorating mental health early and to provide support. Unfortunately, however, many leaders lack training in the management of workplace mental health and thus are ill-equipped to support affected individuals adequately [12]. Third, utilization rates of EAPs are generally low [5,13]. Despite the availability of effective mental health interventions, the majority of affected individuals choose not to seek help [14]. A major barrier that strongly contributes to the underuse of EAPs is the stigma associated with mental illness [15,16].

    Stigma is defined as (1) the lack of knowledge of mental health problems and treatments, (2) prejudicial attitudes, and (3) the lack of supportive behavior, or anticipated or real acts of discrimination against people with mental health problems [14]. Despite its far-reaching impact on employees’ willingness to seek help, current practices in mental health promotion largely fail to address stigma specifically [17]. Therefore, as of yet, efforts in mental health promotion do not seem to reach their primary aim, early and effective treatment, satisfactorily [18]. Instead, raising awareness of the importance of mental health, reducing stigma, and creating an organizational culture of acceptance, diversity, and respect may be a necessary prerequisite for the acceptance, use, and, thus, effectiveness of mental health interventions such as EAPs [19].

    While the majority of stigma reduction programs targeted the general population—for example, in public health campaigns—there is growing interest in the effectiveness of workplace antistigma interventions [20,21]. A systematic review [17] found that workplace antistigma interventions can have a positive impact on employees’ knowledge, attitudes, and supportive behavior toward people with mental health problems. However, limitations became apparent: (1) most interventions targeted the public sector, (2) half of the studies included did not target all 3 dimensions of stigma, which is key in achieving ultimate behavioral change, (3) there is a lack of evidence concerning the sustainability of workplace antistigma interventions due to insufficient follow-up beyond pre- and postintervention assessments, and (4) most interventions were delivered face-to-face, thus having only a limited reach and impact on stigma among the wider workforce.

    The dissemination of digital interventions, however, could be a powerful strategy to facilitate widespread behavioral and cultural change in organizations [22]. Compared with face-to-face interventions, digital interventions have many advantages, such as greater reach, reduced barriers to access, increased participant engagement and adherence to treatment, and flexible and self-paced learning, as well as being more cost effective [23]. However, most digital health promotion efforts so far have targeted physical rather than mental health and mainly focused on the treatment of specific disorders in a subgroup (eg, depression in teenagers) [24-26]. Evidence on digital interventions aiming to prevent mental health problems is still scarce and even more so with regard to the workplace setting [27,28]. This study, therefore, aimed to address some of the limitations of current practices in mental health promotion and of research on stigma reduction. We followed 2 objectives: (1) to develop a digital game-based intervention to train leaders of a private sector organization to effectively manage employee mental health by addressing all 3 dimensions of stigma in order to prevent mental health problems and promote an open, inclusive, and supportive working culture, and (2) to evaluate the intervention in terms of its effectiveness and mid-term sustainability in a pilot study.

    Specifically, we hypothesized that our digital game-based intervention, called Leadership Training in Mental Health Promotion (LMHP), would lead to (1) improved mental health knowledge, (2) increased positive attitudes toward people with mental health problems, (3) increased self-efficacy to deal with mental health situations at work, and (4) improved intentions to promote employee mental health at work in managers undertaking the training.


    Objective 1: Intervention Development

    The intervention was developed in a collaborative effort between the department of psychosocial health and well-being of a large global private sector company, which employed around 348,000 employees in more than 100 countries in 2015, and the Chair for Public Health and Health Services Research of Ludwig-Maximilians-University (LMU) in Munich, Germany.


    In developing LMHP, we followed a systematic approach similar to intervention mapping [29] for designing theory- and evidence-based health promotion programs. Specifically, we took several steps, from analyzing the problem of mental illness stigmatization and effective change methods [17], to assessing the needs for managerial training on mental health, and, finally, to developing the training, as well as an implementation and evaluation plan.


    We developed training content based on a review of workplace training programs on mental health [30-33] and on consultations with subject matter experts in the field of health management, human resources, and training and development. Furthermore, we carried out a needs assessment via 14 semistructured interviews (7 managers, 7 employees) in the participating organization, investigating managerial training needs in terms of preferred content and mode of delivery (unpublished data). Results indicated a particular need for managers to be trained in spotting warning signs of mental distress, and in how to interact with and support affected employees.


    While e-learning is well established in larger enterprises, Web-based training in its most common form, animated slidecasts, is losing more and more in attractiveness and acceptance [34]. To counteract low participant engagement [35], LMHP was developed as a simulation game, a Web-based training program combining elements of both games and simulations [36]. By creating a real in-person environment with all the complexities of the formal and particularly social interactions typically found in the workplace, the program provides managers with the opportunity to directly apply what they learned about people management and to practice new skills in a safe virtual environment [37]. This way, managers can get a sense of the potential impact of different leadership styles on employee mental health without having to worry about real-world consequences.


    To facilitate an innovative and engaging learning experience [35], we used a subtle form of gamification in LMHP to fit the sensitivity of the training content. Gamification is defined as “the use of game design elements in non-game contexts” [38]. For example, while we refrained from providing badges for achievements or enabling competition between players, we did include several gamification strategies that were found to increase engagement and learning [39]. Those involved providing a storyline and clear goals, including the capacity to overcome challenges by learning; providing feedback on performance; showing progress (in terms of how leader behavior affects employee mental health over time); and reinforcing learning by allocating points (eg, for quiz questions answered correctly).

    Objective 2: Intervention Evaluation

    The goal of this pilot study was to evaluate the effectiveness of a digital game-based training program for managers, which we developed to promote employee mental health and reduce mental health-related stigma at work, using a 1-group pre-post design and a 3-month follow-up. The pilot study was carried out at a defined site of the participating organization near Oxford, United Kingdom.


    All managers of this site were invited to take part in LMHP and its associated research study. To be included, participants had to be of working age (between 18 and 65 years) and be managing at least one employee at the time of the training. Informed consent was obtained from all individual participants included in the study.


    Invitations to participate in LMHP were sent out by email approximately a week in advance of the scheduled Web-based training. This invitation notified participants about the study’s objectives, potential risks, data protection, etc.

    Participants were then sent a personal link that allowed (1) participants to give their informed consent to participate in this study, (2) participants to access the training program for a limited time period of 3 weeks, (3) participants to access the pre- and postquestionnaire immediately before (T1) and after (T2) completion of the training, and (4) the researchers to allocate responses at T1, T2, and T3 to an individual. However, the link did not include any information that could be used to identify participants. At T3 (12 weeks after training completion), participants were resent their personal link in order to fill in a follow-up questionnaire to evaluate the first mid-term effects of the intervention.

    Any communication about the training initiative (eg, invitations), as well as personal links to training and questionnaires, was sent out via email by a human resources staff member of the participating organization, who was not involved in the study. Questionnaires were completed anonymously online, and responses were tracked and stored safely at the external training provider. The external training provider then replaced participants’ email addresses with a random, unique 3-digit identifier and posted the data back to the researchers at LMU Munich. To increase response rates, the external training provider informed the human resources staff member of the participating organization about any nonresponders so that he could send out reminders. The researchers were never told the names of individual respondents, and the human resources staff member in the participating organization never saw any completed questionnaires or individually identifiable data.


    Ethical approval for the study was given by the Ethics Committee of LMU Munich, Germany. All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional or national research committee and with the 1964 Declaration of Helsinki and its later amendments or comparable ethical standards.

    Outcome Measures

    Demographic questions included age, sex, level of education, marital status, whether they currently lived alone, and whether they knew someone with a mental health problem and had been diagnosed with or treated for a mental health problem themselves.

    Other outcome measures matched the knowledge, attitudinal, and behavioral dimensions of stigma as defined above. We administered 4 validated instruments. To all of them, a 5-point Likert scale ranging from 1 (“strongly disagree”) to 5 (“strongly agree”) was applied. We calculated global scores on all instruments using sum scores, with higher scores indicating a better outcome, with the exception of stigmatizing attitudes. All measures were administrated at all 3 time points.


    We assessed knowledge about mental health problems using the first 6 items, which are related to stigma, of the 12-item Mental Health Knowledge Schedule (MAKS) [40]. An example item is “Psychotherapy can be an effective treatment for people with mental health problems.” Sum scores ranged from 6 to 30.

    Additionally, we developed a set of 7 quiz questions to test participants’ knowledge on specific training content of LMHP, with 3 answer options, of which 1 was correct. An example item is “Which statement about business costs related to mental disorders is correct?” In this case, sum scores ranged from 0 to 7.


    We assessed attitudes in the workplace toward coworkers who may have a mental illness using the 23-item Opening Minds Scale for Workplace Attitudes (OMS-WA), an adapted version of the Opening Minds Scale for Health Care Providers (OMS-HC) [41]. OMS-WA consists of 5 subscales: 6 items on avoidance, 5 on perceived dangerousness, 5 on work beliefs and competencies, 4 on helping, and 3 on responsibility of people with mental health problems. During evaluation, we considered attitudes as a whole, with sum scores ranging from 23 to 115, as well as the individual subscales, with sum scores ranging from 6 to 30 for avoidance, 5 to 25 for perceived dangerousness, 5 to 25 for work beliefs and competencies, 4 to 20 for helping, and 3 to 15 for responsibility. An example item is “I would try to avoid a coworker with a mental illness.”


    To assess behavioral change in leaders, we used proxy variables (eg, self-efficacy to deal with mental health situations at work and intentions to promote employee mental health), since in a 3-month period not very many mental health situations are likely to arise at work where leaders could possibly demonstrate actual support. However, prior research found that enhanced intentions and high self-efficacy increase the likelihood that a person will engage in newly learned behaviors [42].

    In this study, we measured self-efficacy with regard to managing employee mental health by a previously adapted version of the 9-item New General Self-Efficacy Scale [30,43]. Items included “When facing difficulties related to employee mental health, I am certain that I will handle them appropriately.” Sum scores ranged from 9 to 45.

    To assess participants’ intentions to promote employee mental health, we used a previously adapted 3-item version of a safety scale designed to assess managers’ safety promotion intentions [30,44]. An example item is “I want to apply what I learn about employee mental health to my work setting.” Sum scores ranged from 3 to 15.

    Statistical Methods

    We used descriptive statistics (mean, median, SD) to describe the study population. Multilevel growth models (with random intercept) were applied to investigate change over time in the dependent variables knowledge, attitudes, self-efficacy, and intentions to promote employee mental health [45]. An advantage of multilevel growth models is that missing data can be handled flexibly (using likelihood-based estimation) and thus allowed incorporation of all available data. First, we used time as a fixed factor in the models, as pre- and postmeasurements were collected on the same day for each participant and variability in time from post- to follow-up measurements was very low across participants. Second, we investigated whether selected participant characteristics (age, educational level) predicted initial status. We applied the forward modelling approach, starting with models without any predictors (model A) and adding potential explanatory variables as fixed effects at subsequent steps (models B and C). To select the best model, we considered reductions of deviance (–2*log likelihood) and of Akaike information criterion and Bayesian information criterion values, with smaller values indicating a better-fitting model. We computed change as the difference in relation to the baseline (T1) score. Parameter estimates and standard errors (SE) are reported. Effects were judged significant at alpha≤.05, unless otherwise noted. Statistical analyses were performed using IBM SPSS 23.0 and SPSS MIXED (IBM Corporation).


    Objective 1: Intervention Development

    Taking all formative research described above into consideration, we designed LMHP in a way to train managers in (1) understanding mental health and mental illness, (2) spotting warning signs, (3) taking early and appropriate action, and (4) monitoring and self-monitoring.

    Digital Game-Based Learning

    The training consisted of one single session, which took between 1.5 and 2 hours to complete, thereby meeting managers’ expectations of a particularly concise and time-efficient training format as expressed during interviews (see formative research described above). The setting was the office hub where, over a virtual time period of 7 weeks, the player was put into the position of a manager. During that time period, it was the manager’s task to supervise a virtual team and manage employee mental health effectively.

    Table 1. Outline of content and psychological constructs covered in the virtual scenarios of the Leadership Training in Mental Health Promotion program.
    View this table

    The virtual team consisted of 4 employees showing diverse psychological profiles; thus, each represented a different mental health scenario likely to appear in real office life. Scenarios contained examples of the promotion of mental health, the prevention of mental illness, and the rehabilitation of employees with common mental health problems such as anxiety or depressive disorders (see Table 1). Due to their relatively low prevalence rates, more severe mental disorders such as psychosis were not addressed in this workplace training. All scenarios required managers to develop and practice their skills in spotting warning signs, taking (early) action, and monitoring employees while building knowledge of mental health and mental illness and more positive attitudes toward employees with mental health problems at the same time (see Table 1).

    For example, to sensitize managers in the recognition and identification of warning signs, certain hints were placed into the virtual work environment (eg, medication, uneaten lunch, or work piling up on an employee’s desk) that may or may not signal a growing underlying mental imbalance. Once the manager had spotted something unusual or alarming, he or she could choose to engage in a conversation with the respective employee. Different dialogue options were provided to choose from, which were more or less appropriate given the sensitivity of a certain topic. Depending on how the manager behaved, the respective employee chose to either shut down and end the conversation or open up and share further information the manager needed to be able to offer appropriate and effective support.

    To ensure continuous learning and improved self-efficacy to manage mental health situations at work, the player was provided with instant feedback regarding his or her actions after the end of each conversation. Furthermore, a video of an actual affected employee of the participating organization sharing his or her experience with burnout was shown automatically to every player. The personal testimonial was presented in a way to counter prominent stereotypes of people with mental health problems and with a strong focus on the road toward recovery and well-being, thus involving many features considered fundamental to reducing stigma [46]. This video formed a very powerful part of the training, since contact with people with lived experience (face-to-face or video-based) is argued to be the strongest method to tackle mental illness stigma [47].

    Mental Health Toolbox

    Next to scenario-based learning, LMHP also offered a mental health toolbox that provided managers with practical information on topics found to be relevant to manage a given scenario successfully. The toolbox was presented in a way to improve managers’ knowledge of mental health and mental illness, improve their attitudes toward employees with mental health problems, and train them in skills to deal with mental health situations at work effectively. Topics of the mental health toolbox focused on 4 main areas: what mental health and mental illness mean, how to recognize signs of mental distress, how to start a conversation, and how to support affected employees effectively (see Table 2). Furthermore, the toolbox aimed to facilitate the application of newly learned skills in real everyday office life. For example, checklists with warning signs or guidelines for conversations on mental health could be downloaded as pdf files and serve as useful aids in interactions with employees.

    Table 2. Outline of content and psychological constructs covered in the Mental Health Toolbox of the Leadership Training in Mental Health Promotion program.
    View this table
    Theoretical Foundation and Underlying Models

    The idea behind the training—for example, the progression of employees’ mental state in scenarios—followed the principles of the mental health continuum model [48,49]. This model postulates that mental health is spread out along a continuum, meaning that people are not either mentally healthy or mentally ill, but that they can move in and out of further phases in between.

    In LMHP, we used an adapted version of the mental health continuum model to suit our specific needs. Each phase of this continuum (health, acute stress, chronic stress, and illness) is assigned certain warning signs and recommended actions to take as an affected individual but also as a manager supporting affected employees. In this way, mental health becomes more concrete, which, in turn, facilitates managers’ understanding of mental health and warning signs.

    On several occasions during the training, the manager was asked to assess each employee’s mental state along the phases of the mental health continuum model. Afterward, the player was given feedback on an employee’s actual mental state and on other parameters the manager influenced with his or her behavior, such as perceived managerial support or an employee’s willingness to seek professional help. This exercise was designed to improve managers’ self-efficacy in identifying warning signs and to strengthen their intentions to promote employee mental health.

    Objective 2: Intervention Evaluation


    Figure 1 shows the flow of participants at each stage of the study. Of 54 managers working at the site, 48 (89%) accepted our invitation, completed the baseline questionnaire, and took part in the training. Of the 48 participants, 47 (98%) completed the postquestionnaire immediately after the training and 38 (79%) responded to the follow-up questionnaire 3 months later. Complete data from 3 waves were available for 37 (77%) participants and from at least two waves for 47 (98%) respondents.

    Descriptive Analysis

    Table 3 presents baseline demographic characteristics of the sample population: 92% of participants were male (44/48). Participants ranged in age from 32 to 58 (mean 46.0, SD 7.2) years. Among the 48 participants, 48% (23/48) had a university degree, 77% (37/48) were married, and 88% (42/48) were not living alone. Furthermore, 63% (30/48) knew someone with a mental health problem and 10% (5/48) had been diagnosed with or treated for a mental health problem themselves. Finally, 17% (8/48) received further training on mental health between the postevaluation and follow-up evaluation.

    Figure 1. Flow diagram showing progress through the phases of the trial.
    View this figure
    Table 3. Baseline demographic characteristics of the sample population (n=48).
    View this table
    Table 4. Descriptive statistics for respondents who participated at all 3 time pointsa (n=37).
    View this table
    Multilevel Analysis

    Table 4 shows the mean scores of knowledge, attitudes, self-efficacy, and intentions to promote employee mental health at the 3 time points. In general, observed baseline scores indicated that, before the intervention, managers had quite good knowledge of mental health, fairly positive attitudes toward people with mental illness, and a high level of self-efficacy, as well as intentions to promote employee mental health.

    Table 5 and Table 6 show the results of the multilevel analysis. Adding age and education (refer to Table 3) to the models neither showed significant effects regarding initial status nor improved the goodness of fit. Thus, in the following, we focused on results of model A intercept and, particularly, model B intercept and time. Overall, the B models had good fit. These models indicated that knowledge of mental health and mental illness (measured by MAKS and the quiz) and self-efficacy to deal with mental health situations at work significantly increased over time and that this effect remained significant over the 3-month period (see Table 5). Regarding stigmatizing attitudes, attitudes (total scale; Table 5) and attitude subscales related to avoidance, perceived dangerousness, and responsibility (Table 6) significantly decreased over time with these effects also being sustained 3 months later. However, attitudes related to work and competency beliefs and to helping people with mental health problems did not change over time (Table 6). Moreover, managers’ intentions to promote employee mental health did not change over time (Table 5).

    Table 5. Mixed models (with random intercept) considering knowledge assessed by MAKSa, knowledge assessed by quiz, attitude (total), self-efficacy, and intentions to promote employee mental health as the dependent variable (n=48).
    View this table
    Table 6. Mixed models (with random intercept) considering attitudes regarding avoidance, dangerousness, workability, helping, and responsibility as the dependent variable (n=48).
    View this table


    Principal Results

    In this study we targeted the development and pilot evaluation of a digital game-based training program for managers to promote employee mental health and reduce mental illness stigma at work. Our study contributes to strengthen the evidence base that interventions targeting leaders may be effective in improving mental health literacy and reducing mental illness stigma in the workplace. In line with prior research and our hypotheses, we found statistically significant improvements in managers’ knowledge of mental health and mental illness, attitudes toward people with mental health problems, and self-efficacy to deal with mental health situations at work, with the exception of intentions to promote employee mental health [50-52]. While these results can only be considered preliminary until replicated in a controlled trial, they nevertheless highlight some interesting findings that will help inform, first, the future development of effective antistigma interventions in the workplace and, second, relevant stakeholders such as personnel in human resources or health management about the benefits of investing in stigma reduction efforts.

    Knowledge of mental health and mental illness is a key stigma component and a common target of antistigma interventions, as it enables recognition and is thus essential to the prevention of mental health problems [47]. In line with previous studies [53,54], we found improvements in managers’ knowledge of mental health and mental illness (MAKS and quiz). Research shows that improved knowledge of mental health problems strongly influences a person’s ability not only to recognize signs of mental illness, but also to seek help and support others in seeking help, and to accept treatment [55].

    Evidence of the potential impact of workplace antistigma interventions on managers’ attitudes toward people with mental health problems is generally mixed [17]. While some studies did not find any significant change in overall attitudes toward people with mental health problems [53,54], others reported improvements [56,57]. In our study, we evaluated not only overall attitude but also specific aspects of attitude, namely avoidance, perceived dangerousness, beliefs about workability and competencies, helping, and responsibility. While we found decreasing overall stigmatizing attitudes in managers over time, this did not apply to attitudes related to beliefs about workability and competency of people with mental health problems, nor to attitudes related to helping. An important finding of our study is therefore that a more thorough evaluation of attitudes considering specific themes, such as perceived dangerousness or social avoidance, is necessary and may be crucial to a better understanding of the effectiveness of antistigma interventions.

    Behavioral change is key to creating an open and supportive work environment [58]. While public health efforts have often failed to change behavior, antistigma interventions in the workplace were suggested to be particularly promising because they allow for clear instructions with regard to how one is expected to behave in specific situations at work [21]. In line with prior studies, we found LMHP to have a positive impact on managers’ self-efficacy to deal with mental health situations at work (eg, provide support) [51,59]. This is very important, since, even more so than knowledge, the level of self-efficacy strongly influences whether a person will engage in learned behaviors [42,60].

    An open question is why LMHP did not lead to improvements in attitudes related to beliefs about workability and competency of people with mental health problems, and in managers’ intentions to promote employee mental health. One potential reason might be that managers in our sample already had quite positive attitudes at baseline regarding workability and competency of people with mental health problems, as well as intentions to promote employee mental health, which left little room for improvement postintervention. Moreover, even though people with mental health problems can function productively at work, the literature shows that employers’ beliefs about the workability and competency of people with mental health problems are often poor and may be particularly hard to change [61]. Somewhat surprisingly, attitudes related to helping employees with mental health problems if they, for example, got behind in their work were and remained relatively negative despite the training. This could be related to managers’ concerns about the equity of the distribution of responsibilities and meeting productivity pressures [62]. Having in mind how important these outcomes are to reduce stigma and given that many people with mental health problems are either unemployed but want to work or are working [63,64], we recognize that LMHP and other future workplace antistigma interventions might need to incorporate modules that address those aspects more specifically.

    Due to a lack of sufficient follow-up in relevant prior studies, conclusions regarding the effectiveness of workplace antistigma interventions over the long term are limited [17]. However, the few studies that conducted a follow-up reported that changes achieved in people’s knowledge, attitudes, and behavior were, in part, sustained over time [30,53,54,56,65,66]. We also found that effects of LMHP on managers’ knowledge, attitudes, and self-efficacy were largely sustained over a 3-month period (Table 5 and Table 6). While still being significantly different from baseline values, scores seemed to slightly decrease again from post- to follow-up assessment, indicating a potential need for booster sessions and further measures.

    While the use of digital game-based interventions in mental health promotion is scarce and especially so in the workplace, research in other settings such as schools shows promising effects, including significant improvements in students’ psychological well-being and increased engagement in a learning program [27,28,67]. While existing efforts, however, mainly focus on risk prevention [67,68], LMHP trained managers equally in how they can contribute to reducing symptoms of mental illness in employees and in how to enhance their psychological well-being. Digital mental health promotion interventions need to shift their traditional focus on treatment and risk prevention of mental health problems to emphasizing positive psychology, healthy leadership, and the strengthening of individual resources in healthy people in order to be of greater relevance and applicability for organizations. Compared with other nongamified workplace mental health interventions with often low participant rates [27,66], this study confirmed the growing evidence that digital game-based interventions may increase user engagement and learning attainment, thus making it an attractive strategy to facilitate widespread behavioral and cultural change in organizations [34].

    Strengths and Limitations

    This pilot study contributes to strengthen the evidence base of (digital) workplace antistigma interventions. Previous efforts in mental health promotion have largely neglected the role of leaders and instead have focused on employee-level interventions to address stress at work [7,10]. A marked strength of this study is therefore its focus on managers. Additionally, it addressed (1) a lack of research in private sector organizations, (2) a lack of interventions targeting all 3 dimensions of stigma, and (3) a lack of long-term follow-up that characterizes the available literature. Furthermore, this study could help explain prior mixed findings on attitudinal change by investigating the impact of LMHP on attitudes related to specific themes rather than on a single attitude scale [17]. To the best of our knowledge, LMHP is the first digital game-based training for managers aiming to promote employee mental health and reduce mental illness stigma at work. Thus, this pilot adds to the small pool of digital workplace mental health promotion and antistigma interventions [33], providing further evidence suggesting, first, that brief Web-based interventions can be as effective as more time-consuming face-to-face equivalents, which often do not match business demands [22], and second, that incorporating gamification into the learning strategy can increase participant engagement [34].

    This pilot study has some limitations that must be mentioned. First, the study lacked a control group due to formal restrictions of the participating site. To what extent observed changes were due to the intervention is therefore questionable. To account for that, we recorded whether managers participated in further interventions during the study time, and the majority did not (30/48, 63%). Second, to measure knowledge, we developed our own quiz, which was not validated. Therefore, we used a second standardized instrument (MAKS, see Methods) and found similar change patterns in knowledge over time with both instruments. Third, while the OMS-WA as an adapted version of the OMS-HC [25] has been used extensively in program evaluations [66], an evaluation of the psychometric properties of this measure has yet to be published. However, a validation study of OMS-WA is under review. Fourth, the intervention was carried out in the United Kingdom and, thus, participants might have been presensitized as a result of increased stigma reduction efforts that have been going on in the United Kingdom in the past decade [31,69-71]. This might explain the good baseline values and small changes over time and ultimately may have led to an underestimation of the real training impact. Future evaluations should aim to investigate the effectiveness of LMHP in countries where mental illness stigma might be particularly strong and prevailing and where evidence about the effectiveness of antistigma interventions is scarce [72]. Fifth, we collected no data from employees on mental health, intentions to seek help, and perceived management support, nor on actual help-seeking in this study. However, in this pilot, we specifically wanted to gain first evidence on the effectiveness of LMHP before investigating any potential indirect effects on employees. Sixth, we collected no information on user satisfaction with the digital game-based training that would allow us to make objective inferences about acceptance of and engagement with the training. However, some pretests were done to rule out any technical obstacles that could possibly undermine user satisfaction, and the digital game-based training solution was developed based on suggestions made by employees of the participating organization during semistructured interviews upfront. Furthermore, we received a vast amount of positive feedback on LMHP unofficially on completion of the pilot trial, which seems to be mirrored in the high participation rate of 89% (48/54).

    Implications for Future Research

    Future analysis of data on employees and on EAP utilization, sickness absence rates, or the frequency and duration of disability claims before and after using the training program is essential in evaluating the full impact of LMHP. As the ultimate goal of the training was to create an inclusive and supportive working culture where employees feel comfortable to talk about mental health openly and seek help (early), it would be valuable to include employees’ perceptions on whether they feel supported by leaders, and whether and how that changed after the training. Investigating a change in objective data related to employee help-seeking would help establish the business case of investing in antistigma interventions in the workplace.

    Even though we cannot be certain, it is very unlikely that a single intervention may be sufficient to end mental illness stigma and change the working culture in an organization. Hence, future research should explore whether training managers is an effective means of supporting employees with mental health problems or whether other interventions targeting employees instead or dual approaches (eg, campaign and training) may be more efficient to achieve cultural change in the long term. Finally, to increase the generalizability of our findings, workplace antistigma interventions targeting employees of different hierarchies in different types of workplaces are needed. Another appealing contribution of future research would be to compare different training formats (game-based vs standard Web-based vs face-to-face) and their effect on user engagement and learning attainment. In general, more digital workplace mental health interventions are needed that incorporate elements of positive psychology and focus on keeping employees healthy, motivated, and productive.


    This pilot study provides first evidence on the effectiveness of LMHP, demonstrating its ability to positively affect managers’ knowledge, attitudes, and self-efficacy to deal with mental health situations at work. Further evaluation is needed to investigate potential beneficial effects on employees’ perceptions of management support, on their acceptance and use of existing mental health interventions (eg, EAP), and on the working culture in an organization. The benefits of digital game-based learning, such as increased participant engagement and reach, make it an effective strategy to facilitate widespread behavioral and cultural change in organizations.


    The project received funding from the People Programme (Marie Curie Actions) of the European Union’s Seventh Framework Programme FP7/2007 – 2013 under REA grant agreement no. 316795. The authors alone are responsible for the content and writing of the paper.

    We would like to thank Ovos Media GmbH, which supported us in the development of the game-based training program with regard to the technical solution and use of gamification elements. Moreover, we would like to thank Matthias Strack for his expert advice concerning the development of the intervention content, and Hans Bauer for his expert advice on data analysis.

    Conflicts of Interest

    None declared.


    1. Levav I, Rutz W. The WHO World Health Report 2001 new understanding--new hope. Isr J Psychiatry Relat Sci 2002;39(1):50-56. [Medline]
    2. Henderson M, Madan I, Hotopf M. Work and mental health in the UK. BMJ 2014 Mar 21;348(mar21 6):g2256-g2256. [CrossRef] [Medline]
    3. The Sainsbury Centre for Mental Health. Mental health at work: developing the business case. 2007.   URL: [accessed 2017-06-27] [WebCite Cache]
    4. Malachowski C, Kirsh B. Workplace antistigma initiatives: a scoping study. Psychiatr Serv 2013 Jul 01;64(7):694-702. [CrossRef] [Medline]
    5. Reynolds GS, Lehman WEK. Levels of substance use and willingness to use the employee assistance program. J Behav Health Serv Res 2003 Apr;30(2):238-248. [CrossRef] [Medline]
    6. Richardson KM, Rothstein HR. Effects of occupational stress management intervention programs: a meta-analysis. J Occup Health Psychol 2008 Jan;13(1):69-93. [CrossRef] [Medline]
    7. Czabała C, Charzyńska K, Mroziak B. Psychosocial interventions in workplace mental health promotion: an overview. Health Promot Int 2011 Dec;26 Suppl 1:i70-i84. [CrossRef] [Medline]
    8. Glazer S. A new vision for the journal. ‎Int J Stress Manag 2011;18(1):1-4. [CrossRef]
    9. Michie S. Causes and management of stress at work. Occup Environ Med 2002 Jan;59(1):67-72 [FREE Full text] [CrossRef] [Medline]
    10. Briner R, Amati C, Lardner R. Development of Internal Company Standards of Good Management Practice and a Task-Based Risk Assessment Tool for Offshore Work Related Stressors. Research Report No. RR107. Sudbury, ON: HSE Books; 2003.
    11. De Lange AH, Taris TW, Kompier MAJ, Houtman ILD, Bongers PM. The relationships between work characteristics and mental health: examining normal, reversed and reciprocal relationships in a 4-wave study. Work Stress 2004 Apr;18(2):149-166. [CrossRef]
    12. Thorpe K, Chénier L. Building mentally healthy workplaces: perspectives of Canadian workers and front-line managers. Toronto, ON: Conference Board of Canada; 2011 Jun.   URL: [accessed 2017-06-27] [WebCite Cache]
    13. Azzone V, McCann B, Merrick EL, Hiatt D, Hodgkin D, Horgan C. Workplace stress, organizational factors and EAP utilization. J Workplace Behav Health 2009;24(3):344-356 [FREE Full text] [CrossRef] [Medline]
    14. Thornicroft G. Most people with mental illness are not treated. Lancet 2007 Sep 08;370(9590):807-808. [CrossRef] [Medline]
    15. Vogt D. Mental health-related beliefs as a barrier to service use for military personnel and veterans: a review. Psychiatr Serv 2011 Feb;62(2):135-142. [CrossRef] [Medline]
    16. Walton L. Exploration of the attitudes of employees towards the provision of counselling within a profit-making organisation. Couns Psychother Res 2003 Mar;3(1):65-71. [CrossRef]
    17. Hanisch SE, Twomey CD, Szeto ACH, Birner UW, Nowak D, Sabariego C. The effectiveness of interventions targeting the stigma of mental illness at the workplace: a systematic review. BMC Psychiatry 2016 Jan 06;16:1 [FREE Full text] [CrossRef] [Medline]
    18. Masi D, Altman L, Benayon C, Healy H, Jorgensen D, Kennish R, et al. Employee assistance programs in the year 2002. In: Manderscheid RW, Henderson MJ, editors. Mental Health, United States. DHHS pub no SMA-3938. Rockville, MD: Substance Abuse and Mental Health Services Administration; 2002.
    19. Kirsh B. Organizational culture, climate and person-environment fit: relationships with employment outcomes for mental health consumers. Work 2000;14(2):109-122. [Medline]
    20. Clement S, Lassman F, Barley E, Evans-Lacko S, Williams P, Yamaguchi S, et al. Mass media interventions for reducing mental health-related stigma. Cochrane Database Syst Rev 2013 Jul 23(7):CD009453. [CrossRef] [Medline]
    21. Szeto AC, Dobson KS. Reducing the stigma of mental disorders at work: a review of current workplace anti-stigma intervention programs. Appl Prev Psychol 2010 Jun;14(1-4):41-56. [CrossRef]
    22. Mohr DC, Burns MN, Schueller SM, Clarke G, Klinkman M. Behavioral intervention technologies: evidence review and recommendations for future research in mental health. Gen Hosp Psychiatry 2013;35(4):332-338 [FREE Full text] [CrossRef] [Medline]
    23. Lal S, Adair CE. E-mental health: a rapid review of the literature. Psychiatr Serv 2014 Jan 01;65(1):24-32. [CrossRef] [Medline]
    24. De Cocker CK, De Bourdeaudhuij I, Cardon G, Vandelanotte C. The effectiveness of a web-based computer-tailored intervention on workplace sitting: a randomized controlled trial. J Med Internet Res 2016 May 31;18(5):e96 [FREE Full text] [CrossRef] [Medline]
    25. Riper H, Andersson G, Christensen H, Cuijpers P, Lange A, Eysenbach G. Theme issue on e-mental health: a growing field in internet research. J Med Internet Res 2010 Dec 19;12(5):e74 [FREE Full text] [CrossRef] [Medline]
    26. Christensen H, Griffiths KM, Korten A. Web-based cognitive behavior therapy: analysis of site usage and changes in depression and anxiety scores. J Med Internet Res 2002;4(1):e3 [FREE Full text] [CrossRef] [Medline]
    27. Muuraiskangas S, Harjumaa M, Kaipainen K, Ermes M. Process and effects evaluation of a digital mental health intervention targeted at improving occupational well-being: lessons from an intervention study with failed adoption. JMIR Ment Health 2016 May 11;3(2):e13 [FREE Full text] [CrossRef] [Medline]
    28. Kennedy CM, Powell J, Payne TH, Ainsworth J, Boyd A, Buchan I. Active assistance technology for health-related behavior change: an interdisciplinary review. J Med Internet Res 2012 Jun 14;14(3):e80 [FREE Full text] [CrossRef] [Medline]
    29. Bartholomew LK, Parcel GS, Kok G. Intervention mapping: a process for developing theory- and evidence-based health education programs. Health Educ Behav 1998 Oct;25(5):545-563. [CrossRef] [Medline]
    30. Dimoff JK, Kelloway EK, Burnstein MD. Mental health awareness training (MHAT): the development and evaluation of an intervention for workplace leaders. ‎Int J Stress Manag 2016;23(2):167-189. [CrossRef]
    31. Henderson C, Thornicroft G. Stigma and discrimination in mental illness: Time to Change. Lancet 2009 Jun 06;373(9679):1928-1930. [CrossRef] [Medline]
    32. Matyssek A. Förderung psychischer Gesundheit als Führungsaufgabe. 2012.   URL: [accessed 2017-06-27] [WebCite Cache]
    33. Stuart H, Chen S, Christie R, Dobson K, Kirsh B, Knaak S, et al. Opening minds in Canada: targeting change. Can J Psychiatry 2014 Oct;59(10 Suppl 1):S13-S18 [FREE Full text] [Medline]
    34. Muntean C. Raising engagement in e-learning through gamification. 2011 Presented at: 6th International Conference on Virtual Learning ICVL; Oct 28-29, 2011; Cluj-Napoca, Romania p. 323-329.
    35. Richter G, Raban D, Rafaeli S. Studying gamification: the effect of rewards and incentives on motivation. In: Reiners T, Wood LC, editors. Gamification in Education and Business. Cham, Switzerland: Springer International Publishing Switzerland; 2015:21-46.
    36. Wilson KA, Bedwell WL, Lazzara EH, Salas E, Burke CS, Estock JL, et al. Relationships between game attributes and learning outcomes. Simulation Gaming 2008 May 07;40(2):217-266. [CrossRef]
    37. Landers R, Callan R. Casual social games as serious games: the psychology of gamification in undergraduate education and employee training. In: Ma M, Oikonomou A, Jain LC, editors. Serious Games and Edutainment Applications. London, UK: Springer; 2011:399-423.
    38. Deterding S, Dixon D, Khaled RN, Nacke L. Gamification: toward a definition. 2011 Presented at: CHI Gamification Workshop; May 7-12, 2011; Vancouver, BC, Canada.
    39. Cugelman B. Gamification: what it is and why it matters to digital health behavior change developers. JMIR Serious Games 2013 Dec 12;1(1):e3 [FREE Full text] [CrossRef] [Medline]
    40. Evans-Lacko S, Little K, Meltzer H, Rose D, Rhydderch D, Henderson C, et al. Development and psychometric properties of the Mental Health Knowledge Schedule. Can J Psychiatry 2010 Jul;55(7):440-448. [CrossRef] [Medline]
    41. Kassam A, Papish A, Modgill G, Patten S. The development and psychometric properties of a new scale to measure mental illness related stigma by health care providers: the Opening Minds Scale for Health Care Providers (OMS-HC). BMC Psychiatry 2012 Jun 13;12:62 [FREE Full text] [CrossRef] [Medline]
    42. Vuori J, Toppinen-Tanner S, Mutanen P. Effects of resource-building group intervention on career management and mental health in work organizations: randomized controlled field trial. J Appl Psychol 2012 Mar;97(2):273-286. [CrossRef] [Medline]
    43. Chen G, Gully SM, Eden D. Validation of a new general self-efficacy scale. Organ Res Methods 2016 Jun 29;4(1):62-83. [CrossRef]
    44. Mullen J, Kelloway E. Safety leadership: a longitudinal study of the effects of transformational leadership on safety outcomes. J Occup Organ Psychol 2009;82(2):253-272. [CrossRef]
    45. Singer J, Willett J. Applied Longitudinal Data Analysis: Modeling Change and Event Occurrence. New York, NY: Oxford University Press; 2003:-19.
    46. Corrigan PW, Penn DL. Lessons from social psychology on discrediting psychiatric stigma. Am Psychol 1999 Sep;54(9):765-776. [Medline]
    47. Corrigan P. On the Stigma of Mental Illness: Practical Strategies for Research and Social Change. Washington, DC: American Psychological Association; 2005:11-44.
    48. Keyes CLM. The mental health continuum: from languishing to flourishing in life. J Health Soc Behav 2002 Jun;43(2):207-222. [Medline]
    49. National Defence and the Canadian Armed Forces. Mental health continuum model. 2011.   URL: http:/​/www.​​en/​caf-community-health-services-r2mr-deployment/​mental-health-continuum-model.​page [accessed 2017-06-27] [WebCite Cache]
    50. Kitchener BA, Jorm AF. Mental health first aid training in a workplace setting: a randomized controlled trial [ISRCTN13249129]. BMC Psychiatry 2004 Aug 15;4:23 [FREE Full text] [CrossRef] [Medline]
    51. Knifton L, Walker A, Quinn N. Workplace interventions can reduce stigma. J Public Ment Health 2009 Jun 22;7(4):40-50. [CrossRef]
    52. Krameddine YI, Demarco D, Hassel R, Silverstone PH. A novel training program for police officers that improves interactions with mentally ill individuals and is cost-effective. Front Psychiatry 2013;4:9. [CrossRef] [Medline]
    53. Nishiuchi K, Tsutsumi A, Takao S, Mineyama S, Kawakami N. Effects of an education program for stress reduction on supervisor knowledge, attitudes, and behavior in the workplace: a randomized controlled trial. J Occup Health 2007 May;49(3):190-198. [Medline]
    54. Svensson B, Hansson L. Effectiveness of mental health first aid training in Sweden. A randomized controlled trial with a six-month and two-year follow-up. PLoS One 2014 Jun;9(6):e100911 [FREE Full text] [CrossRef] [Medline]
    55. Jorm AF, Christensen H, Griffiths KM. The impact of beyondblue: the national depression initiative on the Australian public's recognition of depression and beliefs about treatments. Aust N Z J Psychiatry 2005 Apr;39(4):248-254. [CrossRef] [Medline]
    56. Gould M, Greenberg N, Hetherton J. Stigma and the military: evaluation of a PTSD psychoeducational program. J Trauma Stress 2007 Aug;20(4):505-515. [CrossRef] [Medline]
    57. Moffitt J, Bostock J, Cave A. Promoting well-being and reducing stigma about mental health in the fire service. J Public Ment Health 2014 Jun 10;13(2):103-113. [CrossRef]
    58. Corrigan PW, Shapiro JR. Measuring the impact of programs that challenge the public stigma of mental illness. Clin Psychol Rev 2010 Dec;30(8):907-922 [FREE Full text] [CrossRef] [Medline]
    59. Stuart H, Koller M, West AA. Opening minds in a post-secondary environment: results of an online contact-based anti-stigma intervention for college staff - starting the conversation.: Mental Health Commission of Canada; 2013.   URL: http:/​/www.​​English/​system/​files/​private/​document/​Stigma_OM_Algonquin%20College_Starting_the_Conversation.​pdf [accessed 2017-06-27] [WebCite Cache]
    60. Bouffard-Bouchard T, Parent S, Larivee S. Influence of self-efficacy on self-regulation and performance among junior and senior high-school age students. Int J Behav Dev 2016 Jun 30;14(2):153-164. [CrossRef]
    61. Mangili E, Ponteri M, Buizza C, Rossi G. [Attitudes toward disabilities and mental illness in work settings: a review]. Epidemiol Psichiatr Soc 2004;13(1):29-46. [Medline]
    62. Krupa T, Kirsh B, Cockburn L, Gewurtz R. Understanding the stigma of mental illness in employment. Work 2009;33(4):413-425. [CrossRef] [Medline]
    63. Cook JA, Razzano L. Vocational rehabilitation for persons with schizophrenia: recent research and implications for practice. Schizophr Bull 2000;26(1):87-103. [Medline]
    64. Marwaha S, Johnson S. Views and experiences of employment among people with psychosis: a qualitative descriptive study. Int J Soc Psychiatry 2005 Dec;51(4):302-316. [CrossRef] [Medline]
    65. Jorm AF, Kitchener BA, Sawyer MG, Scales H, Cvetkovski S. Mental health first aid training for high school teachers: a cluster randomized trial. BMC Psychiatry 2010 Jun 24;10:51 [FREE Full text] [CrossRef] [Medline]
    66. Luong D, Kirsh B, Krupa T. Region of Peel Report. Ottawa, ON: Mental Health Commission of Canada; 2013 May.   URL: https:/​/www.​​sites/​default/​files/​Region_Of_Peel_Report_May_2013_Final_ENG_0.​pdf [accessed 2017-07-28] [WebCite Cache]
    67. Huen JM, Lai ES, Shum AK, So SW, Chan MK, Wong PW, et al. Evaluation of a digital game-based learning program for enhancing youth mental health: a structural equation modeling of the program effectiveness. JMIR Ment Health 2016 Oct 07;3(4):e46 [FREE Full text] [CrossRef] [Medline]
    68. Wang J, Patten SB, Lam RW, Attridge M, Ho K, Schmitz N, et al. The effects of an e-mental health program and job coaching on the risk of major depression and productivity in Canadian male workers: protocol for a randomized controlled trial. JMIR Res Protoc 2016 Nov 15;5(4):e218 [FREE Full text] [CrossRef] [Medline]
    69. Henderson C, Evans-Lacko S, Thornicroft G. Mental illness stigma, help seeking, and public health programs. Am J Public Health 2013 May;103(5):777-780. [CrossRef] [Medline]
    70. Pinfold V, Toulmin H, Thornicroft G, Huxley P, Farmer P, Graham T. Reducing psychiatric stigma and discrimination: evaluation of educational interventions in UK secondary schools. Br J Psychiatry 2003 Apr 01;182(4):342-346. [CrossRef] [Medline]
    71. Thornicroft G, Brohan E, Kassam A, Lewis-Holmes E. Reducing stigma and discrimination: candidate interventions. Int J Ment Health Syst 2008 Apr 13;2(1):3 [FREE Full text] [CrossRef] [Medline]
    72. Lauber C, Rössler W. Stigma towards people with mental illness in developing countries in Asia. Int Rev Psychiatry 2007 Apr;19(2):157-178. [CrossRef] [Medline]


    EAP: employee assistance program
    LMHP: Leadership Training in Mental Health Promotion
    LMU: Ludwig-Maximilians-University Munich
    MAKS: Mental Health Knowledge Schedule
    OMS-HC: Opening Minds Scale for Health Care Providers
    OMS-WA: Opening Minds Scale on Workplace Attitudes

    Edited by A Powell; submitted 27.02.17; peer-reviewed by A AlMarshedi, A Szeto, S Knaak, J Apolinário-Hagen; comments to author 17.04.17; revised version received 30.06.17; accepted 14.07.17; published 04.08.17

    ©Sabine Elisabeth Hanisch, Ulrich Walter Birner, Cornelia Oberhauser, Dennis Nowak, Carla Sabariego. Originally published in JMIR Mental Health (, 04.08.2017.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Mental Health, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.