This paper is in the following e-collection/theme issue:

Depression and Mood Disorders; Suicide Prevention (1363) e-Learning and Medical Education (691) Undergraduate Education for Allied Health Professionals (40)

Published on in Vol 7, No 1 (2020): January

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/14623, first published .
E-Learning to Improve Suicide Prevention Practice Skills Among Undergraduate Psychology Students: Randomized Controlled Trial

E-Learning to Improve Suicide Prevention Practice Skills Among Undergraduate Psychology Students: Randomized Controlled Trial

E-Learning to Improve Suicide Prevention Practice Skills Among Undergraduate Psychology Students: Randomized Controlled Trial

Authors of this article:

Marie-Louise J Kullberg1 Author Orcid Image ;   Joanne Mouthaan1 Author Orcid Image ;   Maartje Schoorl1 Author Orcid Image ;   Derek de Beurs2 Author Orcid Image ;   Robin Maria Francisca Kenter3 Author Orcid Image ;   Ad JFM Kerkhof4 Author Orcid Image
Article Authors Cited by (5) Tweetations (4) Metrics

    Original Paper

    1Department of Clinical Psychology, Leiden University, Leiden, Netherlands

    2Netherlands Institute for Health Services Research, Utrecht, Netherlands

    3Department of Clinical Psychology, Faculty of Psychology, University of Bergen, Bergen, Norway

    4Vrije University Amsterdam, Amsterdam, Netherlands

    Corresponding Author:

    Marie-Louise J Kullberg, MSc

    Department of Clinical Psychology

    Leiden University

    Wassenaarseweg 52

    Leiden, 2333 AK

    Netherlands

    Phone: 31 715276676

    Email: m.j.kullberg@fsw.leidenuniv.nl


    Background: Despite increasing evidence of the effectiveness of digital learning solutions in higher vocational education, including the training of allied health professionals, the impact of Web-based training on the development of practical skills in psychiatry and psychology, in general, and in suicide prevention, specifically, remains largely understudied.

    Objective: This study aimed to determine the effectiveness of an electronic learning (e-learning) module on the adherence to suicide prevention guidelines, knowledge of practical skills, and provider’s confidence to have a conversation about suicidal behavior with undergraduate psychology students.

    Methods: The e-learning module, comprising video recordings of therapist-patient interactions, was designed with the aim of transferring knowledge about suicide prevention guideline recommendations. The program’s effects on guideline adherence, self-evaluated knowledge, and provider’s confidence were assessed using online questionnaires before the program (baseline and at 1 month [T1] and 3 months after baseline). The eligible third- and fourth-year undergraduate psychology students were randomly allocated to the e-learning (n=211) or to a waitlist control condition (n=187), with access to the intervention after T1.

    Results: Overall, the students evaluated e-learning in a fairly positive manner. The intention-to-treat analysis showed that the students in the intervention condition (n=211) reported higher levels of self-evaluated knowledge, provider’s confidence, and guideline adherence than those in the waitlist control condition (n=187) after receiving the e-learning module (all P values<.001). When comparing the scores at the 1- and 3-month follow-up, after both groups had received access to the e-learning module, the completers-only analysis showed that the levels of knowledge, guideline adherence, and confidence remained constant (all P values>.05) within the intervention group, whereas a significant improvement was observed in the waitlist control group (all P values<.05).

    Conclusions: An e-learning intervention on suicide prevention could be an effective first step toward improved knowledge of clinical skills. The learning outcomes of a stand-alone module were found to be similar to those of a training that combined e-learning with a face-to-face training, with the advantages of flexibility and low costs.

    JMIR Ment Health 2020;7(1):e14623

    doi:10.2196/14623

    Keywords

    e-learning; suicide prevention; digital learning; skills training; randomized controlled trial; undergraduate students 


    Background

    Suicide is currently one of the most common causes of death in the Netherlands, with rates of approximately 11 cases per 100,000 inhabitants since 2013 [1,2]. With suicide being an obvious public health concern, health care professionals need to direct efforts toward reducing suicide rates. However, health professionals are not well equipped because they receive little training that specifically prepares them to address suicidal behavior. For example, a common misperception prevails that asking about suicidality could evoke suicidal thoughts [3] even though studies have shown that asking patients about suicidality can evoke feelings of relief and openness, which in turn can ensure timely treatment and prevent exacerbation of symptoms [4,5]. Therefore, the clinician’s proficiency in discussing suicidal thoughts is crucial for estimating the risk for suicide attempts, and so is the clinician’s confidence in exercising these specific skills.

    The advances in technology, rising costs in health care, and need for continuous education of allied health professionals have made electronic learning (e-learning) a popular educational method, specifically in skills education [6-8]. In both undergraduate and graduate medical programs, the use of e-learning modules is widespread [9-11]. It is used successfully in several branches of medicine and specialties, such as dermatology and surgery [12], as these modules allow for the teaching of concrete and measurable skills such as skin examination. In comparison, e-learning has been applied less frequently in the field of psychiatry [9,11] because direct patient contact is deemed most instructive to master interpersonal professional skills [13]. However, the results of a systematic review on the effectiveness of e-learning programs on clinician behavior and patient outcomes showed that e-learning programs are as effective as traditional learning approaches [14] and show superior results in improving health care professional behavior of nurses and nursing students when compared with no instruction [15]. More recently, the results of a pilot randomized controlled trial in medical students showed that an e-learning course was successful in enhancing mental health first-aid intentions in medical students, suggesting the potential of the e-learning course to improve mental health first-aid skills [16]. Furthermore, positive results in cost-effectiveness have made e-learning a more popular tool for therapists in mental health [17]. The University of Oxford has developed a Web-centered training for enhanced cognitive behavioral therapy for eating disorders, which successfully trained a large number of therapists [18]; for example, of the 102 therapists who commenced the training program, more than 80% completed the training, and their scores increased substantially on the validated measure of competence [17].

    This Study

    Suicide is known to be a complex issue that requires a range of prevention initiatives and methods of evaluation [19]. In the Netherlands, mental health institutions are joining forces to make e-learning for mental health care professionals (in training) on therapeutic skills widely available [20]. As part of a proven effective e-learning–supported Train-the-Trainer intervention named Professionals in Training to STOP (PITSTOP) suicide, an e-learning module was developed to appropriately discuss suicidality in a professional context and offered in conjunction with face-to-face training [21-23]. Considering the positive results of PITSTOP suicide on professionals and the benefits of e-learning in mental health education [24,25], this study aimed to determine the effectiveness of the singular use of the e-learning module on knowledge of suicide prevention, provider’s confidence, and adherence to the Dutch guideline on the assessment of suicidal behavior [26] in undergraduate psychology students.


    Design and Procedure

    A randomized controlled trial was conducted among the third- and fourth-year undergraduate clinical (neuro-) psychology students of the Vrije Universiteit (VU) University in Amsterdam, The Netherlands, who were fluent in Dutch. Students at that particular stage in their curriculum were deemed to be eligible to follow an e-learning module on suicidal behavior because they would be preparing for their internship and their professional career as a health care professional. They were recruited via the student administration office. All participants were informed about the study and provided online informed consent. The study was approved by the ethical committee of the Department of Clinical Psychology at the VU University.

    Baseline (T0) questionnaires were sent to each participant. Subsequently, participants were randomized to either the intervention or the waitlist control condition using the randomize subset option in Qualtrics. In the intervention condition, participants received both a message in Qualtrics and an email with the link and the log-in code for the e-learning module. Follow-up questionnaires were sent 1 and 3 months after the baseline questionnaires (T1 and T2). The participants in the waitlist control condition received a message via Qualtrics, thanking them for the completion of the first questionnaire (T0) and informing them about a follow-up questionnaire after 1 month. After completing the T1 follow-up, participants in the waitlist control group also received access to the module, with a final follow-up at T2. All participants were blind to their condition. As an incentive to finish the module, all participants could request a certificate when they finished at least two assessments and had followed the e-learning module for at least half an hour.

    Intervention

    The e-learning module was originally developed to support a face-to-face training called PITSTOP suicide [21]. The content of both the face-to-face training and the e-learning module was based on the Dutch multidisciplinary guideline on the assessment and treatment of suicidal behavior [26]. The core of the module consisted of 5 videos in which experienced nurses, psychologists, and psychiatrists interacted with different patients with suicidal tendencies, played by actors, to teach the guideline recommendations. The patients were of various ages, gender, and diagnostic categories and displayed various prototypical suicidal symptoms, cognitions, and interaction problems. In designing the video content, the dialogues were purposely unscripted to enable more realistic scenes and to prevent the professionals from acting. In addition, the videos included vignettes, guideline topics, and recommendations that were explained by experts on the topic. The psychiatrists, psychologists, and nurses were selected from the clinical staff of departments participating in the PITSTOP study to act as a role model in the module. During the PITSTOP study, trainees had personalized access to the e-learning module that could be viewed repeatedly. The total running time of the module was 60 min.

    Outcome Measures

    To assess guideline adherence, students rated 5 video vignettes of 30 seconds, in which experienced nurses, psychologists, and psychiatrists interact with suicidal characters, played by actors, on the likelihood that they would respond with any of the 25 different interventions on a visual analog scale (ranging from 1 to 100, with 1=very unlikely and 100=very likely). The examples of interventions are “Ask whether the patient thinks about suicide” and “Ask how hopeless the patient is feeling.” At T0, T1, and T2, participants rated the same vignettes. The mean item score of the 125 items (25 items × 5 vignettes) was used as an estimate for guideline adherence, which ranged from 0 to 100, with higher scores representing stronger guideline adherence. The internal consistency of the guideline adherence scale appeared to be excellent (Cronbach alpha=.96) in this sample.

    The knowledge of suicidal behavior was measured by the 7-item subscale self-evaluation of knowledge on suicidal behavior of the 14-item Question-Persuade-Refer questionnaire [27]. The students were asked to rate the extent to which their knowledge increased on items including “Suicide warning signs,” “How to ask someone who may be suicidal,” and “Persuading someone to get help” on a 5-point scale from 1 (very little) to 5 (a lot). Cronbach alpha was .86, reflecting a good internal consistency.

    The provider’s confidence was calculated by summing the scores of the 2 items “I am confident in my ability to successfully assess suicidal patients” and “I am confident in my ability to successfully treat suicidal patients” [28]. The response options were strongly disagree, disagree, neutral, agree, and strongly agree (ranging from 1 to 5).

    For the evaluation of the e-learning module, participants were asked regarding the extent to which they agreed with the following statements (range 1-10), rating from totally disagree to totally agree: “I learned a lot about prevention of suicide,” “The module is a good preparation for my internship/future/job as psychologist,” “I would recommend the module to my colleagues and fellow students,” “I feel confident dealing with people with suicidal thoughts,” and “I appreciate the e-learning module.” The mean score of the 5 items was used to express the evaluation of the e-learning.

    Sample Size

    To calculate the sample size, we used the effect size (Cohen d) of data from the PITSTOP suicide study on provider’s confidence. The effect size in PITSTOP suicide study was 0.5. To be able to detect this effect size, assuming an alpha of .05 and the statistical power of 1−beta=.80, we set the total number of participants to be included to 128. To allow for attrition, more than 128 participants were included.

    Analysis

    Baseline characteristics were compared between groups using independent sample t tests and chi-square tests. A missing values analysis was conducted to assess the patterns of missing values. Data were imputed using multiple imputation with 10 imputations and 100 iterations, using the mice-package [29]. To assess the effect of the module on the levels of guideline adherence, self-evaluation of knowledge, and provider’s confidence, we performed linear regression analyses with scores at T1 as outcome, scores at baseline as covariate, and condition as between-subjects variable.

    To assess the follow-up effect of the module (ie, when both conditions had received the module), we performed paired sample t tests between T1 and T2 separately for each condition on the completers-only data as a sensitivity analysis. All analyses were performed with SPSS Statistics 25.0 (IBM Corp, Armonk, New York, 2019) and R version 3.5.0 (R Core Team, Vienna, Austria, 2018).


    Sample Characteristics

    T0 were filled in by 398 participants, from which 211 were randomized to the intervention group and 187 to the waitlist control group. Participants in the intervention group reported higher levels of self-evaluated knowledge (t396=2.1; P=.03), and in the waitlist control group, more participants had already completed their practical clinical internship (χ²1,398=4.5; P=.04). No difference regarding other baseline characteristics was noted in the groups (see Table 1).

    Table 1. Group differences at baseline and at 1 month and 3 months after baseline on characteristics and outcome measures (N=398).
    CharacteristicsIntervention group (n=211)Waitlist control group (n=187)P value
    Female gender, n (%)180 (85.3)156 (83.4).68
    Age (years), mean (SD)22.1 (4.0)22.1 (3.0).99
    Completed internship, yes, n (%)58 (27.5)70 (37.4).04a
    T0b Guideline adherence, mean (SD)57.4 (11.5)58.9 (11.4).19
    T1c Guideline adherence, mean (SD)62.1 (15.6) 59.8 (14.7)<.001a
    T2d Guideline adherence, mean (SD)62.0 (13.4)63.7 (10.0).47
    T0 Self-evaluation of knowledge, mean (SD)17.3 (4.6)16.3 (4.6).03a
    T1 Self-evaluation of knowledge, mean (SD)22.3 (4.6)16.4 (5.5)<.001a
    T2 Self-evaluation of knowledge, mean (SD)22.9 (3.6)22.1 (3.9).26
    T0 Provider’s confidence, mean (SD)5.1 (1.8)5.0 (1.7).61
    T1 Provider’s confidence, mean (SD)6.5 (1.7)4.6 (1.9)<.001a
    T2 Provider’s confidence, mean (SD)6.5 (1.5)6.0 (1.7).23

    aP values<.05 were considered significant.

    bT0: baseline.

    cT1: 1 month after baseline.

    dT2: 3 months after baseline.

    Dropout and Missing Data

    Figure 1 shows the flow of students through the trial. Of the 211 intervention group participants, 141 (66.8%) completed the T1 follow-up, and 59 (28%) completed the T2 follow-up. In the waitlist control group (n=187), 113 participants (60.5%) completed the T1 follow-up. After T1, the participants in the waitlist control condition were offered the intervention, of which 46 (25%) used the e-learning and completed the T2 follow-up.

    Dropout rates at T1 in the intervention condition were higher than those in the waitlist control condition and were comparable in both conditions at T2. Values at T1 and T2 were not missing at random according to the Little’s missing completely at random test (χ211=22.8; P=.02). Baseline characteristics were used to predict missing values at follow-up. Completers were younger than noncompleters at T1 (t180=–2.4; P=.02) and T2 (t330=–2.2; P=.026). Completers showed higher levels of provider’s confidence than noncompleters at T1 (t301=2.2; P=.03). For all other baseline characteristics, noncompleters did not differ from completers (all P values>.05).

    ‎
    Figure 1. Flow of students through the trial. T0: baseline; T1: 1 month after baseline; T2: 3 months after baseline.

    Imputed Data: Effect of Module on Guideline Adherence, Confidence, and Knowledge

    After receiving the e-module (T1), participants in the intervention group reported higher levels of self-evaluated knowledge (beta=5.3; SE=0.43; d=1.6), guideline adherence (beta=3.5; SE=1.0; d=0.15), and provider’s confidence (beta=1.7; SE=0.17; d=1.1) than those in the waitlist control group. Baseline levels of knowledge, confidence, and guideline adherence were included as covariates in the regression analyses.

    Completer Analysis

    When completers-only data (intervention group: n=141 and waitlist control group: n=113) was used at T1, the intervention group scored higher on guideline adherence (beta=3.5; 95% CI 1.7-5.8; d=0.5), knowledge (beta=5.3; 95% CI 4.5-6.1; d=1.4), and confidence (beta=1.7; 95% CI 1.3-2.0; d=1.1) than the waitlist control group. Provider’s confidence in dealing with suicidal behavior (t140=9.9; P<.001), guideline adherence (t140=6.19; P<.001), and self-reported knowledge (t140=15.42; P<.001) significantly increased in the participants in the intervention group after completion of the e-learning module.

    Comparison of the scores at T1 and T2 follow-up that were made using completers-only data showed that levels of knowledge, guideline adherence, and confidence within the intervention group stayed constant (all P values>.05), and the levels of the outcome variables, such as knowledge (t45=7.6; P<.001), guideline adherence (t45=2.5; P<.018), and provider’s confidence (t45=2.5; P=.002), within the waitlist control group improved significantly after receiving the e-learning module.

    Evaluation of the Module by Completers

    Participants who completed the intervention (N=102) filled in the evaluation questionnaires at T2, indicating the extent to which they agreed with the statements evaluating the e-learning module. The results show evaluation scores ranging from 6.2 for the statement “I feel confident dealing with people with suicidal thoughts” (SD 1.9) to 7.5 for the statement “I appreciate the e-learning module” (SD 1.5). In addition, the participants rated the other statements in a moderately positive manner, for example, “I learned a lot about prevention of suicide” (mean 6.8, SD 1.4), “The module is a good preparation for my internship/future/job as psychologist” (mean 6.9, SD 1.4), and “I would recommend the module to my colleagues and fellow students” (mean 7.1, SD 1.5).


    Principal Findings

    This study aimed to test the effectiveness of a suicide e-learning module in undergraduate students with respect to the knowledge of suicide prevention, provider’s confidence to have a dialogue about suicide, and adherence to the national guidelines on the assessment of suicidal behavior [26]. One month after completing the e-learning module, the students in the intervention condition reported feeling more confident and knowledgeable compared with those in the waitlist control group who did not have access to the e-learning module. Completers in the intervention condition demonstrated better guideline adherence, knowledge, and provider’s confidence than waitlist controls. The learning effects of the e-learning module were maintained at the 3-month follow-up. These results were replicated in participants in the waitlist control condition who completed the e-learning module after T1. The results were comparable with the improvement in provider’s confidence, guideline adherence, and self-reported knowledge in mental health care professionals who received both e-learning and face-to-face training [22,24]. These findings are surprising as well as encouraging because we expected smaller effect sizes in students compared with professionals owing to the intense 1-day face-to-face training that accompanied the e-learning for the professionals [23]. Owing to less clinical experience, the baseline skills and knowledge level of students were lower than that of the professionals and may have provided more room for improvement. Hence, the effects of additional face-to-face training in this group of undergraduate students above those of the e-learning module are worth investigating.

    Furthermore, the results were also comparable with those of a study evaluating an e-learning module on suicide prevention in mental health providers in the United States [30]. In this study, the effects of e-learning were similar to that of an in-person training on suicide prevention. The effect sizes of guideline adherence were considered small; however, in this study, approximately 10% change in guideline adherence, provider’s confidence, and knowledge was found, which is consistent with other studies in general medicine and psychiatry on the effect of educative interventions that show an average of 10% improvement [31,32].

    Strengths and Limitations

    The randomized controlled design with follow-up is rare in this field of research and can be considered a strength in this study [33]. However, dropout rate in our study, although comparable with a previous study among students [16] and yet still higher than in a study involving professionals [22], was a significant limitation and may represent diminished student satisfaction regarding e-learning [34]. In addition, differential dropout occurred across conditions, possibly because of motivational reasons. Furthermore, the evaluations of guideline adherence, confidence, and knowledge were self-reported, which are subject to the effects of social desirability and demand characteristics, although fairly reliable [35,36]. To get a more objective outcome, one could use role-plays in which a participant interacts with a patient with suicidal tendencies; however, this is much more time consuming. Although this study showed the effectiveness of e-learning in a waitlist control design, its relative effectiveness to alternative research-validated suicide prevention training programs [37] remains unclear and worth studying, especially considering the potential cost- and resource-efficiency of e-learning. Finally, it remains important to question how well the results obtained in a volunteer student sample generalize to working professionals required to take the course. However, as mentioned previously, similar results were found for mental health professionals in our previous study [23], indicating promising generalizability for the workforce.

    Implications and Future Research

    E-learning modules, as the one used in this study, may be a beneficial, accessible way to provide training in suicide prevention skills to both students and professionals [38]. We propose to use e-learning modules in training suicide prevention skills in a wide range of higher education disciplines, such as curricula in nursing, social work, psychology, and psychiatry, to improve the knowledge and skills of clinicians-in-training regarding suicide prevention on a greater scale and consequently improve the care of patients with suicidal thoughts. To evaluate the transfer of knowledge and skills, and competencies of the clinicians (in training) to deal with clients’ suicidal behavior and subsequently reduce suicidality rates, future studies should include (objective) measures for clinician behavior. In addition, careful logging of content exposure and participant adherence to the program is advised for future studies to evaluate the extent to which the effects are attributable to the actual intervention as well as to evaluate the acceptability and feasibility of the intervention for future implementation. Because evidence of the cost-effectiveness of Web-based education in students is still lacking [14], future studies on the benefits of e-learning should consider (the reduction in) costs of personnel, education, and implementation to evaluate the cost-effectiveness. This study, with a randomized design and a feasible sample size of undergraduate students, provides first evidence of the positive effects of acquainting students on a large scale with the suicide prevention guidelines by means of an e-learning module. An e-learning module on the prevention of suicide could be an effective first step toward the improvement of clinical skill knowledge [39]. The learning outcomes of a stand-alone module were found to be similar to those of a training that combined e-learning with a face-to-face training, with the advantages of flexibility and low costs [15,30]. Furthermore, the students found the effects of the module to be reasonably positive; they believed that it prepared them well for their internship or job, and they were willing to recommend the module to colleagues and fellow students.

    Acknowledgments

    The development of the e-learning modules was funded by The Netherlands Organisation for Health Research and Development (ZonMw).

    Authors' Contributions

    DdB designed the e-learning modules. DdB and RMFK performed the study. MLJK conducted all analyses and drafted the manuscript. JM, MS, and AJFMK contributed to writing of the manuscript. All authors contributed to the execution of the study and approved the final draft.

    Conflicts of Interest

    None declared.

    1. de Beurs DP, Hooiveld M, Kerkhof AJ, Korevaar JC, Donker GA. Trends in suicidal behaviour in Dutch general practice 1983-2013: a retrospective observational study. BMJ Open. May 10, 2016;6(5):e010868. [FREE Full text] [CrossRef] [Medline]
    2. StatLine - CBS. URL: https://opendata.cbs.nl/statline/#/CBS/nl/dataset/7022gza/table?ts=1542371154616 [accessed 2018-11-16] [WebCite Cache]
    3. Hjelmeland H, Knizek BL. The general public's views on suicide and suicide prevention, and their perception of participating in a study on attitudes towards suicide. Arch Suicide Res. 2004;8(4):345-359. [CrossRef] [Medline]
    4. Biddle L, Cooper J, Owen-Smith A, Klineberg E, Bennewith O, Hawton K, et al. Qualitative interviewing with vulnerable populations: individuals' experiences of participating in suicide and self-harm based research. J Affect Disord. Mar 5, 2013;145(3):356-362. [CrossRef] [Medline]
    5. Dazzi T, Gribble R, Wessely S, Fear NT. Does asking about suicide and related behaviours induce suicidal ideation? What is the evidence? Psychol Med. Dec 2014;44(16):3361-3363. [CrossRef] [Medline]
    6. Ruiz J, Mintzer MJ, Leipzig RM. The impact of E-learning in medical education. Acad Med. Mar 2006;81(3):207-212. [CrossRef] [Medline]
    7. Cook DA, Levinson AJ, Garside S, Dupras DM, Erwin PJ, Montori VM. Internet-based learning in the health professions: a meta-analysis. J Am Med Assoc. Sep 10, 2008;300(10):1181-1196. [CrossRef] [Medline]
    8. Maloney S, Chamberlain M, Morrison S, Kotsanas G, Keating JL, Ilic D. Health professional learner attitudes and use of digital learning resources. J Med Internet Res. Jan 16, 2013;15(1):e7. [FREE Full text] [CrossRef] [Medline]
    9. George PP, Papachristou N, Belisario JM, Wang W, Wark PA, Cotic Z, et al. Online eLearning for undergraduates in health professions: a systematic review of the impact on knowledge, skills, attitudes and satisfaction. J Glob Health. Jun 2014;4(1):010406. [FREE Full text] [CrossRef] [Medline]
    10. Schopf T, Flytkjaer V. Impact of interactive web-based education with mobile and email-based support of general practitioners on treatment and referral patterns of patients with atopic dermatitis: randomized controlled trial. J Med Internet Res. Dec 5, 2012;14(6):e171. [FREE Full text] [CrossRef] [Medline]
    11. Means B, Toyama Y, Murphy R, Baki M. The effectiveness of online and blended learning: a meta-analysis of the empirical literature. Teach Coll Rec. 2013;115(3):1-47.
    12. Corrigan M, Reardon M, Shields C, Redmond H. 'SURGENT' -- student e-learning for reality: the application of interactive visual images to problem-based learning in undergraduate surgery. J Surg Educ. 2008;65(2):120-125. [CrossRef] [Medline]
    13. Hare E. E-learning for psychiatrists. Psychiatr Bull. 2009;33(3):81-83. [FREE Full text] [CrossRef]
    14. McCall M, Spencer E, Owen H, Roberts N, Heneghan C. Characteristics and efficacy of digital health education: an overview of systematic reviews. Health Educ J. 2018;77(5):497-514. [FREE Full text] [CrossRef]
    15. Sinclair PM, Kable A, Levett-Jones T, Booth D. The effectiveness of Internet-based e-learning on clinician behaviour and patient outcomes: a systematic review. Int J Nurs Stud. May 2016;57:70-81. [CrossRef] [Medline]
    16. Davies EB, Beever E, Glazebrook C. A pilot randomised controlled study of the mental health first aid eLearning course with UK medical students. BMC Med Educ. Mar 21, 2018;18(1):45. [FREE Full text] [CrossRef] [Medline]
    17. Fairburn CG, Allen E, Bailey-Straebler S, O'Connor ME, Cooper Z. Scaling up psychological treatments: a countrywide test of the online training of therapists. J Med Internet Res. Jun 16, 2017;19(6):e214. [FREE Full text] [CrossRef] [Medline]
    18. O'Connor M, Morgan KE, Bailey-Straebler S, Fairburn CG, Cooper Z. Increasing the availability of psychological treatments: a multinational study of a scalable method for training therapists. J Med Internet Res. Jun 8, 2018;20(6):e10386. [FREE Full text] [CrossRef] [Medline]
    19. Hawton K, Pirkis J. Suicide is a complex problem that requires a range of prevention initiatives and methods of evaluation. Br J Psychiatry. Jun 2017;210(6):381-383. [CrossRef] [Medline]
    20. Setkowski K, Mokkenstorm J, van Balkom AJ, Franx G, Verbeek-van Noord I, Dongelmans DA, et al. Feasibility and impact of data-driven learning within the suicide prevention action network of thirteen specialist mental healthcare institutions (SUPRANET Care) in the Netherlands: a study protocol. BMJ Open. Aug 20, 2018;8(8):e024398. [FREE Full text] [CrossRef] [Medline]
    21. de Beurs DP, de Groot MH, de Keijser J, Verwey B, Mokkenstorm J, Twisk JW, et al. Improving the application of a practice guideline for the assessment and treatment of suicidal behavior by training the full staff of psychiatric departments via an e-learning supported Train-the-Trainer program: study protocol for a randomized controlled trial. Trials. Jan 9, 2013;14:9. [FREE Full text] [CrossRef] [Medline]
    22. de Beurs DP, de Groot MH, de Keijser J, Mokkenstorm J, van Duijn E, de Winter RF, et al. The effect of an e-learning supported Train-the-Trainer programme on implementation of suicide guidelines in mental health care. J Affect Disord. Apr 1, 2015;175:446-453. [FREE Full text] [CrossRef] [Medline]
    23. de Groot M, de Beurs DP, de Keijser J, Kerkhof AF. An e-learning supported Train-the-Trainer program to implement a suicide practice guideline. Rationale, content and dissemination in Dutch mental health care. Internet Interv. 2015;2(3):323-329. [FREE Full text] [CrossRef]
    24. de Beurs DP, de Groot MH, de Keijser J, van Duijn E, de Winter RF, Kerkhof AJ. Evaluation of benefit to patients of training mental health professionals in suicide guidelines: cluster randomised trial. Br J Psychiatry. May 2016;208(5):477-483. [CrossRef] [Medline]
    25. de Beurs DP, de Groot MH, de Keijser J, van Duijn E, de Winter RF, Kerkhof AJ. Semantic Scholar. 2015. URL: https://pdfs.semanticscholar.org/8135/0e019465486384f64a6d358b231583b7984e.pdf [accessed 2019-11-12]
    26. Lucassen P, Kerkhof AJ, van Hemert AM, Steendam M, van Boven C, de Groot MH, et al. Multidisciplinary Guideline for Diagnosis and Treatment of Suicidal Behavior. Utrecht. de Tijdstroom; 2012.
    27. Tompkins T, Witt J. The short-term effectiveness of a suicide prevention gatekeeper training program in a college setting with residence life advisers. J Prim Prev. Mar 2009;30(2):131-149. [CrossRef] [Medline]
    28. Oordt MS, Jobes DA, Fonseca VP, Schmidt SM. Training mental health professionals to assess and manage suicidal behavior: can provider confidence and practice behaviors be altered? Suicide Life Threat Behav. Feb 2009;39(1):21-32. [CrossRef] [Medline]
    29. van Buuren S. Multiple imputation of discrete and continuous data by fully conditional specification. Stat Methods Med Res. Jun 2007;16(3):219-242. [CrossRef] [Medline]
    30. Magruder K, York J, Knapp R, Yeager D, Marshall E, DeSantis M. RCT evaluating provider outcomes by suicide prevention training modality: in-person vs e-learning. J Ment Health Train Educ Prac. 2015;10(4):207-217. [FREE Full text] [CrossRef]
    31. Grol R, Grimshaw J. From best evidence to best practice: effective implementation of change in patients' care. Lancet. Oct 11, 2003;362(9391):1225-1230. [CrossRef] [Medline]
    32. Wensing M, Grol M. Implementation: Effective Improvement of Patient Care. Houten. Bohn Stafleu van Loghum; 2017.
    33. Weinmann S, Koesters M, Becker T. Effects of implementation of psychiatric guidelines on provider performance and patient outcome: systematic review. Acta Psychiatr Scand. Jun 2007;115(6):420-433. [CrossRef] [Medline]
    34. Levy Y. Comparing dropouts and persistence in e-learning courses. Comput Educ. 2007;48(2):185-204. [FREE Full text] [CrossRef]
    35. Ross J. The reliability, validity, and utility of self-assessment. Pract Assess Res Eval. Nov 2006;11(10). [FREE Full text]
    36. Falchikov N, Boud D. Student self-assessment in higher education: a meta-analysis. Rev Educ Res. 1989;59(4):395-430. [FREE Full text] [CrossRef]
    37. Zalsman G, Hawton K, Wasserman D, van Heeringen K, Arensman E, Sarchiapone M, et al. Suicide prevention strategies revisited: 10-year systematic review. Lancet Psychiatry. Jul 2016;3(7):646-659. [CrossRef] [Medline]
    38. Gryglewicz K, Chen JI, Romero GD, Karver MS, Witmeier M. Online suicide risk assessment and management training. Crisis. May 2017;38(3):186-194. [CrossRef] [Medline]
    39. McCutcheon K, Lohan M, Traynor M, Martin D. A systematic review evaluating the impact of online or blended learning vs face-to-face learning of clinical skills in undergraduate nurse education. J Adv Nurs. Feb 2015;71(2):255-270. [CrossRef] [Medline]

    ‎
    e-learning: electronic learning
    PITSTOP: Professionals in Training to STOP
    T0: baseline
    T1: 1 month after baseline
    T2: 3 months after baseline
    VU: Vrije Universiteit

    Edited by C Eichenberg; submitted 06.05.19; peer-reviewed by A Pisani, F Shand, R Melia, M Yap; comments to author 15.07.19; revised version received 02.09.19; accepted 07.10.19; published 22.01.20.

    Copyright

    ©Marie-Louise J Kullberg, Joanne Mouthaan, Maartje Schoorl, Derek de Beurs, Robin Maria Francisca Kenter, Ad JFM Kerkhof. Originally published in JMIR Mental Health (http://mental.jmir.org), 22.01.2020.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Mental Health, is properly cited. The complete bibliographic information, a link to the original publication on http://mental.jmir.org/, as well as this copyright and license information must be included.