Suicide is the second leading cause of death for adolescents, and the rate of suicide in the United States has increased in recent years [1,2]. Suicidal thoughts and behaviors (including suicidal ideation, nonsuicidal self-injury, and suicide attempts) increase dramatically during adolescence and are unfortunately common. Data suggest that 17% of US high school students seriously consider suicide each year, and approximately 9% report a suicide attempt [3]. Despite this being a common health concern, our ability to predict suicide is poor [4].

High schools provide a convenient and accessible setting to promote mental health, as most children attend high schools, removing practical barriers for mental health services and promoting care for traditionally underserved groups [5-9]. Thus, schools have the potential to play an essential role in supporting the identification and treatment of youth with mental health difficulties. School is also the most common community setting where suicidal ideation is identified [10]; however, the majority of schools demonstrate poor adherence to gold standard practices related to the identification and treatment of youth at risk [11,12]. Many high schools do not adopt recommended suicide prevention strategies because of practical concerns, such as the capacity to manage false positives, lack of knowledge of evidence-based practices, fear, stigma, as well as legal and ethical issues [11,13].

Technology-enabled services (TESs) hold promise for adolescent suicide prevention because of their capacity to support best practices while achieving population health and may help address many of the practical concerns that act as barriers to the adoption of suicide prevention strategies (ie, cognitive load, burden, and costs). TESs are characterized by having both a human service component (eg, therapist-delivered psychotherapies) and a digital component (eg, dashboard app) that supports, or is supported by, the service [14]. A broad range of support falls under the category of TES, including web- or app-based supports for posttraumatic stress disorder treatment (eg, Prolonged Exposure Coach), interventions for insomnia (Cognitive Behavioral Therapy for Insomnia Coach), and substance use disorders (reSET) [15,16]. Efficacy trials indicate that TESs yield effects commensurate with well-established psychological interventions for depression, anxiety, and suicidal ideation/behavior [17-19]. Nearly all teens are heavy users of smartphones and other digital technologies and use these technologies for health-related concerns [20]. Furthermore, adolescents are seeking TES to manage their health [21]. TES via text-based extensions (eg, Text4Strength with Sources of Strength) of suicide prevention programs show initial feasibility, safety, and utility from the adolescent perspective [22].

High schools represent a key setting for which TES can be effectively adapted and applied. An important suicide prevention research priority is to evaluate how TES may address many of the challenges that schools face in adopting suicide prevention strategies. For example, TES that automatically and continuously monitors social media (SM) data and requires virtually no staff or resources such as time in the classroom to execute may be valuable in reducing the burden of universal screening. Furthermore, strategies that rely on student-generated SM data may reduce common concerns about specific suicide or emotional health screening tools [23,24]. Furthermore, as TES does not rely on explicit reports of suicidal ideation or behavior, the potential for student stigmatization is substantially reduced. A platform that allows for passive data aggregation and monitoring is ongoing rather than linked to one particular assessment time point, facilitating an identification approach that aligns with the episodic nature of suicidality [23-25]. For these reasons, strategic monitoring of SM, has the potential to have a considerable impact on public health via scalable early detection and intervention to decrease adolescent suicide rates.

Although research on the efficacy of TES for mental health broadly—and suicide prevention specifically—is promising [21], school settings have experienced few benefits from TES. Much of this is likely because of insufficient attention to (1) end users’ priorities and experiences regarding aspects of the technology and the human service and (2) the implementation strategies that promote their adoption and sustained use [14,26]. To improve TES implementation, developers are increasingly turning to the methods of human-centered design (HCD) to identify and address problematic system design and its impact on otherwise appealing and effective products [27]. HCD includes a set of approaches that ground the development process in information about the needs and desires of people who will ultimately use a product, with the goal of creating compelling, intuitive, and effective innovations [28,29]. Usability testing, a hallmark of HCD, provides an opportunity for representative end users to interact with the technology, complete specific tasks, and generate information about the functionality and presentation. Owing to the potential impact of TES usability on implementation outcomes [30], we prioritized this determinant in the study design.

The goal of this research was to evaluate the implementation outcomes of acceptability, feasibility, and appropriateness from the primary users of a TES (Quinn Therapeutic) for school-based suicide prevention (ie, school staff and students). These perceived implementation outcomes are critical precursors to adoption and use [30] and can be most effectively assessed at early project stages before actual implementation occurs [31]. As articulated by Proctor et al [31], acceptability is defined as perceptions that an innovation is agreeable, palatable, or satisfactory; feasibility is the extent to which a new innovation can be successfully used or carried out within a given agency or setting; and appropriateness refers to the innovation’s fit, relevance, and compatibility with the setting, staff, and target problem. We also evaluated implementation determinants, including usability factors, driving our primary implementation outcomes. Our specific research questions were as follows: (1) What key, multilevel factors in the school context should drive the adaptation and implementation of a student-monitoring dashboard interface of the Quinn Therapeutic and (2) What changes to the digital dashboard interface are needed to maximize its acceptability, feasibility, appropriateness, and ultimate usability for school systems? We focus on the dashboard interface for applying HCD as the most salient and user-facing feature of TES.

Participants were drawn from an urban area in the Pacific Northwest. Recruitment of participants occurred through past and ongoing research partnerships. Interested principals and/or school counselor leads were presented with the information regarding study procedures in an initial meeting and then recontacted if they were interested in having students and school personnel the opportunity to participate. Recruitment of clinician participants was done in the winter of 2018, with user testing procedures completed in February and March; student recruitment and study student data collection procedures were performed in May 2018. All procedures were approved by our institutional review board (study 3246).

Quinn Therapeutic is a TES that aggregates patient-generated SM data to detect and monitor suicide risk, visualize data over time, and provide feedback to clinicians, which may be particularly suitable in the high school context [37-39]. Quinn Therapeutic’s core digital and human service features are outlined below (Table 1). The technology relies on deep learning (a subset of machine learning algorithms), which enable a computer to discover and use patterns in data, trained, and optimized using SM data [40]. The aggregated SM data and risk ratings based on machine learning algorithms are presented in a student-monitoring dashboard interface, which clinicians log into to view estimated student risk status, data over time, and SM content driving ratings of risk. The dashboard’s purpose is to allow school personnel to monitor student suicide risk over time. Core functions are represented in the web-based supplement. A timeline shows the overall data contribution for the population; there is a search function for finding specific students, and tabs categorize students with high risk for ease of viewing the at-risk population with past suicidal ideation and self-harm. School personnel can also view individuals, including their suicide risk level over time (ie, risk level graph), source SM data that generate the risk ratings, the strength and valence of sentiment (ie, sentiment graph), and students self-reported diagnostic and suicide risk information. Quinn Therapeutic’s predictive algorithms have been developed and evaluated in users who donate their data to an online portal as well as publicly available data from Twitter and demonstrated impressive accuracy in distinguishing those with identifying self-reported suicide attempts from those who did not report this history [41]. The algorithms demonstrated the capability to separate users who would attempt suicide from neurotypical controls. Evaluating SM data from the month before a suicide attempt, the area under the curve (AUC) from receiver operating characteristics for this binary decision task was 0.89, and for all available SM data, AUC was 0.94 (an AUC of 1 is perfect prediction) [42]. Quinn Therapeutic human service components include measurement-based care, crisis prevention planning, and risk management, all of which align with recommendations for the identification and management of adolescent suicide risk [43]. As a first step in evaluating the application of Quinn Therapeutic TES in the school setting, the current project focused only on an early prototype of Quinn Therapeutic’s digital technology component.

For user testing sessions, school personnel viewed the student-monitoring dashboard populated with dummy data. The dashboard allows for data visualization of the posterior probability (range 0-1) representing the likelihood that each individual SM post was written by someone at risk of suicide and status updates that were analyzed with machine learning algorithms developed in prior work [42]. Cohort data, individual monitoring data (time series and risk rating), and source content (SM posts/behavior) that generated ratings were viewed via the dashboard.

Mutltimedia Appendix 1 provides information regarding the measures by the reporter as well as the sample items.

Descriptive statistics, including means and SDs, were calculated for quantitative measures. Qualitative content was coded using the Consolidated Framework for Implementation Research (CFIR) [46]. The CFIR is a commonly used framework that organizes constructs that have been associated with effective implementation. It has been widely used as a practical guide to evaluate implementation efforts in preparation for or during active studies [46]. The codebook template was used to understand the multilevel determinants of implementation. Determinants include aspects of the innovation (eg, evidence strength and relative advantage), outer context (eg, external policies and incentives), the inner organizational context (eg, implementation climate and tension for change ), characteristics of the individuals operating within target settings (eg, attitudes and efficacy), and process of change in the organization (eg, engagement strategies and change agents) [47-51]. School personnel interviews were audio recorded, transcribed, and coded with directed content analysis. In total, 4 coders were trained to conduct directed content analysis based on the CFIR codebook by reviewing the codebook and example codes, reviewing the school personnel’ s responses to each question from the same two transcripts, identifying potential codes, and independently coding. Consensus among the four coders for the two transcripts was achieved through open dialog [52]. Following consensus on the two transcripts, two coding team members who had completed the consensus coding were split into two groups. The remaining transcripts were coded independently and then the two groups met to review codes in consensus meetings. A consensus coding process was used to reduce biases, groupthink, and errors [53]. This coding approach was used, as many qualitative researchers consider it to be more valid for analyzing human communication, as it explicitly uses coding ambiguities to prompt discussion and increases confidence in complex data compared with interrater reliability [54].

The demographic characteristics of the participants are included in Tables 2 and 3.

School personnel gave the student-monitoring dashboard moderate scores, on average, for acceptability (mean 3.69, SD 0.85; range 2.25-5.00), appropriate (mean 3.72, SD 1.09; range 2.00-5.00), and feasibility (mean 3.78, SD 0.75; range 2.25-4.75) of implementation in their setting, indicating school personnel viewed the student-monitoring dashboard as moderately appropriate for the school setting.

To understand the reasons for ratings of core implementation outcomes, qualitative themes at the level of the innovation, outer setting, inner setting, individual characteristics, and engagement were summarized (Multimedia Appendix 2). The following three themes emerged: (1) compatibility with culture, values, and norms in the school setting; (2) additional attention needed to confidentiality and privacy; and (3) flexibility in the way to support students. The majority of the qualitative codes related to the first theme, that is, the organizational context, culture, resources, and structure (161/350, 46.0% of school personnel comments and 118/222, 53.2% of student comments). Specific comments highlight positive aspects of the system being compatible with culture and values/norms in the school setting. For example, a student indicated:

However, both participant groups reported difficulty in managing confidentiality and privacy within this context and adequately managing the workflow. For example, 1 clinician stated:

A student’s perspective highlighted:

The second theme highlighted the need for careful attention to how information would be used within the school setting and remain confidential. Some expressed uncertainty about the extent to which machine learning can discern the complexities of unstructured text and nuanced communication occurring on SM platforms:

School personnel and students noted wanting clarity on how the TES would impact internal communications and other external systems outside the school, including the district and outside resources (eg, therapists outside of the school and crisis responding).

The third theme relates to the potential for an approach similar to this to expand options for youth at risk. Overall, 34.0% of school personnel comments and 18.0% of student comments highlight the innovative aspects of using passive and ongoing data collection in this way. Themes of positive comments related to the relative perceived advantage of a technology-based solution compared with the status quo as well as the ability to provide individualized solutions and options for youth who appear distressed and/or suicidal. One student noted:

Another student stated the asset of flexibility:

In this study, high school students and school personnel provided feedback on implementation determinants and outcomes to facilitate the redesign of a TES to support suicide risk identification and prevention. Universal emotional health screening is recognized as an essential component of a multitier system of support and behavioral health framework [55]. Universal emotional health screening may facilitate the identification of undetected difficulties [56]; however, emotional health screening is rarely conducted in school settings because of feasibility, burden on school personnel, and lack of knowledge of best practices. A solution that supports accurate, ongoing, and passive screening for youth risk, clinical decision making, and improved communication and that fits within the school context would be a great asset toward facilitating identification and triage of students at risk for suicide. Through mixed qualitative and quantitative approaches, our study identified a number of strengths of the digital component of the Quinn Therapeutic TES. We additionally identified several challenges related to the school context and concerns regarding fit within workflow and the network of communications around protected health information such as suicidality. Three primary themes identified by students and school personnel were (1) compatibility with culture, values, and norms in the school setting; (2) additional attention needed to confidentiality and privacy; and (3) flexibility in the way to support students. With regard to compatibility with the culture, students, and school personnel highlighted that this approach aligned with the school’s value to support well-being and help achieve goals related to caring for the whole student. However, both stakeholders reported a need for additional information about the data and processes for analysis, interpretation, communication, and human responses were requested. Student concerns regarding confidentiality were centered on how school personnel would manage communications, not sharing data with a company for purposes of suicide prevention. Along with other researchers, before the widespread acceptance of a system that was supported by existing data sources such as SM, additional education regarding the validity of psychologically relevant data can be measured via SM language. Finally, both school personnel and students found appeal in TES flexibility, including the ability for multiple options for enrollment and strategies to support students.

Several researchers have suggested that digital innovations that rely on machine learning strategies similar to the one evaluated in this research would provide significant advances in the field [4,57-60]. In addition, programs that rely on suicide risk prediction algorithms have been deployed in the Veteran’s administration. This program, called Recovery Engagement and Coordination for Health—Veterans Enhanced Treatment, uses medical record data and applies machine learning to identify those at a statistically elevated risk for suicide or other adverse outcomes. At present, there is an active clinical trial of the program (NCT03280225). The evaluation of the application of machine learning algorithms to medical records for the prediction of suicide attempts has demonstrated good performance [61]. Few programs have been designed to provide rigorous evaluation of the usability and other implementation determinants for the use of a technology-based solution to universal suicide screening in the school context. Several limitations must be considered when interpreting the results. First, the sample of users was small, and not representative, as it was limited to participants from a small private school in an urban area. Second, we coded perceptions of the intervention implementation following scenario-based user testing, not actual implementation of the intervention, and, therefore, may not be valid for real-world implementation. Finally, we only included primary end users, that is, students and school personnel who would be involved in responding to suicide risk directly and, therefore, did not include other important stakeholders such as teachers and parents.

Strategies to make suicide prevention efforts in high schools scalable, sustainable, and supportive may benefit from attention to how technology can facilitate and aid human efforts. This research evaluated a system that aggregates existing data sources—SM data—to provide ongoing monitoring of suicide risk based on machine learning algorithms. Primary users—high school students and school personnel—highlighted the potential advantages of providing individualized solutions and options for youth compared with the current suicide prevention strategy within the school (which included gatekeeper training, mental health awareness group, and onsite counseling support). However, the management of private and sensitive communications in the school context and limited functionality of the prototype dashboard dampened enthusiasm for widespread implementation. Although further investment in an improved user interface may improve some of the concerns, the large fundamental challenge facing this and similar TES is a lack of understanding and policy surrounding the privacy and use of sensitive communications in the school context. Widespread agreement on community norms and commonly accepted guidelines for how and when to use this sort of data will be necessary for the widespread adoption of any similar TES.