Published on in Vol 10 (2023)

Preprints (earlier versions) of this paper are available at, first published .
Comparing the Acceptability and Quality of Intervention Modalities for Suicidality in the Emergency Department: Randomized Feasibility Trial

Comparing the Acceptability and Quality of Intervention Modalities for Suicidality in the Emergency Department: Randomized Feasibility Trial

Comparing the Acceptability and Quality of Intervention Modalities for Suicidality in the Emergency Department: Randomized Feasibility Trial

Original Paper

1Department of Emergency Medicine, University of Massachusetts Chan Medical School, Worcester, MA, United States

2The Business School, Worcester Polytechnic Institute, Worcester, MA, United States

Corresponding Author:

Celine Larkin, PhD

Department of Emergency Medicine

University of Massachusetts Chan Medical School

55 Lake Avenue North

Worcester, MA, 01655

United States

Phone: 1 5084211436


Background: Emergency departments (EDs) manage many patients with suicide risk, but effective interventions for suicidality are challenging to implement in this setting. ReachCare is a technology-facilitated version of an evidence-based intervention for suicidal ED patients. Here, we present findings on the acceptability and quality of ReachCare in the ED, as well as a comparison of these measures across 3 potential delivery modalities

Objective: Our aim was to test the feasibility of the ReachCare intervention in its entirety through conducting a pilot study with patients presenting with suicidality to the ED. We tested three different ways of receiving the ED-based components of ReachCare: (1) self-administered on the tablet app using a chatbot interface, (2) administered by an in-person clinician, or (3) administered by a telehealth clinician.

Methods: In total, 47 ED patients who screened positive for suicide risk were randomly allocated to receive one of three delivery modalities of ReachCare in the ED: (1) self-administered on the patient-facing tablet app with a chatbot interface, (2) delivered by an in-person clinician, or (3) delivered by a telehealth clinician, with the latter two using a clinician-facing web app. We measured demographic and clinical characteristics, acceptability and appropriateness of the intervention, and quality and completeness of the resulting safety plans.

Results: Patients assigned high ratings for the acceptability (median 4.00/5, IQR 4.00-4.50) and appropriateness (median 4.00/5, IQR 4.00-4.25) of ReachCare’s ED components, and there were no substantial differences across the 3 delivery modalities [H(acceptability)=3.90, P=.14; H(appropriateness)=1.05, P=.59]. The self-administered modality took significantly less time than the 2 clinician modalities (H=27.91, P<.001), and the usability of the self-administered version was in the “very high” range (median 93.75/100, IQR 80.00-97.50). The safety plans created across all 3 modalities were high-quality (H=0.60, P=.74).

Conclusions: Patients rated ReachCare in the ED as highly acceptable and appropriate regardless of modality. Self-administration may be a feasible way to ensure patients with suicide risk receive an intervention in resource constrained EDs. Limitations include small sample size and demographic differences between those enrolled versus not enrolled. Further research will examine the clinical outcomes of patients receiving both the in-ED and post-ED components of ReachCare.

Trial Registration: NCT04720911;

JMIR Ment Health 2023;10:e49783



After a brief decline in 2019-2020 [1,2], suicide rates have again begun to rise in the United States [1]. Each year, there are over 1 million suicide-related visits to emergency departments (EDs) in the United States [3], and EDs represent a feasible venue to screen and assess suicide risk [4,5]. However, once risk is detected, many ED patients with suicide risk receive suboptimal care [6].

Several effective interventions have been developed for this patient population [7]. The Emergency Department Safety Assessment and Follow-up Evaluation (ED-SAFE) intervention [8], for example, featured paper-based self-administered safety planning and resource lists in the ED; this was followed by intensive telephone outreach to patients and a chosen significant other in the 12 months after discharge. The telephone counseling involved safety planning, outpatient treatment engagement, values clarification and life planning, and family communication and coordination. The ED-SAFE intervention was associated with a 30% reduction in the total number of suicide attempts in the intervention group compared to treatment-as-usual [8]. Despite its promise and cost-effectiveness [9], the ED-SAFE intervention was resource-intensive and difficult to sustain without research support. Such interventions routinely struggle to “cross the chasm” to become routine care [10].

Clinicians and leadership recognize that ED-based care for suicidality needs to improve but often face substantial constraints in effecting change [11,12]. The dearth of effective care for suicidality in the ED can be attributed to a wide range of factors, including a hectic milieu focused on patient flow [13], a shortage of behavioral health (BH) providers [14], and a shortfall in suicide-related skills and confidence among ED clinicians [15,16]. With these barriers in mind, we set out to adapt the ED-SAFE intervention to become “ReachCare,” leveraging technology to make the intervention scalable in resource-constrained EDs, while avoiding a corresponding loss in fidelity.

We previously outlined the user-centered development and testing of the patient-facing technology aspects of the ReachCare intervention [17]. These technologies include a tablet-based app for use within the ED (that allows a patient to build a high-quality safety plan in the absence of a BH clinician) and a patient- and family-facing mobile app (that houses the patient’s safety plan and life plan, helpline resources, psychoeducation, distractions, and a BH provider search engine). In that study, user-testing with current ED patients presenting with suicidality demonstrated high usability and acceptability of the technologies [17]. ReachCare is further supported by interoperable clinician-facing software that provides tips, call structure, and note-taking for clinicians, as well as allowing them to update and share patients’ therapeutic tools during follow-up phone calls. This clinician-facing technology was designed to be used in-person or via telehealth.

We conducted a feasibility trial with patients presenting with suicidality to the ED to develop and test implementation strategies for the ReachCare intervention that will be used in a future effectiveness trial. We tested three different modalities of delivering the ED-based components of ReachCare, randomizing patients to receive: (1) self-administered on the tablet app using a chatbot interface, (2) administered by an in-person clinician, or (3) administered by a telehealth clinician. Here, we present the findings from the ED-based portion of the feasibility trial, namely the acceptability and appropriateness of ReachCare in the ED overall, the quality and completeness of safety plans created in the ED, intervention duration, and a comparison of these measures across the 3 potential delivery modalities.

Ethical Considerations

This study was approved by the institutional review board of the University of Massachusetts Chan Medical School on May 19, 2020, under docket number H00020238.

Study Design

This study was a feasibility trial of the ReachCare intervention, in which eligible patients were randomized to receive ReachCare in the ED in one of three ways: (1) self-administered on the tablet app using a chatbot interface, (2) administered by an in-person clinician, or (3) administered by a telehealth clinician. Randomization was on a 1:1:1 ratio based on blocks of 6: randomization tables were prepared by a team member who was not involved in data collection and was uploaded to the Research Electronic Data Capture (Vanderbilt University) data collection system. Allocation was done after the baseline research interview to preserve blinding of the research assistant collecting the baseline demographic and clinical measures. Further, because the research assistant had to know the allocation in order to facilitate the intervention, blinding was not possible for the acceptability measures, which were completed after the intervention. The quality rating of the safety plans was completed after completion of the trial by an assessor blinded to allocation (see more detail under the Measures section). The target sample size of 45 (15 participants per arm) was set based on previous literature recommending sample sizes between 24 and 50 for feasibility studies [18,19].


This study took place in the ED of a large teaching hospital in the northeastern United States. Over a 6-month period, we sought to enroll patients who presented to the ED and screened positive for suicidality on the Patient Safety Screener (PSS) [20]. The PSS is used universally and routinely at the enrollment ED: a positive screen is defined as active ideation in the past 2 weeks or a suicide attempt in the past 6 months. Patients were considered for inclusion if they (1) screened positive for suicidality on the PSS, (2) were aged 18 years or older, (3) were cognitively and medically able to consent and participate, (4) owned a smartphone, and (5) had a stable mailing address to receive compensation. Patients were excluded if they were (1) a prisoner or in police custody, (2) overly agitated or violent, or (3) on precautions for infectious disease, such as COVID-19.

Potentially eligible patients were approached by a research assistant in the ED’s medical or psychiatric area and were screened to confirm eligibility. The research assistant then explained the study: patients were informed as to what the study entailed, procedures to protect confidentiality and privacy, their right to withdraw at any time, and the potential risks and benefits of the study. If the patient was still interested in participating, the research assistant then administered a consent mini-quiz to test the patient’s understanding of the information and invited them to read and sign an informed consent form. Participants then completed baseline research measures, were randomly allocated to 1 of 3 arms, participated in the ED portion of the intervention, and then completed acceptability measures. Participants received a US $30 gift-card for completion of the baseline research interview.


The ED-based portion of the ReachCare intervention focuses on creating a 6-step safety plan [21]. The safety plan involves working with the patient to identify warning signs of an impending suicidal crisis, internal coping strategies, social supports and places for distraction, social supports for help, professionals for help, and making the environment safer by limiting access to lethal means (Figure 1). In the self-administered arm of the trial, patients engaged with a tablet-based app: they watched an introductory video (with the choice of watching a clinician, community member, or animated character deliver the same information; Figure 2) and then proceeded to create a safety plan by interacting with a chatbot-style interface that presented cues and questions, to which the patient typed in personalized answers [17]. Once the safety plan had been created, the patient had the opportunity to review and edit it, before watching an outro video explaining the next steps in their care. In contrast, in the clinician in-person and telehealth arms, the patient completed safety planning with the clinician. In those arms, the clinician used the ReachCare clinician portal, which provides clinician-facing cues, suggested phrasing, and structure to introduce the intervention to the patient and to create a safety plan with them.

Figure 1. Screenshot of the ReachCare tablet-based app's safety planning functionality.
Figure 2. Screenshot of the ReachCare tablet-based app’s video functionality.


Our measures included a variety of modalities, including patient self-report via interview, medical chart review, researcher measurement (of duration of intervention), and researcher rating (of safety plan quality).

We assessed several demographic variables by patient self-report, namely gender, age, race, ethnicity, psychiatric diagnoses, BH treatments received in the past 3 months, access to lethal means, and smartphone usage. We assessed mobile technology self-efficacy by patient self-report using 3 items from Balapour et al [22]. The 15-item version of the Interpersonal Needs Questionnaire [23] was used to assess patient’s self-reported thwarted belonginess and perceived burdensomeness, key drivers of suicidality. The Interpersonal Needs Questionnaire includes a 7-point Likert Scale and has good psychometric properties [24]. We used the Columbia Suicide Severity Rating Scale to classify patient’s self-reported recent and past suicidal ideation and suicidal behavior [25] and suicide-related impulsivity was assessed using 2 items from Mccullumsmith et al [26].

We used the medical record to extract the patient’s insurance status and the circumstances of their current presentation (chief complaint, involuntary hold status, and suicide screening result). We also extracted information regarding the care they received in the ED beyond the research intervention, that is, whether they had received each of the following: a BH evaluation; referral to outpatient BH; a structured suicide risk assessment; constant observation; or hospital admission.

We investigated acceptability and appropriateness, usability (for the self-administered arm only), duration of intervention in minutes, and quality of safety plans. Acceptability and appropriateness were measured using Weiner et al’s [27] 4-item Acceptability of Intervention measure and Intervention Appropriateness Measure respectively. Both instruments are on a 4-point Likert-type scale and have good reliability and validity [27]. Usability was assessed using the System Usability Scale [28]: this scale has 10 items, 5 of which are reverse-scored, and is the standard in usability research. The range of possible scores on the System Usability Scale is from 0 to 100. The duration of the intervention was timed by the research assistant and rounded to the closest minute. The quality of the safety plans was rated using the Safety Planning Intervention Scoring Algorithm-Brief [29], which assesses whether the safety plan has content in each line and whether that content is sufficiently specific. Each line receives a rating of 0 (no content), 1 (poor content), or 2 (satisfactory content), which is summed to create a total quality score. This quality rating was completed by a researcher who was blinded to the allocation of the patient. This scale’s psychometric properties have not been established but, in our sample, the value for Cronbach α for the 19-item scale was α=.70, indicating acceptable internal consistency.


For the first part of the results, we summarized patient and encounter characteristics using descriptive statistics. We then examined differences in implementation measures (median duration, acceptability, appropriateness, and quality) across the 3 arms: these variables were not normally distributed so we used independent samples Kruskal-Wallis tests including post hoc tests as appropriate. Finally, we summarize completeness of the safety plans in each arm using descriptive statistics. All tests were 2-sided with α set at P<.05. Analyses were completed in SPSS (version 28; IBM Corp) [30].

Characteristics of Sample

Figure 3 illustrates the patient enrollment flow. A total of 592 patients screened positive for suicide risk during research shifts and their medical charts were reviewed for initial eligibility: 144 of these were approached. Common reasons for nonapproach included the patient being: too ill, cognitively unable, agitated, without a stable mailing address, and being on infection control precautions. Of the 144 patients approached, 78 (55%) agreed to complete verbal eligibility screening. Of these, 64 were confirmed eligible, of whom 47 patients agreed to participate and were randomized: 16 participants to “arm 1 self-administered,” 15 participants to “arm 2 clinician in-person,” and 16 to “arm 3: telehealth clinician” (Figure 2). In total, 46 participants completed the intervention and all research measures; 1 patient allocated to arm 3 (telehealth clinician) withdrew during the intervention. Comparing those enrolled (N=47) to other patients who were screened for potential participation (n=545), those who were enrolled were more likely to be female (24/47, 51% vs 238/545, 44%), non-Hispanic (40/47, 85% vs 433/545, 79%), White (38/47, 81% vs 388/545, 71%), and younger (median 27.0, IQR 23-37 years vs 39.6, IQR 29-52 years). The characteristics of the enrolled sample are summarized in Tables 1 and 2 and are presented by study arm in Multimedia Appendix 1. Participants demonstrated high prevalence of depressive and anxiety disorders, high levels of smartphone use and self-efficacy, and high levels of perceived burdensomeness and thwarted belongingness.

Features of the participants’ index ED visit are summarized in Table 3, and are presented by study arm in Multimedia Appendix 2. Most participants were enrolled in the psychiatric area of the ED, almost all had a psychiatric chief complaint, and many were in the ED on an involuntary hold. Most participants received a BH evaluation during their ED visit and subsequent admission to a psychiatric facility.

Figure 3. CONSORT (Consolidated Standards of Reporting Trials) flowchart showing progress of participants through the trial.
Table 1. Patient characteristics at baseline (N=47).
Categorical variablesPatients, n (%)

Female24 (51.1)

Male20 (42.6)

Nonbinary3 (6.4)
Age (years)

18-2416 (34.0)

25-3417 (36.2)

35+14 (28.8)
Race (check all that apply)

White38 (80.9)

Other5 (10.6)

Black or African American4 (8.5)

Asian2 (4.3)

American Indian or Alaskan Native1 (2.1)

Native Hawaiian or Pacific Islander0 (0)

Non-Hispanic or Latinx40 (85.1)

Hispanic or Latinx7 (14.9)

Private23 (48.9)

Public (eg, Medicare and Medicaid)19 (40.4)

None or self-pay5 (10.6)
Psychiatric diagnoses (check all that apply)

Depressive disorder40 (85.1)

Anxiety disorder38 (80.9)

Trauma or stress-related24 (51.1)

ADHDa21 (44.7)

Substance use disorder16 (34.0)

Personality disorder12 (25.5)

Eating disorder5 (10.6)

Schizophrenia or psychotic disorder3 (6.4)

Autism spectrum disorder2 (4.3)

Other2 (4.3)
Suicidal behavior

Past-week active ideation37 (78.7)

Lifetime attempt30 (63.8)

Current attempt7 (14.9)
BHb treatments received in past 3 months

Prescription medication37 (78.7)

Individual therapy27 (57.4)

Partial hospitalization or intensive outpatient17 (36.2)

BH EDc visit12 (25.5)

Group support11 (23.4)

Inpatient psychiatric hospitalization9 (19.1)

Helpline support9 (19.1)

Other9 (19.1)

Family therapy or couples counseling4 (8.5)
Lethal means

Access to medication40 (85.1)

Access to firearm0 (0)
Smartphone use in past week per day

>3 h a day35 (74.5)

30 min to 3 h9 (19.1)

<30 min a day1 (2.1)

Rarely or never2 (4.3)

aADHD: attention-deficit/hyperactivity disorder.

bBH: behavioral health.

cED: emergency department.

Table 2. Patient characteristics at baseline (N=47).
Continuous variablesMean (SD)
Smartphone self-efficacy

I can use smartphone technology if there was no one around to tell me what to do (max score 5)4.62 (0.68)

I can use smartphone technology even if I have never used a similar technology before (max score 5)4.15 (1.02)

I am confident that I can effectively open and use an app on my smartphone (max score 5)4.79 (0.46)
Hypothesized mechanisms of action

INQa perceived burdensomeness subscale19.57 (10.04)

INQ thwarted belongingness subscale33.83 (13.24)

BASb-drive subscale11.17 (3.01)

Suicide-related impulsivity3.02 (1.35)

aINQ: Interpersonal Needs Questionnaire.

bBAS: behavioral activation scale.

Table 3. Characteristics of index EDa visit (N=47).

Patients, n (%)
Enrollment location

Psychiatric ED30 (63.8)

Medical ED17 (36.2)
Chief complaint

Suicidality35 (74.5)

Nonsuicide psychiatric11 (23.4)

Nonpsychiatric1 (2.2)
Involuntary BHb hold

Yes41 (87.2)

No6 (12.8)
Patient Safety Screener result (check all that apply)

Positive for active ideation in past 2 wk43 (91.5)

Positive for suicide attempt in past 6 mo15 (31.9)
BH Evaluation

Yes40 (85.1)

No7 (14.9)
Referral to outpatient BH

Yes1 (2.1)

No46 (97.9)
Structured suicide assessment

Yes26 (55.3)

No21 (44.7)
Under observation

Yes45 (95.7)

No2 (4.3)

Psychiatric admission or transfer29 (61.7)

Home16 (34.0)

Other or not documented2 (4.3)

aED: emergency department.

bBH: behavioral health.

Implementation Outcomes of ReachCare in the ED

Table 4 summarizes the response of participants to ReachCare in the ED. Participants rated the acceptability and appropriateness of ReachCare in the ED highly, with no significant difference across the 3 delivery modalities. The usability of the self-administered (chatbot) modality was also rated very highly by participants, and post hoc tests showed that the self-administered arm took significantly less time to complete than the 2 clinician arms. The overall quality of the safety plans generated were not significantly different across the 3 arms (Table 4).

Table 5 summarizes the completeness of each step of the safety plans, compared across the 3 study arms. There were no statistical differences in the completeness of each step across arms, but Step 2 tended to be more complete in the self-administered arm (H=5.14, P=.08) and Step 5 tended to be more complete in the 2 clinician arms (H=4.96, P=.08).

Table 4. Implementation outcomes postintervention (n=46) by treatment arm.

Median (IQR)Kruskal-Wallis HP value
Acceptability of intervention (patient report, range 1-5)3.90.14

Overall4.00 (4.00-4.50)

Arm 1 self4.00 (3.81-4.56)

Arm 2 in-person4.25 (4.00-4.50)

Arm 3 telehealth4.00 (3.75-4.50)

Appropriateness of intervention (patient report, range 1-5)1.05.59

Overall4.00 (4.00-4.25)

Arm 1 self4.25 (3.75-4.25)

Arm 2 in-person4.00 (4.00-4.50)

Arm 3 telehealth4.00 (4.00-4.25)

System Usability Scale (patient report, range 0-100)N/AaN/A

Arm 1 only93.75 (80.00-97.50)

Duration of intervention (researcher-measured, min) 27.91 <.001b

Overall34.50 (16.50-41.75)

Arm 1 self15.00 (14.75-17.75)

Arm 2 in-person38.00 (33.50-41.75)

Arm 3 telehealth42.50 (34.50-53.38)

Quality of safety plan (researcher-rated, range 0-38)0.60.74

Overall30.00 (26.00-33.00)

Arm 1 self30.00 (26.00-32.75)

Arm 2 in-person30.00 (24.00-33.00)

Arm 3 telehealth28.00 (26.00-34.00)

aN/A: not applicable.

bPost hoc tests showed arm 1 had significantly shorter duration than arms 2 and 3 (P<.001).

Table 5. Safety plan completeness by step overall and by treatment arms (n=46).
Safety plan stepNumber of lines with content, median (IQR)HP value

Overall (all 3 arms)Arm 1 self-administeredArm 2 clinician in-personArm 3 clinician telehealth

Step 1 warning signs (out of 3 lines)3.0 (3.0-3.0)3.0 (3.0-3.0)3.0 (3.0-3.0)3.0 (3.0-3.0)1.28.53
Step 2 coping strategies (out of 3 lines)3.0 (2.0-3.0)3.0 (3.0-3.0)3.0 (2.0-3.0)2.0 (2.0-3.0)5.14.08
Step 3 people or places for distraction (out of 4 lines)3.0 (2.8-4.0)4.0 (2.0-4.0)3.0 (3.0-4.0)3.0 (2.0-4.0)1.02.60
Step 4 people for help (out of 3 lines)2.0 (1.0-2.3)2.0 (1.3-3.0)2.0 (1.0-3.0)2.0 (1.0-2.0)1.12.57
Step 5 professionals for help (out of 4 lines)3.0 (3.0-4.0)3.0 (2.0-3.0)3.0 (3.0-4.0)3.0 (3.0-4.0)4.96.08
Step 6 making environment safe (out of 2 lines)2.0 (1.0-2.0)2.0 (2.0-2.0)2.0 (1.0-2.0)2.0 (1.0-2.0)2.84.24

Many patients who present to the ED with suicidality do not receive effective intervention [6]. EDs struggle with shortages in BH clinician availability and suicide-specific skills [15,16] and patients often describe their experiences of suicide-related care in the ED as negative [31]. ReachCare was developed to initiate an acceptable and feasible form of evidence-based care in the ED and support the patient in the high-risk period after discharge [17]. In this paper, we summarized initial implementation outcomes of the in-ED components of ReachCare across a variety of modalities. We found that patients rated the acceptability and appropriateness of the intervention highly in the context of other studies that have used the same measures [32-34], and there were no substantial differences across the various modalities. At just over 15 minutes, the self-administered, chatbot-style modality took less than half the time of the clinician modalities and did not require BH specialist input. The self-administered safety plans tended to have more content and the usability of the patient-facing tablet app was in the “very high” range. The safety plans that were created with ReachCare, whether self-administered with a chatbot or delivered by a clinician, were of high quality and completeness. Given that higher safety plan quality is associated with reduced risk of prospective suicidal behavior [35,36] and of subsequent psychiatric hospitalization [37], ReachCare has the potential to positively impact patient outcomes in a fully powered trial, even when it is implemented using self-administered modality.

Technology is becoming a key implementation strategy for BH interventions in health care settings [38]. Brief interventions for suicidality in the ED are recommended by the Joint Commission [39] and the American College of Emergency Physicians [40] but in practice have proven difficult to implement. In EDs, where there is a dearth of BH clinicians and the focus is on evaluation and disposition, engaging methods of self-administration may be one of the few ways to ensure that patients receive any intervention. Self-guided digital interventions for suicidality have been found to have a significant, if small, effect [41]. This study contributes to growing evidence that, when delivered through engaging digital platforms, self-guided versions of effective interventions can be efficient, feasible, and acceptable. For example, the tablet-based intervention “Jaspr” was associated with better care process and decreased distress in patients presenting with suicidality [42]. Similarly, a self-administered tablet-based safety planning website designed for use in the ED was associated with good usability and reduced suicidal intensity [43]. Finally, “Lock to Live,” is a promising tablet-based intervention for counseling on access to lethal means in the ED [44,45]. These encouraging results support the idea that self-administration could offer unique advantages in terms of timely, tailored, and scalable suicide intervention in resource-constrained EDs.

Like the broader population of people presenting with suicidality, our sample displayed a high prevalence of depression, anxiety disorder, trauma, and attention-deficit/hyperactivity disorder, as well as elevated levels of thwarted belongness and perceived burdensomeness. Given that those who are experiencing suicidality may be feeling particularly disconnected [46], there are ethical implications around simply handing a patient a tablet computer as a panacea to suicide-related care in the ED. The development of such technology-facilitated intervention should start with careful mapping of user needs from a variety of stakeholders and apply an iterative approach to testing and refinement [17,47]. Although self-administration holds promise as a way of scaling evidence-based interventions, programs that propagate loneliness and disempowerment in suicidal patients have the potential to do more harm than good. Any self-administered software for this population should be engaging but easy to navigate, foster a sense of connection as much as possible (for example by using videos and interaction), and anticipate and seek to ameliorate feelings of burdensomeness and disconnection. When done well, it is possible that technology could increase empowerment by introducing flexibility and choice to fit with patients’ timeline and needs [48,49]. Self-administered intervention in this population may be thoughtfully introduced by a clinician, such as a physician, nurse, or a medical assistant: the patient could be offered a choice, to complete a safety plan with a BH clinician (in-person or via telehealth if necessary) or for themselves if they prefer. A third option is a hybrid approach, where a patient initiates the safety plan using software and a BH clinician reviews it with them, ideally before discharge or as part of telephone follow-up. Intervention is ideally not limited to the ED but carries on out into the community in the form of caring contacts or outpatient engagement that can help to address the underlying drivers of the individual’s suicidality [50]. It is also important to note that technology is not a sufficient implementation strategy in itself: digital interventions require careful implementation planning, training, and support if they are to result in improved reach and sustainability [51].

Finally, our study led to some interesting observations around digitally supported clinician delivery. There appeared to be a slight trend toward lower acceptability of the telehealth clinician modality compared to the in-person clinician modality in our small trial. Larger trials that have compared telehealth to in-person delivery for a variety of psychosocial interventions have found little difference in satisfaction or effectiveness between the 2 modalities [52-54]. Moreover, telehealth evaluations can help to improve access and timeliness of mental health care in ED settings [55,56]. Further research is required to examine whether telemental health interventions specifically are less acceptable in an emergency medicine context.

This study has several limitations. Although the broader trial involved 3-month follow-up of clinical outcomes, the current analyses were cross-sectional only and our sample size was relatively small. In addition, the patients who were successfully enrolled tended to be younger, female, White, and non-Hispanic than those who were not enrolled. This may reflect the fact that the intervention was only available in English and trial enrollment required the patient to own a smartphone: additional user-centered research is needed to identify the needs of underserved populations and design systems that can adapt to their needs and preferences. For example, future research should explore Spanish-language adaptation of the intervention, as well as adaptations of the post-ED components that can be delivered without a smartphone. Notably, the majority of patients who screened positive for suicide risk during the trial were not eligible to participate because of medical or cognitive issues, which interfered with their ability to give informed consent: it remains to be seen what the reach of this intervention would be under naturalistic clinical conditions, where the capacity threshold for participation may be more flexible.

Our findings suggest that patients considered the ReachCare intervention in the ED to be acceptable and appropriate across a variety of modalities. Technology facilitation of evidence-based interventions in the ED holds much promise, with the caveat that we must take account of the particular and diverse needs of those in suicidal crisis in this setting. There are several potential directions for future research. These include possible adaptations for older patients, those whose preferred language is not English, and patients who do not own a smartphone. Other potential research questions include: which intervention modality patients or clinicians tend to select when given a choice; whether that choice is affected by patient characteristics such as age and medical condition; whether the selected modality affects longitudinal clinical outcomes; and rates of uptake under more naturalistic implementation conditions.


This research was supported by the National Institute of Mental Health of the National Institutes of Health (award R34MH123578). The content is the sole responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. We gratefully acknowledge the contribution of our participants and consultants.

Data Availability

Data are available in the National Institute of Mental Health Data Archive.

Conflicts of Interest

The authors of this paper include the developers of the ReachCare app.

Multimedia Appendix 1

Patient characteristics at baseline (N=47) by study arm.

DOCX File , 21 KB

Multimedia Appendix 2

Characteristics of index ED visit (N=47) by arm.

DOCX File , 16 KB

Multimedia Appendix 3

CONSORT-eHEALTH checklist (V 1.6.1).

PDF File (Adobe PDF File), 1264 KB

  1. Stone DM, Jones CM, Mack KA. Changes in suicide rates—United States, 2018-2019. MMWR Morb Mortal Wkly Rep. 2021;70(8):261-268. [FREE Full text] [CrossRef] [Medline]
  2. Ehlman DC, Yard E, Stone DM, Jones CM, Mack KA. Changes in suicide rates—United States, 2019 and 2020. MMWR Morb Mortal Wkly Rep. 2022;71(8):306-312. [FREE Full text] [CrossRef] [Medline]
  3. Owens PL, McDermott KW, Lipari RN, Hambrick MM. Emergency Department Visits Related to Suicidal Ideation or Suicide Attempt, 2008-2017. Rockville (MD). Agency for Healthcare Research and Quality (US); 2020.
  4. Ballard ED, Cwik M, Van Eck K, Goldstein M, Alfes C, Wilson ME, et al. Identification of at-risk youth by suicide screening in a pediatric emergency department. Prev Sci. 2017;18(2):174-182. [FREE Full text] [CrossRef] [Medline]
  5. Grumet JG, Boudreaux ED. Universal suicide screening is feasible and necessary to reduce suicide. Psychiatr Serv. 2023;74(1):81-83. [CrossRef] [Medline]
  6. Olfson M, Marcus SC, Bridge JA. Emergency treatment of deliberate self-harm. Arch Gen Psychiatry. 2012;69(1):80-88. [FREE Full text] [CrossRef] [Medline]
  7. Inagaki M, Kawashima Y, Kawanishi C, Yonemoto N, Sugimoto T, Furuno T, et al. Interventions to prevent repeat suicidal behavior in patients admitted to an emergency department for a suicide attempt: a meta-analysis. J Affect Disord. 2015;175:66-78. [FREE Full text] [CrossRef] [Medline]
  8. Miller IW, Camargo CA, Arias SA, Sullivan AF, Allen MH, Goldstein AB, et al. Suicide prevention in an emergency department population: the ED-SAFE study. JAMA Psychiatry. 2017;74(6):563-570. [FREE Full text] [CrossRef] [Medline]
  9. Dunlap LJ, Orme S, Zarkin GA, Arias SA, Miller IW, Camargo CA, et al. Screening and intervention for suicide prevention: a cost-effectiveness analysis of the ED-SAFE interventions. Psychiatr Serv. 2019;70(12):1082-1087. [FREE Full text] [CrossRef] [Medline]
  10. Ruffolo MC, Capobianco J. Moving an evidence-based intervention into routine mental health care: a multifaceted case example. Soc Work Health Care. 2012;51(1):77-87. [CrossRef] [Medline]
  11. True G, Pollock M, Bowden CF, Cullen SW, Ross AM, Doupnik SK, et al. Strategies to care for patients being treated in the emergency department after self-harm: perspectives of frontline staff. J Emerg Nurs. 2021;47(3):426-436. [FREE Full text] [CrossRef] [Medline]
  12. Cullen SW, Diana A, Olfson M, Marcus SC. If you could change 1 thing to improve the quality of emergency care for deliberate self-harm patients, what would it be? A national survey of nursing leadership. J Emerg Nurs. 2019;45(6):661-669. [FREE Full text] [CrossRef] [Medline]
  13. Kirk JW, Nilsen P. Implementing evidence-based practices in an emergency department: contradictions exposed when prioritising a flow culture. J Clin Nurs. 2016;25(3-4):555-565. [FREE Full text] [CrossRef] [Medline]
  14. Richmond JS, Dragatsi D, Stiebel V, Rozel JS, Rasimas JJ. American association for emergency psychiatry recommendations to address psychiatric staff shortages in emergency settings. Psychiatr Serv. 2021;72(4):437-443. [FREE Full text] [CrossRef] [Medline]
  15. Betz ME, Sullivan AF, Manton AP, Espinola JA, Miller I, Camargo CA, et al. Knowledge, attitudes, and practices of emergency department providers in the care of suicidal patients. Depress Anxiety. 2013;30(10):1005-1012. [FREE Full text] [CrossRef] [Medline]
  16. Rayner G, Blackburn J, Edward KL, Stephenson J, Ousey K. Emergency department nurse's attitudes towards patients who self-harm: a meta-analysis. Int J Ment Health Nurs. 2019;28(1):40-53. [CrossRef] [Medline]
  17. Larkin C, Djamasbi S, Boudreaux ED, Varzgani F, Garner R, Siddique M, et al. ReachCare mobile apps for patients experiencing suicidality in the emergency department: development and usability testing using mixed methods. JMIR Form Res. 2023;7:e41422. [FREE Full text] [CrossRef] [Medline]
  18. Sim J, Lewis M. The size of a pilot study for a clinical trial should be calculated in relation to considerations of precision and efficiency. J Clin Epidemiol. 2012;65(3):301-308. [CrossRef] [Medline]
  19. Julious SA. Sample size of 12 per group rule of thumb for a pilot study. Pharmaceut Statist. 2005;4(4):287-291. [CrossRef]
  20. Boudreaux ED, Jaques ML, Brady KM, Matson A, Allen MH. The patient safety screener: validation of a brief suicide risk screener for emergency department settings. Arch Suicide Res. 2015;19(2):151-160. [CrossRef] [Medline]
  21. Stanley B, Brown GK. Safety planning intervention: a brief intervention to mitigate suicide risk. Cogn Behav Pract. 2012;19(2):256-264. [CrossRef]
  22. Balapour A, Reychav I, Sabherwal R, Azuri J. Mobile technology identity and self-efficacy: implications for the adoption of clinically supported mobile health apps. Int J Inf Manag. 2019;49:58-68. [FREE Full text] [CrossRef]
  23. Van Orden KA, Cukrowicz KC, Witte TK, Joiner TE. Thwarted belongingness and perceived burdensomeness: construct validity and psychometric properties of the interpersonal needs questionnaire. Psychol Assess. 2012;24(1):197-215. [FREE Full text] [CrossRef] [Medline]
  24. Hill RM, Rey Y, Marin CE, Sharp C, Green KL, Pettit JW. Evaluating the interpersonal needs questionnaire: comparison of the reliability, factor structure, and predictive validity across five versions. Suicide Life Threat Behav. 2015;45(3):302-314. [CrossRef] [Medline]
  25. Posner K, Brown GK, Stanley B, Brent DA, Yershova KV, Oquendo MA, et al. The Columbia-suicide severity rating scale: initial validity and internal consistency findings from three multisite studies with adolescents and adults. Am J Psychiatry. 2011;168(12):1266-1277. [FREE Full text] [CrossRef] [Medline]
  26. Mccullumsmith CB, Williamson DJ, May RS, Bruer EH, Sheehan DV, Alphs LD. Simple measures of hopelessness and impulsivity are associated with acute suicidal ideation and attempts in patients in psychiatric crisis. Innov Clin Neurosci. 2014;11(9-10):47-53. [FREE Full text] [Medline]
  27. Weiner BJ, Lewis CC, Stanick C, Powell BJ, Dorsey CN, Clary AS, et al. Psychometric assessment of three newly developed implementation outcome measures. Implement Sci. 2017;12(1):108. [FREE Full text] [CrossRef] [Medline]
  28. Brooke J. SUS-a quick and dirty usability scale. In: Jordan PW, Thomas B, Weerdmeester BA, McClelland IL, editors. Usability Evaluation in Industry. Milton Park, Abingdon-on-Thames, Oxfordshire United Kingdom. Taylor & Francis; 1996;4-7.
  29. Goodman M, Brown GK, Galfalvy HC, Spears AP, Sullivan SR, Kapil-Pair KN, et al. Group ("Project Life Force") versus individual suicide safety planning: a randomized clinical trial. Contemp Clin Trials Commun. 2020;17:100520. [FREE Full text] [CrossRef] [Medline]
  30. IBM Corp. Released 2021. IBM SPSS Statistics for Windows, Version 28.0. Armonk, NY. IBM Corp URL: [accessed 2023-09-21]
  31. MacDonald S, Sampson C, Turley R, Biddle L, Ring N, Begley R, et al. Patients' experiences of emergency hospital care following self-harm: systematic review and thematic synthesis of qualitative research. Qual Health Res. 2020;30(3):471-485. [FREE Full text] [CrossRef] [Medline]
  32. Hofmann G, Hampanda K, Harrison MS, Fasano M, Nacht A, Yeoman M. Virtual prenatal and postpartum care acceptability among maternity care providers. Matern Child Health J. 2022;26(7):1401-1408. [FREE Full text] [CrossRef] [Medline]
  33. Rosenberg M, Kianersi S, Luetke M, Jozkowski K, Guerra-Reyes L, Shih PC, et al. Wearable alcohol monitors for alcohol use data collection among college students: feasibility and acceptability. Alcohol. 2023;111:75-83. [FREE Full text] [CrossRef] [Medline]
  34. Lai-Kwon J, Kelly B, Lane S, Biviano R, Bartula I, Brennan F, et al. Feasibility, acceptability, and utility of a nurse-led survivorship program for people with metastatic melanoma (MELCARE). Support Care Cancer. 2022;30(11):9587-9596. [FREE Full text] [CrossRef] [Medline]
  35. Green JD, Kearns JC, Rosen RC, Keane TM, Marx BP. Evaluating the effectiveness of safety plans for military veterans: do safety plans tailored to veteran characteristics decrease suicide risk? Behav Ther. 2018;49(6):931-938. [CrossRef] [Medline]
  36. Chalker SA, Parrish EM, Ceren CSM, Depp CA, Goodman M, Doran N. Predictive importance of social contacts on U.S. veteran suicide safety plans. Psychiatr Serv. 2023;74(3):244-249. [CrossRef] [Medline]
  37. Gamarra JM, Luciano MT, Gradus JL, Stirman SW. Assessing variability and implementation fidelity of suicide prevention safety planning in a regional VA healthcare system. Crisis. 2015;36(6):433-439. [FREE Full text] [CrossRef] [Medline]
  38. Hasselberg MJ. The digital revolution in behavioral health. J Am Psychiatr Nurses Assoc. 2020;26(1):102-111. [FREE Full text] [CrossRef] [Medline]
  39. Joint Commission. Suicide prevention resources to support Joint Commission accredited organizations implementation of NPSG 15.01. 01, revised July 2020. URL: https:/​/www.​​-/​media/​tjc/​documents/​resources/​patient-safety-topics/​suicide-prevention/​suicide_prevention_compendium_5_11_20_updated-july2020.​pdf [accessed 2023-04-19]
  40. Wilson MP, Moutier C, Wolf L, Nordstrom K, Schulz T, Betz ME. ED recommendations for suicide prevention in adults: The ICAR2E mnemonic and a systematic review of the literature. Am J Emerg Med. 2020;38(3):571-581. [FREE Full text] [CrossRef] [Medline]
  41. Torok M, Han J, Baker S, Werner-Seidler A, Wong I, Larsen ME, et al. Suicide prevention using self-guided digital interventions: a systematic review and meta-analysis of randomised controlled trials. Lancet Digit Health. 2020;2(1):e25-e36. [FREE Full text] [CrossRef] [Medline]
  42. Dimeff LA, Jobes DA, Koerner K, Kako N, Jerome T, Kelley-Brimer A, et al. Using a tablet-based app to deliver evidence-based practices for suicidal patients in the emergency department: pilot randomized controlled trial. JMIR Ment Health. 2021;8(3):e23022. [FREE Full text] [CrossRef] [Medline]
  43. Boudreaux ED, Brown GK, Stanley B, Sadasivam RS, Camargo CA, Miller IW. Computer administered safety planning for individuals at risk for suicide: development and usability testing. J Med Internet Res. 2017;19(5):e149. [FREE Full text] [CrossRef] [Medline]
  44. Betz ME, Knoepke CE, Simpson S, Siry BJ, Clement A, Saunders T, et al. An interactive web-based lethal means safety decision aid for suicidal adults (lock to live): pilot randomized controlled trial. J Med Internet Res. 2020;22(1):e16253. [FREE Full text] [CrossRef] [Medline]
  45. Betz ME, Knoepke CE, Siry B, Clement A, Azrael D, Ernestus S, et al. 'Lock to live': development of a firearm storage decision aid to enhance lethal means counselling and prevent suicide. Inj Prev. 2019;25(Suppl 1):i18-i24. [FREE Full text] [CrossRef] [Medline]
  46. Joiner T. Why People Die by Suicide. Cambridge, MA. Harvard University Press; 2005.
  47. Akeel AU, Mundy D. Re-thinking technology and its growing role in enabling patient empowerment. Health Informatics J. 2019;25(4):1278-1289. [FREE Full text] [CrossRef] [Medline]
  48. Lau F. Patient empowerment: the role of technology. In: Lau F, Bartle-Clar JA, Bliss G, Borycki EM, Courtney KL, Kuo AMH, et al, editors. Improving Usability, Safety and Patient Outcomes with Health Information Technology: From Research to Practice. Amsterdam, Berlin, Washington, DC. IOS Press; 2019.
  49. Calvillo J, Román I, Roa LM. How technology is empowering patients? A literature review. Health Expect. 2015;18(5):643-652. [FREE Full text] [CrossRef] [Medline]
  50. Jobes DA, Gregorian MJ, Colborn VA. A stepped care approach to clinical suicide prevention. Psychol Serv. 2018;15(3):243-250. [CrossRef] [Medline]
  51. Graham AK, Lattie EG, Powell BJ, Lyon AR, Smith JD, Schueller SM, et al. Implementation strategies for digital mental health interventions in health care settings. Am Psychol. 2020;75(8):1080-1092. [FREE Full text] [CrossRef] [Medline]
  52. Herbert MS, Afari N, Liu L, Heppner P, Rutledge T, Williams K, et al. Telehealth versus in-person acceptance and commitment therapy for chronic pain: a randomized noninferiority trial. J Pain. 2017;18(2):200-211. [FREE Full text] [CrossRef] [Medline]
  53. Morland LA, Mackintosh MA, Greene CJ, Rosen CS, Chard KM, Resick P, et al. Cognitive processing therapy for posttraumatic stress disorder delivered to rural veterans via telemental health: a randomized noninferiority clinical trial. J Clin Psychiatry. 2014;75(5):470-476. [CrossRef] [Medline]
  54. Choi NG, Hegel MT, Marti N, Marinucci ML, Sirrianni L, Bruce ML. Telehealth problem-solving therapy for depressed low-income homebound older adults. Am J Geriatr Psychiatry. 2014;22(3):263-271. [FREE Full text] [CrossRef] [Medline]
  55. Fairchild RM, Ferng-Kuo SF, Laws S, Rahmouni H, Hardesty D. Telehealth decreases rural emergency department wait times for behavioral health patients in a group of critical access hospitals. Telemed E Health. 2019;25(12):1154-1164. [FREE Full text] [CrossRef] [Medline]
  56. Southard EP, Neufeld JD, Laws S. Telemental health evaluations enhance access and efficiency in a critical access hospital emergency department. Telemed E Health. 2014;20(7):664-668. [CrossRef] [Medline]

BH: behavioral health
ED: emergency department
ED-SAFE: Emergency Department Safety Assessment and Follow-up Evaluation
PSS: Patient Safety Screener

Edited by J Torous; submitted 09.06.23; peer-reviewed by C King, C Chung; comments to author 29.06.23; revised version received 25.07.23; accepted 27.07.23; published 24.10.23.


©Celine Larkin, Bengisu Tulu, Soussan Djamasbi, Roscoe Garner, Fatima Varzgani, Mariam Siddique, John Pietro, Edwin D Boudreaux. Originally published in JMIR Mental Health (, 24.10.2023.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Mental Health, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.