Background: Emergency departments (EDs) manage many patients with suicide risk, but effective interventions for suicidality are challenging to implement in this setting. ReachCare is a technology-facilitated version of an evidence-based intervention for suicidal ED patients. Here, we present findings on the acceptability and quality of ReachCare in the ED, as well as a comparison of these measures across 3 potential delivery modalities
Objective: Our aim was to test the feasibility of the ReachCare intervention in its entirety through conducting a pilot study with patients presenting with suicidality to the ED. We tested three different ways of receiving the ED-based components of ReachCare: (1) self-administered on the tablet app using a chatbot interface, (2) administered by an in-person clinician, or (3) administered by a telehealth clinician.
Methods: In total, 47 ED patients who screened positive for suicide risk were randomly allocated to receive one of three delivery modalities of ReachCare in the ED: (1) self-administered on the patient-facing tablet app with a chatbot interface, (2) delivered by an in-person clinician, or (3) delivered by a telehealth clinician, with the latter two using a clinician-facing web app. We measured demographic and clinical characteristics, acceptability and appropriateness of the intervention, and quality and completeness of the resulting safety plans.
Results: Patients assigned high ratings for the acceptability (median 4.00/5, IQR 4.00-4.50) and appropriateness (median 4.00/5, IQR 4.00-4.25) of ReachCare’s ED components, and there were no substantial differences across the 3 delivery modalities [H(acceptability)=3.90, P=.14; H(appropriateness)=1.05, P=.59]. The self-administered modality took significantly less time than the 2 clinician modalities (H=27.91, P<.001), and the usability of the self-administered version was in the “very high” range (median 93.75/100, IQR 80.00-97.50). The safety plans created across all 3 modalities were high-quality (H=0.60, P=.74).
Conclusions: Patients rated ReachCare in the ED as highly acceptable and appropriate regardless of modality. Self-administration may be a feasible way to ensure patients with suicide risk receive an intervention in resource constrained EDs. Limitations include small sample size and demographic differences between those enrolled versus not enrolled. Further research will examine the clinical outcomes of patients receiving both the in-ED and post-ED components of ReachCare.
Trial Registration: ClinicalTrials.gov NCT04720911; https://clinicaltrials.gov/ct2/show/NCT04720911
After a brief decline in 2019-2020 [, ], suicide rates have again begun to rise in the United States [ ]. Each year, there are over 1 million suicide-related visits to emergency departments (EDs) in the United States [ ], and EDs represent a feasible venue to screen and assess suicide risk [ , ]. However, once risk is detected, many ED patients with suicide risk receive suboptimal care [ ].
Several effective interventions have been developed for this patient population . The Emergency Department Safety Assessment and Follow-up Evaluation (ED-SAFE) intervention [ ], for example, featured paper-based self-administered safety planning and resource lists in the ED; this was followed by intensive telephone outreach to patients and a chosen significant other in the 12 months after discharge. The telephone counseling involved safety planning, outpatient treatment engagement, values clarification and life planning, and family communication and coordination. The ED-SAFE intervention was associated with a 30% reduction in the total number of suicide attempts in the intervention group compared to treatment-as-usual [ ]. Despite its promise and cost-effectiveness [ ], the ED-SAFE intervention was resource-intensive and difficult to sustain without research support. Such interventions routinely struggle to “cross the chasm” to become routine care [ ].
Clinicians and leadership recognize that ED-based care for suicidality needs to improve but often face substantial constraints in effecting change [, ]. The dearth of effective care for suicidality in the ED can be attributed to a wide range of factors, including a hectic milieu focused on patient flow [ ], a shortage of behavioral health (BH) providers [ ], and a shortfall in suicide-related skills and confidence among ED clinicians [ , ]. With these barriers in mind, we set out to adapt the ED-SAFE intervention to become “ReachCare,” leveraging technology to make the intervention scalable in resource-constrained EDs, while avoiding a corresponding loss in fidelity.
We previously outlined the user-centered development and testing of the patient-facing technology aspects of the ReachCare intervention . These technologies include a tablet-based app for use within the ED (that allows a patient to build a high-quality safety plan in the absence of a BH clinician) and a patient- and family-facing mobile app (that houses the patient’s safety plan and life plan, helpline resources, psychoeducation, distractions, and a BH provider search engine). In that study, user-testing with current ED patients presenting with suicidality demonstrated high usability and acceptability of the technologies [ ]. ReachCare is further supported by interoperable clinician-facing software that provides tips, call structure, and note-taking for clinicians, as well as allowing them to update and share patients’ therapeutic tools during follow-up phone calls. This clinician-facing technology was designed to be used in-person or via telehealth.
We conducted a feasibility trial with patients presenting with suicidality to the ED to develop and test implementation strategies for the ReachCare intervention that will be used in a future effectiveness trial. We tested three different modalities of delivering the ED-based components of ReachCare, randomizing patients to receive: (1) self-administered on the tablet app using a chatbot interface, (2) administered by an in-person clinician, or (3) administered by a telehealth clinician. Here, we present the findings from the ED-based portion of the feasibility trial, namely the acceptability and appropriateness of ReachCare in the ED overall, the quality and completeness of safety plans created in the ED, intervention duration, and a comparison of these measures across the 3 potential delivery modalities.
This study was approved by the institutional review board of the University of Massachusetts Chan Medical School on May 19, 2020, under docket number H00020238.
This study was a feasibility trial of the ReachCare intervention, in which eligible patients were randomized to receive ReachCare in the ED in one of three ways: (1) self-administered on the tablet app using a chatbot interface, (2) administered by an in-person clinician, or (3) administered by a telehealth clinician. Randomization was on a 1:1:1 ratio based on blocks of 6: randomization tables were prepared by a team member who was not involved in data collection and was uploaded to the Research Electronic Data Capture (Vanderbilt University) data collection system. Allocation was done after the baseline research interview to preserve blinding of the research assistant collecting the baseline demographic and clinical measures. Further, because the research assistant had to know the allocation in order to facilitate the intervention, blinding was not possible for the acceptability measures, which were completed after the intervention. The quality rating of the safety plans was completed after completion of the trial by an assessor blinded to allocation (see more detail under the Measures section). The target sample size of 45 (15 participants per arm) was set based on previous literature recommending sample sizes between 24 and 50 for feasibility studies [, ].
This study took place in the ED of a large teaching hospital in the northeastern United States. Over a 6-month period, we sought to enroll patients who presented to the ED and screened positive for suicidality on the Patient Safety Screener (PSS) . The PSS is used universally and routinely at the enrollment ED: a positive screen is defined as active ideation in the past 2 weeks or a suicide attempt in the past 6 months. Patients were considered for inclusion if they (1) screened positive for suicidality on the PSS, (2) were aged 18 years or older, (3) were cognitively and medically able to consent and participate, (4) owned a smartphone, and (5) had a stable mailing address to receive compensation. Patients were excluded if they were (1) a prisoner or in police custody, (2) overly agitated or violent, or (3) on precautions for infectious disease, such as COVID-19.
Potentially eligible patients were approached by a research assistant in the ED’s medical or psychiatric area and were screened to confirm eligibility. The research assistant then explained the study: patients were informed as to what the study entailed, procedures to protect confidentiality and privacy, their right to withdraw at any time, and the potential risks and benefits of the study. If the patient was still interested in participating, the research assistant then administered a consent mini-quiz to test the patient’s understanding of the information and invited them to read and sign an informed consent form. Participants then completed baseline research measures, were randomly allocated to 1 of 3 arms, participated in the ED portion of the intervention, and then completed acceptability measures. Participants received a US $30 gift-card for completion of the baseline research interview.
The ED-based portion of the ReachCare intervention focuses on creating a 6-step safety plan . The safety plan involves working with the patient to identify warning signs of an impending suicidal crisis, internal coping strategies, social supports and places for distraction, social supports for help, professionals for help, and making the environment safer by limiting access to lethal means ( ). In the self-administered arm of the trial, patients engaged with a tablet-based app: they watched an introductory video (with the choice of watching a clinician, community member, or animated character deliver the same information; ) and then proceeded to create a safety plan by interacting with a chatbot-style interface that presented cues and questions, to which the patient typed in personalized answers [ ]. Once the safety plan had been created, the patient had the opportunity to review and edit it, before watching an outro video explaining the next steps in their care. In contrast, in the clinician in-person and telehealth arms, the patient completed safety planning with the clinician. In those arms, the clinician used the ReachCare clinician portal, which provides clinician-facing cues, suggested phrasing, and structure to introduce the intervention to the patient and to create a safety plan with them.
Our measures included a variety of modalities, including patient self-report via interview, medical chart review, researcher measurement (of duration of intervention), and researcher rating (of safety plan quality).
We assessed several demographic variables by patient self-report, namely gender, age, race, ethnicity, psychiatric diagnoses, BH treatments received in the past 3 months, access to lethal means, and smartphone usage. We assessed mobile technology self-efficacy by patient self-report using 3 items from Balapour et al . The 15-item version of the Interpersonal Needs Questionnaire [ ] was used to assess patient’s self-reported thwarted belonginess and perceived burdensomeness, key drivers of suicidality. The Interpersonal Needs Questionnaire includes a 7-point Likert Scale and has good psychometric properties [ ]. We used the Columbia Suicide Severity Rating Scale to classify patient’s self-reported recent and past suicidal ideation and suicidal behavior [ ] and suicide-related impulsivity was assessed using 2 items from Mccullumsmith et al [ ].
We used the medical record to extract the patient’s insurance status and the circumstances of their current presentation (chief complaint, involuntary hold status, and suicide screening result). We also extracted information regarding the care they received in the ED beyond the research intervention, that is, whether they had received each of the following: a BH evaluation; referral to outpatient BH; a structured suicide risk assessment; constant observation; or hospital admission.
We investigated acceptability and appropriateness, usability (for the self-administered arm only), duration of intervention in minutes, and quality of safety plans. Acceptability and appropriateness were measured using Weiner et al’s  4-item Acceptability of Intervention measure and Intervention Appropriateness Measure respectively. Both instruments are on a 4-point Likert-type scale and have good reliability and validity [ ]. Usability was assessed using the System Usability Scale [ ]: this scale has 10 items, 5 of which are reverse-scored, and is the standard in usability research. The range of possible scores on the System Usability Scale is from 0 to 100. The duration of the intervention was timed by the research assistant and rounded to the closest minute. The quality of the safety plans was rated using the Safety Planning Intervention Scoring Algorithm-Brief [ ], which assesses whether the safety plan has content in each line and whether that content is sufficiently specific. Each line receives a rating of 0 (no content), 1 (poor content), or 2 (satisfactory content), which is summed to create a total quality score. This quality rating was completed by a researcher who was blinded to the allocation of the patient. This scale’s psychometric properties have not been established but, in our sample, the value for Cronbach α for the 19-item scale was α=.70, indicating acceptable internal consistency.
For the first part of the results, we summarized patient and encounter characteristics using descriptive statistics. We then examined differences in implementation measures (median duration, acceptability, appropriateness, and quality) across the 3 arms: these variables were not normally distributed so we used independent samples Kruskal-Wallis tests including post hoc tests as appropriate. Finally, we summarize completeness of the safety plans in each arm using descriptive statistics. All tests were 2-sided with α set at P<.05. Analyses were completed in SPSS (version 28; IBM Corp) .
Characteristics of Sample
illustrates the patient enrollment flow. A total of 592 patients screened positive for suicide risk during research shifts and their medical charts were reviewed for initial eligibility: 144 of these were approached. Common reasons for nonapproach included the patient being: too ill, cognitively unable, agitated, without a stable mailing address, and being on infection control precautions. Of the 144 patients approached, 78 (55%) agreed to complete verbal eligibility screening. Of these, 64 were confirmed eligible, of whom 47 patients agreed to participate and were randomized: 16 participants to “arm 1 self-administered,” 15 participants to “arm 2 clinician in-person,” and 16 to “arm 3: telehealth clinician” ( ). In total, 46 participants completed the intervention and all research measures; 1 patient allocated to arm 3 (telehealth clinician) withdrew during the intervention. Comparing those enrolled (N=47) to other patients who were screened for potential participation (n=545), those who were enrolled were more likely to be female (24/47, 51% vs 238/545, 44%), non-Hispanic (40/47, 85% vs 433/545, 79%), White (38/47, 81% vs 388/545, 71%), and younger (median 27.0, IQR 23-37 years vs 39.6, IQR 29-52 years). The characteristics of the enrolled sample are summarized in and and are presented by study arm in . Participants demonstrated high prevalence of depressive and anxiety disorders, high levels of smartphone use and self-efficacy, and high levels of perceived burdensomeness and thwarted belongingness.
Features of the participants’ index ED visit are summarized in, and are presented by study arm in . Most participants were enrolled in the psychiatric area of the ED, almost all had a psychiatric chief complaint, and many were in the ED on an involuntary hold. Most participants received a BH evaluation during their ED visit and subsequent admission to a psychiatric facility.
|Categorical variables||Patients, n (%)|
|Race (check all that apply)|
|Black or African American||4 (8.5)|
|American Indian or Alaskan Native||1 (2.1)|
|Native Hawaiian or Pacific Islander||0 (0)|
|Non-Hispanic or Latinx||40 (85.1)|
|Hispanic or Latinx||7 (14.9)|
|Public (eg, Medicare and Medicaid)||19 (40.4)|
|None or self-pay||5 (10.6)|
|Psychiatric diagnoses (check all that apply)|
|Depressive disorder||40 (85.1)|
|Anxiety disorder||38 (80.9)|
|Trauma or stress-related||24 (51.1)|
|Substance use disorder||16 (34.0)|
|Personality disorder||12 (25.5)|
|Eating disorder||5 (10.6)|
|Schizophrenia or psychotic disorder||3 (6.4)|
|Autism spectrum disorder||2 (4.3)|
|Past-week active ideation||37 (78.7)|
|Lifetime attempt||30 (63.8)|
|Current attempt||7 (14.9)|
|BHb treatments received in past 3 months|
|Prescription medication||37 (78.7)|
|Individual therapy||27 (57.4)|
|Partial hospitalization or intensive outpatient||17 (36.2)|
|BH EDc visit||12 (25.5)|
|Group support||11 (23.4)|
|Inpatient psychiatric hospitalization||9 (19.1)|
|Helpline support||9 (19.1)|
|Family therapy or couples counseling||4 (8.5)|
|Access to medication||40 (85.1)|
|Access to firearm||0 (0)|
|Smartphone use in past week per day|
|>3 h a day||35 (74.5)|
|30 min to 3 h||9 (19.1)|
|<30 min a day||1 (2.1)|
|Rarely or never||2 (4.3)|
aADHD: attention-deficit/hyperactivity disorder.
bBH: behavioral health.
cED: emergency department.
|Continuous variables||Mean (SD)|
|I can use smartphone technology if there was no one around to tell me what to do (max score 5)||4.62 (0.68)|
|I can use smartphone technology even if I have never used a similar technology before (max score 5)||4.15 (1.02)|
|I am confident that I can effectively open and use an app on my smartphone (max score 5)||4.79 (0.46)|
|Hypothesized mechanisms of action|
|INQa perceived burdensomeness subscale||19.57 (10.04)|
|INQ thwarted belongingness subscale||33.83 (13.24)|
|BASb-drive subscale||11.17 (3.01)|
|Suicide-related impulsivity||3.02 (1.35)|
aINQ: Interpersonal Needs Questionnaire.
bBAS: behavioral activation scale.
|Patients, n (%)|
|Psychiatric ED||30 (63.8)|
|Medical ED||17 (36.2)|
|Nonsuicide psychiatric||11 (23.4)|
|Involuntary BHb hold|
|Patient Safety Screener result (check all that apply)|
|Positive for active ideation in past 2 wk||43 (91.5)|
|Positive for suicide attempt in past 6 mo||15 (31.9)|
|Referral to outpatient BH|
|Structured suicide assessment|
|Psychiatric admission or transfer||29 (61.7)|
|Other or not documented||2 (4.3)|
aED: emergency department.
bBH: behavioral health.
Implementation Outcomes of ReachCare in the ED
summarizes the response of participants to ReachCare in the ED. Participants rated the acceptability and appropriateness of ReachCare in the ED highly, with no significant difference across the 3 delivery modalities. The usability of the self-administered (chatbot) modality was also rated very highly by participants, and post hoc tests showed that the self-administered arm took significantly less time to complete than the 2 clinician arms. The overall quality of the safety plans generated were not significantly different across the 3 arms ( ).
summarizes the completeness of each step of the safety plans, compared across the 3 study arms. There were no statistical differences in the completeness of each step across arms, but Step 2 tended to be more complete in the self-administered arm (H=5.14, P=.08) and Step 5 tended to be more complete in the 2 clinician arms (H=4.96, P=.08).
|Median (IQR)||Kruskal-Wallis H||P value|
|Acceptability of intervention (patient report, range 1-5)||3.90||.14|
|Arm 1 self||4.00 (3.81-4.56)|
|Arm 2 in-person||4.25 (4.00-4.50)|
|Arm 3 telehealth||4.00 (3.75-4.50)|
|Appropriateness of intervention (patient report, range 1-5)||1.05||.59|
|Arm 1 self||4.25 (3.75-4.25)|
|Arm 2 in-person||4.00 (4.00-4.50)|
|Arm 3 telehealth||4.00 (4.00-4.25)|
|System Usability Scale (patient report, range 0-100)||N/Aa||N/A|
|Arm 1 only||93.75 (80.00-97.50)|
|Duration of intervention (researcher-measured, min)||27.91||<.001b|
|Arm 1 self||15.00 (14.75-17.75)|
|Arm 2 in-person||38.00 (33.50-41.75)|
|Arm 3 telehealth||42.50 (34.50-53.38)|
|Quality of safety plan (researcher-rated, range 0-38)||0.60||.74|
|Arm 1 self||30.00 (26.00-32.75)|
|Arm 2 in-person||30.00 (24.00-33.00)|
|Arm 3 telehealth||28.00 (26.00-34.00)|
aN/A: not applicable.
bPost hoc tests showed arm 1 had significantly shorter duration than arms 2 and 3 (P<.001).
|Safety plan step||Number of lines with content, median (IQR)||H||P value|
|Overall (all 3 arms)||Arm 1 self-administered||Arm 2 clinician in-person||Arm 3 clinician telehealth|
|Step 1 warning signs (out of 3 lines)||3.0 (3.0-3.0)||3.0 (3.0-3.0)||3.0 (3.0-3.0)||3.0 (3.0-3.0)||1.28||.53|
|Step 2 coping strategies (out of 3 lines)||3.0 (2.0-3.0)||3.0 (3.0-3.0)||3.0 (2.0-3.0)||2.0 (2.0-3.0)||5.14||.08|
|Step 3 people or places for distraction (out of 4 lines)||3.0 (2.8-4.0)||4.0 (2.0-4.0)||3.0 (3.0-4.0)||3.0 (2.0-4.0)||1.02||.60|
|Step 4 people for help (out of 3 lines)||2.0 (1.0-2.3)||2.0 (1.3-3.0)||2.0 (1.0-3.0)||2.0 (1.0-2.0)||1.12||.57|
|Step 5 professionals for help (out of 4 lines)||3.0 (3.0-4.0)||3.0 (2.0-3.0)||3.0 (3.0-4.0)||3.0 (3.0-4.0)||4.96||.08|
|Step 6 making environment safe (out of 2 lines)||2.0 (1.0-2.0)||2.0 (2.0-2.0)||2.0 (1.0-2.0)||2.0 (1.0-2.0)||2.84||.24|
Many patients who present to the ED with suicidality do not receive effective intervention . EDs struggle with shortages in BH clinician availability and suicide-specific skills [ , ] and patients often describe their experiences of suicide-related care in the ED as negative [ ]. ReachCare was developed to initiate an acceptable and feasible form of evidence-based care in the ED and support the patient in the high-risk period after discharge [ ]. In this paper, we summarized initial implementation outcomes of the in-ED components of ReachCare across a variety of modalities. We found that patients rated the acceptability and appropriateness of the intervention highly in the context of other studies that have used the same measures [ - ], and there were no substantial differences across the various modalities. At just over 15 minutes, the self-administered, chatbot-style modality took less than half the time of the clinician modalities and did not require BH specialist input. The self-administered safety plans tended to have more content and the usability of the patient-facing tablet app was in the “very high” range. The safety plans that were created with ReachCare, whether self-administered with a chatbot or delivered by a clinician, were of high quality and completeness. Given that higher safety plan quality is associated with reduced risk of prospective suicidal behavior [ , ] and of subsequent psychiatric hospitalization [ ], ReachCare has the potential to positively impact patient outcomes in a fully powered trial, even when it is implemented using self-administered modality.
Technology is becoming a key implementation strategy for BH interventions in health care settings . Brief interventions for suicidality in the ED are recommended by the Joint Commission [ ] and the American College of Emergency Physicians [ ] but in practice have proven difficult to implement. In EDs, where there is a dearth of BH clinicians and the focus is on evaluation and disposition, engaging methods of self-administration may be one of the few ways to ensure that patients receive any intervention. Self-guided digital interventions for suicidality have been found to have a significant, if small, effect [ ]. This study contributes to growing evidence that, when delivered through engaging digital platforms, self-guided versions of effective interventions can be efficient, feasible, and acceptable. For example, the tablet-based intervention “Jaspr” was associated with better care process and decreased distress in patients presenting with suicidality [ ]. Similarly, a self-administered tablet-based safety planning website designed for use in the ED was associated with good usability and reduced suicidal intensity [ ]. Finally, “Lock to Live,” is a promising tablet-based intervention for counseling on access to lethal means in the ED [ , ]. These encouraging results support the idea that self-administration could offer unique advantages in terms of timely, tailored, and scalable suicide intervention in resource-constrained EDs.
Like the broader population of people presenting with suicidality, our sample displayed a high prevalence of depression, anxiety disorder, trauma, and attention-deficit/hyperactivity disorder, as well as elevated levels of thwarted belongness and perceived burdensomeness. Given that those who are experiencing suicidality may be feeling particularly disconnected , there are ethical implications around simply handing a patient a tablet computer as a panacea to suicide-related care in the ED. The development of such technology-facilitated intervention should start with careful mapping of user needs from a variety of stakeholders and apply an iterative approach to testing and refinement [ , ]. Although self-administration holds promise as a way of scaling evidence-based interventions, programs that propagate loneliness and disempowerment in suicidal patients have the potential to do more harm than good. Any self-administered software for this population should be engaging but easy to navigate, foster a sense of connection as much as possible (for example by using videos and interaction), and anticipate and seek to ameliorate feelings of burdensomeness and disconnection. When done well, it is possible that technology could increase empowerment by introducing flexibility and choice to fit with patients’ timeline and needs [ , ]. Self-administered intervention in this population may be thoughtfully introduced by a clinician, such as a physician, nurse, or a medical assistant: the patient could be offered a choice, to complete a safety plan with a BH clinician (in-person or via telehealth if necessary) or for themselves if they prefer. A third option is a hybrid approach, where a patient initiates the safety plan using software and a BH clinician reviews it with them, ideally before discharge or as part of telephone follow-up. Intervention is ideally not limited to the ED but carries on out into the community in the form of caring contacts or outpatient engagement that can help to address the underlying drivers of the individual’s suicidality [ ]. It is also important to note that technology is not a sufficient implementation strategy in itself: digital interventions require careful implementation planning, training, and support if they are to result in improved reach and sustainability [ ].
Finally, our study led to some interesting observations around digitally supported clinician delivery. There appeared to be a slight trend toward lower acceptability of the telehealth clinician modality compared to the in-person clinician modality in our small trial. Larger trials that have compared telehealth to in-person delivery for a variety of psychosocial interventions have found little difference in satisfaction or effectiveness between the 2 modalities [- ]. Moreover, telehealth evaluations can help to improve access and timeliness of mental health care in ED settings [ , ]. Further research is required to examine whether telemental health interventions specifically are less acceptable in an emergency medicine context.
This study has several limitations. Although the broader trial involved 3-month follow-up of clinical outcomes, the current analyses were cross-sectional only and our sample size was relatively small. In addition, the patients who were successfully enrolled tended to be younger, female, White, and non-Hispanic than those who were not enrolled. This may reflect the fact that the intervention was only available in English and trial enrollment required the patient to own a smartphone: additional user-centered research is needed to identify the needs of underserved populations and design systems that can adapt to their needs and preferences. For example, future research should explore Spanish-language adaptation of the intervention, as well as adaptations of the post-ED components that can be delivered without a smartphone. Notably, the majority of patients who screened positive for suicide risk during the trial were not eligible to participate because of medical or cognitive issues, which interfered with their ability to give informed consent: it remains to be seen what the reach of this intervention would be under naturalistic clinical conditions, where the capacity threshold for participation may be more flexible.
Our findings suggest that patients considered the ReachCare intervention in the ED to be acceptable and appropriate across a variety of modalities. Technology facilitation of evidence-based interventions in the ED holds much promise, with the caveat that we must take account of the particular and diverse needs of those in suicidal crisis in this setting. There are several potential directions for future research. These include possible adaptations for older patients, those whose preferred language is not English, and patients who do not own a smartphone. Other potential research questions include: which intervention modality patients or clinicians tend to select when given a choice; whether that choice is affected by patient characteristics such as age and medical condition; whether the selected modality affects longitudinal clinical outcomes; and rates of uptake under more naturalistic implementation conditions.
This research was supported by the National Institute of Mental Health of the National Institutes of Health (award R34MH123578). The content is the sole responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. We gratefully acknowledge the contribution of our participants and consultants.
Data are available in the National Institute of Mental Health Data Archive.
Conflicts of Interest
The authors of this paper include the developers of the ReachCare app.
Patient characteristics at baseline (N=47) by study arm.DOCX File , 21 KB
Characteristics of index ED visit (N=47) by arm.DOCX File , 16 KB
CONSORT-eHEALTH checklist (V 1.6.1).PDF File (Adobe PDF File), 1264 KB
- Stone DM, Jones CM, Mack KA. Changes in suicide rates—United States, 2018-2019. MMWR Morb Mortal Wkly Rep 2021;70(8):261-268 [https://www.cdc.gov/mmwr/volumes/70/wr/mm7008a1.htm?s_cid=mm7008a1_w] [CrossRef] [Medline]
- Ehlman DC, Yard E, Stone DM, Jones CM, Mack KA. Changes in suicide rates—United States, 2019 and 2020. MMWR Morb Mortal Wkly Rep 2022;71(8):306-312 [https://www.cdc.gov/mmwr/volumes/71/wr/mm7108a5.htm?s_cid=mm7108a5_w] [CrossRef] [Medline]
- Owens PL, McDermott KW, Lipari RN, Hambrick MM. Emergency Department Visits Related to Suicidal Ideation or Suicide Attempt, 2008-2017. Rockville (MD): Agency for Healthcare Research and Quality (US); 2020.
- Ballard ED, Cwik M, Van Eck K, Goldstein M, Alfes C, Wilson ME, et al. Identification of at-risk youth by suicide screening in a pediatric emergency department. Prev Sci 2017;18(2):174-182 [https://europepmc.org/abstract/MED/27678381] [CrossRef] [Medline]
- Grumet JG, Boudreaux ED. Universal suicide screening is feasible and necessary to reduce suicide. Psychiatr Serv 2023;74(1):81-83 [CrossRef] [Medline]
- Olfson M, Marcus SC, Bridge JA. Emergency treatment of deliberate self-harm. Arch Gen Psychiatry 2012;69(1):80-88 [https://jamanetwork.com/journals/jamapsychiatry/fullarticle/1107376] [CrossRef] [Medline]
- Inagaki M, Kawashima Y, Kawanishi C, Yonemoto N, Sugimoto T, Furuno T, et al. Interventions to prevent repeat suicidal behavior in patients admitted to an emergency department for a suicide attempt: a meta-analysis. J Affect Disord 2015;175:66-78 [https://www.sciencedirect.com/science/article/abs/pii/S0165032714008349?via%3Dihub] [CrossRef] [Medline]
- Miller IW, Camargo CA, Arias SA, Sullivan AF, Allen MH, Goldstein AB, et al. Suicide prevention in an emergency department population: the ED-SAFE study. JAMA Psychiatry 2017;74(6):563-570 [https://jamanetwork.com/journals/jamapsychiatry/fullarticle/2623157] [CrossRef] [Medline]
- Dunlap LJ, Orme S, Zarkin GA, Arias SA, Miller IW, Camargo CA, et al. Screening and intervention for suicide prevention: a cost-effectiveness analysis of the ED-SAFE interventions. Psychiatr Serv 2019;70(12):1082-1087 [https://ps.psychiatryonline.org/doi/10.1176/appi.ps.201800445] [CrossRef] [Medline]
- Ruffolo MC, Capobianco J. Moving an evidence-based intervention into routine mental health care: a multifaceted case example. Soc Work Health Care 2012;51(1):77-87 [CrossRef] [Medline]
- True G, Pollock M, Bowden CF, Cullen SW, Ross AM, Doupnik SK, et al. Strategies to care for patients being treated in the emergency department after self-harm: perspectives of frontline staff. J Emerg Nurs 2021;47(3):426-436 [https://europepmc.org/abstract/MED/33610311] [CrossRef] [Medline]
- Cullen SW, Diana A, Olfson M, Marcus SC. If you could change 1 thing to improve the quality of emergency care for deliberate self-harm patients, what would it be? A national survey of nursing leadership. J Emerg Nurs 2019;45(6):661-669 [https://europepmc.org/abstract/MED/31495507] [CrossRef] [Medline]
- Kirk JW, Nilsen P. Implementing evidence-based practices in an emergency department: contradictions exposed when prioritising a flow culture. J Clin Nurs 2016;25(3-4):555-565 [https://onlinelibrary.wiley.com/doi/10.1111/jocn.13092] [CrossRef] [Medline]
- Richmond JS, Dragatsi D, Stiebel V, Rozel JS, Rasimas JJ. American association for emergency psychiatry recommendations to address psychiatric staff shortages in emergency settings. Psychiatr Serv 2021;72(4):437-443 [https://ps.psychiatryonline.org/doi/10.1176/appi.ps.201900501] [CrossRef] [Medline]
- Betz ME, Sullivan AF, Manton AP, Espinola JA, Miller I, Camargo CA, et al. Knowledge, attitudes, and practices of emergency department providers in the care of suicidal patients. Depress Anxiety 2013;30(10):1005-1012 [https://europepmc.org/abstract/MED/23426881] [CrossRef] [Medline]
- Rayner G, Blackburn J, Edward KL, Stephenson J, Ousey K. Emergency department nurse's attitudes towards patients who self-harm: a meta-analysis. Int J Ment Health Nurs 2019;28(1):40-53 [CrossRef] [Medline]
- Larkin C, Djamasbi S, Boudreaux ED, Varzgani F, Garner R, Siddique M, et al. ReachCare mobile apps for patients experiencing suicidality in the emergency department: development and usability testing using mixed methods. JMIR Form Res 2023;7:e41422 [https://formative.jmir.org/2023/1/e41422] [CrossRef] [Medline]
- Sim J, Lewis M. The size of a pilot study for a clinical trial should be calculated in relation to considerations of precision and efficiency. J Clin Epidemiol 2012;65(3):301-308 [CrossRef] [Medline]
- Julious SA. Sample size of 12 per group rule of thumb for a pilot study. Pharmaceut Statist 2005;4(4):287-291 [CrossRef]
- Boudreaux ED, Jaques ML, Brady KM, Matson A, Allen MH. The patient safety screener: validation of a brief suicide risk screener for emergency department settings. Arch Suicide Res 2015;19(2):151-160 [CrossRef] [Medline]
- Stanley B, Brown GK. Safety planning intervention: a brief intervention to mitigate suicide risk. Cogn Behav Pract 2012;19(2):256-264 [CrossRef]
- Balapour A, Reychav I, Sabherwal R, Azuri J. Mobile technology identity and self-efficacy: implications for the adoption of clinically supported mobile health apps. Int J Inf Manag 2019;49:58-68 [https://www.sciencedirect.com/science/article/abs/pii/S0268401218310867?via%3Dihub] [CrossRef]
- Van Orden KA, Cukrowicz KC, Witte TK, Joiner TE. Thwarted belongingness and perceived burdensomeness: construct validity and psychometric properties of the interpersonal needs questionnaire. Psychol Assess 2012;24(1):197-215 [https://europepmc.org/abstract/MED/21928908] [CrossRef] [Medline]
- Hill RM, Rey Y, Marin CE, Sharp C, Green KL, Pettit JW. Evaluating the interpersonal needs questionnaire: comparison of the reliability, factor structure, and predictive validity across five versions. Suicide Life Threat Behav 2015;45(3):302-314 [CrossRef] [Medline]
- Posner K, Brown GK, Stanley B, Brent DA, Yershova KV, Oquendo MA, et al. The Columbia-suicide severity rating scale: initial validity and internal consistency findings from three multisite studies with adolescents and adults. Am J Psychiatry 2011;168(12):1266-1277 [https://ajp.psychiatryonline.org/doi/full/10.1176/appi.ajp.2011.10111704] [CrossRef] [Medline]
- Mccullumsmith CB, Williamson DJ, May RS, Bruer EH, Sheehan DV, Alphs LD. Simple measures of hopelessness and impulsivity are associated with acute suicidal ideation and attempts in patients in psychiatric crisis. Innov Clin Neurosci 2014;11(9-10):47-53 [https://europepmc.org/abstract/MED/25520888] [Medline]
- Weiner BJ, Lewis CC, Stanick C, Powell BJ, Dorsey CN, Clary AS, et al. Psychometric assessment of three newly developed implementation outcome measures. Implement Sci 2017;12(1):108 [https://implementationscience.biomedcentral.com/articles/10.1186/s13012-017-0635-3] [CrossRef] [Medline]
- Brooke J. SUS-a quick and dirty usability scale. In: Jordan PW, Thomas B, Weerdmeester BA, McClelland IL, editors. Usability Evaluation in Industry. Milton Park, Abingdon-on-Thames, Oxfordshire United Kingdom: Taylor & Francis; 1996:4-7
- Goodman M, Brown GK, Galfalvy HC, Spears AP, Sullivan SR, Kapil-Pair KN, et al. Group ("Project Life Force") versus individual suicide safety planning: a randomized clinical trial. Contemp Clin Trials Commun 2020;17:100520 [https://www.sciencedirect.com/science/article/pii/S2451865420300041?via%3Dihub] [CrossRef] [Medline]
- IBM Corp. Released 2021. IBM SPSS Statistics for Windows, Version 28.0. Armonk, NY: IBM Corp URL: https://www.ibm.com/support/pages/how-cite-ibm-spss-statistics-or-earlier-versions-spss [accessed 2023-09-21]
- MacDonald S, Sampson C, Turley R, Biddle L, Ring N, Begley R, et al. Patients' experiences of emergency hospital care following self-harm: systematic review and thematic synthesis of qualitative research. Qual Health Res 2020;30(3):471-485 [https://journals.sagepub.com/doi/10.1177/1049732319886566] [CrossRef] [Medline]
- Hofmann G, Hampanda K, Harrison MS, Fasano M, Nacht A, Yeoman M. Virtual prenatal and postpartum care acceptability among maternity care providers. Matern Child Health J 2022;26(7):1401-1408 [https://link.springer.com/article/10.1007/s10995-022-03412-7] [CrossRef] [Medline]
- Rosenberg M, Kianersi S, Luetke M, Jozkowski K, Guerra-Reyes L, Shih PC, et al. Wearable alcohol monitors for alcohol use data collection among college students: feasibility and acceptability. Alcohol 2023;111:75-83 [https://www.sciencedirect.com/science/article/abs/pii/S0741832923002252?via%3Dihub] [CrossRef] [Medline]
- Lai-Kwon J, Kelly B, Lane S, Biviano R, Bartula I, Brennan F, et al. Feasibility, acceptability, and utility of a nurse-led survivorship program for people with metastatic melanoma (MELCARE). Support Care Cancer 2022;30(11):9587-9596 [https://link.springer.com/article/10.1007/s00520-022-07360-4] [CrossRef] [Medline]
- Green JD, Kearns JC, Rosen RC, Keane TM, Marx BP. Evaluating the effectiveness of safety plans for military veterans: do safety plans tailored to veteran characteristics decrease suicide risk? Behav Ther 2018;49(6):931-938 [CrossRef] [Medline]
- Chalker SA, Parrish EM, Ceren CSM, Depp CA, Goodman M, Doran N. Predictive importance of social contacts on U.S. veteran suicide safety plans. Psychiatr Serv 2023;74(3):244-249 [CrossRef] [Medline]
- Gamarra JM, Luciano MT, Gradus JL, Stirman SW. Assessing variability and implementation fidelity of suicide prevention safety planning in a regional VA healthcare system. Crisis 2015;36(6):433-439 [https://europepmc.org/abstract/MED/26648231] [CrossRef] [Medline]
- Hasselberg MJ. The digital revolution in behavioral health. J Am Psychiatr Nurses Assoc 2020;26(1):102-111 [https://journals.sagepub.com/doi/10.1177/1078390319879750] [CrossRef] [Medline]
- Joint Commission. Suicide prevention resources to support Joint Commission accredited organizations implementation of NPSG 15.01. 01, revised July 2020. URL: https://www.jointcommission.org/-/media/tjc/documents/resources/patient-safety-topics/suicide-prevention/suicide_prevention_compendium_5_11_20_updated-july2020.pdf [accessed 2023-04-19]
- Wilson MP, Moutier C, Wolf L, Nordstrom K, Schulz T, Betz ME. ED recommendations for suicide prevention in adults: The ICAR2E mnemonic and a systematic review of the literature. Am J Emerg Med 2020;38(3):571-581 [https://www.sciencedirect.com/science/article/abs/pii/S0735675719304115?via%3Dihub] [CrossRef] [Medline]
- Torok M, Han J, Baker S, Werner-Seidler A, Wong I, Larsen ME, et al. Suicide prevention using self-guided digital interventions: a systematic review and meta-analysis of randomised controlled trials. Lancet Digit Health 2020;2(1):e25-e36 [https://www.thelancet.com/journals/landig/article/PIIS2589-7500(19)30199-2/fulltext] [CrossRef] [Medline]
- Dimeff LA, Jobes DA, Koerner K, Kako N, Jerome T, Kelley-Brimer A, et al. Using a tablet-based app to deliver evidence-based practices for suicidal patients in the emergency department: pilot randomized controlled trial. JMIR Ment Health 2021;8(3):e23022 [https://mental.jmir.org/2021/3/e23022] [CrossRef] [Medline]
- Boudreaux ED, Brown GK, Stanley B, Sadasivam RS, Camargo CA, Miller IW. Computer administered safety planning for individuals at risk for suicide: development and usability testing. J Med Internet Res 2017;19(5):e149 [https://www.jmir.org/2017/5/e149/] [CrossRef] [Medline]
- Betz ME, Knoepke CE, Simpson S, Siry BJ, Clement A, Saunders T, et al. An interactive web-based lethal means safety decision aid for suicidal adults (lock to live): pilot randomized controlled trial. J Med Internet Res 2020;22(1):e16253 [https://www.jmir.org/2020/1/e16253/] [CrossRef] [Medline]
- Betz ME, Knoepke CE, Siry B, Clement A, Azrael D, Ernestus S, et al. 'Lock to live': development of a firearm storage decision aid to enhance lethal means counselling and prevent suicide. Inj Prev 2019;25(Suppl 1):i18-i24 [https://europepmc.org/abstract/MED/30317220] [CrossRef] [Medline]
- Joiner T. Why People Die by Suicide. Cambridge, MA: Harvard University Press; 2005.
- Akeel AU, Mundy D. Re-thinking technology and its growing role in enabling patient empowerment. Health Informatics J 2019;25(4):1278-1289 [https://journals.sagepub.com/doi/10.1177/1460458217751013] [CrossRef] [Medline]
- Lau F. Patient empowerment: the role of technology. In: Lau F, Bartle-Clar JA, Bliss G, Borycki EM, Courtney KL, Kuo AMH, et al, editors. Improving Usability, Safety and Patient Outcomes with Health Information Technology: From Research to Practice. Amsterdam, Berlin, Washington, DC: IOS Press; 2019.
- Calvillo J, Román I, Roa LM. How technology is empowering patients? A literature review. Health Expect 2015;18(5):643-652 [https://onlinelibrary.wiley.com/doi/10.1111/hex.12089] [CrossRef] [Medline]
- Jobes DA, Gregorian MJ, Colborn VA. A stepped care approach to clinical suicide prevention. Psychol Serv 2018;15(3):243-250 [CrossRef] [Medline]
- Graham AK, Lattie EG, Powell BJ, Lyon AR, Smith JD, Schueller SM, et al. Implementation strategies for digital mental health interventions in health care settings. Am Psychol 2020;75(8):1080-1092 [https://europepmc.org/abstract/MED/33252946] [CrossRef] [Medline]
- Herbert MS, Afari N, Liu L, Heppner P, Rutledge T, Williams K, et al. Telehealth versus in-person acceptance and commitment therapy for chronic pain: a randomized noninferiority trial. J Pain 2017;18(2):200-211 [https://www.jpain.org/article/S1526-5900(16)30292-9/fulltext] [CrossRef] [Medline]
- Morland LA, Mackintosh MA, Greene CJ, Rosen CS, Chard KM, Resick P, et al. Cognitive processing therapy for posttraumatic stress disorder delivered to rural veterans via telemental health: a randomized noninferiority clinical trial. J Clin Psychiatry 2014;75(5):470-476 [CrossRef] [Medline]
- Choi NG, Hegel MT, Marti N, Marinucci ML, Sirrianni L, Bruce ML. Telehealth problem-solving therapy for depressed low-income homebound older adults. Am J Geriatr Psychiatry 2014;22(3):263-271 [https://www.liebertpub.com/doi/10.1089/tmj.2018.0227] [CrossRef] [Medline]
- Fairchild RM, Ferng-Kuo SF, Laws S, Rahmouni H, Hardesty D. Telehealth decreases rural emergency department wait times for behavioral health patients in a group of critical access hospitals. Telemed E Health 2019;25(12):1154-1164 [https://www.liebertpub.com/doi/10.1089/tmj.2018.0227] [CrossRef] [Medline]
- Southard EP, Neufeld JD, Laws S. Telemental health evaluations enhance access and efficiency in a critical access hospital emergency department. Telemed E Health 2014;20(7):664-668 [CrossRef] [Medline]
|BH: behavioral health|
|ED: emergency department|
|ED-SAFE: Emergency Department Safety Assessment and Follow-up Evaluation|
|PSS: Patient Safety Screener|
Edited by J Torous; submitted 09.06.23; peer-reviewed by C King, C Chung; comments to author 29.06.23; revised version received 25.07.23; accepted 27.07.23; published 24.10.23Copyright
©Celine Larkin, Bengisu Tulu, Soussan Djamasbi, Roscoe Garner, Fatima Varzgani, Mariam Siddique, John Pietro, Edwin D Boudreaux. Originally published in JMIR Mental Health (https://mental.jmir.org), 24.10.2023.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Mental Health, is properly cited. The complete bibliographic information, a link to the original publication on https://mental.jmir.org/, as well as this copyright and license information must be included.