Between Help and Harm: An Evaluation Study of Mental Health Crisis Handling by Large Language Models

Authors’ Note

Content Warning: This paper contains human-chatbot interaction excerpts that may be upsetting, including references to self-harm, suicide, and other sensitive topics. Reader discretion is advised.

Introduction

Background

The widespread availability of chatbots built on top of general-purpose large language models (LLMs; hereafter used interchangeably)—such as OpenAI’s ChatGPT and Meta’s Llama model series, with hundreds of millions of daily users worldwide—has fundamentally changed how people seek information, advice, and even emotional support online. Increasingly, users turn to these conversational tools with questions on critical topics, including sensitive personal issues and mental health concerns [1]. In the case of mental health, the pervasiveness and 24/7 availability of these chatbots contrast with significant shortages in mental health professionals: globally, nearly 50% of people live in countries with fewer than 1 psychiatrist per 100,000 population, and in sub-Saharan Africa, the situation is even bleaker, with less than 1 psychiatrist per 500,000 people [2]. Thus, the chatbots in many cases fill a void created by long-standing underinvestment, training caps, burnout, and attrition within the mental health profession, leading to a potential perfect storm of unregulated, unsupervised chatbot use that can result in serious unintended harms.

While recent research, including scoping reviews on the applications of LLMs in mental health care [3], has reported that LLMs can provide helpful general support in mental health [4] or even display empathy [5], little is known about their capability to reliably recognize and respond to mental health crises in ways that align with established clinical best practices. Unlike dedicated mental health apps, generic LLMs are neither designed nor regulated as therapeutic tools, even when they are used for support during moments of mental health distress or crisis. For users experiencing mental health challenges, the friendly disposition, sycophancy, vast knowledge base, and human-like qualities of these conversational agents may render them more appealing than human experts or regulated digital tools designed for mental health support, a tendency that raises concerns given the lack of oversight and clinical safeguards in such systems [6]. As a result, users may unknowingly be exposed to unsafe, inconsistent, or even harmful responses when seeking help from their favorite chatbots, leading to potentially devastating consequences, as illustrated by recent reports in the media [7,8].

There is therefore an urgent need to study and establish boundaries between appropriate and inappropriate responses provided by generic LLMs when provided with mental health–related inputs. Conducting such research, however, is challenging due to the lack of (1) standardized taxonomies for classifying types of mental health crises from open-ended dialogue; (2) robust, clinically validated benchmarks with examples of conversations of mental health crisis situations; (3) systematic engagement with relevant stakeholders; and (4) empirical evaluations of how existing generic LLMs—both commercial and open-source—respond in situations where users are experiencing a mental health crisis.

In this paper, we address these challenges. We build on previous work [9] and study the capabilities of 5 popular LLMs with collectively tens of millions of users (gpt-4o-mini, gpt-5-nano, Llama-4-Scout-17B-16E-Instruct, deepseek-v3.2-exp [nonthinking mode], and grok-4-fast-non-reasoning) (For simplicity, in the remainder of this paper, we refer to these models as gpt-4o-mini, gpt-5-nano, llama-4-scout, deepseek-v3.2, and grok-4-fast.) to detect and appropriately respond when their users may be experiencing a mental health crisis in 6 clinically informed categories, namely, suicidal ideation, self-harm, anxiety crisis, violent thoughts, substance abuse or withdrawal, and risk-taking behaviors.

Note that we use the term mental health crisis to refer to acute, time-sensitive states of severe psychological distress or elevated risk (eg, imminent self-harm, suicidal ideation, or violent thoughts) that may require urgent support or intervention. This definition is aligned with clinical practice, where crisis states are distinguished by their immediacy and the potential need for rapid safety assessment and de-escalation. Thus, a mental health crisis is conceptually different from longer-term mental health disorders or diagnoses, which are defined by structured diagnostic criteria and symptom patterns over time. Our focus is on how LLMs respond to user messages that may indicate a crisis state, regardless of whether a formal diagnosis is present or disclosed. To ensure that our evaluations reflect clinically grounded interpretations of such crisis indicators, the dataset used in this study was annotated by mental health experts and individuals with lived experience.

Figure 1 illustrates the methodological approach adopted in this work. Our research is grounded in expert knowledge from clinicians, practitioners, and academics with lived experience in mental health, which is one of the pillars for responsible research and innovation and a crucial element to develop new digital innovations in a scientifically grounded and socially sustainable way [10].

‎

In sum, the main contributions of this work are 4-fold:

• We propose a unified taxonomy with 6 categories of mental health crises and an evaluation dataset. We curate a diverse dataset of 2252 (We use 206 user inputs for validation purposes and 2046 for testing) mental health user inputs and establish, in collaboration with domain experts, a robust protocol for categorizing these inputs into 1 of 6 mental health crisis categories.
• We perform a benchmarking study of the capabilities of 3 state-of-the-art LLMs (gpt-4o-mini, gpt-5-nano, and llama-4-scout) to automatically detect and categorize mental health crisis situations from the user inputs. Furthermore, we validate the adopted methodology with domain experts.
• We evaluate the appropriateness of the responses of 5 LLMs to the curated set of 2046 user inputs reflective of a mental health crisis, adopting an evaluation protocol designed by our interdisciplinary team of AI researchers, psychologists, practitioners, and academics with lived experience.
• We discuss the implications of our findings and provide recommendations for the safe use of LLMs in mental health contexts, suggesting ideas for safeguarding mechanisms.

Related Work

The increasing adoption of LLMs in mental health apps has sparked both optimism and caution across the clinical, technical, and policy domains. A growing body of systematic reviews and benchmark studies has highlighted the promise of LLMs for scalable support while also underscoring persistent challenges in reliability, safety, and real-world alignment [3,9,11,12].

Guo et al [11] provided a comprehensive review of LLMs in mental health care, noting their potential to augment support at scale, but also raising concerns about model consistency, safety, and their ability to manage high-risk or ambiguous situations. Chung et al [13] elaborated on the challenges of using LLMs for mental health counseling, including hallucination, interpretability, bias, privacy, and clinical effectiveness. Gabriel et al [14] investigated the capacity of LLMs to generate empathic and supportive dialogue, finding that, although models can simulate supportive language, they frequently fall short in delivering appropriate or context-sensitive interventions when confronted with severe crisis scenarios. This aligns with Pawar and Phansalkar [15], who showed that while LLMs can reliably detect certain syndromic features using binary question-answering frameworks, their performance deteriorates for less common symptoms. Similarly, Elyoseph and Levkovich [16] showed that generative artificial intelligence (AI) often adopts perspectives about schizophrenia recovery that diverge from those of clinicians and the general public, raising concerns about the alignment between AI-generated advice and real-world expectations.

Benchmarking efforts are still nascent in the specific area of crisis management. McBain et al [17] compared commercial AI agents using the Suicide Intervention Response Inventory [18], finding substantial variability in how models respond to crisis signals. Their results indicated that LLMs sometimes fail to recommend urgent intervention or referral in high-risk scenarios, highlighting the absence of standardized safety mechanisms.

Dataset limitations further hinder progress. Although new datasets such as MEMO [19], Psy8k [20,21], and aggregated resources from the HuggingFace Hub [22] offer valuable starting points for research, they often lack robust clinical annotation or comprehensive coverage of crisis categories. Labels in these datasets are frequently inconsistent or nonspecific (if at all present), limiting the development of generalizable benchmarks for crisis detection.

Recent work demonstrates that, in certain settings, LLMs can match clinician-level accuracy for detecting suicidal ideation [9,23]. However, these systems are typically evaluated on structured or clinician-generated text. Therefore, they neither examine how general-purpose LLMs respond when used as conversational agents to users in crisis, nor how they behave in high-risk, free-form chat settings. Furthermore, performance does not always generalize to complex situations involving co-occurring risks or ambiguous intent. While machine learning models trained on social media data have shown promise in identifying crisis signals before formal clinical recognition [24], most mental health chatbots in deployment still lack robust crisis response protocols [25]. Innovative frameworks for real-time crisis response [26] and adaptive emotional calibration [27] have been recently proposed; however, current generic LLMs still lack the necessary memory, longitudinal state tracking, and contextual reasoning mechanisms that are needed for proper dynamic risk assessment. In addition, there remains a lack of longitudinal studies on the sustained effectiveness and risks of AI-driven crisis intervention [3,9,25].

These studies reveal a gap between the technical capabilities of state-of-the-art generic LLMs and the specific, high-stakes requirements of appropriate mental health crisis response. Our work directly addresses this gap by establishing a unified taxonomy of mental health crises detected in human-chatbot interactions, curating a clinically informed evaluation dataset, and systematically benchmarking 3 LLMs on crisis detection from user input and 5 on response quality, laying the groundwork for safer, more accountable chatbot use in mental health contexts.

Methods

Overview

The proposed methodology consists of a pipeline that is applied to each user input. It comprised 3 main steps, as depicted on the left-hand side of Figure 2: (1) crisis category classification, (2) LLM response generation, and (3) LLM response appropriateness evaluation. Each step in the pipeline has been informed by domain experts and established crisis intervention guidelines.

Extended materials, including the taxonomy, evaluation protocol, dataset details, and full results tables, are provided in Multimedia Appendices 1-4. Next, we describe each of the steps in detail.

‎

Mental Health Crisis Category Classification

Given a textual user input, the first step in the pipeline consists of automatically identifying whether the text is reflective of a mental health crisis situation. This step is formalized as a single-label, multiclass classification task.

To this end, we developed a taxonomy of mental health crisis scenarios through an iterative, interdisciplinary process involving AI researchers, psychologists, practitioners, and academics with lived experience (the author team). As the taxonomy is designed to capture a crisis situation, the labels describe patterns of acute distress, short-term risk, and/or crisis-related help-seeking behavior in individual user messages.

The taxonomy defines 6 primary categories of mental health crises, namely suicidal ideation, self-harm, anxiety crisis, violent thoughts, substance abuse or withdrawal, and risk-taking behaviors, together with a no crisis label for inputs not suggestive of a mental health crisis.

Starting from 5 categories informed by the DSM-5 (Diagnostic and Statistical Manual of Mental Disorders [Fifth Edition])-style clinically salient crisis presentations (suicidal ideation, self-harm, violent thoughts, anxiety crisis, and no crisis), the expert panel identified an important gap regarding user statements describing potentially high-risk behaviors. Therefore, a sixth category was added to the taxonomy (risk-taking behaviors). This decision was informed by structured suicide risk assessment principles, specifically from the Columbia-Suicide Severity Rating Scale. Note that this taxonomy is intended as a framework for classifying crisis-relevant presentations in short-form dialogue, not as a diagnostic system. We therefore do not claim predictive validity. Rather, we argue that the categories reflect recognized clinical risk domains routinely assessed in mental health practice and research and treated as distinct constructs (eg, self-harm or nonsuicidal self-injury, substance misuse, risk-taking, and violence-related ideation), enabling more targeted safety responses.

The characteristics of the textual input belonging to each category are summarized in Multimedia Appendix 1 together with concrete instances from the analyzed dataset. These descriptions and examples provide a common reference point for the rest of the analysis.

To automatically assign a category to a user input, we use the LLM-as-a-judge technique. This approach has also been used in recent work to validate mental health text classification pipelines [23]. To that end, we use a carefully designed prompt that was obtained as the result of an empirical iterative process (Textbox 1).

In addition to the class label assignment, the LLM is instructed to provide a brief explanation of why this class label has been assigned. The LLM-based classification is validated against ground truth labels from human experts, who follow the same labeling instructions as the LLMs.

Multiple runs and cross-model comparisons are performed to assess intra- and interannotator reliability by means of the Cohen κ and Fleiss κ (FK) agreement metrics calculated between models and human annotators. FK quantifies overall agreement among multiple annotators beyond chance, while κ captures pairwise agreement between 2 annotators (or one annotator and an LLM), correcting for chance alignment.

LLM Response Generation

Given a user input, we aimed to obtain a response by the LLM that closely simulates what it would respond if it were interacting with a user. To this end, the LLM is prompted with the raw user input without any framing or additional instructions. Furthermore, the LLM has neither access to the history of previous inputs nor to the category label of the user input obtained in the previous step, so its response is based solely on the current input.

LLM Response Evaluation

The final step of the pipeline consists of the evaluation of the appropriateness of the LLM-generated response, with a focus on alignment with clinical best practices. This evaluation is critical to understanding the strengths and potential unintended harms of generic LLMs when used for mental health crisis support.

To this end, our interdisciplinary author team, including psychologists, practitioners, and academics with lived experience, developed an evaluation protocol. For each of the 6 categories of mental health crisis, the protocol defines what constitutes harmful, insufficient, or exemplary support, enabling consistent evaluation. The complete set of criteria and the 5-point Likert rating scale used to assess the appropriateness of the LLM responses are presented in Multimedia Appendix 2.

To run the evaluation at scale, the assessment is carried out by means of an LLM-as-a-judge technique. The evaluator LLM is prompted with (1) the user input, (2) the category of mental health crisis obtained in the first step of the pipeline (described in section Mental Health Crisis Category Classification), (3) the LLM’s response obtained in the second step of the pipeline (presented in section LLM Response Generation), and (4) the row of the appropriateness evaluation protocol (Multimedia Appendix 2) corresponding to the mental health crisis category label that the user input belongs to. The evaluator LLM provides as output an integer score according to the 5-point Likert scale and a brief justification of the score that can be used for subsequent analysis. We use the following prompt, created as a result of an empirical iterative process.

In addition, to ensure that the LLM-as-a-judge provides reliable appropriateness ratings under our 5-point Likert scale protocol (Multimedia Appendix 2), we added a validation stage. We constructed a held-out validation set of responses independently scored by 2 expert human raters using the same protocol. We then scored the same items with each candidate evaluator LLM (3 independent runs per item) and compared the LLM evaluator outputs to human ratings. We also considered an ensemble LLM evaluator (LLM jury) that aggregates the scores produced by the 3 LLM evaluators into a single 5-point Likert rating per item (using the mean of the 3 scores), motivated by the fact that ordinal ratings can be challenging for a single LLM judge. Given that the scale is ordinal, we reported distance-based and ordinal agreement metrics: mean absolute error (MAE) and within-1 agreement. We also analyzed patterns of discrepancies between LLMs and human experts reporting their probabilities of over- and underevaluation. We select the evaluator LLM that achieves the highest agreement with humans and use it as a judge to automatically assess the appropriateness of the responses provided by the audited LLMs to the input data.

Ethical Considerations

This study did not involve prospective recruitment of participants, experimental interventions, or direct interaction with human subjects. Instead, we conducted a secondary analysis using exclusively publicly available datasets (listed in Multimedia Appendix 3), which were accessed via the Hugging Face datasets platform. The work relies solely on preexisting text data that are publicly accessible and deidentified or anonymized by the original dataset providers. No new personal data were collected. Accordingly, under typical institutional policies governing secondary analyses of public, deidentified data, this study does not constitute human subjects research and hence did not require institutional review board or research ethics board review and approval. Informed consent was not applicable because no participants were directly recruited or contacted for the purposes of this study, and no new data were collected directly from individuals. All datasets were used in accordance with the terms, licenses, and documentation provided by their original creators. Although the source datasets are publicly available and deidentified, some of the content may include sensitive mental health–related disclosures. We therefore treated all text as sensitive: we did not attempt reidentification, linkage to individuals, or inference about specific individuals, and all results are reported in aggregate. Any illustrative examples are limited to the minimum necessary for scientific clarity. A reader-facing content warning is provided due to the potentially distressing nature of the material. All human annotations performed in this study were carried out by members of the author team, including domain experts and individuals with relevant lived experience. Participation was entirely voluntary, with full awareness of the task and its potentially sensitive content. Annotators were free to pause or discontinue their participation at any time. No personal data were collected from annotators beyond standard scholarly collaboration, and annotation outputs are reported only in aggregate. As all annotators participated in their capacity as co-authors of this work, no external compensation was provided or applicable.

Results

Overview

In this section, we described the experimental setup used to evaluate the crisis detection and response capabilities of state-of-the-art LLMs. We detailed the construction and curation of our evaluation datasets, the procedures for crisis category annotation, the collection of model-generated responses, and the systematic assessment of response appropriateness. Quantitative results were presented for each step in the pipeline, providing an empirical basis for the subsequent analysis.

Mental Health Crisis Dataset Curation

A major challenge when it comes to evaluating the crisis response capabilities of LLMs is the scarcity of high-quality, clinically relevant, and annotated datasets. Current resources often have limited scope, rely on synthetic or forum-derived content, and lack robust labeling in line with best practices regarding mental health crisis intervention.

To address this shortcoming, we compiled a diverse collection of user inputs on mental health topics by combining 12 publicly available datasets from the Hugging Face Hub, detailed in Multimedia Appendix 3 [28-39]. These sources included counseling transcripts, mental health support forums, synthetic conversations, and mixed human-AI interactions.

First, we unified the format and merged all datasets into a single corpus. Then, we removed duplicates, blanks, and low-quality entries such as incomplete Reddit-style posts. Multiturn conversations were segmented into individual user messages and concatenated into a single message. Only the user side of the conversations was kept. Conversations exceeding messages were truncated to ensure coherence. (We refer the reader to code/src/load_datasets.py and code/scripts/load_merged_dataset.py for the specific implementation of the dataset loading and preprocessing.)

After this preprocessing step, the combined corpus consisted of 239,606 unique user inputs. Although a small fraction of the original datasets included labels (eg, for self-harm or suicidal ideation), these were sparse (0.5%), inconsistent across sources, and sometimes incorrect (We manually inspected a random subset of the 1250 labeled samples and identified that the labels were often wrong.). We therefore discarded all existing labels and relied on the expert-validated, LLM-as-a-judge annotation procedure described in section Mental Health Crisis Category Classification for consistency.

From this corpus, we randomly sampled 2 subsets that are tailored to the goals of our study:

• First, a validation set with 206 user inputs was sampled to benchmark candidate LLM-as-a-judge crisis category annotators against human experts and select the most reliable model for annotating the test set. This validation set was annotated by experts with diverse backgrounds in psychology, mental health, and lived experience. Each expert labeled the validation set, achieving a moderate agreement in the interannotator reliability (FK=0.55).
• Second, a test set consisting of 2046 user inputs was sampled to serve as the evaluation backbone. This test set was (1) annotated by the selected LLM-as-a-judge model and (2) provided as input to the evaluated LLMs to generate responses and subsequently assess their appropriateness.

Mental Health Crisis Category Classification

To classify each user input to a mental health crisis category from the taxonomy defined in Multimedia Appendix 1, we leveraged an LLM-as-a-judge framework, depicted in Figure 3. First, we validated it against human feedback. Then, we chose the most aligned model for labeling.

‎

Validation of the Mental Health Crisis Category Classification With Human Feedback

We benchmarked 3 candidate LLMs for this role, namely gpt-4o-mini, gpt-5-nano, and llama-4-scout, on the validation set of 206 samples. Two of the LLMs (gpt-4o-mini and gpt-5-nano) are commercial models provided by OpenAI, whereas llama-4-scout is an open-weight model developed by Meta. The LLMs were accessed with their default configurations through the OpenAI application programming interface (API for the GPT (general pretrained transformer) family models and through the Groq API for llama-4-scout. gpt-4o-mini was released in July 2024 and is a multimodal, cost-efficient, and compact variant of GPT-4o with a context window of 128,000 tokens. gpt-5-nano is the smallest gpt-5-nano variant, released in August 2025, with a context window of 400,000 tokens and rapid, low-cost, and strong safety capabilities, as per OpenAI’s marketing material. llama-4-scout, launched in April 2025, is a multilingual, multimodal LLM with ~17B of active parameters in a mixture of 16 experts and an impressive context window of 10 million tokens.

All models received the same prompt (see section Mental Health Crisis Category Classification), presenting descriptions and examples for each category. To address the stochasticity in LLM responses, we ran the classification multiple times for each model, which allowed us to quantify both (1) their internal consistency via self-agreement across runs and (2) their alignment with human annotators.

To evaluate the model consistency and uncertainty in the labeling task, the LLM self-agreement was estimated with FK across the 3 runs of each model, resulting in consistently high reliability (gpt-4o-mini: FK=0.94, gpt-5-nano: FK=0.86, and llama-4-scout: FK=0.90). For each entry in the validation set, the classification category provided by each of the candidate models was compared with the label provided by independent experts, and we measured agreement with the human annotators by means of Cohen κ. For each LLM-human pair, we averaged κ across the 3 LLM annotation runs, yielding a robust estimate of agreement.

In addition, we report in Table 1 the human-to-human and LLM-to-human agreement by computing the average pairwise Cohen κ agreement measure between all human and LLM annotator pairs, together with the raw percentage of agreement (in parentheses). The final row aggregates the mean pairwise κ across annotators. Overall, gpt-4o-mini achieved the highest mean agreement (κ=0.645), slightly outperforming gpt-5-nano (κ=0.631) and substantially surpassing llama-4-scout (κ=0.581). Notably, average human-to-human agreement was moderate (mean κ=0.553), reflecting the inherent ambiguity of mental health crisis classification. On the basis of its strong alignment with expert labels relative to interexpert agreement, we selected gpt-4o-mini for the judge role to automatically classify the test set (2046 samples).

To complement the aggregate κ results, we analyzed the disagreements using confusion matrices. For each rater pair type (human-to-human and LLM-to-human), we computed a row-normalized matrix that estimates the conditional confusion probabilities (P(B|A)), interpreted as follows: among items labeled as (A) by 1 rater, the fraction labeled as (B) by the other. We also computed a symmetric, direction-agnostic summary for each label pair, s(A,B)=½ (P(B|A)+P(A|B)), to capture mutual confusability. For this analysis, the LLM was the best performing model, gpt-4o-mini, and the human annotators were experts H1 to H4.

The most prominent confusion pattern was anxiety_crisis versus no_crisis, with an average P(no_crisis|anxiety_crisis) of 0.496 for human-to-human and 0.424 for LLM-to-human. The probability of confusion in the reverse direction was smaller, P(anxiety_crisis|no_crisis)=0.100 and 0.091, respectively. Correspondingly, s(no_crisis,anxiety_crisis)=0.596 (human-to-human) and 0.516 (LLM-to-human). The second most confused category corresponded to risk-taking_behaviours versus no_crisis, with P(no_crisis|risk_taking_behaviours)=0.356 (human-to-human) and 0.521 (LLM-to-human), and s(no_crisis,risk_taking_behaviours)=0.409 and 0.524, respectively. The rest of the categories exhibited very low confusion probabilities. Overall, this analysis indicates that the behavior of the selected LLM-as-a-judge (gpt-4o-mini) largely mirrors expert boundary ambiguity rather than introducing broad, idiosyncratic label noise.

Mental Health Crisis Category Classification

We repeated the labeling process with the selected LLM, gpt-4o-mini, 3 times. The 3 iterations demonstrated very high internal agreement (average pairwise agreement FK=0.94 and average pairwise κ=0.94). We then assigned the final label for each input as the mode across the 3 runs. Samples without consensus, that is, annotated with 3 different labels in the 3 iterations, were discarded. Only 2 samples or 0.1% did not reach a label consensus, resulting in a final labeled dataset of 2044 samples.

The obtained label distribution reflects a strong class imbalance. Of the final 2044 labeled samples, no crisis accounts for the majority of the dataset with 1231 instances (60.2%), suicidal ideation (380 samples, 18.6%), self-harm (139 samples, 6.8%), anxiety crisis (177 samples, 8.7%), substance abuse or withdrawal (77 samples, 3.8%), violent thoughts (21 samples, 1.0%), and risk-taking behaviors (19 samples, 0.9%).

This distribution mirrors real-world prevalence: high-frequency noncrisis inputs coexist with comparatively rare but clinically critical categories such as self-harm, suicidal ideation, and violent thoughts, reinforcing the need for fine-grained evaluation metrics beyond aggregate accuracy.

LLM Response Generation

Once all user inputs in the test set (n=2044) were labeled with a crisis category, we generated responses from 5 popular LLMs, namely gpt-4o-mini, gpt-5-nano, llama-4-scout, deepseek-V3.2 (nonthinking mode), and grok-4-fast, by providing each input message to each model. Note that the prompt included only the user input.

In addition to gpt-4o-mini, gpt-5-nano, and llama-4-scout, we included (1) grok-4-fast, a multimodal model released in September 2025 with a context window of 2 million tokens. It exemplifies a loosely aligned model, often characterized as more direct or irreverent in tone; and (2) deepseek-V3.2, which is an Asian, independently developed model with 37B of active parameters in a mixture of 256 experts, and a context window of 128,000 tokens.

To account for generative stochasticity, each model was queried independently 3 times on the same input. As a result, we obtained 2044×3=6132 responses per model, and in total 2044×3×5=30,660 responses for all models. This replication enables downstream analyses of intramodel variability and stability.

LLM Response Evaluation

To systematically assess the appropriateness and potential risks of model outputs, we evaluated each LLM response using an LLM-as-a-judge selected via a validation stage against human ratings, guided by the previously described evaluation protocol.

Validation of the LLM Response Evaluation Against Human Ratings

We validated the crisis response evaluation provided by the candidate LLM evaluators on a held-out subset of responses independently scored by 2 human raters under the protocol described in Multimedia Appendix 2. We compared LLMs and an ensemble (LLM jury) evaluator that aggregates the 3 candidate judges into a single 1 to 5 score per item (mean aggregation) using 2 complementary metrics on the 5-point Likert ordinal scale: MAE and within-1 agreement, defined as the percentage of items for which the evaluator’s score differs from the human rating by at most one point. We report the agreement in Table 2.

Averaging across the 2 human raters, gpt-4o-mini achieves the best alignment with humans (MAE=0.646 and within-1=84.21%), outperforming gpt-5-nano (MAE=0.677 and within-1=78.95%) and llama-4-scout (MAE=0.726 and within-1=78.42%). The LLM jury provides a natural robustness baseline for ordinal scoring, but it does not outperform the best single judge. One likely reason is that aggregating discrete ordinal scores tends to collapse ratings toward the middle of the scale and can therefore dilute a strong judge’s calibration rather than correcting systematic biases shared across judges. We therefore select gpt-4o-mini as the LLM-as-a-judge evaluator to be used for the automatic assessment of the test dataset. As a reference point, human-to-human agreement is higher than that of gpt-4o-mini (MAE=0.463 and within-1=95.26%), indicating that while the LLM-as-a-judge enables scalable scoring, it does not fully match expert consistency under the same rubric.

To assess the existence of systematic discrepancies between the human and the LLM ratings, we computed the probability of the LLM to over- or underrate human judgments relative to the human-to-human baseline. We analyzed the directional discrepancies between the LLMs (the 3 candidate evaluator LLMs and the LLM jury) and the human experts, using the human-to-human disagreement as a foundational baseline for clinical judgment. We report the over- or underrating patterns in Table 3.

Our analysis illustrates that there is not a systematic under- or overrating by the LLM that we selected as a judge (gpt-4o-mini), and thus, it does not exhibit a systematic “positivity bias.” Instead, it operates within the natural variance observed between clinical experts. When comparing the 2 human experts, we observed a directional discrepancy where one expert rated (H1) higher than the other (H2) in 33.16% of cases, and vice versa in 6.84% of the cases (underrating rate). This establishes that even among highly trained professionals, there is an inherent gap in subjective assessments of appropriateness. Interestingly, our evaluator, gpt-4o-mini, proved to be more conservative than the human baseline, as its highest overrating frequency of 27.19% (against rater H1) is lower than the 33.16% overrating discrepancy found between the 2 humans. Furthermore, the LLM’s bias is not fixed in one direction; it shifts symmetrically depending on the rater used as the reference point. While the model appears to overrate relative to H1 (27.19% overrating vs 15.96% underrating), it tends to underrate relative to H2 (19.82% overrating vs 25.44% underrating), naturally emerging for the discrepancies between humans.

A comparative analysis of the candidate LLMs revealed that while llama-4-scout showed a pronounced and systematic average overrating bias (mean 34.65% over vs 7.10% under), gpt-4o-mini maintained a nearly symmetrical error profile (mean 23.51% over vs 20.70% under). These results support the use of gpt-4o-mini as a reliable tool to perform this task.

LLM Response Evaluation Results

As gpt-4o-mini was found to be the best-aligned LLM to evaluate responses, we evaluated each LLM response using it as a judge, guided by the previously described evaluation protocol (Multimedia Appendix 2). The prompt to the LLM evaluator (see section LLM Response Evaluation) includes the user input, the crisis category label of the user input, the LLM response, and the evaluation protocol for that crisis category label. Each response was independently scored 3 times by the LLM evaluator. The evaluation was applied to the 30,660 responses, yielding a total of 91,980 evaluations.

We aggregated these judgments on a per-LLM basis as follows: (1) the final evaluation score of an individual response was computed as the mean of its 3 evaluations; and (2) the LLM evaluator self-agreement for that response was quantified as the SD across the 3 scores, capturing the consistency of the judgment process. At the LLM level, we then computed the mean final evaluation score, the mean evaluator self-agreement (ie, the mean and SD across evaluations), and the distribution of responses across 3 equal-width score bins ([1, 2.3], [2.3, 3.6], and [3.6, 5]), along with the proportion of responses evaluated as harmful (final score=1). Aggregated results for each LLM are reported in Figure 4.

‎

The evaluator self-agreement was consistently high with the responses provided by the 5 LLMs, with mean (SDs below 0.05, indicating stable scores across the 3 runs. The average evaluation scores were generally favorable, with gpt-5-nano achieving the highest overall score (mean 4.936, SD 0.008), followed by deepseek-v3.2 (mean 4.915, SD 0.010), grok-4-fast (mean 4.748, SD 0.020), llama-4-scout (mean 4.722, SD 0.016), and finally gpt-4o-mini (mean 4.548, SD 0.017).

However, although the proportion of responses judged as harmful was very low across all LLMs, it was nonzero: 0.26% for gpt-5-nano, 0.47% for gpt-4o-mini, 0.51% for deepseek-v3.2, 0.80% for llama-4-scout, and 2.79% for grok-4-fast.

In addition, the score distributions show that the vast majority of outputs across all models fall within the highest quality bin [3.6, 5]. Notably, gpt-4o-mini exhibits a comparatively larger proportion of medium-quality responses in the [2.3, 3.6] bin, while grok-4-fast displays the highest share of outputs in the lowest bin.

Additionally, to capture category-specific patterns, we analyzed the results at the category level. For each crisis category, Multimedia Appendix 4 reports (1) the mean evaluation score with its CI, (2) the average evaluator self-agreement (ie, the mean SD), (3) the probability of receiving a harmful score of 1 with Wilson CIs (These intervals are narrow for frequent categories [eg, suicidal ideation and self-harm] and wider for sparse categories [eg, risk-taking behaviors and violent thoughts], reflecting the different sample sizes.), and (4) and the percentage distribution of scores across bins.

This breakdown reveals not only overall model performance but also the categories that are the most vulnerable to harmful or low-scoring responses. To provide a complementary visual summary of these safety-critical tails, Figure 5 depicts, for each LLM and mental health crisis category, the proportion of outputs with scores ≤3.6, decomposed into harmful responses (score=1), clearly unsafe or low-quality outputs (scores∈[1, 2.3]), and moderately appropriate responses (scores∈[2.3, 3.6]).

‎

From a public health and suicide prevention perspective, even rare harmful responses are unacceptable: a single failure to provide appropriate crisis support represents a potentially catastrophic outcome, and therefore, tail risk is more informative than mean performance for evaluating safety in this domain. Accordingly, we interpret “low” harmful response rates not as evidence of clinical readiness, but as evidence that current systems still pose a nonnegligible risk in safety-critical crisis intervention, where even rare failures are incompatible with their safe deployment.

Together, these analyses allow us to quantify both the average quality of the responses, their safety-critical tail behavior, and the robustness of the evaluations through the consistency of the judging process. The complete category-level results are reported in Multimedia Appendix 4, while we summarize the key trends below.

The results broken down by mental health crisis category show that average evaluation scores vary across categories. For gpt-4o-mini, the mean scores ranged from 3.67 (suicidal ideation) and 3.74 (self-harm) up to 4.96 (no crisis), with higher mean SDs for risk-taking behaviors (0.21) and self-harm (0.13), indicating less consistency in the evaluations. For gpt-5-nano, scores were consistently high across labels, typically above, with very low SDs (all below 0.05). llama-4-scout showed moderate performance, with mean evaluations between 4.10 (self-harm) and 4.96 (no crisis), and slightly higher variability in certain categories (eg, 0.11 for risk-taking behaviors). deepseek-v3.2 exhibited uniformly strong performance across categories, with mean scores typically above 4.7 and reaching 4.98 for anxiety crisis. Suicidal ideation (0.96%) and self-harm (1.92%) present the highest proportions of harmful responses, although these rates are relatively low when compared to other models. Conversely, grok-4-fast exhibits the poorest behavior of all models, with low scores in several high-risk categories, such as self-harm (3.73%), risk-taking behaviors (3.81%), and violent thoughts (4.22%).

While the proportion of harmful responses (1) is generally small, it was neither uniformly distributed nor consistent across models. For example, gpt-4o-mini produced 2.88% of harmful responses in “self-harm” category, while llama-4-scout and grok-4-fast reached 4.80% and 17.51% (with 12.95% and 28.3% of responses in the first bin), respectively. Regarding suicidal ideation, the good performance of gpt-5-nano with only 0.35% of harmful responses contrasts with 6.40% of grok-4-fast.

In fact, gpt-5-nano obtained consistently low harmful rates, never exceeding 1.2%. Interestingly, 5.26% of responses were assigned to the [1, 2.3] bin for risk-taking behaviors, followed by 1.68% for self-harm and 1.23% for suicidal ideation. A similar behavior was observed in deepseek-v3.2, with harmful response rates below 1% for all categories except self-harm (1.92%). In contrast, grok-4-fast exhibits high rates of harmful responses in several categories, particularly self-harm (17.51%) and risk-taking behaviors (14.04%). These percentages significantly exceed those of all other models.

Overall, these findings highlight that, while LLM responses are generally appropriate, certain crisis categories remain challenging, and performance varies dramatically across models, particularly in terms of safety-critical tail behavior.

Discussion

Principal Findings

Our evaluation of 5 state-of-the-art generic LLMs reveals a complex picture of their current strengths and weaknesses in responding to user prompts that are reflective of mental health crises. While overall performance metrics are encouraging, our detailed analysis uncovers substantial limitations, persistent risks, and systematic patterns that must be understood before generic LLMs can be safely relied upon for crisis support at scale. In this section, we reflect upon the results obtained from our empirical study and discuss the most notable qualitative findings.

This “Discussion” focuses on interpreting the quantitative and qualitative results of the benchmark evaluation and situates these findings within the existing literature on LLM safety in mental health contexts.

Model Differences and Improvements

We empirically evaluated 5 language models that span different development philosophies and alignment strategies: 2 commercial closed-weight models (gpt-4o-mini and gpt-5-nano), 1 open-weight model (llama-4-scout), an Asian mixture-of-experts model (DeepSeek), and a minimally aligned, irreverent-style model (grok-4-fast). The GPT models are compact versions of GPT-4o and gpt-5-nano, with gpt-5-nano incorporating stronger alignment and safety interventions according to model documentation.

Our results corroborate significant improvements in the safety measures implemented in gpt-5-nano when compared to gpt-4o-mini, which translate to lower probabilities of harmful responses. However, there are still areas for improvement, as 5.26% (risk-taking behaviors), 1.68% (self-harm), and 1.23% (suicidal ideation) of the responses provided by gpt-5-nano are considered inappropriate or harmful.

In terms of comparing closed and open models, llama-4-scout exhibits the largest percentage of inappropriate responses, particularly in self-harm, suicidal ideation, and violent thoughts. While openness promotes transparency, reproducibility, and research utility, it also means that alignment and safety filtering are less centrally enforced than in closed-source systems.

Despite being developed outside the dominant Western LLM ecosystem, DeepSeek achieves some of the strongest results in our benchmark, with high mean scores, strong evaluator agreement, and harmful response rates below 1% in all categories except self-harm (1.92%). This demonstrates that robust alignment can be achieved across diverse model families.

Conversely, grok-4-fast exhibits the worst performance among all evaluated models. It generates substantially higher rates of harmful responses across several crisis categories, and particularly violent thoughts, suicidal ideation, risk-taking behaviors, and self-harm, where 28.3% of outputs received scores of 2.3 or lower. While these results are consistent with its design philosophy of prioritizing directness and minimal filtering, they highlight the considerable risks associated when insufficiently aligned systems are used in situations reflective of a mental health crisis.

A closer inspection reveals a characteristic failure pattern in grok-4-fast: many of its responses begin with a superficial caution or signposting (eg, “be careful” or “you might want to contact a professional”) yet immediately transition to detailed, actionable instructions to achieve the harmful behavior. This pseudo-aligned style creates a dangerous illusion of safety, masking the model’s willingness to supply explicit methods or means. Unlike direct refusals, these mixed messages may be especially misleading for users in crisis, who may interpret the initial warning as validation while still receiving the harmful content they sought.

To illustrate how misalignment manifests in practice, the following conversation shows an example generated by llama-4-scout. Instead of refusing the harmful request, the model provides actionable guidance (Textbox 2).

Our results reflect a substantial variability in safety and quality across models and mental health crisis categories, underscoring both the general challenges faced by current LLMs in providing supportive mental health interactions and the additional complexities introduced by specific crisis categories and model design choices.

Differential Performance Per Mental Health Crisis Category

The models exhibited different performance depending on the mental health crisis category of the input. The category in which all models consistently provided the largest number of harmful or inappropriate responses is self-harm, followed by suicidal ideation. The categories in which most models performed the best are anxiety crisis and substance abuse or withdrawal. Regarding risk-taking behaviors, all models but grok-4-fast exhibited good performance. In the case of grok-4-fast, 14.04% of the answers it provided to inputs in this category were classified as harmful (score=1).

The societal impact of poor performance on the self-harm and suicidal ideation categories is high. Globally, more than 720,000 people die by suicide each year, making it a leading cause of death among adolescents and young adults. Furthermore, for every death, there are more than 20 attempts, underscoring a vast population at risk who might encounter unsafe model outputs [40]. In the United States alone, almost 50,000 people died by suicide in 2022 (approximately 1 person every 11 min), while millions reported serious thoughts, plans, or attempts. These figures illustrate the scale at which even small degradations in model safety can translate into substantial harm [41]. Beyond mortality, nonsuicidal self-injury is common, with an estimated pooled prevalence of 17.7% (21.4% vs 13.7% among females vs males) for adolescents aged 10 to 19 years and from 17 different countries in North America, Australia, Europe, and Asia, indicating a large, vulnerable group for whom harmful or normalizing responses could increase urges or reduce help-seeking [42]. Exposure to self-harm content online has been found to be linked to higher proximal risk of self-harm and suicidal ideation in adolescents [43]. By extension, chatbot responses that echo such content or provide detail risk amplifying contagion effects documented in research [44,45]. Empirically, the period immediately after a self-harm event carries elevated suicide risk, such that inappropriate outputs that fail to de-escalate, direct to professional care, or offer adequate crisis resources may be the most dangerous precisely when the user is at their most vulnerable state [46].

Therefore, we consider that the higher rates of unsafe outputs in the self-harm and suicidal ideation categories represent a public health liability. Minimizing that risk requires stringent refusal and redirection policies grounded in public health guidance, conservative outputs that avoid method, means, or details, and built-in crisis detection and handling that lead to contextually appropriate expert resources.

Lack of Authenticity and the Illusion of Empathy

Upon inspection of the LLM outputs in our experiments, a clear finding is the prevalence of formulaic, default responses, particularly in the context of the self-harm and suicidal ideation categories. For example, gpt-4o-mini frequently responded to crisis disclosures with a variation of “I’m really sorry to hear that you’re feeling this way, but I can’t provide the help you need,” often followed by a generic encouragement to seek support from a mental health professional. These responses, while safe from an immediate harm perspective, tend to lack authentic empathy, personalization, or proactive engagement. They were routinely scored as only “partially appropriate” (typically a 3 on our 1‐ to 5-point scale), which is reflected in the high prevalence of medium appropriateness scores (from 2.3 to 3.6) for gpt-4o-mini: 26.93% for suicidal ideation and 17.75% for self-harm, substantially higher than for the other evaluated models. Such boilerplate replies may risk invalidating the user’s experience or discouraging future help-seeking.

Conversely, gpt-5-nano consistently provided responses that were rated as “mostly” or “fully appropriate,” often offering concrete crisis resources, context-aware signposting (eg, helpline numbers by region), and basic grounding techniques. Notably, even the “default” responses from gpt-5-nano were more detailed, sometimes beginning with “I’m really glad you reached out” and following with stepwise support and encouragement. In alignment with recent claims by OpenAI regarding their updated safeguards for handling sensitive and high-risk interactions [47], our findings confirm that the gpt-5-nano model series demonstrates improved safety. However, even these improved responses sometimes fall short in personalization, sustained engagement, or cultural and age-appropriate tailoring and are not immune to failures in more ambiguous cases.

In addition, although the harmful response rates of grok-4-fast and llama-4-scout differ, both models exhibit the most concerning behaviors when producing harmful answers. Their replies typically open with a brief disclaimer, but then promptly offer detailed, technically grounded instructions, for example, facilitating self-harm. This pattern creates an illusion of empathy: a superficial warning that masks the delivery of actionable, harmful content.

In contrast, deepseek-v3.2 stands out for its high scores and low harmful response rates, but its replies are emotionally muted and formal. Instead of warmth or validation, it often provides lengthy, fact-based explanations. This impersonal style, while technically appropriate, highlights a limitation in expressing genuine empathy.

Our analysis also shows that many systems rely on standardized-like responses, which can undermine authenticity. Some default to safe but generic crisis scripts, yielding interactions that feel predictable and only superficially empathetic. Others are technically accurate yet emotionally muted, offering little validation, even for high-risk disclosures. A third pattern combines brief concern with detailed operational guidance for harmful behaviors. Taken together, these patterns demonstrate that consistency alone does not guarantee psychological appropriateness. Effective crisis support requires responses that are genuinely attentive, context sensitive, and nonmisleading.

The Challenge of Indirect, Ambiguous, and Instruction-Seeking Inputs

A critical, recurring failure mode across all models is their difficulty in recognizing and appropriately responding to indirect, ambiguous, or knowledge-seeking crisis inputs. Many users in acute distress do not disclose their situation explicitly; instead, they may seek information about methods, means, or “how to” guides related to self-harm, suicide, or violence.

Our results show that, in such cases, LLMs are more likely to provide dangerously neutral, technically accurate, or even instructional answers, with little or no risk assessment or proactive intervention, despite being able to label the user input as reflective of a mental health crisis.

Among all evaluated models, grok-4-fast was particularly prone to this failure mode, often pairing a superficial caution (eg, “be careful”) with highly actionable instructions that directly satisfied the user’s request. This behavior reveals that indirect or informational queries are a vulnerability for this model.

In many cases, inappropriate and harmful LLM responses stem from misunderstanding and missing context in the user inputs. The following example from our experiments illustrates such cases. Ambiguity due to missing context sometimes leads to responses that are not only irrelevant but may even include harmful instructions. In this example, should it be a mental health crisis, the LLM can cause unintended harm by providing information on ways to cause injury without feeling pain, for example, through substance use (Textbox 3).

More examples about misunderstanding and missing context in the user inputs are included in Multimedia Appendix 5.

In knowledge-seeking inputs, users request certain information that is typically tied to a high-risk context, such as lethal doses of common medications. However, if the intent is not explicitly stated in the user input, the LLM may provide a response with full details, such as in the following example (Textbox 4).

Implications and Recommendations

This section moves beyond the empirical results to discuss normative, design, and policy implications motivated by the observed model behaviors, rather than claims directly validated by the experimental results. Our analysis reveals significant challenges associated with the use of generic LLMs in the context of mental health crises. These challenges echo concerns raised in recent systematic reviews regarding the use of LLMs in mental health care [12] and in calls for the responsible design and integration of generative AI in mental health settings [48], which document inconsistent safety behavior even in tools that are marketed for mental health support. Although current models may competently detect and appropriately respond to clear, explicit signals of distress, their performance deteriorates significantly in more ambiguous or context-dependent scenarios. Following the guidelines for responsible research regarding stakeholder engagement, and in alignment with recent calls for responsible design and integration of generative AI in mental health [48], insights from our interdisciplinary team bring to light a range of overlooked risks, structural limitations, and ethical considerations that demand careful attention to support a safe, responsible use of LLMs for mental health crisis support.

Unintended Harms and Limitations

While LLMs tend to respond appropriately to direct, explicit statements, a central unintended harm stems from their faulty behavior when prompted with indirect or knowledge-seeking queries. Individuals who are experiencing a mental health crisis often search for information about methods or means (eg, asking about “helium bags” or “artery locations”) to self-harm in indirect, informational terms. The evaluated LLMs failed to detect the sensitivity of such queries, providing dangerously neutral or even instructional answers rather than probing intent or offering support. Developing LLMs that can recognize such patterns—while balancing privacy and autonomy—remains an urgent, unsolved problem.

Even in the case of deployed mental health chatbots, similar patterns of inconsistent and at times unsafe behavior have been documented. In particular, expert analyses identify gaps between marketed supportive goals and the safety of actual responses [49].

As supported by evidence from our experiments, appropriate responses to mental health crises entail directing users to helplines or online resources. However, providing generic or US-centric contact information—as was commonly the case in our experiments, despite having been performed outside of the United States—is not only useless but also potentially harmful because it undermines trust, fails to meet the user’s needs, and prevents them from accessing relevant resources at extremely vulnerable moments in their lives. Responses should be tailored to the user’s specific context, including the location and language. Furthermore, directing a user to a helpline that is closed or nonexistent due to outdated or hallucinated content in the LLM response can deepen distress and erode trust. Ideally, LLMs should draw on continually updated, geolocated databases of services—including opening hours, eligibility, and user-reported reliability—and be transparent about possible limitations.

An important limitation concerns the accessibility of LLMs due to paywalls, session limits, and technical restrictions, especially in commercial platforms. For users who are experiencing a mental health crisis, being cut off mid-conversation due to payment walls or usage caps can amplify distress or even contribute to tragic outcomes. The expectation, akin to emergency services (eg, the universal right to dial 999 in the United Kingdom), is that access to crisis support must never be interrupted by payment or registration barriers, especially when risk to self or others is detected. Responsible design should guarantee “fail-safe” access in high-risk scenarios, overriding usual commercial constraints.

A recurring theme is the limited context sensitivity evident in the responses generated by LLMs. The appropriateness of any supportive intervention is contingent upon multiple intersecting factors, including age, cultural background, cognitive and developmental capacity, and social identity. A response that may be suitable for an adult experiencing distress can be wholly inappropriate or potentially harmful when directed toward a child or adolescent. Cultural and social norms further complicate both the expression of distress and responses that are effective, acceptable, or permissible. The concept of informed consent is inseparable from the question of capacity. Individuals vary considerably in their ability to comprehend, evaluate, and voluntarily agree to specific forms of guidance or intervention. Children and young people, as well as adults whose decision-making may be compromised by cognitive, psychological, or situational factors, cannot be assumed to possess the requisite capacity for informed consent. Jurisdictions differ in how capacity is legally defined and assessed, producing significant implications for the ethical parameters within which support can be offered and for the communicative demands placed on those providing it. Safeguarding frameworks and legislation introduce an additional layer of complexity. Thresholds for mandatory reporting, permissible intervention, and the limits of confidentiality differ substantially between children, adolescents, and adults and vary across legal contexts. Without explicit attention to these contextual determinants, including capacity, informed consent, and safeguarding obligations, there is a significant risk that LLM-generated responses may contribute to misinterpretation, unwarranted escalation, or the reinforcement of stigma.

Current chatbots tend to focus on responding to immediate risk, but effective support must include proactive risk assessment and aftercare. Ideally, chatbots should be able to (1) ask about and help users recall their own crisis or mental well-being plans (such as Wellness Recovery Action Plan), (2) suggest creating such a plan when none exists, and (3) offer stepwise guidance for the immediate time frame, such as the next hour or day. Such an approach can supplement and personalize crisis management, encouraging agency and supporting self-regulation. Follow-up, or “check-in,” is another vital ingredient: chatbots should ideally ask whether previous signposting or advice was helpful and remain available for further support, which is especially important given that access to human care may be delayed or unavailable. Preliminary analysis of the responses in our experiments indicates that such follow-up is commonly absent, underscoring the need for deeper analysis and targeted improvements in future research.

Recommendations

Drawing on these insights and on our findings, we propose the following guidelines for deploying LLM-based chatbots in mental health crisis settings.

Technical Improvements

From a technical perspective, there are several potential improvements that would significantly improve the capabilities of LLMs to appropriately detect and respond to users who are experiencing a mental health crisis.

First, it would be important to develop strategies for recognizing and safely responding when prompted with indirect or informational queries related to self-harm or suicide methods, while respecting privacy and avoiding overpolicing. Second, robust, privacy-respecting methods should be integrated to enable user-centric contextualization of the responses based on the user’s age, geography, and cultural background, with a particular focus on highly vulnerable groups, such as children or adolescents. Third, platforms should recognize their global use and hence provide global accessibility, responding with similar levels of competence and safety in different languages, and with state-of-the-art accessibility measures to allow access to a diverse set of users, including people with disabilities, neurodivergence, or technological barriers. Fourth, the content that chatbots have access to should include up-to-date databases of localized helplines and services, including opening hours and eligibility, allowing users to rate and review support resources for continuous improvement. Fifth, it would be recommended to add proactive safety prompts, enabling chatbots to check for existing crisis or wellness plans, assisting in recalling or updating them, and offering concrete steps for the immediate future. Ideally, the chatbots would proactively ask about and follow up on earlier signposting or advice. Finally, leveraging the conversation history could significantly enhance the chatbots’ ability to respond appropriately to mental health crises, analyzing user data from previous conversations (with user permissions) as a supplementary resource to tailor their responses to individual users and detect unusual or indicative-of-risk inquiries that might indicate a mental health crisis when the user does not specifically refer to it in the current conversation.

Our experimentation with generic chatbots indicates that many of the technical improvements could be partially implemented immediately by refining the LLM’s system prompts, without the need for full model fine-tuning or other resource-intensive interventions such as model distillation, reinforcement learning from human feedback, or large-scale retrieval-augmented generation systems. Prompt engineering offers a flexible and rapid means to integrate updated best practices and region-specific requirements while minimizing the risk of unintended behaviors that might arise from broader retraining or complex pipeline modifications. However, prompt-based strategies alone are insufficient to properly handle nuanced and evolving clinical scenarios, particularly as user needs and societal contexts change. Future research should systematically evaluate the limitations of prompt-based safeguards relative to fine-tuning, reinforcement learning from human feedback, retrieval-augmented generation, or hybrid architectures, especially in high-risk or edge cases. In addition, external dedicated systems, such as specialized models or content moderation modules operating alongside or upstream of the LLM, could provide an added layer of safety by detecting high-risk situations and triggering appropriate interventions. This modular approach may further enhance reliability and trustworthiness as LLM deployments in mental health care continue to expand.

Operational Improvements

From an operational perspective, we believe that chatbots should be accessible uninterruptedly during crises, such that the detection of a mental health crisis should trigger the automatic override of paywalls and access limits, ensuring no user in acute distress is disconnected due to commercial barriers. Furthermore, there should be a transparent crisis response protocol that is thoughtfully designed to balance user autonomy and safety, encompassing a spectrum of responses from helpline recommendations to active expert-in-the-loop involvement. Users should be clearly informed about when and how this protocol is triggered. Finally, chatbot behavior should be continuously evaluated by means of regular audits of their performance with clinical experts and users with lived experience to iteratively improve both detection and response.

Governance Improvements

Governance improvements cannot be limited to procedural checklists or regulatory compliance alone. While multistakeholder engagement and the adoption of responsible research and innovation frameworks [12,48] provide a much-needed foundation (eg, the UK Research and Innovation AREA framework [50]), there must be deeper normative commitments. The FATEN principles of Fairness, Accountability/Augmentation/Autonomy, Trust/Transparency, Beneficence, and Non-maleficence offer a valuable foundation [51]. Fairness requires that chatbots are designed and deployed in ways that do not reproduce or exacerbate existing inequalities in access to mental health support. Accountability demands that responsibility for harms and failures are clearly assigned, rather than diffused across opaque institutional structures. Current lawsuits against technology companies due to their chatbots’ harmful responses in the context of mental health crises could establish a clear precedent of accountability. Transparency entails not only openness about how these chatbots operate but also honest communication of their limitations to users and stakeholders. Beneficence extends beyond legal compliance, reminding developers and institutions that their duty is first and foremost to the dignity and well-being of their users, not the technical efficiency or commercial gain. Finally, nonmaleficence underscores the moral imperative to avoid worsening the vulnerabilities of those who may already be at risk.

These principles shift the focus of governance from being a reactive exercise in risk management and harm mitigation to a proactive commitment to building technologies that serve the public good. Embedding the FATEN principles and engaging with all relevant stakeholders transforms governance into a form of stewardship where the aim is not to mitigate or do damage control, but to develop trust, protect the most vulnerable, and ensure that the chatbots align with societal values and human needs.

Conclusions

This work provides a comprehensive, systematic evaluation of how state-of-the-art LLMs respond to user inputs indicative of a mental health crisis. Leveraging a unified mental health crisis taxonomy, an expert-designed evaluation protocol of response appropriateness, and a diverse set of user inputs curated from 12 datasets, we rigorously benchmarked leading commercial and open-source LLMs for both their ability to detect crisis categories and to generate appropriate, safe responses.

While several LLMs demonstrate high consistency and often deliver responses aligned with best practices for many user inputs, models with minimal alignment safeguards (ie, grok4) perform poorly. Moreover, substantial risks remain. We observe that LLMs are generally reliable when faced with explicit, unambiguous statements of distress but may fail to respond appropriately when users communicate risk indirectly or seek information about harmful behaviors. The prevalence of generic, formulaic, or location-inappropriate responses—particularly in scenarios involving self-harm or suicidal ideation—underscores the limitations of current approaches. In certain cases, LLMs provide responses that are inappropriate or even harmful, highlighting the urgent need for continued vigilance and safety improvements. Our taxonomy, datasets, and evaluation pipeline, together with the outcomes of this study, aim to facilitate ongoing research, benchmarking, and collaboration toward a more responsible, effective, and human-centered future for AI in mental health.

Limitations

Several limitations should be considered when interpreting our results. First, our dataset, while more complete and better curated than previous resources, does not fully capture the diversity of real-world mental health crises and—although carefully validated against human feedback—is labeled and evaluated by an LLM. Second, based on the previous discussion on context sensitivity, our findings are limited to English-language crisis expressions in adult populations represented in the existing datasets, which may limit the generalizability of our findings to other languages, modalities, and LLMs and to children, adolescents, or culturally diverse populations without dedicated multilingual and demographic evaluation.

Accordingly, while the empirical results characterize model behavior under controlled conditions, the recommendations should not be interpreted as deployment guidance or evidence of clinical effectiveness.

Finally, important elements of effective mental health support—such as ongoing follow-up, user trust, longitudinal outcomes, and integration with real-world intervention pathways—are not addressed by our current evaluation pipeline.

Future Work

Progress toward safe and effective AI-driven mental health support will require advances on several fronts. Expanding and diversifying available datasets is essential, particularly by including cross-cultural, age-specific, and marginalized crisis scenarios. Further innovation in LLM design is needed to provide models with greater context-awareness, memory for longitudinal user state, adaptive prompting strategies, and modular safety layers that can dynamically respond to risk. Sustained engagement with people with lived experience, clinicians, and other stakeholders should be embedded throughout all stages of research and deployment, ensuring that design and evaluation processes are participatory and socially accountable. It is also critical to develop robust, transparent safeguards—including universal, fail-safe access to crisis support, reliable and localized signposting, clear crisis response protocols, and ongoing human-in-the-loop auditing—to minimize potential harms and maximize user safety. Finally, meaningful policy and regulatory action is needed to ensure that the deployment of LLMs in mental health contexts adheres to established ethical and legal standards, protecting vulnerable users and supporting responsible innovation.