Suicide exacts a staggering societal cost. Recent World Health Organization (WHO) estimates indicate that more than 700,000 individuals die of suicide each year and its worldwide mortality exceeds that of malaria, HIV/AIDS, war, and homicide [1]. Suicide is the 10th leading cause of death in the United States, where its economic costs are estimated to be over US $70 billion per year in lifetime medical expenses and loss of productivity alone, with similar numbers across the globe [2-7]. Moreover, suicide is a leading cause of death among young people in the Netherlands. Increasing suicide rates among young people have been observed [8].

Beyond deaths from suicide, many more individuals experience suicidal thoughts and intent. US Centers for Disease Control and Prevention (CDC) statistics indicate that, in 2018, a total of 10.7 million, 3.3 million, and 1.4 million American adults either seriously thought about suicide, made a plan, and/or attempted suicide, respectively [9]. The burden on minors and young adults is equally high. In 2019, a total of 18.8% of high school students reported seriously considering suicide and 8.9% reported an attempt [10].

Given these indicators of a global mental health crisis [11], the death toll of suicide, and the large population contemplating or attempting suicide, scalable and effective mitigation is of the utmost importance [12-14]. Here, we focus on suicide prevention helplines, which aim to provide immediate assistance and access to important resources [15] to those experiencing an imminent risk of suicide. Many telephone helplines have recently been augmented with online chat services to extend access, in particular for the young [16] who may prefer chat services over telephone conversations. Given their use of ubiquitous online chat applications, chat helplines can offer time-sensitive, real-time, cost-effective, and anonymous assistance to individuals reaching out at a time of urgent need [17]. They are for these reasons preferred over other channels, such as email, text messaging, voice calls, and face-to-face counseling [18]. A large-scale study of a national suicide prevention helpline showed that the chat functionality had been helpful for almost 70% of the users and that 45% felt less suicidal after the interaction(s) [19]. In addition, short-term improvements were recently observed in suicidal ideation and psychological risk factors after a chat conversation in another national helpline [20].

Thus far, research has focused on studying suicidal ideation from language, for example, through linguistic features from social media content [21-24], poetry [25], forums [23,24,26-28], or suicide notes [29-31] using tools, such as Linguistic Inquiry and Word Count [32]. Examples of such linguistic features that are extracted from these texts are self-to-relational word counts [33], linguistic accommodation [26], hopelessness [28], social disengagement [28], or categories of stress [31].

By definition, chat-based suicide helplines rely on text-driven communication, which, although it takes place in real time, does not include verbal cues, body language, intonation, and other important aspects of in-person communication. They require that respondents interpret the written language of help seekers, detect cues and indications of the help seeker’s state, and respond accordingly. However, it is not clear what these written language cues or features may be in the context of chat-driven suicide helplines; this remains an important knowledge gap. Recent efforts that have started to address this gap focus on providing suggestions for counselors based on previous chat sessions [34] or training counselors in motivational interviewing [35].

There exists an extensive literature on language features associated with depression and other internalizing disorders, which are known to be strongly comorbid with suicidal ideation and depression [36-38]. For example, Eichstaedt et al [39] showed that language features indicative of emotional distress, interpersonal dissatisfaction, and rumination can be leveraged by machine-learning algorithms to predict depression in Facebook users at a reasonable level of accuracy. Rude et al [40] showed that the language of individuals with depression is indeed marked by the prevalence of specific identifiable terms. However, the language of individuals with depression is not only marked by the presence of discrete features that may be useful in diagnostic classification tasks, but also by a style of thinking that is deeply implicated with the development of internalizing disorders such as depression.

According to the theory underlying cognitive behavioral theory, the cognitive triad and the cognitive mediation model [41-43], individuals with depression exhibit self-reinforcing thought patterns in which unrealistic and overly negative views about the world, the future, and oneself cause a cascade of detrimental cognitive, affective, and behavioral changes associated with internalizing disorders. These thought patterns are referred to as cognitive distortions and frequently described in terms of 12 general types, eg, “I will never amount to anything,” which in this case amounts to “fortune-telling,” “dichotomization,” and “personalizing.” Regardless of type, all cognitive distortions share the characteristic that they amount to unrealistic, exaggerated, and overly rigid thinking patterns.

Cognitive behavioral therapy (CBT), now the leading treatment of depression, anxiety, and suicide [44-49], considers cognitive distortions an important intervention target. In fact, the very effectiveness of CBT demonstrates the importance of the identification and mitigation of cognitive distortions in an individual seeking help with depression or anxiety. Previous research [50,51] demonstrated that the online language of individuals with depression is indeed marked by a higher prevalence of cognitive distortions than a random control sample. Since depression is strongly comorbid with suicide [36-38], here we investigate whether the written language of individuals seeking help with suicidal thoughts in a chat-based helpline is similarly impacted by the presence of cognitive distortions. If so, as indicated by the effectiveness of CBT [48,49], addressing cognitive distortions may be an important driver and intervention target for suicidality in this context.

We analyzed 6,349,202 chat messages recorded from July 2017 to June 2021, which were split between those written by either help seekers or the professional respondents of the 113 suicide helpline in the Netherlands. We found that markers of some types of cognitive distortions are 20-fold more prevalent in the language of help seekers than the 113 counselors. In fact, a comparison with the online language of individuals with depression and a random sample recorded on Twitter revealed a similar difference and indicates that Twitter language may be particularly enriched with cognitive distortions even in the absence of any diagnosed mental health issues.

This study was approved by the Indiana University Bloomington Institutional Review Board under protocol 2007591998 “Retrospective language analysis of suicide prevention online chat content (Dutch 113 service),” which was reviewed specifically for this study and its research team. Help seekers did not receive any compensation for the inclusion of their sessions in our analysis. By participating in the chat sessions at 113, help seekers approve that these sessions can be used for research purposes, as stated in the terms and conditions.

As our analysis pertains to individual health-related information, we took additional steps to ensure the privacy of all individuals involved in the chat sessions. We deidentified all data by assigning each chat session a unique but anonymous label, for example, cl123456, to remove all identifying information. All times are reported in terms of time elapsed from the start of the conversation. All raw data are stored in a protected environment at 113 suicide prevention. Finally, all analyses were performed and reported in the aggregate, that is, without reference to the individuals that constitute the specific cohorts of the study.

We performed our analysis on chat sessions recorded by the secure chat platform of the 113 suicide prevention helpline in the Netherlands. On this platform, help seekers can reach out to 113 employees to discuss any topics related to suicide. For our analysis, we selected all chat sessions that passed the initial triage stage determining whether the help seeker was in a safe location before being passed to a professional counselor (a psychologist or a volunteer). Chat sessions have 3 types of participants: help seekers, 113 counselors, and the chat service system itself. The chat service system posts automated messages to indicate that it has taken a specific action; we exclude these messages from our analysis. Applying this criterion resulted in a large-scale dataset consisting of 71,148 chat sessions that were recorded between July 1, 2017, and June 1, 2021. In total, these chat sessions contain 3,316,873 messages posted by help seekers and 3,032,329 messages posted by 113 counselors. The average monthly number of forwarded chat sessions that we have included in our analysis is 1513 (95% CI 1199-1874), showing high levels of engagement with the 113 helpline.

This study investigated the differences between the written language of a cohort of 113 help seekers who are seeking urgent help with suicidal ideation or intent and the written language of a control cohort with little or no suicidality in the context of an online chat environment. Given the nature of the 113 helpline, we did not expect to find many help seekers with low levels of suicidality that could serve as a control cohort. In fact, an analysis of the responses to the questionnaire [35,52] that help seekers are required to fill in showed insufficient discrimination between low and high levels of suicidality to construct a control cohort (see section “Types of Help seekers as control” in Multimedia Appendix 1). Here, we chose to use the 113 counselors as a control group since they are expected to be least affected by suicidality and depression, while conversing with help seekers on the same platform about the same topic. For the remainder of the analysis, h denotes the help seeker cohort and O denotes the Counselor cohort.

In addition, we compared the written content of the 113 chat sessions with 2 very large samples of Dutch texts, the latter providing control data that are independent from a suicide prevention helpline. First, we included the CC100-Dutch dataset, a large collection of Dutch sentences (N=229,914,131), which were extracted from websites that were included in the January to December 2018 CommonCrawl snapshots from the CC-Net repository [53]. Second, we included a comparison to the language of a large sample of users on Twitter (now X), a popular microblogging platform. Using an open-access repository of COVID-19 messages [54], cohorts of active Twitter users in Dutch municipalities with more than 100,000 inhabitants in 2020 [55] were constructed based on the user-provided location information on their own profile page. Concretely, users were included who indicated that they live in one of the following cities: Amsterdam, Rotterdam, The Hague, Utrecht, Eindhoven, Groningen, Tilburg, Almere, Breda, Nijmegen, Apeldoorn, Haarlem, Arnhem, Hoofddorp, Amersfoort, Enschede, Zaanstad, Den Bosch, Zwolle, Leiden, Leeuwarden, Zoetermeer, Maastricht, Ede, Dordrecht, Alphen aan den Rijn, Alkmaar, Delft, Emmen, Venlo, and Deventer. To ensure that these Twitter users were indeed individuals, we used the M3 system [56] (a deep learning classifier trained on large collections of Twitter profiles) to classify accounts according to 3 categories: gender, age, and organization. Using the outcomes of the organization classifier, we only kept accounts that had a likelihood score of more than 0.8 of being an individual in our analysis. In total, this dataset consists of 4829 Twitter users for which the individual timelines (3200 most recent tweets, time-sorted by date posted by individual) were collected using Twitter’s application programming interface for an analytical sample of 4,600,054 tweets that were posted between July 1, 2017, and October 31, 2020. Using 0.8 as a threshold for gender and 0.6 as a threshold for age (as this category contains 4 labels rather than 2), we can assess the demographics of the Twitter users we have included in our analysis. In total, our sample contains 3119 male individuals (64 of whom were aged 18 years and younger, 117 aged 19-29 years, 455 aged 30-39 years, and 1959 aged 40 years and older) and 1522 female individuals (39 of whom were aged 18 years and younger, 145 aged 19-29 years, 202 aged 30-39 years, and 839 aged 40 years and older). Note that the totals do not necessarily have to sum up to the total amount of included Twitter users, as the outcomes of the M3 system are below the used threshold (0.8 for gender and 0.6 for age).

Cognitive distortions are maladaptive patterns of thought that individuals with depression exhibit in which unrealistic and overly negative views about the world, the future, and oneself cause a cascade of detrimental cognitive, affective, and behavioral changes associated with internalizing disorders [41-43]. These cognitive distortions are frequently described in terms of 12 general types, for example, “I will never amount to anything,” which in this case amounts to “fortune-telling,” “dichotomization,” and “personalizing.” Regardless of type, all cognitive distortions share the characteristic that they amount to unrealistic, exaggerated, and overly rigid thinking patterns.

CBT, now the leading treatment of depression, anxiety, and suicide [44-49], considers cognitive distortions an important intervention target. In fact, the very effectiveness of CBT demonstrates the importance of the identification and mitigation of cognitive distortions in an individual seeking help with depression or anxiety.

The detection of cognitive distortions in written language, in open-form online chat sessions, remains an open question. Some recent advances suggest natural language processing and machine learning can be leveraged to detect the presence of cognitive distortions in text [57,58] at reasonable levels of accuracy, even matching the performance of human raters, but it is not clear how well these methods generalize to online chat data, nor whether the lexical features involved in the classification correspond to our understanding of cognitive distortions [59]. In fact, some research indicates that sophisticated machine-learning models, such as deep learning, may in fact be outperformed by more transparent models that capture a small number of specific expressions associated with cognitive distortions [60].

Here, we used a theory-driven method proposed by Bathina et al [50], namely a lexicon of 241 one- to five-grams that was specifically designed and vetted by a panel of experts in CBT, originally introduced by Beck [41-43], to capture the sequences of words involved with the expression of 12 common types of cognitive distortions (see Table 1). For example, the 1-gram “never,” among other 1-grams, is deemed to be indicative of “dichotomization,” and the 3-grams “they will think” or “I am a” are deemed to be indicative of “mindreading” and “labeling,” respectively. A message that contains either one of the n-grams in the cognitive distortion schemata (CDS) lexicon is deemed to contain an indication of the expression of the specific cognitive distortion type, regardless of its topic or context. For example, the message “I will never be loved” contains the 3-gram “I will never,” which is associated with the expression of a cognitive distortion of the “fortune telling” type. Counting the number of matches of the entire CDS lexicon against the content of the chat-session messages (by help seekers or counselors) provides an estimate of the prevalence of cognitive distortion expressions in their language.

The CDS lexicon is grouped in 12 distortion types following a common taxonomy, but its use does not rely on claims with respect to the component structure of cognitive distortions in general. Each n-gram is matched independently. The number of literal matches of the n-grams of the CDS lexicon in the messages of either help seekers or 113 counselors is normalized by message volume to determine CDS prevalence (see section “Measuring CDS in chat messages”).

The original CDS n-grams were translated from English to Dutch by 4 bilingual native language speakers of Dutch, namely 2 Dutch-speaking coauthors, and in-house 113 helpline psychologists Wilco Janssen and Jeroen Gomes, to match the language in which the 113 helpline conversations were conducted. The resulting set of 4 translations (for each CDS) was combined to a single list of translated CDS n-grams through a process of iterative design. Finally, 2 Dutch experts in CBT (Claudi Bockting and Marcus Huibers) were asked to validate whether the translations still captured the distorted thinking.

The translations focused on retaining the distorted thinking aspect of the schemata, respecting grammar (eg, gender) and idiomatic expressions. For that reason, the number of Dutch schemata (N=265) does not equal the number of English schemata (N=241), since one-on-one translation was not always feasible. Moreover, due to differences in Dutch word ordering (eg, verb at the end of the sentence), many of the schemata had to be formulated as regular expressions instead of n-grams. The number of schemata per category in the Dutch CDS set along with the number of observed schemata per category can be found in Table 1. The complete set of CDS is freely available in Table 2.

Let S denote the set of all chat sessions. Each chat session s∈S in our sample set consists of a collection of k time-ordered messages, s={m₁, m₂, ..., m_k}. Each message m_i in the session is marked by an indicator of the role of who posted that message denoted by rmi, that is, counselor, help seeker, or system, denoted by o, h, or b, respectively, and the time at which the message was posted (in s from the start of the session), denoted by tmi.

Additionally, as mentioned, we have defined a set of 265 Dutch CDS n-grams, denoted as C. Each CDS, denoted as c, is intended to represent the lexical building blocks of expressing cognitive distortions. The notation for the collection of CDS within a distortion type is indicated in the column “Notation” of Table 1 (eg, ℂc denotes all CDS belonging to the category Catastrophizing). We define a CDS matching function g_C(m_i)→{0,1}, which assigns each message m_i to 0 or 1 based on whether that message contains at least 1 CDS that is contained in the set C⊆C.

As we are interested in the prevalence of cognitive distortions in messages, we measured the fraction of messages posted that contain a CDS. Thus, we calculated the CDS prevalence for a given set of CDS C, a role r (either help seeker or counselor), and a set of chat session S as the ratio of messages that contain a CDS marker over the total number of messages.

For the NL cohort, S can be seen as the set of timelines of the individuals in the cohort, in which s∈S represents the timeline of an individual that consists of a collection of tweets rather than messages. Finally, for the NL web dataset, S consists of a single element, that is, a collection of sentences rather than messages.

We calculated the prevalence ratio between roles as follows:

We employed a theory-driven approach to investigate the prevalence of expressions of cognitive distortions, that is, maladaptive thought patterns that characterize internalizing disorders, such as depression and anxiety [41-43], in a large collection of chat sessions of a suicide prevention helpline in the NL between help seekers and counselors and 2 baselines extracted from large-scale corpora of Dutch social media and web content.

Our analysis investigated the CDS prevalence in the recorded chat sessions of the Dutch suicide prevention helpline. Our aim was to quantify the levels of cognitive distortions in the language of online help seekers versus 2 controls that include the 113 respondents. Given the comorbidity between suicide and depression [36-38], and the fact that the latter is associated with cognitive distortions [41-43], we expected high levels of cognitive distortions in the language of those who contact suicide helplines.

The monthly CDS prevalence was quite stable during the entire observation period, even during the 2 years of a devastating worldwide pandemic, which according to some analyses led to topical changes in Dutch suicide helpline conversations (eg, an 808% increase of the topic of “COVID-19”) [61]. This shows that even though the topics that are discussed in the chats change over time, the degree of distorted expressions does not seem to change. This is surprising given the backdrop of a global pandemic. Future work may explore the stability of CDS prevalence across time in online environments further. There are some indications of a minor trend toward increased session volume starting in early 2021. However, this does not seem to affect the monthly CDS prevalence significantly, which is in line with other research studies that showed that the prevalence of high anxiety and depression symptom levels did not increase [62]. This is the case across all separate CDS types (see Figure S1 in Multimedia Appendix 1).

In general, certain types of CDS n-grams are more prevalent than others; the CDS prevalence for fortune-telling remained low at 0.04%, while the CDS prevalence for dichotomous reasoning was 2 orders of magnitude more prevalent at 12.57% across all chat sessions, as indicated in Figure 2. In general, categories that relate to black-and-white thinking (eg, dichotomous reasoning and overgeneralizing) were most frequent among both help seekers and counselors, whereas categories that relate to emotion and affect (eg, catastrophizing, emotional reasoning, and disqualifying the positive) were less frequent. This could reflect how some distortion types are generally more or less common in Dutch or how CDS n-grams of a specific type are worded in the lexicon. In fact, many n-grams contain some of the most frequent Dutch words (eg, “ik” (I) or “ben” (am)), since the CDS lexicon was designed to capture common, context-free patterns of language. It is expected that shorter n-grams occur more frequently in the language than longer n-terms.

We compared the CDS prevalence ratio between help seekers and counselors, since the latter is least likely to be affected by suicidality and depression. We observe that nearly all types of cognitive distortions are significantly more prevalent in help seeker messages than those of the 113 counselors. CDS of the personalizing, emotional reasoning, and mental filtering types are, respectively, 20.22, 7.87, and 4.53 times more prevalent in the help seeker cohort than the counselor cohorts. In fact, 5 categories are used at least twice as frequently in the help seeker cohort as in the counselor cohort, that is, personalizing, emotional reasoning, mental filtering, catastrophizing, and disqualifying the positive. The fact that the differences for other categories are less pronounced could be explained by counselors mimicking the phrasings of help seekers in their chat in their attempt to build connection with the help seeker, a phenomenon called lexical entrainment [63]. The CDS n-grams have in previous research been shown not to have significant valence loadings [50] nor is their prevalence in the language of individuals with depression driven by the presence of personal pronouns. Hence, the expected higher level of negativity or self-referential nature of the help seekers' messages is not likely to explain the higher prevalence of the set of CDS n-grams in the language of help seekers.

Our analysis of the within-session CDS prevalence shows clear differences in CDS prevalence between the help seeker and counselor cohorts, with the help seeker cohort having significantly a higher mean CDS prevalence than the counselor cohort. This may be expected because the counselor cohort has been professionally trained to communicate with help seekers, which possibly included the avoidance of cognitive distortions in their communications with help seekers. To verify the degree to which counselors exhibit exceptionally low levels of cognitive distortions in their language, we compared the prevalence of CDS in their language and that of the help seekers to a cohort of Dutch individuals (referred to as the NL cohort) on Twitter, since the latter as the microblogging platform may closely approximate the format and vernacular of online chat helplines.

As expected, the counselor cohort has significantly a lower mean CDS prevalence than the NL cohort, that is, their language contains fewer markers of cognitive distortions than a random sample of Dutch Twitter users. As mentioned, this lower usage of CDS in the counselor cohort is likely a result of the training they have received in handling these chat sessions at 113. Another fact that stands out in Figure 3 is the difference in variability between the help seeker and the counselor cohorts, which indicates that there are significant differences between individual chat sessions and that the 113 service is reaching a wide audience of individuals with varying degrees of urgency and the need of assistance. These results highlight the usability of the CDS annotations in the chat sessions to further analyze these conversations, for example, by providing the counselors with real-time feedback during chat sessions, or by highlighting important changes in language during the sessions that could give an indication of the perceived quality of the conversation.

Our data were collected by an anonymous helpline, which in the context of providing urgent, low-threshold care did not collect information on neurodevelopmental diagnoses. Certain neurodevelopmental conditions have been associated with differences in cognitive styles, which may relate to variations in the expression of cognitive distortions. Additionally, some neurodivergent populations have been shown to experience higher levels of suicidal ideation and suicide attempts [64,65]. However, the relationship between such differences and the prevalence of cognitive distortions remains insufficiently characterized. Future research should examine how these individual factors may interact with the prevalence of cognitive distortions in the context of suicide helplines.

Nevertheless, we have converging evidence across 4 different online samples (113 help seekers, 113 counselors, a large cohort of random Dutch Twitter users, and a large collection of Dutch online texts), indicating that the prevalence of cognitive distortion markers in the language of 113 help seekers is exceptionally high.

Several limitations of our approach should be considered.

Given the sensitive nature of both the data and the possible applications of our considered technique, there are several ethical reflections that we want to address explicitly. First, given the sensitive nature of the content of the chat sessions, all data were processed in the aggregate, without personal identifiers, ensuring the anonymity of all analyses and research outcomes. Next, we stress that our aim is not to diagnose individuals with cognitive distortions or suicidality nor is it our intention to underpin replacements of human counselors by automated chatbots [66,67]. Based on our current results, we believe that our research has the potential to provide additional guidance or support to the counselors of the suicide prevention helpline by highlighting important changes in language that are known to be implicated in the development and recovery trajectories of internalizing disorders. However, careful considerations must be made before such techniques are put into practice [67,68]. Future work will focus on more accurate methods to detect cognitive distortions in language, such as the use of more sophisticated natural language processing and artificial intelligence, including ground truth obtained from human raters. Our analysis did not investigate changes in language over the course of a chat session as the conversation between the help seeker and the counselor unfolds, which may provide indicators of the effectiveness of the exchange. Future research should examine whether the changes of CDS prevalence during a conversation, in subsequent utterances, may shape its perceived quality and effectiveness.

We presented converging evidence across 4 different online samples (113 help seekers, 113 counselors, a large cohort of random Dutch Twitter users, and a large collection of Dutch online texts), indicating that the prevalence of cognitive distortion markers in the language of 113 help seekers is exceptionally high. Our results point to the value of lexical analysis as an instrument to detect the cognitive and lexical markers of suicidality.