Assessing the Impact of Conversational Artificial Intelligence in the Treatment of Stress and Anxiety in Aging Adults: Randomized Controlled Trial

Introduction

Background

The multiplicity of issues related with active aging has been on the agenda of national institutions and health agencies for many years worldwide. The European Union framework directive on health and safety at work (89/391/ EEC) [1] indicates that practicable adjustments to physical and social working environments are necessary to prevent or reduce excessive physical and mental demands on aging workers. Many studies have identified high levels of stress in the workplace as a major factor for developing age-related health risks, including cardiovascular diseases, sickness absence, anxiety, depression, and burnout syndrome [2-5]. As a consequence, several interventions have been implemented and evaluated for the prevention of physical diseases and mental disorders, and the strengthening of older employees, as reported in a recent systematic review [6]. Although this review did not focus only on the older population of workers, it reported some interesting relevant findings for our research. Based on moderate evidence that emerged from the review, cognitive behavioral therapy (CBT) and stress management programs are expected to reduce perceived stress. Nevertheless, the persistence and sustainability of these interventions were insufficient or limited.

Another systematic review analyzed the results of studies providing evidence for digital psychological interventions in the workplace [7]. The authors reviewed digital interventions aimed to address the well-known problem of accessibility of mental health care for the working population in general, due to limited resources in the occupational health sector and to stigma. The adjective “digital” in the reviewed studies stands for interventions whose primary modality of delivery was a website, where participants could access different types of assignments and receive feedback after completing the assignments from a coach or therapist by email, text, or phone call. All the summarized studies were randomized controlled trials (RCTs), but only one study reported data about a mobile app, and no study mentioned artificial intelligence (AI)-empowered treatments.

The demand for accessible and large-scale mental health care support has been previously pointed out [8] and aggravated by the COVID-19 pandemic and its consequences [9,10]. A growing number of studies have indicated that the development of conversational AI systems (also known as chatbots) as applications in the mental health domain can improve access to mental health care support in an easy and inexpensive manner [8,11,12]. Even though traditional in-person therapy sessions remain the most frequent framework for support provision, conversational AI agents have been shown to be an effective alternative regarding various mental disorders, such as stress, anxiety, and depression [8]. In particular, during the COVID-19 pandemic, the problem of accessibility to mental health treatments increased users’ appreciation of remote therapy, thus providing video therapy an opportunity to develop its potential in a world where these kinds of communications represent the new normal [13].

TEO (Therapy Empowerment Opportunity) is a mobile personal health care agent (m-PHA) designed to provide CBT support for the prevention and treatment of stress and anxiety [12]. It has been designed and developed in collaboration with CBT therapists [12]. In the course of the intervention, TEO converses with users through text-based dialogues. From these conversations, TEO recognizes users’ emotional states, beliefs, and personal events, and implements strategies designed by professionals.

Objective

The observational study discussed in this paper was designed for evaluating the impact of introducing AI technology in the psychological treatment of aging workers presenting a variety of stress symptoms hypothetically related with moderate to high levels of perceived stress in the workplace. The experimental protocol was designed to answer the following questions: (1) whether the use of AI-empowered conversational technologies could contribute to people’s psychological well-being; (2) whether there are differences in terms of symptom reduction between receiving support from an AI-empowered conversational technology and traditional psychotherapy; (3) whether the observed changes are different when comparing a group of people receiving treatment or not receiving it; and (4) whether there are differences compared with a group of people receiving a standard course with a psychologist in a remote setting.

Methods

Design

The experimental design included comparison of the presence of several different symptoms, like anxiety and depression, and psychological attitudes, measured by standardized self-assessed psychological questionnaires. We applied these metrics before treatment (T1) and at the end of treatment (T2). An additional measurement was performed 3 months after the end of treatment to longitudinally assess the effects (T3). The self-assessment scales we applied were Symptom Checklist-90-Revised (SCL-90-R), Occupational Stress Indicator (OSI), and Perceived Stress Scale (PSS). SCL-90-R is a self-administered questionnaire that assesses a broad spectrum of psychopathological symptoms like depression, anxiety, psychoticism, and others. OSI is a questionnaire for the evaluation of psychosocial stress in organizations. PSS is a brief questionnaire for the detection of generalized psychological stress. In addition, 2 brief versions of Patient Health Questionnaire-8 (PHQ-8) and General Anxiety Disorders-7 (GAD-7) were administered at the beginning of treatment (T1), after 4 weeks (T2), at the end of treatment (T3), and after 3 months (T4). PHQ-8 is an 8-item questionnaire for assessing and monitoring depression severity [14], while GAD-7 is a short questionnaire for assessing and monitoring generalized anxiety disorders [15].

The treatment involved administering 8 weeks of cognitive behavioral psychotherapy, specifically oriented toward the acquisition of stress management skills. In addition, the experimental design included the possibility of supporting stress management training-CBT with the continuous assistance of an AI-based conversational agent for mental health care (TEO) [12]. The experimental design included 4 groups of subjects as follows: group 1 received traditional psychotherapy from CBT therapists in a remote setting; group 2 received both traditional therapy and the support of the conversational AI agent; group 3 received only the support of the conversational AI agent; and group 4 was the control group not receiving any treatment. Participants assigned to group 4 were also assigned to a waiting list and received treatment at the end of the 8 weeks of the experiment.

IDEGO (Digital Psychology srl, Rome, Italy) carried out the psychometric tests and their data analysis. The experimental design of the RCT, training, and evaluation of the AI algorithms and systems were performed by the University of Trento.

Ethics Approval

This methodology was approved by the Ethics Committee of the University of Trento within the context of the research activities of the HORIZON2020 CO-ADAPT project, and the experimental protocol has been registered on ClinicalTrials.gov (NCT04809090).

Recruitment

We collected the data of this study between Spring and Fall 2021, when the third wave of the COVID-19 pandemic was hitting Italy, starting from the Northern regions of the country. The traditional recruitment strategies were inadequate or limited owing to social distancing measures. To overcome these difficulties, we designed new strategies on social media with recruiting campaigns involving engaging posts and graphics. Comparing the usage statistics of the 2 social networks Facebook and Instagram, Facebook provided the highest percentage of users in our target group (21.3% and 11.7% for Facebook and Instagram, respectively) [16]. The campaigns were widespread throughout Italy, with the goal to motivate people to reach our website [17] and enroll in our research. The site included all the information about the research and a form where the users could request to participate. Moreover, the users could ask for more information, resulting in one-to-one interviews to answer all the questions. In order to select eligible participants, several questionnaires and a clinical interview with each subject were conducted. Exclusion criteria were the presence of severe depression (PHQ-8 score ≥20), suicidal thoughts, substance abuse, and mild cognitive impairment (Montreal Cognitive Assessment score <26) [18].

Participants

The characteristics of the samples are described in Table 1. A total of 65 potential participants were examined, and of these, 60 were recruited. A code was assigned to each participant, and through a random generator of numbers, the selected subjects were distributed into 4 groups. After the assignment, 2 subjects (1 in group 3 and 1 in group 4) showed mental health issues that made it necessary to reassign them to groups 1 and 2 to provide more accurate monitoring, where they could receive psychological support throughout the experiment. Other subjects showed a critical profile during the experiment, and they were directed to a standard psychological support service. Subsequently, these subjects were excluded from the analyses (Figure 1). Only 45 subjects were considered for the analysis. Group 1 included 27% (4/15) men and 73% (11/15) women, with a mean age of 54.08 (SD 4.11; median 54) years. Group 2 included 17% (2/12) men and 83% (10/12) women, with a mean age of 55.17 (SD 3.69; median 55) years. Group 3 included 25% (2/8) men and 75% (6/8) women, with a mean age of 55.63 (SD 4.50; median 55.5) years. Group 4 included 20% (2/10) men and 80% (8/10) women, with a mean age of 57.20 (SD 7.96; median 60) years.

‎View this figure

TEO

TEO is an m-PHA [19], a type of AI conversational agent, in the form of a mobile app that supports input/output interactions with users via natural language. Many PHAs currently developed for the mental health domain demonstrate limited flexibility of interactions with users, with system-directed interactions and a predefined dialogue flow [11]. As a result, the user has no control over the flow of the conversation and can only follow the system directives throughout the conversation. These limitations lead to shallow conversations and weak user engagement [8].

TEO allows users to share their thoughts and emotions using free-form natural language and engages users in personalized interactions about the events that are specific to each user. TEO can engage users in 2 types of dialogues. For the first type, TEO is designed to facilitate ABC (Activation, Belief, and Consequence) note writing for users. ABC notes are worksheets used by CBT therapists to help their patients in the identification of activating events (A), their beliefs related to the events (B), and the consequences of the events (C). Upon initiatives from a user to share a moment he/she is experiencing, TEO engages the user in dialogues where it asks a controlled set of questions designed by CBT therapists and collects an ABC note from the user in the form of a personal narrative about the event and his/her emotions. For the second type of dialogue (follow-up), TEO notifies the user about the ABC note written the day before and asks the user how he/she feels about the events, whether the issue is resolved, or whether the user is experiencing a different emotion [20]. TEO then tends to engage the user in a short personalized dialogue where it detects the recurrence of emotions and life events the user is experiencing [21], and provides helpful suggestions to ensure a healthier mental state.

Furthermore, TEO benefits from a knowledge base of therapeutic suggestions, recommendations, and exercises, which have been collected from therapists and domain experts. Users receive personalized tips and exercises weekly based on their progress of the therapy intervention. All the interactions with TEO are provided to the therapist weekly prior to the therapy session, so that the therapist can provide necessary support regarding the events and emotions expressed in the recollections and notes.

Measures

According to the findings by Sullivan and Artino [22] about the power of parametric versus nonparametric tests to detect differences between small-size samples, parametric analysis with repeated measures ANOVA (with a mixed within and between-subjects design) was performed to assess the differences between times (T1, T2, and T3) and groups (group 1, group 2, group 3, and group 4), and their interaction effect related to the results obtained in the PSS, SCL-90-R, and OSI tests. Multiple comparisons were corrected by using Bonferroni adjustment. The same analysis was conducted on PHQ-8 and GAD-7, which were administered before the intervention (T1), at mid-term (T2), at the end of the intervention (T3), and after 3 months (T4) to assess the differences between times (T1, T2, T3, and T4) and groups (group 1, group 2, group 3, and group 4). Regarding the OSI test, only a few scales were considered for the analysis, that is, the ones regarding coping strategies (social support, home-work relationship, task oriented, logic, time, and involvement), mental health, and physical health.

Results

PSS and SCL-90-R Results

The results obtained by administering the PSS and SCL-90-R tests are reported in Table 2. For the PSS, Global Severity Index (GSI), Positive Symptom Total (PST), Positive Symptom Distress Index (PSDI), obsessiveness-compulsiveness, interpersonal hypersensitivity, and depression scales/subscales, the assumption of sphericity had not been violated; otherwise, for the hostility and psychoticism subscales, the assumption had been violated (Multimedia Appendix 1 presents the results of the Mauchly test).

For the PSS, lower scores indicate lower stress levels and better well-being. Significant differences within groups between times were found for group 2 between T1 (mean 22.4, standard error [SE] 1.97) and T2 (mean 11.6, SE 2.36) (SE 2.52; P<.001), between T2 (mean 11.6, SE 2.36) and T3 (mean 16.6, SE 1.90) (SE 1.80; P=.03), and between T1 (mean 22.4, SE 1.97) and T3 (mean 16.6, SE 1.90) (SE 2.01; P=.02; Table 2). Further comparisons conducted within times between groups revealed a significant difference between groups at T1 (F_3,32=3.34; P=.03; η²p=0.24), specifically between group 2 (mean 22.4, SE 1.97) and group 4 (mean 13.88, SE 2.20) (SE 2.95; P=.04) at T1.

For the GSI subscale of the SCL-90-R, lower values indicate less psychological distress. Significant differences within groups between times were found for group 2 between T1 (mean 59.4, SE 2.64) and T2 (mean 48.9, SE 3.99) (SE 2.83; P=.002; Table 2). Further comparisons conducted within times between groups did not highlight any significant difference.

For the PST subscale, lower scores indicate fewer reported symptoms. Significant differences within groups between times were found for group 2 between T1 (mean 59.7, SE 2.50) and T2 (mean 51.9, SE 3.17) (SE 2.24; P=.004; Table 2). Further comparisons conducted within times between groups did not highlight any significant difference.

For the PSDI subscale, lower scores indicate lower intensity of distress. Significant differences within groups between times were found for group 2 between T1 (mean 57, SE 2.51) and T2 (mean 45.1, SE 3.94) (SE 3.37; P=.004; Table 2). Further comparisons conducted within times between groups did not highlight any significant difference.

For the obsessiveness-compulsiveness subscale, lower scores indicate less symptomatology. Significant differences within groups between times were found for group 2 between T1 (mean 57.6, SE 2.29) and T2 (mean 47.9, SE 3.53) (SE 3.02; P=.009; Table 2). Further comparisons conducted within times between groups did not highlight any significant difference.

For the interpersonal hypersensitivity subscale, lower scores indicate less presence of feelings of inadequacy and inferiority. Significant differences within groups between times were found for group 2 between T1 (mean 54.9, SE 2.05) and T2 (mean 48, SE 2.36) (SE 2.21; P=.01; Table 2). Further comparisons conducted within times between groups did not highlight any significant difference.

For the depression subscale, lower scores indicate less depression symptoms. Significant differences within groups between times were found for group 2 between T1 (mean 63.1, SE 3.29) and T2 (mean 51.8, SE 4.79) (SE 3.58; P=.01; Table 2). Further comparisons conducted within times between groups did not highlight any significant difference.

For the hostility subscale, lower scores indicate the presence of fewer anger-related personal characteristics. Significant differences within groups between times were found for group 2 between T1 (mean 57.6, SE 4.22) and T2 (mean 45.4, SE 2.21) (SE 4.49; P=.03; Table 2). Further comparisons conducted within times between groups did not highlight any significant difference.

For the psychoticism subscale, lower scores indicate less tendency of isolation and less presence of symptoms. Significant differences within groups between times were found for group 2 between T1 (mean 56.7, SE 3.29) and T2 (mean 50.2, SE 3.58) (SE 2.18; P=.02; Table 2). Further comparisons conducted within times between groups did not highlight any significant difference.

The results of the somatization, anxiety, phobic anxiety, and paranoid ideation (PAR) subscales are shown in Multimedia Appendix 1.

OSI Results

The main results of the OSI are reported in Table 3. For the task-oriented, logic, mental health, and physical health subscales, the assumption of sphericity had not been violated (Multimedia Appendix 1 presents the results of the Mauchly test).

For the task-oriented subscale, lower scores indicate criticality. Significant differences within groups between times were found for group 2 between T1 (mean 5.2, SE 0.56) and T2 (mean 6.9, SE 0.55) (SE 0.62; P=.04; Table 3). Further comparisons conducted within times between groups did not highlight any significant difference.

For the logic subscale, lower scores indicate criticality. Significant differences within groups between times were found for group 1 between T1 (mean 3.88, SE 0.59) and T2 (mean 5.63, SE 0.72) (SE 0.52; P=.01; Table 3). Further comparisons conducted within times between groups did not highlight any significant difference.

For the mental health subscale, lower scores indicate a higher level of mental well-being. Significant differences within groups between times were found for group 2 between T2 (mean 3.0, SE 0.72) and T3 (mean 5.1, SE 0.87) (SE 0.56; P=.004; Table 3). Further comparisons conducted within times between groups did not highlight any significant difference.

For the physical health subscale, lower scores indicate a higher level of physical well-being. Significant differences within groups between times were found for group 2 between T1 (mean 7.6, SE 0.77) and T2 (mean 5.7, SE 0.65) (SE 0.66; P=.03; Table 3). Further comparisons conducted within times between groups did not highlight any significant difference.

The results of the social support, home-work relationship, time, and involvement subscales are shown in Multimedia Appendix 1.

PHQ-8 and GAD-7 Results

The main results of PHQ-8 and GAD-7 are reported in Table 4. For the PHQ-8 test, lower scores indicate lower levels of depression. The only significant difference found was the one between groups at T1 (F_3,31=3.85; P=.02; η²p=0.27), specifically between group 1 (mean 9.42, SE 1.16) and group 4 (mean 3.43, SE 1.51) (SE 1.91; P=.02).

For the GAD-7 test, lower scores indicate lower levels of generalized anxiety. Significant differences within groups between times were found for group 1 between T1 (mean 9.5, SE 1.21) and T4 (mean 4.83, SE 1.1) (SE 1.24; P=.004; Table 4). Furthermore, comparisons conducted within times between groups revealed significant differences between groups at T1 (F_3,31=3.53; P=.03; η²p=0.25), specifically between group 1 (mean 9.5, SE 1.21) and group 4 (mean 3.14, SE 1.58) (SE 1.99, P=.02). 

The PSS, SCL-90-R, OSI, PHQ-8, and GAD-7 results of the interaction effects between time and group can be found in Multimedia Appendix 1.

Participant Feedback

At the end of treatment, feedback was collected from all the participants through the administration of a satisfaction questionnaire conceived for this study. For each item of the questionnaire, users were asked to indicate their degree of agreement on a 5-point Likert scale, from 1 (strongly disagree) to 5 (strongly agree). To assess satisfaction across all groups, 1 item of the questionnaire asked the users if they were satisfied overall with the received treatment. In the same way, to assess usefulness, they were asked if they felt that the treatment was useful. General results of satisfaction and perceived usefulness are shown in Table 5.

In addition to the general questions available for all groups, some specific questions were asked to assess the experience of the participants who could interact with TEO (ie, groups 2 and 3), focusing on the participants’ experiences with the conversational agent. The results are shown in Table 6. “Easy to use” was used to refer to the ease of interaction with TEO, and “usefulness” was used to refer to the perceived usefulness of the app. “Personal usage” was intended to investigate if, in case the conversational agent was available on app stores (iOS or Android), the users would use it (using the question “If TEO was available on the Android/iOS store, would you use/download it for your personal use?”). Statistical analysis with one-way ANOVA was conducted to assess whether there were significant differences between groups for the above variables. No significance was detected. Specific results are reported in Multimedia Appendix 1.

Discussion

Principal Findings

Given the small number of subjects per group, the results concerning the differences between groups and between times within each group are discussed. The statistical analysis seemed to show significant differences between groups as follows: at T1, group 2 and group 4 differed in terms of the PSS, and group 1 and group 4 differed in terms of the GAD-7 and PHQ-8 scales. More specifically, group 2 reported higher levels of stress than group 4, group 1 reported higher levels of anxiety than group 4, and group 1 reported higher levels of depression than group 4. Although randomization of the groups was performed (explained in the Participants subsection in the Methods section), the differences at T1 for the GAD-7, PSS, and PHQ-8 scales could be due to a reduced sample size and a nonuniform distribution in the groups of subjects from different geographical zones of Italy as shown in Table 7. Overall, there was a worsening trend in almost all scales across all groups (Tables 2 and 3) between T2 and T3, although it did not appear to be significant. When we compared the interviews with some subjects and the Italian COVID-19 epidemiological statistics, we could observe an increase in COVID-19 positive cases and a general concern arising from the Delta variant of the virus in conjunction when the T3 statistics were collected from the participants after several months of general stability. This may be the reason for the overall deterioration observed from T2 to T3.

Analysis conducted separately within each group showed that there were many significant differences between times in group 2. Altogether, group 2 seemed to show improvements in the PSS, GSI, PST, PSDI, obsessiveness-compulsiveness, interpersonal hypersensitivity, depression, hostility, and psychoticism scores (Table 2), as well as the task-oriented, mental health, and physical health scores (Table 3). Despite significant improvements in group 2, for the PSS, there was significant worsening between T2 and T3, and there was an increase in stress at T3, although it was lower than that at T1. There was also worsening of psychological symptoms generally related to stress for the mental health scale of the OSI questionnaire (see Table 3 above). Despite significant worsening of the PSS and mental health (OSI) scores from T2 to T3, which generally detect similar symptoms of stress, it emerged that the physical health (OSI) score improved. This could indicate that subjects in group 2 were more susceptible to sudden changes, that is, increased cases of the contagious disease at T3 in their residential areas (in particular in Northern Italy) that could have increased psychological stress. However, the results for other scales suggest that the participants living in that geographical area could sufficiently cope with increased COVID-19–related worries, without developing higher levels of stress-related physical symptoms.

Group 3 reported significant improvements in the scores of the interpersonal hypersensitivity and PAR scales of the SCL-90-R questionnaire between T2 and T3 (see Table 2). Group 1 reported improved PAR (Table 2), logic (Table 3), and GAD-7 (Table 4) scores between times. Furthermore, group 4 showed a significant improvement in the PAR score between times (Table 2). In group 3, several individuals withdrew their consent to participate. Among them, 2 withdrew their consent owing to organizational complications that emerged and 2 withdrew their consent owing to very high expectations of the conversational AI agent that were not maintained. They judged that it was a waste of time to participate in this research compared with the perceived benefits.

The feedback questionnaires administered to understand the users’ experiences showed that group 2 experienced higher levels of satisfaction and perceived the usefulness of the received treatment more than the other groups (ie, psychological support and use of the mobile health [mHealth] agent), as shown in Table 5. Moreover, comparing groups 2 and 3, participants in group 2 showed a greater ease of interaction with TEO, and they found it more useful than those in group 3 (ie, the group that interacted with the conversational AI agent without human psychological support). Indeed, the “personal usage” scores revealed a greater inclination of group 2 participants to use TEO.

A few specific questions were administered per group to explore some expectations. In group 1, the aim was to understand whether users would accept or find useful the use of a mHealth app together with traditional treatment. The results showed a score of 3.07. In group 3, the aim was to understand whether users would accept or find useful the use of a mHealth app together with traditional treatment. The results showed a score of 4.57. Overall, positive expectations related to combining traditional treatment with a mHealth app were found, considering the fact that participants in group 3, who used the app, had higher expectations. In group 2, the aim was to understand not the expectations but how effectively, for the subjects, the use of the mHealth app facilitated traditional treatment. The results showed a score of 3.93.

Altogether, these results suggest that a psychological treatment, characterized by the presence of human contact, along with a conversational mHealth agent would improve the impact of treatment in terms of satisfaction and usefulness.

A further aspect to be considered in the evaluation of these results is that this experiment was performed during the third wave of the COVID-19 pandemic in Italy, as mentioned in the Methods section. Our results indicate that although this event had an impact on the levels of stress and on the general psychological well-being of the participants, the observed and perceived improvements were maintained over time in terms of the reduction of physical stress–related symptoms.

Limitations

Following the recruitment process, the number of active participants involved in this study was small, and this may weaken the inferences and conclusions. Besides, although the recruitment campaign was conducted through social media platforms to reach out to all Italian regions, the majority of our participants were from the center of Italy. The participants were mainly women, and the fact that women tend to seek psychological help more often than men has been studied previously [23,24]. Nevertheless, the observed gender predominance weakens the generalization of the drawn inferences for both genders.

Conclusions

The aim of this study was to evaluate the possible improvements related to the introduction of an AI-based mHealth app in psychological interventions aiming to reduce stress-related physical and psychological symptoms in aging workers. We administered different standard psychological tests to measure the levels of perceived stress, generalized anxiety, and depression, along with other psychological dimensions. We could not observe statistically significant differences between the participants who used the mHealth app alone and those who used it within the traditional setting of psychological treatment. On the contrary, we could observe significant within-group differences, with improvements in subjects who received treatment. Moreover, we observed greater levels of satisfaction and subjective perception of usefulness in participants who were supported by a human therapist as well as the mHealth conversational agent.