The prevalence of depressive symptoms within the United States drastically increased from 17 million to 21 million, a nearly 25% increase, from 2018 to 2020 during the COVID-19 pandemic [1] with young adults and women disproportionately affected [2]. To address depressive symptoms, mental health apps have emerged to offer assistance and therapeutic techniques to the public. Cognitive behavioral therapy (CBT)-based mental health apps represent a viable option to improve access to mental health resources [3]. A form of CBT, brief cognitive behavioral therapy (bCBT) has been suggested for depressive individuals as a means of maintaining the user’s attention while not requiring large amounts of the user’s time or energy. This form of CBT has successfully delivered therapeutic interventions in a time-efficient manner, around 4‐16 brief sessions [4,5] in both subclinical [6-8] and clinical populations [9]. Several bCBT-based apps, such as MoodMission [10], Pacifica [11], and SuperBetter [12], have demonstrated effectiveness in reducing depressive symptoms. Despite their effectiveness, it is unclear how specific app features may enhance user experience to maximize therapeutic benefits.

The use of virtual agents represents one avenue that may enhance mental health user experience, as virtual agents can be leveraged to mimic realistic human interactions and model social connection [13-15]. The term “virtual agent” refers to a noncontrollable, artificial intelligence (AI)-driven virtual entity, such as chatbots and embodied conversational agents (ECAs) designed to interact with users [16-18]. Chatbots communicate with the user through a textual or voice-based interface design but typically lack a visual embodiment [19]. ECAs are characterized by a human-like visual presence and have the capability to include both verbal or nonverbal communication behaviors [14,15]. While chatbots have demonstrated potential in numerous bCBT-based mental health apps, such as Woebot, Wysa, and Tess [20-22], ECAs offer a richer, more natural social presence [15], making them particularly suited for mental health interventions. Surprisingly, exceptionally few studies have evaluated the effectiveness of bCBT-based mental health apps with ECAs [6]. This study addresses this gap by incorporating an ECA-style virtual agent into the app design.

Given that a key component of ECA-style virtual agents is visual embodiment, the physical characteristics of a virtual agent may impact user experience. Research shows that similarity between a user’s demographics and an agent’s characteristics, such as gender and voice, fosters positive interactions by building trust and enhancing user motivation [23-26]. This aligns with the similarity-attraction effect, where users often prefer agents that mirror their own demographics, appearance, and voice [24]. In mental health contexts, such similarity has been shown to significantly increase users’ willingness to engage in support activities [25]. To leverage these benefits, the mental health app in this study includes customization options for the agent’s physical characteristics, aiming to create a greater sense of connection and comfort during interactions.

Beyond visual embodiment, 2 key features can be embedded into ECA-style virtual agents to convey the realism of human face-to-face interactions: conversational and animation features. Conversational behaviors, including lip-sync and speech, are used to replicate natural, verbal communication actions [15,27,28]. Virtual agent verbal cues that align with social norms, such as greetings, small talk, and thanking, foster trust and perceived knowledgeability [29,30], particularly when the agent uses a formal, familiar voice quality and style [31]. In addition, conversational agents can engage in turn-taking and provide feedback, mimicking the natural flow of human conversation [15]. Turn-taking allows users to feel that they are actively participating in the interaction, while feedback conveys that the agent is attentive and responsive [15]. A systematic review of mental health interventions leveraging conversational agents observed a significant reduction in psychological distress postintervention compared with baseline [32]. These findings underscore the preliminary efficacy of virtual agents with conversational features on mental health symptoms and suggests that virtual agent conversational feature may afford empathy and interactivity that mimics therapeutic dynamics [32]. However, few studies reviewed were empirical randomized controlled trials [32], and the variability in mental health symptoms limits understanding of how conversational features affect individuals with and without depressive symptoms.

On the other hand, animation supports natural communication by conveying nonverbal behaviors, such as facial expressions, co-speech gestures, body movements, and eye gaze [14,15]. Nonverbal cues, such as nodding and eye gaze indicate active listening and foster rapport [33-35], while facial expressions can convey emotional responsiveness [36]. It is critical for animations to appear natural, as overly expressive facial animations can seem unrealistic [37]. Natural animations encourage positive attributions toward virtual agents, such as greater acceptance, trust, credibility, and task appropriateness [31,38]. Natural animations also elicit stronger emotional responses and a greater sense of social presence compared with static or partially animated agents [39,40]. While natural animation cues, such as body movements and facial expressions, can enhance social presence, they are not always effective in conveying trustworthiness [31]. Factors like the user’s age, the relevance of the animation to the task, and the context (eg, interviews, learning, or commerce) influence how animation impacts perceived trust [31,41], and the effectiveness of such animation features in mental health contexts remains underexplored.

In human-human therapeutic interactions, body language, tone, and other social cues are critical to conveying empathy and can influence therapeutic outcomes in individuals with depression [42-44]. Research with chatbots [13] and ECAs [45] has demonstrated that individuals experiencing depressive symptoms report high perceived virtual agent empathy and user-agent working alliance with levels mirroring that of CBT-based human interventions. These findings suggest that virtual agents may be able to mirror human-human therapeutic interactions by encouraging users to feel understood and supported. Such characteristics may be especially critical for individuals with depression, who often experience negative perceptions of themselves, others, and their environment [46]. However, no studies have directly compared how these virtual agent features (eg, conversational vs animation) may influence mental health outcomes and user experience in users with and without depressive symptoms.

This study builds on previous work [5-13,20-22,45] in several ways. First, this study directly compares how virtual agent conversational and animation features influence user experience in the context of mental health apps using a randomized controlled trial design. Second, this study assesses whether either of these features within a bCBT-based mental health app can reduce symptoms of depression, stress, and rumination over 2 weeks. Third, this study compares these features in a sample of users with and without depressive symptoms, addressing gaps in understanding how conversational versus animation features uniquely contribute to mental health outcomes in this population.

The study hypotheses for the quantitative analyses are outlined below:

H1: Individuals will exhibit significantly lower symptoms of depression, stress, and rumination after 2 weeks. This reduction will be more pronounced in the conversational and animated conditions.

H2: Individuals will have a more positive user experience with the agent in the conversational and animated conditions.

In addition to these quantitative analyses, we will query participants’ rationale in designing their virtual agents in terms of gender and similarity to people they know through qualitative methods.

The goal of this 4-arm randomized controlled trial was to determine the effect of virtual agent conversational and animation features within a bCBT-based mental health app on user experience and change in depressive symptoms over a 2-week intervention period. The virtual agent conversational feature reflects dialogue-based interaction between the user and agent, and the virtual agent animation embodies dynamic body movements and facial expressions. Participants completed a baseline training and setup session along with baseline questionnaires through a face-to-face assessment; thereafter, participants completed the intervention and 2-week postintervention questionnaires remotely. In this section, we describe the design of the overall app and virtual agent with a focus on the manipulation of the conversational and animation features. We also describe the methodology and analytic approach used to evaluate these features in a sample of college students with and without depressive symptoms. We note that this study is based partially on dissertation work by lead author SS.

Following previous research evaluating AirHeart [6], an a priori power analysis (F test, repeated measures ANOVA (analysis of variance), within-between interaction) was conducted for H1. The analysis aimed for 80% power to detect effects at P=.05 with 4 groups and 2 time points, based on previous effect sizes [47]. Results indicated a required minimum sample of 136 participants. Multimedia Appendix 1 provides detailed information. We sought to recruit ≥25% (170 minimum) over the minimum to account for attrition and data exclusions.

A total of 209 college students completed the study and were randomized to one of the 4 experimental conditions (n= 209; mean age 19.97 years, SD 2.19; Table 1). Participants were incentivized to participate with compensation in the form of course credit, extra credit, or a $20 Amazon gift card, depending on their choice. Participants were excluded if they were outside the 18‐30 years age range or did not have daily access to a smartphone. Data were excluded for 3 reasons: (1) the participant completed less than 2 CBT-based modules, (2) failed more than 1 attention check, or (3) did not submit the postintervention survey.

The AirHeart mental health app was designed using Unity 2021 and contains all themes and features of a version published in previous work [6] but included new features, such as a help section, additional customization options for the virtual agent, and an additional resources section. The virtual agent was introduced to participants as their “virtual coach“ who joined them on their journey and guided them through CBT topics. Given the importance of customization features to foster user-agent similarity [24-26], users were able to customize numerous features of their agent, including facial features, body shape, and clothing. The majority of the conversational and animation feature design choices were motivated by past research describing the importance of both verbal (ie, lip sync animation and co-speech gesturing) and nonverbal (ie, head nods and backchanneling) behaviors in conveying natural social communication information [14,15]. All user-agent communication was conducted through natural dyadic verbal exchanges. Users initiated verbal input using a speech-to-text engine, the speech recognition system plugin, while audio-based agent dialogue was created using the following text-to-speech (TTS) engines: RTVoice Native for Android+ AmazonWeb Services Polly Standard for iOS. Additional technical details regarding app development and virtual agent customization, conversational feature, and animation feature development can be found in Multimedia Appendix 2.

The current experiment included 4 experimental conditions differing based on the presence or absence of conversational and animation features (conversational, animated; conversational, nonanimated; animated, conversational; nonconversational, nonanimated). All conditions had access to all app features (ie, CBT modules, journaling, mood tracker, agent customization, help section, and additional resources section).

The animation feature involved dynamic body movements and facial expressions exhibited by the virtual agent. The animated condition included human-like nonverbal body movements, mouth movements, and gestures in association with the information provided by the virtual agent. The nonanimated condition displayed a static, nonmoving virtual agent with a blank facial expression.

The conversational feature was characterized by user-agent interactivity in whcih question-and-response style dialogue was embedded within the CBT modules. The virtual agent asked questions or instructed the participant to complete activities aloud. A microphone icon provided a visual cue for users to engage in conversation with the agent when red, the microphone is on, and white when off. The nonconversational condition did not allow the user to add their input or respond to questions. Figure 1 shows the visualization of the virtual agent in the 4 different conditions.

At the baseline assessment (Time 1), participants were first randomized to one of the 4 virtual agent conditions that varied in conversational and animation features. After providing written informed consent, they completed the mental health questionnaires (PHQ-8, PSS-10, and RRS). Users then downloaded and piloted the AirHeart app using TestFlight, a beta-testing app required for iPhones due to their additional security measures, while Android users could install the app directly. Next, they created an account, followed a tutorial to personalize their virtual agent, and completed the first CBT module. They used the app every other day for 2 weeks, which included a minimum of 8 times for full completion, but additional usage was encouraged. When participants logged into the app for the first time that day, they were prompted to complete the daily questionnaire, view their mood tracker, and then taken to the home page where they had access to the CBT modules. After the 2-week intervention, participants were contacted through email to complete postintervention questionnaires. At this assessment (Time 2), participants completed the mental health (PHQ-8, PSS-10, RRS) questionnaires again as well as the user experience questionnaire (MAU-Q) and open-ended user experience questions.

To investigate H1, separate 2 (conversational status: present vs absent) × 2 (animation status: present vs absent) × 2 (time: baseline vs postintervention symptoms) mixed effects ANOVAs was used to analyze changes in depressive, stress, and rumination symptoms, respectively. Conversational status and animation status were between-subjects factors; time was a within-subjects repeated measures factor. Sensitivity analyses were conducted that focused only on participants who reported experiencing depressive symptoms (PHQ-8 scores >4).

To assess H2 for the user experience predictions, separate 2 (conversational status: present vs absent) × 2 (animation status: present vs absent) × 2 (depressive status: depressive vs nondepressive state) multifactorial ANOVAs were performed for each of the 3 MAUQ subscales. Using the validated cutoff scores established in previous work [48,49], PHQ-8 scores ranging from 0‐4 were considered normal (or nondepressive) and scores of 5 and above were considered in a depressive state. Inclusion of this factor allowed for distinguishing whether individuals with and without current depressive symptoms had user experience preferences for the virtual agent characteristics.

For quantitative data, we conducted parametric ANOVA analyses after verifying that the data were normally distributed and error variances were equivalent [54,55]. Box’s test confirmed equality of covariance matrices, and Levene test verified homogeneity of variance. Mauchly test ensured sphericity. When appropriate, posthoc pairwise tests were conducted using the Tukey honestly significant difference for between-subjects variables and the Bonferroni adjusted alpha method for within-subjects variables. These methods are widely used in user studies and human factors research in computing [6,39,56-58]. For the open-ended qualitative questions, a reflexive thematic analysis was performed in order with the procedure specified by Braun and Clarke [59] which has been used in numerous user studies evaluating virtual agents [26,60-63]. Two researchers independently reviewed deidentified responses, manually created initial codes, and then grouped codes into categories. For each of the 3 qualitative questions, percentage agreement for categories between researchers was >85%. Researchers then reviewed the independently-generated categories, consolidated duplicates, and refined and labeled themes. Next, the study conditions (ie, conversational and animated) and depressive group (depressive vs nondepressive) were reattached to the responses to create a frequency data table.

The study procedures were approved by the Clemson University Institutional Review Board (IRB2021-0879) before procedures were implemented. All participants provided written informed consent before participating in the study. They were given the option to opt-out of participating. All data are deidentified. Participants were incentivized to participate with compensation in the form of course credit, extra credit, or a US $20 Amazon gift card, depending on their choice.

Participants were asked to explain the reason they selected the gender of their virtual agent. Responses were collected from all 209 participants, but 3 were excluded for failing to supply a usable response. Two key themes emerged: relatability (89/206, 43.2%) and trust or comfort in talking with a particular gender about one’s mental health concerns (160/206; 77.7%); note that some participants listed both reasons. Example quotes to illustrate the relatability theme are listed below:

Quotes describing the comfortability preference with a particular gender are included below:

Suggestions for improving the virtual agent were collected from all 209 participants, but 39 of them failed to provide a viable answer. The 170 responses resulted in 4 different themes: (1) robotic voice or interaction, (2) lack of personalization or customization, (3) more engagement or realism, and (4) technical issues. Similar to the previous free response question, z score proportion tests were conducted for the depressive and nondepressive participants in each category. The robotic voice/interaction (z=3.36, P<.001) was the sole categories to reach significance. A frequency data table was created to help visualize this information (Table 8).

The current randomized controlled trial sought to investigate how conversational and animated components of a virtual agent within a bCBT-based mental health app might affect change in depressive symptoms and perceived user experience. Given that individuals experiencing depressive symptoms may have negative views of themselves or others and may struggle with anhedonia, low energy, amongst other symptoms [64], it is reasonable that individuals experiencing depressive symptoms may have different intervention needs or preferences compared with those who are not experiencing such symptoms. The results demonstrated that bCBT delivered through a virtual agent within a mental health app significantly reduced symptoms of depression, stress and rumination over a 2-week period, regardless of whether the agent included conversational or animation features. Consequently, these results partially support H1. The animation feature did enhance user experience, while the conversation feature had no significant impact.

While several empirically-evaluated bCBT-based mental health apps like Woebot, Wysa, Tess, and Fido [20-22,65] include virtual agents, these existing mental health apps leverage a text-based chatbot design. Such design does not allow for animation features and certain conversational feature components like natural speech dynamics and nonverbal behaviors. In addition, while Tess displays a static picture of a smiling Caucasian female in the text-based chat dialogue box [22], Woebot, Wysa, and Fido do not feature a human-like graphic and instead use animals or robots [20,21,65]. Furthermore, none of these apps feature a customizable virtual agent. In contrast, the AirHeart mental health app included a human-like, customizable virtual agent, and the conversational animated app condition featured both speech and text-based verbal capabilities, nonverbal behaviors, and dynamic animations.

Small pilot studies on virtual agent-based self-monitoring technologies have shown promise in demonstrating the feasibility and preliminary efficacy in reducing depressive symptoms [66-68]. The current study advances this work by demonstrating that virtual agent-based bCBT technology can effectively reduce depressive symptoms through a moderate-size randomized controlled trial. While conversational and animation features were expected to enhance the effectiveness of the intervention, particularly among those experiencing depressive symptoms, no added benefit of these features was observed on changes in depressive symptoms, stress, or rumination. Past work has shown that ECA-style virtual agents that mimic human-human interactions may enhance perceived empathy and working alliance with the user [45,69]. The results of the present study suggest that conversational and animation features may not be critical for establishing a meaningful connection between the virtual agent and the user in the context of bCBT mental health apps for depression. Instead, the social presence of the human-like virtual agent alone may be sufficient.

Study results indicated that users in the animated agent conditions reported higher ratings for system information arrangement (MAUQ-system information arrangement), ease of use (MAUQ-ease of use and satisfaction), and usefulness (MAUQ-usefulness) compared with those in nonanimated conditions. There was no significant difference in conversational versus nonconversational conditions; therefore, H2 is partially supported. These results suggest that animation can enhance the user experience in mental health interventions, which aligns with previous research showing that the inclusion of both nonverbal behaviors can create more human-like interactions and improve user impressions in mental health contexts [34,35]. In addition, the inclusion of such animation design has previously demonstrated a strong connection to higher levels of agent acceptance, trust, credibility, and task appropriateness [38]. These findings are crucial for developers of mental health interventions, as they underscore the importance of integrating virtual agents with natural animations, such as body, mouth, and gesture movements, to enhance user satisfaction and foster human-like interactions.

Consistent with the similarity-attraction effect [24,25], most participants (>90%) selected an agent of the same gender as themselves and designed it to resemble themselves or someone familiar, such as a friend, family member, or therapist. This preference aligns with research showing that familiarity provides comfort, particularly during vulnerability [70]. Many participants reported feeling more comfortable discussing mental health with females, citing greater relatability on emotional matters. This increased relatability may be more attributable to similarity than stereotypes of females as more emotionally aware and empathetic than men [71-73]. Indeed, while females often self-report higher empathy, a meta-analysis showed no objective gender differences in empathy [74]. The findings support research demonstrating stronger therapeutic alliances when clients and counselors share the same gender, particularly among female clients [75], and users’ preference for same-gender virtual agents [76-79]. In mental health contexts, gender synchrony has been shown to enhance trust in virtual agents, especially when paired with similar age [79]. These results highlight the importance of virtual agent gender customization for relatability in mental health app design. However, past research suggests that developers often rely on stereotypical binary gender cues, which can reinforce societal gender expectations [23]. Thus, mental health app developers and researchers should be cognizant of the limitations of stereotypical binary gender cues and enhance features that support diverse gender representation, especially in verbal and nonverbal animations.

The qualitative analysis revealed that most participants found the virtual agent’s voice robotic and suggested improvements to voice quality. It is possible that the quality of the virtual agent’s voice may have impacted the results of the conversational feature. The app used AmazonWeb Services Polly Standard Voice (iOS) and RTVoice Native (Android) for TTS, both of which can sound synthetic, similar to Siri (Apple) or Google Assistant (android). Previous research has shown that synthetic, artificial voices induce an eerie feeling [80,81], and similar results were found using a TTS agent for CBT-based emotional regulation, where participants also noted the robotic speech [82]. Future studies should explore higher quality TTS or prerecorded human voices to enhance user interactions with the virtual agent.

Furthermore, the study included pre- and 2-week postintervention measurements, but long-term follow-ups assess whether the effects of the intervention are sustained over time were not included in the study design. Additional research is needed to determine the duration of the benefits from the virtual agent-delivered bCBT mental health intervention following the conclusion of app use.

This study is among the first to compare the effectiveness and user experience of a virtual agent bCBT-based mental health app in both users with and without depressive symptoms. The key findings from the study demonstrated that the app intervention was effective in reducing mental health symptoms, regardless of whether the agent included conversational or animation features, but animation features enhanced user experience. These effects were observed in both users with and without depressive symptoms. This work suggests that college students experiencing depressive symptoms may not have unique user experience requirements in mental health apps, and such findings may apply more broadly to wellness apps. The finding that virtual agent animation improves user experience in mental health apps but does not affect the intervention’s effectiveness offers valuable insight for optimizing app design, which can help guide future development of digital mental health tools that are both effective and user-friendly.