The preregistered study involved a pilot RCT in which participants were randomly assigned to 1 of three groups in a 1:1:1 ratio: (1) the AI therapy group, who engaged in AI-assisted therapy sessions via the ChatMind app with a requirement of 3 sessions per week for 3 weeks; (2) the ChatGPT group, who completed equivalent sessions with a general chatbot in the ChatGPT mobile app; and (3) the AOC group, who received no intervention. Participants were aware of their assigned groups so that they could perform the relevant tasks and were emailed twice a week to remind them about the experiment. Mental health indices were measured before and after the 3-week intervention protocols using an online self-assessment questionnaire hosted on the OQS platform.

The sample size was estimated based on a previous RCT [30] evaluating a 3-week solution-focused brief, human-delivered therapy intervention, as the AI therapy shares similar characteristics. Using the effect sizes from this study, a range of anticipated mean changes was estimated, and a simulation-based power analysis was conducted. Across different sample sizes, specifying a mixed-effects model accounting for repeated measures design and group differences, power simulations indicated that 16 participants per group would achieve ≥80% power to detect a very large effect size (d=1.24). Since we were preparing for a high dropout rate, which is common in mobile app research [31], we set the minimum number of participants in each group to 26. However, given the aims of this pilot-stage study, statistical power to detect a smaller effect was not the priority. More details on this analysis, including a power analysis report, are described in the project preregistration.

Participants were recruited using a convenience sampling method through online advertisements (eg, Instagram), emails, and push notifications in a partnered mental health app VOS. The VOS app shares the same parent company as ChatMind but is not an AI-based therapy chatbot. It targets a general, nonclinical population interested in mental well-being and provides general mental-well-being tools such as a mood tracker, guided journaling, breathing exercises, and meditations. Inclusion criteria were minimal; participants were eligible if they were native English-speaking, aged 18 years or older, were not being treated for any psychiatric condition, and had never used the ChatMind AI chatbot.

Participants were recruited in 2 waves. Enrollment for the first wave took place during November and December 2024 (prospective study registration took place on November 29, 2024); 83 participants were enrolled, but only 76 (92%) completed the baseline survey, of which 9 (12%) records were deleted for completing the questionnaire outside the time schedule or not finishing the questionnaire, resulting in 67 randomized participants in the first wave. To ensure a sufficient sample size for pilot-stage effect size estimates, an additional recruitment wave took place in February 2025. With the same randomization strategy, participants were allocated into groups. Of the 102 participants enrolled in this wave, 88 (86%) completed the baseline survey, of which 8 were incomplete or off-schedule records, resulting in 80 participants in the second wave and 147 participants in total. No safety incidents or adverse events were reported during the study. Consistent with the aims of a pilot RCT, statistical power was not prioritized; instead, the sample size was selected to enable estimation of preliminary effect sizes to provide early-stage evidence and foundational support for forthcoming large-scale trials.

The final analytic sample comprised 85 (58%) participants from the United States, 21 (14%) from Canada, and 41 (28%) from other predominantly English-speaking countries, including the United Kingdom, Ireland, and Australia. Participants ranged in age from 20 to 74 years (mean 38.4, SD 10.8), and 73% were women.

Participation was voluntary, with the only incentive being trial access to the ChatMind AI chatbot (ie, participants in all conditions got access after the 3-wk study period). Informed consent (Multimedia Appendix 1) was obtained from all individuals before the baseline assessment. To ensure privacy and confidentiality, all data were pseudonymized at the point of collection and stored on secure, password-protected servers. No personally identifiable information is reported in this study, and only aggregated data are presented to prevent the identification of individual participants. Individuals currently receiving psychiatric treatment were excluded to minimize the risk of distress or adverse reactions. The study design was approved by the Ethics Committee at Olomouc University Social Health Institute (OUSHI) (approval September 2, 2024).

Group differences in demographic variables and baseline differences in outcome measures were estimated using χ² tests for categorical variables and nonparametric analysis of variance (the Kruskal-Wallis test) for age. A χ² test was also performed to assess for differential attrition across the study groups. This test revealed a strong trend suggesting that dropout rates were dependent on group assignment (χ²=5.14; P=.08). This signal of differential attrition, driven by a substantially higher dropout rate in the AI therapy group, raises concerns about the validity of a per-protocol analysis (PPA), as this approach becomes susceptible to selection bias that can compromise the initial randomization. Thus, our primary analysis followed the intention-to-treat (ITT) principle to provide an unbiased estimate of the intervention’s effectiveness. A secondary PPA, consistent with one of the options outlined in our preregistration, was also conducted to explore the efficacy of the intervention specifically among participants who completed the study. The results of the PPA are presented in supplementary analysis in Multimedia Appendix 4.

To evaluate intervention effects, linear mixed-effects models were used. Separate models were fitted for each outcome variable: anxiety (GAD-7), depressive symptoms (PHQ-9), depression severity and impairment (ODSIS), and mental well-being (WHO-5). For the GAD-7, ODSIS, and WHO-5 outcomes, models were fitted with parameters estimated using restricted maximum likelihood. For the PHQ-9 outcome, initial diagnostic checks revealed significant heteroscedasticity. To address this, a mixed-effects model was refitted with the nonconstant variance explicitly modeled. All models were adjusted for participants’ education level to account for baseline differences across groups.

Each model included fixed effects for time, with measurements taken at baseline and after the 3-week study period (ie, preintervention vs postintervention), experimental group (AI therapy, ChatGPT, and AOC), and the time × group interaction representing the differential change over time by group. Random intercepts for participants accounted for individual differences in baseline scores and changes over time. Initially, AOC was coded as the reference group to derive contrasts between AOC and the 2 active treatment conditions, then we releveled the condition variable coding to directly contrast AI therapy and ChatGPT conditions via planned comparisons. Although the constructs measured together are often statistically related (eg, [37], they are conceptually different. Therefore, we treated the scales as independent variables (depression symptoms, anxiety symptoms, depression-related functional impairment, and positive well-being). Following strict adherence to our preregistration, we applied Holm-Bonferroni correction to the 8 preregistered directional hypotheses: 4 tests comparing AI therapy to control (across 4 outcomes) and 4 tests directly comparing AI therapy to ChatGPT (across 4 outcomes). For these 8 preregistered tests, 1-tailed P-values were computed based on the directional hypotheses (expecting AI therapy to show greater improvements). Comparisons between ChatGPT and control were not preregistered with directional hypotheses and are therefore reported as exploratory analyses with 2-tailed P-values and no correction for multiple testing.

To formally assess the mechanism of missing data, we performed an omnibus test for data being missing completely at random (MCAR). In the MCAR test, the assumptions of multivariate normality and homoscedasticity required for the standard parametric Little’s test were violated (Hawkins test, P<.001). Therefore, we relied on the robust nonparametric alternative. The result of this test was not statistically significant (P=.06), providing insufficient evidence from this omnibus test to reject the MCAR null hypothesis. To examine predictors of intervention completion (fidelity) and retention (completing the follow-up survey), we performed a series of logistic regression analyses. In these analyses, independent variables were outcome measures (ie, PHQ-9, ODSIS, WHO-5), age, gender, and attitude toward AI.

Effect sizes were reported as unstandardized regression coefficients (b) and Cohen d. Because all outcomes were self-reported by participants, blinding of outcome assessors was not feasible. The researchers conducting the data analysis were not blinded to group assignment; however, all code and analytic decisions were independently reviewed by multiple members of the research team to minimize potential bias. All statistical analyses were conducted using R software, version 4.3.0 (R Core Team, 2023) within the RStudio environment, version 2024.04.2.

The Consolidated Standards of Reporting Trials (CONSORT) diagram chart in Figure 1 illustrates the process of respondent enrollment, allocation into groups, assessment, and intervention. Sample demographics and baseline characteristics for the ITT sample are provided in Table 1. Baseline comparisons showed no significant differences across the 3 groups for age or any of the clinical outcome measures (GAD-7, ODSIS, PHQ-9, or WHO-5), with all P values >.05. However, a Pearson χ² test with Monte Carlo simulated p-value revealed a significant difference in the distribution of education levels across the groups (χ² = 16.07, simulated p = .014). (χ²=16.07; P=.01). The demographic characteristics of the sample are summarized in Table 1.

Full fidelity to the intervention protocol was relatively low in both treatment conditions; 17 (39%) of those in the AI therapy condition completed all intervention sessions, and 38 (62%) of those in the ChatGPT condition completed all sessions. Logistic regression revealed that none of the baseline clinical measures were significant predictors (GAD-7: odds ratio [OR]=0.97, P=.456; ODSIS: OR=0.95, P=0.23; PHQ-9: OR=0.98, P=.64). However, older age was significantly associated with a lower likelihood of completing the intervention (OR=0.95, P=.03).

Study retention, in terms of completing the follow-up survey, was 28 (64%) in the AI therapy condition, 36 (60%) in the ChatGPT condition, and 28 (65%) in the AOC condition. Logistic regression revealed that baseline symptom severity was not a significant predictor, although higher baseline ODSIS scores showed a trend toward predicting a lower likelihood of retention (OR=0.92; P=.05). In these models, older age again emerged as a significant predictor of lower retention (OR=0.95; P=.003). Furthermore, a more positive baseline attitude toward AI significantly predicted a higher likelihood of retention (OR=1.48; P=.048).

Descriptive statistics for anxiety, depression, and mental well-being scores at baseline and after the 3-week intervention period, stratified by the experimental groups, are presented in Multimedia Appendix 5. Kruskal-Wallis tests showed no significant group differences in mental health indices at baseline (P-value range: 0.43‐0.94). Although not a major feature of the trial, baseline assessment of attitudes toward AI-based therapy was very favorable (mean 3.05, SD 0.92) on a 4-point scale, where 1=skeptical or negative and 4=enthusiastic).

Table 2 presents the ITT intervention effect estimates, with unstandardized regression coefficients (b) for the interaction terms representing the estimated difference in change between groups from preintervention to postintervention. Table 3 presents the corresponding effect sizes (Cohen d) for these interaction effects, along with their 95% CIs.

During preregistered hypothesis testing, it was found that the AI therapy group exhibited a statistically significant reduction in PHQ-9 depressive symptoms compared to the control group with a 2.67 point greater reduction (d=−0.47, 1-tailed and noncorrected P=.006, corrected P=.046), which remained significant after Holm-Bonferroni correction for the 8 preregistered hypotheses. The AI therapy group also showed trends toward greater improvement in anxiety symptoms (GAD-7: 1.98 point reduction, d=−0.37) and ODSIS depression functional impairment scores (1.13 point reduction, d=−0.25), although these did not reach statistical significance. No significant change in well-being (as measured by the WHO-5) was observed compared to the control group.

During nonpreregistered exploratory analyses, we examined differences between the ChatGPT group and the control group in the outcome measures. It was revealed that the ChatGPT group showed a significant 2.47 point greater reduction in PHQ-9 depressive symptoms (d=-0.44, 2-tailed P=.02), as well as nonsignificant improvements in anxiety symptoms (d=-0.27) and ODSIS scores (d=-0.12). These exploratory results were not subject to multiple comparison correction. As shown in the supplemental analyses in Multimedia Appendix 4, these results were largely consistent with the per-protocol results.

The next set of models was the planned comparisons between the AI therapy and ChatGPT (reference group) conditions, as shown in the lower half of Tables 2 and 3. Although none of the estimates reached statistical significance, the AI therapy group showed small, favorable effect size trends compared to the ChatGPT group for anxiety (d=-0.11), ODSIS depression severity and impairment scores (d=-0.13), PHQ depressive symptoms (d=-0.03), but not in well-being (d=-0.07). In the PPA, the ChatGPT group showed numerically larger reductions in depression scores (ODSIS and PHQ-9) compared to AI therapy, but these differences were very small and not statistically significant (Multimedia Appendix 4).

As digital therapy alternatives rapidly evolve, including GenAI-based therapy chatbots, there is a need for careful, iterative testing of these novel therapeutic modalities. This pilot study provides early-stage proof of concept for a brief 3-week (9 sessions) AI therapy chatbot intervention based on solution-focused principles (ie, ChatMind), relative to a standard untrained chatbot (GPT-4o-based models) condition and an AOC condition. As described below, findings provide foundational support for the overall promise/merits of GenAI chatbot therapy while also elucidating key areas of improvement regarding feasibility.

The multinational sample of nontreatment-seeking adults generally reported favorable attitudes toward AI-based therapy assessed at baseline; however, the suboptimal adherence to intervention protocols may suggest waning interest once involved. Indeed, only 39% of those in the AI therapy condition fully adhered to the 9-session protocol. This pattern may reflect limited engagement among participants without acute treatment needs and is an important consideration in our study of healthy, nontreatment-seeking samples. Fidelity was higher in the ChatGPT group (62%), likely due to several factors. Accessing ChatMind required extra steps, such as obtaining a promo code, which may have discouraged some participants. The structured, solution-focused session format may also have reduced engagement compared with the more flexible ChatGPT experience. Another possible contributing factor could have been differences in overall user-friendliness. Notably, participants with higher baseline depression severity and impairment on the ODSIS scale were more likely to adhere to the full protocol, although this trend did not reach statistical significance. However, this was not the case for higher depression symptom scores measured by the PHQ-9, suggesting greater adherence in those whose symptoms of depression had already begun to interfere with their lives. The relatively rigid structure of the current protocol (3 sessions per wk for 3 wk) may have also contributed to lower protocol adherence. In contrast, real-world applications of AI therapy are likely to be more flexible, allowing individuals to engage when most needed and at a preferred pace. Furthermore, this pilot tested a single therapeutic approach (SFBT), which may not resonate with all users or meet a wide range of needs and preferences. The fixed protocol may have felt repetitive for some participants, further limiting sustained engagement. Looking ahead, AI therapy tools will likely need to emulate the flexibility and adaptability of human therapists by tailoring interactions to patient preferences, symptom severity, and evolving goals to maximize engagement and therapeutic impact. Future feasibility research should focus on clinical, treatment-seeking samples, explore adaptive or flexible treatment models, and further evaluate the optimal number and pacing of AI therapy sessions.

Our study is 1 of the few RCTs to investigate the effectiveness of a generative mental health chatbot. The field of AI is rapidly advancing, and our study focuses on instruments that use the most recent AI models, underscoring its urgency and relevance. To the best of our knowledge, we are the first to compare 2 generative chatbots against a control group, which is the strongest aspect of our study. Furthermore, the study code and data are freely available online so that findings from this study can be easily replicated.

The study also has several limitations. The first one is a high attrition rate, which is a common challenge in short-term CA intervention studies [49]. We addressed this limitation with second participant recruitment, but although the targeted sample size estimated by power analysis was achieved, a larger sample would still likely be needed to detect smaller effects. The second limitation is the intervention length. In addition, the second limitation relates to the nature of the GenAI on which both interventions were based, as it inherently limits control over the conversation flow and content compared to the earlier researched rule or retrieval-based chatbots. However, this also allowed for more naturalistic data and greater ecological validity. The third limitation is a lack of data on the exact length of interventions. We were not able to track the total time participants spent conversing within the AI therapy and ChatGPT groups. Nonetheless, this reflects a common limitation in studies of unsupervised digital interventions. The fourth limitation is that we did not specify in the guidelines that participants should not start any psychological treatment, only that they should not use another chatbot for therapeutic purposes. A final limitation of the present study is the absence of detailed engagement data, such as session duration or use of voice versus text input. Future studies should incorporate more precise tracking of user interaction patterns to better evaluate engagement and adherence.

Future research should investigate the long-term effects of AI-based psychological interventions, as most studies, including ours, assess only short-term outcomes. Extended intervention periods and follow-up assessments are needed to evaluate sustainability and rule out novelty effects. Comparative studies should also explore different AI types, delivery formats (eg, text vs voice), and therapeutic approaches embedded in chatbots.

Regarding implications for practice, our results underscore that not all AI-based mental health tools are equal: therapeutic outcomes may critically depend on how AI is deployed in a particular chatbot, including the use of prompt engineering, conversation design, and alignment with established therapeutic frameworks. For chatbot developers, it will be important to build on development practices as well as psychological foundations that are supported by research, evaluate the effectiveness of specific chatbots through new research, and identify new factors contributing to their effectiveness.

This study evaluated the effectiveness of a structured, generative chatbot rooted in solution-focused brief therapy compared to a general-purpose GenAI chatbot (ChatGPT) and a no-intervention control group over 3 weeks. Both the AI therapy and ChatGPT groups demonstrated a significant reduction in depressive symptoms compared to the control group. These findings support the potential of GenAI interventions for mental health. Further comparative studies are essential to identify the specific design features and therapeutic mechanisms that contribute to the effectiveness of AI-based mental health tools.