This systematic review was registered in PROSPERO (International Prospective Register of Systematic Reviews) on August 2, 2025 (registration number CRD420251072392). The review followed the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) 2020 guidelines and was structured according to the PIO (Population, Intervention, Outcome) framework; the completed PRISMA 2020 checklist is provided in Checklist 1.

.

Multimedia Appendix 1

Studies were selected according to predefined criteria aligned with the research question and the PIO structure:

The bibliographic search was conducted in May 2025 in the following electronic databases: PubMed/MEDLINE, PsycINFO, Scopus, and Web of Science Core Collection. These sources were selected for their relevance to biomedical, psychological, and multidisciplinary fields, ensuring comprehensive coverage of studies on AI and mental health.

For each database, specific search strategies were designed, combining controlled vocabulary terms (MeSH in PubMed and the Thesaurus in PsycINFO) and free-text keywords, adapted to the syntax and Boolean operators of each platform.

The following limits were applied: publications between 2019 and 2025, in English or Spanish. No study design restrictions were applied in the initial search. Additionally, the reference lists of included studies and relevant systematic reviews were manually reviewed to identify additional literature (Table 1).

The selection process was carried out in two phases: (1) initial screening of titles and abstracts and (2) full-text evaluation to determine final eligibility.

Three independent reviewers conducted the screening in parallel (JMG-C, IM-E, and IMG), using the bibliographic management tool Rayyan. Discrepancies were resolved through discussion and, when necessary, with the intervention of a fourth reviewer to reach consensus.

The previously defined inclusion and exclusion criteria (Eligibility Criteria) were applied. The complete process was documented following the PRISMA 2020 guideline and is presented in the Results section (PRISMA diagram), which details the number of identified records, those eliminated due to duplication, those excluded after title/abstract review, and the reasons for exclusion in the full-text phase.

Data extraction was performed independently by 2 reviewers, using a standardized template designed for this review. For each included study, the following variables were collected:

Each study was classified into categories of low risk, moderate risk, or high risk in the evaluated domains. The results were graphically synthesized using stacked bar charts to facilitate the global interpretation of methodological quality.

The initial search identified 583 records in the selected databases. After removing 99 duplicates, 484 titles and abstracts were evaluated, of which 412 were excluded for not meeting the inclusion criteria. Thirty-two full texts were reviewed, excluding 14 studies for the following reasons:

Finally, 18 studies met the criteria and were included in the qualitative synthesis. The complete process is shown in the PRISMA diagram (Figure 1).

The 18 included studies were published between 2021 and 2025, covering diverse geographical contexts (North America, Europe, Asia, and Oceania) and samples ranging from 20 to 3902 participants. Most studies were conducted in clinical, community, or university settings, with adult populations presenting with depressive disorders, anxiety, suicide risk, cognitive impairment, schizophrenia, or PTSD.

Regarding interventions, 2 major categories were identified:

The most commonly used reference instruments were the Patient Health Questionnaire-9 (PHQ-9), Generalized Anxiety Disorder-7 Scale (GAD-7), Social Phobia Inventory (SPIN), suicide risk scales, including the Suicidal Ideation Response Inventory-2 (SIRI-2), structured diagnostic interviews, and cognitive tests. Methodological designs included RCTs, validation studies, cross-sectional studies, qualitative studies, and feasibility pilots, reflecting notable heterogeneity in objectives and metrics.

The findings were grouped into 3 dimensions:

Risk of bias analysis was conducted applying specific tools according to the methodological design of each study. For RCTs, the Cochrane RoB 2 tool was used, while diagnostic accuracy studies were evaluated using QUADAS-2. Qualitative studies were analyzed with the JBI checklist for qualitative research, and mixed methods studies with the MMAT. Finally, cross-sectional studies were evaluated with the JBI checklist for descriptive studies. Each study was classified according to the risk of bias in the different methodological domains, assigning judgments of “low risk,” “moderate risk,” or “high risk.”

The results were visualized using stacked bar charts by domain, facilitating the global interpretation of the methodological quality of the included evidence. Cross-sectional studies presented a higher proportion of moderate bias in sampling strategies and measurement validity. In RCTs, the most frequent bias was observed in the blinding of participants and personnel. Studies evaluated with QUADAS-2 showed moderate bias in participant selection and intervention classification.

Additional per-study risk-of-bias visualizations are provided in Multimedia Appendix 2. Overall, the included studies present heterogeneous methodological quality, with a predominance of moderate bias in critical dimensions of internal validity. This methodological variability should be considered when interpreting the results of efficacy and accuracy of chatbots in psychological assessment.

This systematic review demonstrates that chatbots and AI-based conversational agents achieve clinically relevant performance in mental health screening and assessment. Rule-based systems showed high reliability (Cronbach α>0.85) in administering standardized scales, while generative LLMs achieved sensitivities of 0.84‐0.93 and specificities of 0.80‐0.96 for depression and anxiety. LLMs demonstrated strong correlations with expert clinicians (r values up to 0.96) in suicide risk assessment and exceptional performance when combined with ML (F₁-score=92.1%) in cognitive impairment detection [12]. User acceptability was generally high (≥70%), although barriers existed for older adults and those with lower digital literacy. These findings suggest AI-driven tools can replicate traditional psychometric protocols while offering additional capabilities through natural language processing; however, high-performance metrics must be interpreted cautiously given methodological heterogeneity and moderate risk of bias.

Our findings extend beyond previous systematic reviews, whereas those reviews [9,10] reported promising but limited evidence for chatbot effectiveness in mental health contexts. These reviews focused primarily on therapeutic interventions rather than assessment capabilities. A critical distinction in our review is the emphasis on generative LLMs, which represent a qualitative advance from earlier rule-based systems. Our synthesis captures the rapid evolution from 2023 to 2025, during which LLM-based applications surged from 16% to 45% in mental health chatbot studies [15]. Recent evidence demonstrates that GPT-4 and similar models can pass psychiatric licensing examinations and perform diagnostic reasoning comparable to experienced clinicians [16,17], capabilities not observed in earlier generation systems. Our findings on cognitive impairment detection [12] and suicide risk assessment [21,27] extend the evidence base beyond the depression and anxiety focus of prior reviews.

Included studies used appropriate reference standards, including validated instruments (PHQ-9 and GAD-7) and structured clinical interviews, with several using external validation cohorts. However, significant limitations imply careful interpretation. Risk of bias assessment revealed moderate to high risk in critical domains, particularly participant selection, blinding procedures, and selective reporting. Sample sizes varied considerably (20-3902 participants), and most used cross-sectional designs, precluding longitudinal assessment. Importantly, only 47% of studies focused on clinical efficacy testing, with most LLM-based studies (77%) remaining in early validation phases [15]. Technological advancement has progressed faster than the multistage validation processes necessary for robust clinical evidence, resulting in limited data on long-term effectiveness and real-world implementation. Heterogeneity in reported metrics and a lack of standardized evaluation frameworks complicated cross-study comparisons and precluded meta-analysis for most outcomes.

The evidence suggests several potential applications for AI-driven assessment tools. First, chatbots administering standardized instruments demonstrate equivalence with traditional modes, offering opportunities for remote screening, automated triage, and population scrutiny [35,42]. This could help address treatment gaps in underserved areas and reduce waiting times for initial assessment. Second, LLMs show promise as clinical decision support tools for extracting structured information from clinical narratives [46], identifying at-risk individuals requiring urgent evaluation [27], and augmenting cognitive assessment in resource-limited settings [12].

However, safe implementation requires clear clinical protocols, professional oversight, and integration with existing health care workflows. AI should function as a complementary tool that enhances rather than replaces clinical judgment. Hybrid models combining automated screening with human review for positive or uncertain cases may optimize sensitivity while maintaining specificity. EHR integration is essential for continuity of care, but it requires attention to data security, patient consent, and algorithmic transparency. Critically, the absence of longitudinal studies limits conclusions about sustained accuracy, calibration drift, or adaptation to changing diagnostic criteria.

A major concern emerging from this review is the potential for AI systems to perpetuate or amplify health disparities. Empirical work shows that performance and risk estimates can vary across demographic subgroups, underscoring the importance of bias evaluation and mitigation [28,30]. Adler et al [30] demonstrated that sensed-behavioral patterns predictive of depression vary substantially across demographic subgroups, with AI tools incorrectly ranking individuals with depression from certain groups as lower risk than healthier individuals from other groups. Qualitative analyses reveal that race-explicit or race-implied patient information frequently results in inferior treatment recommendations from LLMs [7], and natural language processing models exhibit measurable biases related to religion, race, gender, nationality, sexuality, and age [31,32].

Moreover, acceptability findings suggest differential engagement by age, education, and technological familiarity, with older adults and those with lower digital literacy reporting lower acceptability [48], potentially expanding access gaps for vulnerable populations. Addressing these inequities requires diverse and representative training datasets, rigorous bias testing across protected characteristics, transparency in algorithmic decision-making, human oversight mechanisms, and continuous monitoring of real-world performance stratified by demographic factors.

Beyond bias, several ethical concerns require attention. The “black box” nature of many LLMs raises questions about accountability when errors occur [49]. Clear governance frameworks delineating roles and liabilities are essential. Informed consent presents unique challenges, as patients must understand when AI is involved in their assessment, how their data will be used, and the limitations of algorithmic predictions. Data privacy and security are paramount, given the sensitivity of mental health information. The risk of overreliance on automated systems could lead to deskilling of clinicians or premature diagnostic closure. AI should augment, not replace, comprehensive clinical assessment that considers contextual factors, patient preferences, and therapeutic relationships. Implementation also faces practical barriers, including EHR integration, evolving regulatory frameworks, underdeveloped reimbursement mechanisms, and the need for clinician training.

Several critical gaps require attention. First, prospective longitudinal studies are needed to evaluate sustained performance, calibration over time, and clinical utility in real-world settings. Second, comparative effectiveness research should directly compare AI-driven assessment against standard care in pragmatic trials, measuring diagnostic accuracy, access, efficiency, and clinical outcomes. Third, research must systematically evaluate performance across diverse populations with prespecified, adequately powered subgroup analyses. Fourth, studies should explore optimal human-AI collaboration models and the required clinician training. Fifth, research on algorithmic fairness and bias mitigation strategies is essential. Sixth, implementation science research should identify facilitators, barriers, and sustainment strategies. Finally, regulatory science research is needed to establish appropriate evaluation frameworks and postmarket surveillance requirements.

Health care systems considering AI adoption should prioritize infrastructure for secure data management, clinical protocols specifying appropriate use cases and oversight requirements, clinician training programs, bias monitoring mechanisms, and patient engagement. Regulatory bodies should develop clear approval pathways requiring evidence of clinical validity, algorithmic transparency, bias testing, and ongoing surveillance. Professional organizations should establish clinical practice guidelines for AI-assisted assessment, addressing appropriate use cases, limitations, documentation requirements, and ethical obligations. Researchers and developers should prioritize open science practices, including sharing algorithms and evaluation frameworks where ethically permissible. Policymakers should ensure that AI deployment does not exacerbate existing disparities and that vulnerable populations maintain access to human-delivered care.

AI-driven conversational agents demonstrate clinically relevant performance in mental health screening and assessment, with potential to expand access and to support clinical decision-making. However, realizing this potential requires addressing substantial methodological, ethical, and practical challenges. The evidence base remains limited by heterogeneity, short-term follow-up, and moderate risk of bias. Algorithmic bias poses serious threats to health equity that demand proactive mitigation. Implementation requires careful attention to clinical integration, human oversight, transparency, and patient engagement. The goal should not be to replace human clinicians but to develop complementary tools that enhance efficiency, access, and quality while preserving the essential human elements of empathy, contextual understanding, and ethical judgment. A collaborative model integrating AI capabilities with human expertise may offer a path toward more accessible, timely, and equitable mental health assessment. However, this vision requires sustained research, thoughtful regulation, and a commitment to equity as technological capabilities continue to evolve.