JMIR Ment Healthmental16JMIR Mental HealthJMIR Ment Health2368-79595688610.2196/56886Impacts of Telehealth Adoption on the Quality of Care for Individuals With Serious Mental Illness: Retrospective Observational Analysis of Veterans Affairs Administrative DataCummingsCamillaPhD1*RajaPushpaMSHPM, MD2*GabrielianSonyaMPH, MD13*DoranNealPhD4*Veterans Affairs Greater Los Angeles Healthcare System, Los AngelesCA, United StatesDepartment of Mental Health, Veterans Affairs Greater Los Angeles Healthcare System, Los AngelesCA, United StatesDavid Geffen School of Medicine, University of California, Los AngelesCA, United StatesOffice of Mental Health, Veterans Affairs San Diego Healthcare System, San DiegoCA, United StatesTorousJohnRotondiArmandoTepperMiriam CCorrespondence to Sonya Gabrielian, MPH, MDsgabrielian@mednet.ucla.edu

all authors contributed equally

202497202411e56886290120242604202427042024© Camilla Cummings, Pushpa Raja, Sonya Gabrielian, Neal Doran. Originally published in JMIR Mental Health (https://mental.jmir.org), 9.7.2024. 2024

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Mental Health, is properly cited. The complete bibliographic information, a link to the original publication on https://mental.jmir.org/, as well as this copyright and license information must be included.

Background

Telehealth implementation can be challenging for persons with serious mental illness (SMI), which may impact their quality of care and health outcomes. The literature on telehealth’s impacts on SMI care outcomes is mixed, necessitating further investigation.

Objective

We examined the impacts of facility-level telehealth adoption on quality of care metrics over time among patients with SMI.

Methods

We analyzed Veterans Affairs (VA) administrative data across 138 facilities from January 2021 to December 2022. We performed longitudinal mixed-effects regressions to identify the relationships between the proportion of facility-level telehealth visits and SMI specialty care quality metrics: engagement with primary care; access and continuity of care across a range of mental health services including psychotherapy or psychosocial rehabilitation, SMI-specific intensive outpatient programs, and intensive case management; and continuity of mental health care after a high-risk event (eg, suicide attempt).

Results

Facilities with a higher proportion of telehealth visits had reduced access and continuity of physical and mental health care for patients with SMI (P<.05). Higher telehealth adoption was associated with reduced primary care engagement (z=−4.04; P<.001), reduced access to and continuity in SMI-specific intensive case management (z=−4.49; P<.001; z=−3.15; P<.002), reductions in the continuity of care within psychotherapy and psychosocial rehabilitation (z=−3.74; P<.001), and continuity of care after a high-risk event (z=−2.46; P<.01). Telehealth uptake initially increased access to intensive outpatient but did not improve its continuity over time (z=−4.47; P<.001). Except for continuity within SMI-specific intensive case management (z=2.62; P<.009), continuity did not improve over time as telehealth became routinized.

Conclusions

Although telehealth helped preserve health care access during the pandemic, telehealth may have tradeoffs with regard to quality of care for some individuals with SMI. These data suggest that engagement strategies used by SMI-specific intensive case management may have preserved quality and could benefit other settings. Strategies that enhance telehealth implementation—selected through a health equity lens—may improve quality of care among patients with SMI.

telemedicinequality of careserious mental illnesstelehealthadoptionmental illnesspatientspatientveteransveteranpsychotherapypsychosocialmental healthcaresuiciderehabilitationmental health careIntroduction

Telehealth accelerated during the COVID-19 pandemic and is likely to persist. Although telehealth is accepted by most psychiatric patients, individuals with primary or co-occurring serious mental illness (SMI; ie, psychotic spectrum disorders and bipolar disorders) show lower engagement [1,2] and are less likely to use video-based services [3,4] than other diagnostic groups. A systematic review found that individuals with (vs without) SMI use telehealth at lower rates, despite much higher in-person mental health care (MH) use among individuals with SMI [2]. A study of patients with SMI who use telehealth during the pandemic found that, although many patients with SMI maintained engagement, individuals with schizophrenia were the least likely to engage in telehealth [3]. Despite lower telehealth engagement among patients with SMI (defined variably in the literature, but often equated to the aforementioned diagnostic categories [5]), few have evaluated the impact of telehealth expansion on this population’s quality of care.

Literature on telehealth’s impacts on care outcomes for individuals with SMI is mixed. Prepandemic, county-level research linked telehealth uptake to modest increases in the proportion of patients with SMI receiving minimum levels of MH. Though telehealth resulted in a greater likelihood of posthospitalization follow-up, no changes in medication adherence and slight increases in acute care use were seen [6]. During the first 6 months of the pandemic, rapid telehealth growth was associated with lower MH use among individuals with SMI [7]. A recent study of specialty mental health practices with higher telehealth use found that Medicare patients with SMI had more visits per year, but no differences in medication refills, postpsychiatric hospitalization follow-up, or all-cause mortality [8].

The Department of Veterans Affairs (VA), which serves more patients with SMI than any other US health care system, is an opportune setting to examine telehealth’s impact on quality of care. VA maintains a comprehensive performance measurement system for outcome monitoring and quality improvement at national and facility levels [9]. This report analyzes national VA administrative data to characterize the impacts of facility-level telehealth adoption on SMI performance metrics, and interaction effects on quality metrics over time as telehealth was routinized.

MethodsOverview

National VA administrative data (from Corporate Data Warehouse) were used to extract all outpatient visits (across specialties) with SMI-specific International Statistical Classification of Diseases, Tenth Revision (ICD-10) codes as primary or secondary diagnostic codes. In total, 6 of 8 SMI-specific VA performance measures were examined; 2 measures had an expanded definition of SMI, encompassing visits with ICD-10 codes for severe depression and posttraumatic stress disorder. Visits were stratified by modality (video telehealth vs face to face) and quarter from January 2021 to December 2022.

SMI-related performance metrics were collated across 138 VA facilities. Eight measures assessing access to care and care continuity for individuals with SMI were examined: (1) engagement with primary care (PC; ie, assigned to PC provider and ≥1 PC visit within the last year); (2) access to psychotherapy or psychosocial services (ie, proportion of patients with SMI who visited ≥1 psychotherapy or psychosocial rehabilitation within the last year); (3) continuity (ie, 5 psychotherapy or psychosocial rehabilitation visits within 10 weeks); access to care within 2 SMI specialty programs: (4) intensive outpatient programs (IOPs; ie, ≥3 visits in the past year); (5) intensive case management (ICM; VA’s brand of Assertive Community Treatment; ie, ≥5 visits in the past year); continuity in these SMI specialty programs, with (6) ≥3 IOP visits over 90 days and (7) ≥12 ICM visits over 90 days; and (8) continuity of MH after a high-risk event like psychiatric hospitalization or suicide attempt (ie, ≥1 visit per 6 months over the past year).

Stata (version 15.0; StataCorp LLC) was used to perform longitudinal mixed-effects regressions evaluating whether facility unstandardized scores changed in relation to total volume of SMI telehealth visits. We controlled for facility size approximated by facility-level volume of SMI visits and time (as telehealth was routinized after the initial pandemic emergency). Separate models were fit for each measure. Models initially included linear and quadratic time to assess whether scores on quality measures varied across the 2-year period. Initial models included all 2-way interactions between linear and quadratic time and total SMI visits and telehealth SMI visits to evaluate whether associations between SMI visit volume and quality measures changed over time. Nonsignificant interactions were not retained.

Ethical Considerations

Procedures were approved by the VA San Diego Institutional Review Board (V201253). As an operations/quality improvement project, these data were collected in routine clinical care. Informed consent was not required, and no compensation was provided.

Results

No interactions were observed between time and SMI visit volume, and interactions were omitted. Table 1 displays our findings. In total, 7 of 8 measures (all but access to psychotherapy) were significantly associated (P<.05) with total SMI volume, telehealth SMI volume, or both.

Four measures were positively associated with facility total SMI volume but negatively associated with facility telehealth volume, including continuity of psychotherapy or psychosocial care (z=−3.74), PC engagement (z=−4.04), and ICM program access and continuity (z=−4.49; z=−3.15). Access to IOPs was negatively associated with total SMI visit volume but positively with telehealth volume. Total SMI visit volume was negatively associated with continuity of MH after high-risk events (z=−2.46) and positively associated with continuity of IOP (z=2.24); neither measure was associated with telehealth visit volume.

In total, 6 of 8 measures changed over time. Access to psychotherapy, PC engagement, and continuity of MH after high-risk events were positively associated with linear time (z=4.39 or 2.54 or 8.18) but negatively with quadratic time (z=−3.78 or −2.34 or −6.37); these 3 measures demonstrated an inverted U shape over time, with initial improvement followed by a decline back toward baseline. Continuity of psychotherapy (z=−2.99) and IOP (z=−4.47) declined significantly and linearly over time, whereas continuity of ICM (z=2.62) exhibited a linear increase over time.

Summary of models testing associations between SMIa visits by modality and unstandardized scores on SMI-related quality measures across 138 Veterans Affairs facilities nationwide. z scores greater than 0 indicate a positive association between the quality measure and the time- or SMI visit volume–based predictor; z scores less than 0 reflect a negative association.

Quality measureDefinitionLinear timeQuadratic timeTotal SMI visitsbTelehealth SMI visits
z scoreP valuez scoreP valuez scoreP valuez scoreP value
Access to psychotherapy≥1 visit within past year4.39<.001−3.78<.001−0.10.920.46.64
Continuity of psychotherapy≥5 visits within 10 weeks past year−2.99.003—c—6.15<.001−3.74<.001
Access to primary careAssigned primary care team +≥1 primary care visit past 12 months2.54.01−2.34.022.12.03−4.04<.001
Continuity of mental health care after high risk event≥1 visit per 6 months in past year8.18<.001−6.37<.001−2.46.010.46.65
Access to ICMd≥5 visits within past year−0.30.76——2.22.03−4.49<.001
Continuity of ICM≥12 visits within 90 days2.62.009——3.88<.001−3.15.002
Access to IOPe≥3 visits past year−0.46.65——−4.62<.0012.66.008
Continuity of IOPf≥3 visits within 90 days−4.47<.001——2.24.02−1.58.11

aSMI: serious mental illness.

bSMI encompassing schizophrenia and other psychotic disorders and bipolar disorders.

cNot applicable.

dICM: intensive case management.

eIOP: intensive outpatient program.

fMeasures for access to and continuity of IOP include a broader definition of SMI including severe depressive disorders and posttraumatic stress disorder.

DiscussionPrincipal Findings

In VA facilities with higher telehealth adoption, patients with SMI had diminished engagement with PC, lower access and care continuity in ICM programs, and reduced psychotherapy or psychosocial continuity. Although telehealth uptake initially increased access to IOP care for patients with SMI, IOP continuity was not improved. Larger sites, measured by total SMI volume, performed better in PC engagement, access and continuity for ICM, and continuity of psychotherapy, adjusted for telehealth use.

Higher telehealth adoption at the facility-level may have reduced access to physical and MH for patients with SMI. Although telehealth helped preserve health care access during COVID-19, these findings suggest potential tradeoffs in quality metrics for individuals with SMI, particularly in PC engagement and continuity of care in psychotherapy or psychosocial rehabilitation settings and ICM programs. Though others report that practice-level telehealth use was associated with more visits per year for individuals with SMI [8], a subset of patients with SMI may require more robust supports (eg, from peers or case managers) to use telehealth without reducing access. Training and technical support may be necessary to overcome barriers such as hardware and software needs, broadband access, technological illiteracy, and condition-related aversions to technology mediated communication (eg, delusions) [10,11].

Access to primary and MH as well as continuity of MH after a high-risk event initially improved but returned to baseline over time. Continuity of psychotherapy and IOP declined over time. These findings suggest quality metrics did not improve as service providers and patients habituated to telehealth. Perhaps flexibility with modality and platform decreased as telehealth was routinized, or patients open to telehealth as a temporary measure were less open to it long term. The only metric to improve over time was continuity of SMI-specific ICM, suggesting more intensive strategies (eg, frequent in-person case management visits) may have facilitated successful engagement. Such strategies could be scaled to support telehealth implementation for SMI populations in other settings.

In addition to condition-specific barriers to care, individuals with SMI face social and economic inequities, including lower socioeconomic status, education, employment, and higher rates of homelessness, affecting their ability to engage in telehealth [12]. Access and continuity of care are crucial for this group because of their impact on health outcomes and mortality [13]. Our data suggest that reliance on telehealth may exacerbate disparities in health care access and quality, raising concerns about health equity for this vulnerable population.

These analyses are limited by a focus on VA, which differs from community systems in services and patient population. VA’s ICM and IOP programs have programmatic elements and performance metrics that may differ from other settings. These analyses used operations data intended for quality improvement; facility-level characteristics (eg, patient complexity) that could explain associations were not available. Future research could explore the impact of characteristics related to facility (eg, urbanicity or rurality), patients (eg, average age or facility), organization (eg, staffing), and provider attitudes regarding telehealth. Such investigations might assess the extent to which quality of care is impacted by telehealth enabling access to individuals previously disengaged from services altogether.

Conclusions

These findings raise concerns about facility-level impacts of telehealth adoption on quality of care and service engagement for individuals with SMI. Though telehealth is effective for persons with SMI [14], it may contribute to quality gaps for some individuals with SMI, negatively impacting facility-level quality metrics. Strategic telehealth supports are likely needed to reduce inequities for individuals with SMI, such as telehealth literacy screening [15], telehealth skills training [16], or in-person support to enable telehealth access. Given existing health disparities among individuals with SMI [10,12], centering health equity in the identification of implementation supports is warranted to ensure equitable and effective service provision for this vulnerable population.

None declared.

AbbreviationsICD-10

International Statistical Classification of Diseases, Tenth Revision

ICM

intensive case management

IOP

intensive outpatient program

MH

mental health care

PC

primary care

VA

Veterans Affairs

ReferencesMatthewsEB LushinV RzewinskiJ Patterns & predictors of telehealth utilization among individuals who use substances: implications for the future of virtual behavioral health servicesCommunity Ment Health J20240160114815310.1007/s10597-023-01166-237410213NeumannA KönigHH BokermannJ HajekA Determinants of patient use and satisfaction with synchronous telemental health services during the COVID-19 pandemic: systematic reviewJMIR Ment Health2023081810e4614810.2196/4614837594785BareisN TepperMC WangR Engagement of individuals with serious mental illness in outpatient mental health services and telehealth use during the COVID-19 pandemicPsychiatry Res20231132911549710.1016/j.psychres.2023.11549737778232RajaPV GabrielianS DoranN Access to care for veterans with serious mental illness during the COVID-19 pandemicPsychiatr Serv202111172111324132710.1176/appi.ps.20200089834030456VHA Handbook 1163.06Department of Veterans Affairs20162023-10-25https://www.va.gov/vhapublications/ViewPublication.asp?pub_ID=3164#:~:text=National%20Psychosis%20Registry.&text=The%20NPR%20is%20an%20ongoing,from%201988%20to%20the%20presentWangB HuskampHA RoseS Association between telemedicine use in nonmetropolitan counties and quality of care received by medicare beneficiaries with serious mental illnessJAMA Netw Open202206156e221873010.1001/jamanetworkopen.2022.1873035759264BuschAB HuskampHA RajaP RoseS MehrotraA Disruptions in care for medicare beneficiaries with severe mental illness during the COVID-19 pandemicJAMA Netw Open202201451e214567710.1001/jamanetworkopen.2021.4567735089352WilcockAD HuskampHA BuschAB Use of telemedicine and quality of care among medicare enrollees with serious mental illnessJAMA Health Forum2023106410e23364810.1001/jamahealthforum.2023.364837889483LemkeS BodenMT KearneyLK Measurement-based management of mental health quality and access in VHA: SAIL mental health domainPsychol Serv20170214111210.1037/ser000009728134552KopelovichSL Monroe-DeVitaM BuckBE Community mental health care delivery during the COVID-19 pandemic: practical strategies for improving care for people with serious mental illnessCommunity Ment Health J20210457340541510.1007/s10597-020-00662-z32562033ClareCA Telehealth and the digital divide as a social determinant of health during the COVID-19 pandemicNetw Model Anal Health Inform Bioinform2021041012610.1007/s13721-021-00300-y33842187ShinnAK VironM Perspectives on the COVID-19 pandemic and individuals with serious mental illnessJ Clin Psychiatry2020042881320com1341210.4088/JCP.20com1341232369691JacobsR AylottL DareC The association between primary care quality and healthcare utilisation, costs and outcomes for people with serious mental illness: retrospective observational studyHealth Serv Deliv Res20200682510.3310/hsdr08250ChakrabartiS Usefulness of telepsychiatry: a critical evaluation of videoconferencing-based approachesWorld J Psychiatry201509225328630410.5498/wjp.v5.i3.28626425443GillieM AliD VadlamuriD CarstarphenKJ Telehealth literacy as a social determinant of health: a novel screening tool to support vulnerable patient equityJ Alzheimers Dis Rep20220261677210.3233/ADR-21002435360275CamachoE TorousJ Impact of digital literacy training on outcomes for people with serious mental illness in community and inpatient settingsPsychiatr Serv202305174553453810.1176/appi.ps.2022020536164771