Patient Satisfaction and Recommendations for Delivering a Group-Based Intensive Outpatient Program via Telemental Health During the COVID-19 Pandemic: Cross-sectional Cohort Study

Introduction

The COVID-19 pandemic has led to increased demand for mental health services worldwide, and most countries are reporting significant disruptions to the delivery of critical mental health services [1]. Early evidence suggests that symptoms of anxiety, depression, and self-reported stress were common responses to COVID-19 in the general population [2]. Concerns that suicide rates during and after the pandemic might increase have been highlighted [3], though data are still limited on the rates and risk of suicide in the context of the current pandemic. Certain populations, such as those with serious mental illness (SMI), may be particularly vulnerable to the stressors and hardships related to COVID-19. Thus, it is pertinent to ensure adequate access to behavioral health services during this pandemic, particularly for adults with SMI.

The COVID-19 pandemic has created significant obstacles to the delivery of mental health services, especially for services delivered in a group setting due to the need for social distancing. However, maintaining access to group-based interventions is essential given their efficiency in treatment delivery to a larger population when resources are limited. Telemental health (TMH), defined as the delivery of mental health care services at a distance through the use of information and telecommunications technology, has emerged during the COVID-19 pandemic as an essential platform to ensure continuous mental health care delivery. TMH has been shown to be highly effective and increases access to care [4]. It has been shown to be an effective mode of health care delivery across different patient populations, diagnoses, and settings, including group interventions [5-7]. The COVID-19 Federal Emergency Order temporarily lifted several administrative barriers to TMH, allowing for its expanded use during the pandemic [8]. As a result, TMH services have been increasing substantially in the wake of COVID-19, with the veterans administration reporting a 500% increase in TMH use in the early stages of the pandemic [9]. Initial TMH studies during the pandemic have shown increased utilization and decreased no-show rates [10]. Though TMH has provided essential mental health care during this time, questions remain regarding how different populations accept and respond to TMH interventions. A study of patient satisfaction related to TMH services during the perinatal period showed that a majority of participants indicated that TMH improved their health care access and that the visit was as effective as in-person visits [11]. Understanding patient satisfaction and engagement with TMH interventions is crucial to the sustainability of TMH programs both during and beyond the pandemic.

Understanding patients’ perspective on the quality of behavioral health services delivered via telehealth is important to ensure their engagement with treatment and to improve outcomes. Several pre–COVID-19 studies indicated that patients had a positive perception toward telehealth and were satisfied with the delivery format [12]. Although the literature is still limited, studies are also finding high patient satisfaction with telehealth programs developed during the pandemic [13,14]. Emerging research during this pandemic were consistent with previous findings indicating that patients were satisfied with the option to continue behavioral health services via telehealth. Most of this research, however, has focused on individual outpatient behavioral health services. A gap in the literature exists on patient satisfaction for group-based intensive outpatient programs (IOP) delivered via telehealth during the pandemic.

The aim of this study was to evaluate patient satisfaction while exploring future recommendations of a group-based IOP for adults with SMI, which was rapidly transformed to a telehealth format during the COVID-19 pandemic. The results from this study can be used to improve the quality of programming and enhance the delivery of services in the future.

Methods

The protocol for this cross-sectional cohort survey research was approved by the Mayo Clinic Institutional Review Board. Data were collected as part of clinical care at the Adult Transitions Program (ATP), a group-based IOP within the Mayo Clinic Department of Psychiatry and Psychology. This program was intended to treat adults with SMI who were recently discharged from psychiatric hospitalization or were at risk of psychiatric hospitalization if not treated in a more intensive level of outpatient care. Inclusion criteria for the present study were patients who were admitted to ATP, were at least 18 years old, and consented for their clinical data to be used for research purposes. The patients completed the satisfaction survey over the phone with research personnel after they were discharged from the program. The phone call took approximately 15 minutes to complete.

ATP was delivered by a multidisciplinary team that included psychologists, a psychiatrist, nurse practitioners or physician assistants, licensed professional clinical counselors, occupational therapists, and registered nurses. The patients received the program 5 days per week, 3 hours a day, for a 3-week period. The programming was mainly group-based and informed by evidence-based cognitive and behavioral interventions such as Behavioral Activation [15], dialectical behavioral therapy (DBT) [16], and acceptance and commitment therapy [17]. The patients were assigned to 1 of the 3 tracks, with 8 patients in each track. The inclusion criteria for the program were adults aged 18 years and older, who were diagnosed with SMI (eg, mood disorders, anxiety disorders, psychosis, personality disorders, and substance use), who had recent psychiatric hospitalization or were at risk for psychiatric hospitalization, and who reported having access to a mobile or computer device to connect to the video teleconference software (ie, Zoom). The exclusion criteria were cognitive impairment and higher symptom severity that did not require a higher level of care with a psychiatric hospitalization or residential settings.

The patient satisfaction questionnaire was developed through a literature review. Some items were generated based on the acceptability of intervention measure, intervention appropriateness measure, and feasibility of intervention measure by Weiner and colleagues [18]. These original measures have Cronbach alphas from .85 to .91, and test-retest reliability coefficients ranged from 0.73 to 0.88. The research team generated and reviewed the initial items, and the suggested changes included adding and removing certain questions and improving grammatical errors and wording. The research team members took each iteration of the survey to ensure the readability of the content items. The final version of the Patient Satisfaction Questionnaire (Multimedia Appendix 1) included 14 quantitative questions answered on a Likert-type scale from 1 to 5 with the higher numbers indicating higher satisfaction. Three open-ended questions assessed the patients’ overall experience with TMH, the most valuable part of the TMH format, and recommendations for future program improvement. In addition, demographic variables were pulled from the electronic health record.

Descriptive statistics were generated to identify the most commonly endorsed items. The open-ended questions were analyzed using summative content analysis [19]. Keywords were identified and quantified to characterize the themes that emerged from the 3 open-ended questions. Two researchers independently read the qualitative responses multiple times to identify the keywords. These keywords were then sorted into categories, and the themes were then quantified using frequency counts. The 2 researchers compared emerging categories for validation purposes.

Results

A total of 76 patients were admitted to the program between March and August of 2020. Of the 76 patients admitted to the program, 40 (53%) completed the survey over the phone with research personnel. The referral source and track attended for those who did and did not complete the survey were similar. The referral source for completers versus noncompleters, respectively, was as follows: inpatient, 42.5% versus 35%; emergency department, 2.5% for both groups; primary care, 30% versus 27.5%; other outpatient programs, 15% versus 32.5%; and other programs, 10% versus 2.5%. The track attended for the completers versus noncompleters, respectively, were as follows: cognitive behavioral therapy morning 25% versus 30%; DBT morning 30% versus 42.5%; and DBT afternoon 45% versus 27.5%. The patients had a mean age of 36.55 (SD 13.43) years. The majority of the patients were female (n=32, 80%) and White (n=33, 82.5%), married (n=14, 35%) or single (n=23, 57.5%), cisgender (n=38, 95%), heterosexual (n=30, 75%), and employed (n=23, 57.5%). The patients had the following psychiatric diagnoses as a primary presenting problem: major depressive disorder (n=29, 72.5%), anxiety disorder (n=2, 5%), borderline personality disorder (n=6, 15%), and suicidal ideation (n=2, 5%). Full baseline characteristics are reported in Table 1.

The complete results for the quantitative portion of the satisfaction survey are presented in Table 2. Overall, the majority of patients reported high satisfaction, comfort, appropriateness, relevance, and compatibility of the TMH format of ATP. Most patients (92.5% [n=37]) reported that they would recommend this service format to a friend or family member. They noted that the TMH format was well organized and executed, user friendly, and not burdensome. We also assessed preference between in-person versus a TMH format. We found a split among the patients where 35% (n=14) preferred to receive an in-person format, 50% (n=20) preferred continuing with a TMH format, and 15% (n=6) were neutral when asked, “Once COVID-19 travel restrictions are lifted, would you still want to continue with video format?” (Table 2).

We also assessed to what extent patients experienced technological difficulties with the TMH format. A portion of the patients (46.15% [n=18]) reported experiencing challenges during the program. We analyzed the qualitative open-ended responses and reported that challenges included problems with slow internet connection, the video camera of their devices, logging into the teleconference room, and being inadvertently removed from the session.

We conducted content analyses of the qualitative questions and extracted themes from each question. The frequency counts for the categories within each question are presented in Table 3. Examples of the qualitative feedback are presented in Table 4.

Regarding the patients’ overall perception of the TMH program, they provided both positive feedback and challenges that they encountered. The patients provided overall positive attitudes toward the TMH format. They noted that TMH provided easier access to treatment and that treatment was effective and beneficial to learning skills and coping with their problems. Some individuals also reported that TMH increased social support during the pandemic. These findings are similar to those found by Ackerman et al [11], which showed increased satisfaction with TMH. Others noted challenges of this delivery format, which included experiencing technological issues, with one patient reporting an overall negative experience with the program. Some patients (18% [n=7]) also expressed preferences to receive services in-person rather than via TMH.

We asked the patients to identify the most valuable part of the program. More than half of the patients stated that they found the convenience of TMH as valuable, with others reporting the benefits from social support and the adequate learning skills to cope with their presenting problems.

Most patients did not provide further recommendations to improve the TMH program format. Some suggested improvements on the TMH delivery process, such as using more features on Zoom. Others suggested that the therapeutic delivery process and materials could be improved. One patient suggested that we offer the in-person format again once the pandemic is over.

Discussion

Principal Findings

Prior to the COVID-19 pandemic, very little information existed in the empirical literature on how to rapidly convert group-based IOPs to a TMH format. This study assessed the acceptability of a group-based IOP delivered via TMH during the COVID-19 pandemic. Our data show that patients were satisfied with the TMH ATP, and IOP, with most reporting that they would recommend these services to a friend or family member. When asked to describe their preference, most patients preferred to continue the TMH format during the pandemic and beyond. These results demonstrate that a “hybrid” model of care, which allows for both approaches (depending upon the patient’s choice and availability of stable internet services in their area) may be a viable alternative. Common technological difficulties experienced by patients included slow or unstable internet connections, malfunctioning cameras, and log-in difficulties. However, for most patients, these technological difficulties did not negatively affect their experience with the program. TMH services are important in reaching patients that are geographically distanced from mental health facilities. It is important to recognize that the infrastructure for stable internet connections within communities and access to devices that can facilitate this type of treatment play a role in who can access TMH.

Content analyses of qualitative data suggest that the patients were willing to effectively address technological problems in the spirit of accessing convenient, in-home services that reduce the risk of health care-associated infections during the COVID-19 pandemic. Further, patients noted that the TMH format facilitated the acquisition of evidence-based coping skills and engendered a sense of social connection despite ongoing social and physical distancing measures. These findings suggest that TMH IOPs are sustainable and acceptable to adults with SMI. Moreover, mental health systems should consider offering both TMH and traditional in-person services to best meet the needs of patients with diverse preferences, technologic capabilities, and learning needs regardless of the state of the pandemic.

The lack of patient-identified quality improvement recommendations is likely due to the high degree of satisfaction reported by the overall sample. Start-point recommendations offered by respondents included expanding platform features (eg, using the virtual whiteboard), improving the use of program handouts (eg, sending documents virtually) and maintaining the availability of in-person IOPs for those who prefer face-to-face treatment.

Limitations

This study used the data gathered through convenience sampling, which limits the generalizability of our findings to other populations. Although TMH IOPs may be helpful for a large proportion of adults with SMI, not all clinics or programs may be prepared to provide such services. This study was performed at a large clinical and academic center with previous experience with telehealth programming. There was also significant administrative and information technology support available, which limits the generalizability of our findings to other clinics. Additionally, to determine patient satisfaction, we used selected items from established measures of acceptability of interventions, which may have influenced internal consistency. Furthermore, the findings may contain positive bias given that not all patients completed the satisfaction survey. Lastly, our sample lacked a comparison, in-person group, and was limited in terms of racial and ethnic diversity. This sample was also limited to those patients who had sufficient technologic knowledge, skills, and resources (eg, high-speed internet, smartphone, and computer) to engage in the TMH platform. Subsequent research should aim to report TMH IOP outcome data, ideally across a broader range of patient characteristics. Despite these limitations, the findings detailed here reinforce the benefits of delivering TMH IOPs during public health emergencies and contribute to the sparse literature available on real-world program adaptations.

Conclusions

The COVID-19 pandemic led to the rapid adoption of TMH services across mental health systems. Our findings indicate that TMH IOPs are feasible and can be an effective, safe, and convenient treatment framework for adults with SMI. High satisfaction with TMH IOP delivery and content can be achieved without compromising ongoing social and physical distancing measures. Additional research is needed to assess the efficacy of TMH IOPs in treating mental health concerns.