This is a retrospective observational study emulating a pragmatic, clustered RCT (see Table 1 for comparison of the specified and emulated target trial design features) via the reporting and analysis of data collected within the medical record system of Kaiser Permanente Northern California (KPNC) and the Recovery Record (RR) mobile ED management program app.

KPNC is a large, integrated health care system serving more than 4.6 million patients (36% of the regional population) insured through commercial, Medicare, Medicaid, and health insurance exchange plans. KPNC patients are highly representative of the ethnic and socioeconomic diversity of the surrounding and statewide population [21].

While individuals who are members of the majority of health plans in the United States typically receive treatment from multiple providers using different medical records, KPNC members receive all health care services exclusively with KPNC providers who use a singular medical record (Epic Systems). This created the unique opportunity to access a complete data set on clinical and health care service utilization outcomes.

All patients using RR consent to the anonymized use of their data for research and quality improvement when they access the software, as per RR’s privacy policy [22]. In addition, the software is fully Health Insurance Portability and Accountability Act compliant. As this paper presents an evaluation of an ongoing health service, no approval of a research ethics committee was obtained. Since the study was classified as a quality improvement study and deemed to be of minimal risk, no approval of a research ethics committee was deemed necessary by KPNC. All data included were deidentified prior to analysis. The research was conducted in accordance with the principles of the Declaration of Helsinki. No compensation was provided to participants.

All included patients received treatment for an ED at 1 of 6 participating KPNC ED treatment programs and were divided into the TAU or ATT group; those in the AAT group were patients who had also successfully linked with a member of their treatment team using the RR app. All patients had received a diagnosis for an ED, per International Classification of Diseases, Tenth Revision (ICD-10) codes in the medical record, and were continuing members of KPNC, to ensure that they met inclusion criteria of having had access to KPNC’s services and complete medical record data for the full run-period of the study (6 months). Members whose health insurance with KPNC was discontinued during the study period were excluded (n=21). Of those continuing members who were identified, we excluded those who were younger than 13 years (n=4); had incomplete data at baseline (n=2666); and those with an ICD-10 code of F98.X, relating to “other behavioral and emotional disorders with onset usually occurring in childhood and adolescence” (n=738). We also excluded those who had a BMI outside the plausible range, that is, less than 8 kg/m² or above 60 kg/m² (n=33).

These comprised 2 large groups: health care service utilization and health outcomes. Health care service utilization measures included encounters relevant to outpatient treatment compliance and retention, namely, outpatient psychiatric visits completed. In addition, we included emergency department visits, with associated ICD-10 ED diagnosis, as a health service utilization measure that may be indicative of escalation of ED acuity. Given that ED often requires medical management, records of outpatient visits with primary care physician including an associated ED diagnosis were also obtained. Health outcomes included changes in BMI, changes in blood pressure (BP), and depressive symptoms, which were measured using the 9-item Patient Health Questionnaire (PHQ-9) [24].

A summary of the baseline characteristics of the study population was conducted, using frequencies and means as appropriate. We tested for differences between AAT and standard of care using chi-square tests for categorical variables and 2-sample t tests for continuous variables.

The primary outcomes included health care service utilization (ie, the number of outpatient psychiatric visits, outpatient primary care visits, and emergency department visits). Secondary outcomes included health outcomes: BMI change and BP (systolic and diastolic) change. An exploratory outcome included depression scores as measured by the PHQ-9.

To address confounding by observed covariates and reduce the selection bias that exists in the absence of randomization, we used inverse probability of treatment weighting (IPW) for estimating causal effects. IPW is a type of weighting method whose goal is to make the observed sample as representative of the population by weighting outcomes according to the inverse of the probability of treatment assignment [25]. Weights are obtained from a treatment selection model, estimated using logistic regression with the treatment assignment as the dependent variables and the baseline characteristics as independent variables, that is, gender, age at ED diagnoses, current age, treatment facility, history of depression, history of anxiety, history of substance abuse, BP, BMI, history of any intensive outpatient admission, any intensive outpatient admission during the study period, and diagnosis type.

The average treatment effect (ATE) was defined as the mean difference between AAT and TAU with respect to number of outpatient and emergency department visits (primary outcome). For the secondary outcome, ATE was defined by the difference in weighted averages of BMI and BP obtained from the treated and TAU groups. For the exploratory outcome of depression scores, the ATE was defined as the mean difference between AAT and TAU with respect to change in PHQ category.

Significance tests and CIs were based on SEs obtained from the bootstrap method (5000 replications) to calculate the variance of the weighted average of outcomes. We used an a value of .05 for significance, and all tests were 2-sided. A complete case analysis was performed on the primary outcomes, meaning that patients who were missing utilization outcomes data were not included in the primary analysis.

The final analytic cohort included 1154 patients who had complete data on pre- and postintervention BMI as well as all demographic variables needed for inclusion in the weighting model. For the analysis on health care utilization, 1104 (AAT: n=165, TAU: n=939) individuals had complete information on outpatient primary care visits, 939 (AAT: n=173, TAU: n=766) individuals had completed outpatient psychiatric visits, and 222 individuals had complete information on emergency department visits. The analysis on health outcomes excluded individuals who had vitals taken less than 30 days apart, resulting in 535 individuals. The exploratory analysis on depression scores excluded individuals who had taken the PHQ-9 fewer than 30 days apart, resulting in 441 individuals. All analyses were performed using R Statistical Software (version 4.1.2; R Core Team 2021).

Baseline characteristics of patient population are shown in Table 2. Patients in the AAT group were younger, with a mean age of 29 (SD 14) years, than those in the TAU group, with a mean age of 38 (SD 18) years (P<.001); they also had a lower mean BMI of 24.1 (SD 8.4) kg/m² compared with 26.8 (SD 9.4) kg/m² (P<.001). More patients in the AAT group were categorized as underweight (P<.001) and having a history of hypotension (P=.004), with a lower mean resting systolic BP (P=.003). In addition, more patients in the AAT group had a history of a depressive disorder diagnosis (P=.05) and anxiety diagnosis (P<.001); engagement with Intensive Outpatient Programs (IOPs; P<.001); and a current diagnosis of AN (P<.001). Conversely, there were more patients in the TAU group with a diagnosis of BN (P<.001) and other EDs (P<.001).

Patients in the AAT group had an average of 28.7 outpatient psychiatric visits compared with patients in the TAU group, who had an average of 18.5 outpatient psychiatric visits, which was a significant difference (treatment effect [TE] 10.2, 95% CI 10.0-10.41; P<.001). Patients in the AAT group had fewer emergency department visits (mean 0.14) than patients in the TAU group (mean 0.50; difference of 0.36, 95% CI –0.55 to –0.18; P<.001; Table 3). There was no significant difference in primary care outpatient utilization, with an average of 4.7 visits in the AAT group and 5.1 visits in the TAU group (P=.30).

Patients in the AAT group experienced a larger increase in BMI after treatment than those in the control group (0.75 compared with 0.31; TE 0.44, 95% CI –0.05 to 0.92; P=.02; Table 4). Furthermore, a significantly higher proportion of patients in the AAT group changed into a lower BMI category compared with patients in the TAU group (TE 0.19, 95% CI 0.05-0.33; P=.01). Notably, patients in the AAT group who were underweight at baseline had nearly a 1 weight category increase (0.89 compared with 0.24 in the TAU group; TE 0.65, 95% CI 0.35-0.95; P<.01). There was a significant difference in resting systolic BP, with the AAT group experiencing a larger change (TE –2.30, 95% CI –2.38 to –2.22; P=.05), although there was no significant difference in diastolic BP (TE –1.29, 95% CI –6.38 to 3.79; P=.11).

Patients in both the AAT group and the TAU group showed significant improvement in depressive symptoms at the end of treatment as measured by the change in raw PHQ-9 scores (pre-post difference of –2.44, 95% CI –3.39 to –1.48 and –1.5, 95% CI –1.72 to –1.24, respectively; P<.001 for both). Moreover, patients in the AAT group showed significantly greater improvement in depressive symptoms than patients engaging in the TAU group (TE –0.96, 95% CI 0.027 to –1.94; P=.03). This was a clinically significant change, with those in the AAT group showing a larger PHQ-9 change in terms of units of category changes compared with TAU patients (difference –0.22, 95% CI –0.417 to 0.015; P=.015).

Our research addresses an important evidence gap regarding the potential of digital therapeutics for EDs to impact health service utilization and health outcomes when implemented in a real-world health system. The results of this study suggest that AAT may increase participation in outpatient ED treatment and result in a significant reduction in disorder-related emergency department visits, which are a large component of associated disease-specific costs [9]. Furthermore, in this study, AAT resulted in significantly and clinically improved health outcomes, with faster weight stabilization in low-weight patients and a greater reduction in depression symptoms.

RR includes several capabilities that automated best practice components of treatment. Patients using the RR mobile app are prompted to complete meal-monitoring entries throughout the day and are systematically introduced to tailored therapeutic coping strategies and clinical goals. The self-monitoring component of the app is provided in an evidence-based CBT format. Meal monitoring is a fundamental component of CBT for EDs and comprises a standard CBT-style question set (what was eaten, with whom, was it sufficient, etc) and some optional additional EDs symptom-focused questions such as urges to engage in compensatory behaviors, current emotional state, compulsive exercise, sleeping patterns, hunger levels, intrusive thoughts, and coping skills used (if any). These questions serve to promote awareness of the nature of the eating problem and identification of contextual and personal triggers that are maintaining disordered behaviors, in addition to the promoters of and barriers to change. Mealtime self-monitoring also creates a structure for routinized eating, which has been found to disrupt dietary restriction and binge eating patterns [26]. In a prior study that included 458 individuals who used the RR app for self-monitoring without any additional support, 35% of users were classified as in remission and 46% achieved a clinically meaningful reduction in ED symptoms, as per the Eating Disorder Examination Questionnaire, after 1 month [27]. Furthermore, a qualitative study on use of the RR app describes how patients reported that it helped them become more aware of their eating habits and feel more in control of their recovery [17].

A major strength of this study is its examination of the real-world effects of the implementation of an ED app on health service utilization and clinical outcomes. This analytic approach was made possible due to a multiorganization collaboration that included academics, researchers at a DTx organization, and clinical researchers from a large HMO. Collaboration with the integrated HMO and integration of the RR app with therapist care created the unique opportunity to access a complete medical record data set on clinical and health care service utilization outcomes, allowing for a retrospective analysis in which a causal TE could be estimated.

Moreover, this study took place within a system that treats racially and diagnostically diverse patients and in which therapists are often tasked with managing large caseloads with constrained resources, while faced with high dropout from outpatient services and a high volume of emergency admissions for ED. Given that the most recent metareviews of the literature on treatment of ED have called out an “urgent need for novel treatments, particularly in AN [28],” the findings from this study shed light on the real-world potential of implementing an innovative digital solution in outpatient and intensive outpatient services, with the ultimate goal of helping patients stick to their evidence-based treatment plans for long enough to achieve clinical outcomes.

Due to the fact that this study was conducted in an observational setting, some baseline differences are due to implementation factors within the HMO. For example, the app was implemented extensively in the IOP during the study period, which explains why the history of IOP is higher among the AAT group. Related, younger patients, as well as patients with AN, were disproportionately represented in the AAT group. It is possible that this difference may be attributed to the greater level of severity or medical complexity typically associated with AN and higher levels of participation of younger patients with AN in intensive treatment programs. Given that an RCT was not feasible in this real-world setting, we used a robust statistical method of IPW to adjust for such baseline between group differences. We explored alternative methodologies such as covariate balancing via propensity score matching, where a treatment subject would be matched to a set of control participants; however, we found that this approach discarded many control participants and thus we ultimately selected a method that would permit use of all available data with the most robust analysis.

While our study provides insight into the potential for digital recovery solutions to be integrated into ED treatment within a real-world health care system, there are limitations to this approach that are worth noting. For example, at the time of the study, KPNC used various partners for residential and partial hospitalization programs with outside vendors that could not be extracted for inclusion in this study. In the absence of available data, we were unable to evaluate the potential impact of AAT on admissions or readmissions to these high-acuity and often high-cost programs. Another limitation of this retrospective research design is the absence of a standardized psychometric measure of ED pathology in the TAU group. Without these data for comparison, we were unable to fully explore the potential impact of AAT on all clinical outcomes of interest. Finally, the study was conducted among a stably insured population within an integrated health care delivery system with a focus on preventive care. In this treatment context, both therapists and patients may have been more likely to have adequate resource to participate in app-augmented therapist–guided care. Thus, the findings may not generalize to the overall US population without similar levels of insurance and access to care.

Future work deriving from this study could include a more in-depth exploration of the demographic and clinical characteristics that influence engagement with and response to the app. Researching the differences in the effectiveness of the app-enhanced program among different age groups, gender, and severity of ED could provide insightful information to clinicians as they consider how to further maximize the benefits of incorporating the technology in their clinical practice. Given the behavioral health outcome findings, further investigation into the AAT’s potential in addressing comorbid mental health conditions associated with EDs such as depression would be of added value. Furthermore, studying the long-term costs and clinical effectiveness would be necessary to gain insights into the effects of the app on maintaining improvement in health outcomes and preventing relapses and readmissions at a sustained cost reduction via lower health care utilization. Finally, it is important to note that not everyone engages with the app at a similar rate, and understanding usage rates and engagement with the different tools and features of the app can help identify to what degree is engagement itself positively associated with health care utilization and clinical outcomes, and how much usage may be needed to gain the benefits suggested by this study.

This study provides evidence that integrating the use of a remote patient engagement app into the treatment plan for patients with ED has beneficial effects. Particularly, AAT led to significant improvements in BMI for underweight patients and in depressive symptoms when compared with conventional treatment methods. The AAT was also associated with fewer emergency department visits and increased participation in outpatient treatment programs. This suggests that digital interventions could play a key role in managing these health conditions and potentially reducing health care service utilization and associated costs. This study serves as a promising point of reference for future research on the long-term efficacy, cost-effectiveness, and potential impact of leveraging digital health interventions to better treat EDs.