Working alliance is often conceptualized as a tripartite construct comprising agreement on therapeutic tasks and goals, as well as the bond between patients and therapists [1]. It has been identified as one of the most robust factors contributing to therapeutic change, with higher working alliance often associated with better treatment outcomes [2]. These effects have been consistently present in in-person psychotherapy as well as synchronous teletherapy, both of which feature direct, face-to-face interactions between clinicians and patients in real time [3]. However, the accessibility of in-person psychotherapy or synchronous teletherapy is limited by the shortage of clinicians, scheduling issues, difficulties with finding therapeutic space for both patients and therapists, transportation challenges (for in-person therapy), instability of internet connection (for synchronous teletherapy), perceived stigma of psychotherapy, and the financial cost of treatment. These barriers to in-person or synchronous teletherapy are especially salient during times of public health crisis, such as the COVID-19 pandemic, when the need for mental health services has surged despite limited supply.

Technology innovations have offered alternative options such as internet-based interventions (IBIs), which provide accessible mental health services that do not require synchronous communication. An unguided IBI, which is characterized by the delivery of a web-based therapeutic program with no support from clinicians, is an example of such interventions. Many unguided IBIs, also known as self-help programs, are based on cognitive behavior therapy (CBT) principles and involve components of psychoeducation, behavioral and cognitive practice, homework, and tracking-related activities for important variables (eg, mood and behaviors) [4,5]. In contrast to the absence of clinician involvement in unguided programs, guided IBIs often include a self-help program and asynchronous support from clinicians. Such integrated interventions are thought to improve treatment outcomes by leveraging the benefits of the therapeutic relationship between patients and clinicians. Clinicians in guided IBIs often provide low-intensity clinical guidance to facilitate the patient’s independent work with the self-help program [6]. Empirical studies have shown effectiveness for both types of IBIs in treating issues such as anxiety, depression, and traumatic stress for a wide range of populations [3,7,8], supporting their flexible use in situations where in-person communication or real-time telecommunication may be limited. However, what remains unclear is the nature and function of working alliance in guided and unguided IBIs.

Previous studies have focused on understanding the level, trajectories, and outcome associations of working alliance in IBIs. A high level of working alliance has been reported by patients across guided IBIs with varying degrees of clinician involvement and different communication modalities [3,9], as well as across unguided programs [10]. Nonetheless, very few studies have directly compared the levels of working alliance between guided and unguided IBIs [11]. Such direct comparisons are needed to clarify (1) the nature of the relationships that patients have established with unguided intervention programs and (2) whether guided IBIs are able to improve outcomes by leveraging the benefits of working alliance with additional clinician support compared with unguided programs [12]. Information regarding these questions will help to determine the contexts in which additional clinician support is needed to improve the outcomes and delivery of IBIs.

However, a conceptual question arises regarding whether we can quantitatively compare working alliance between guided interventions and unguided interventions. Most previous studies have directly taken measures of working alliance from studies of face-to-face therapy with minor adaptations to IBI contexts (eg, replacing the word therapist with the word program in items to refer to the relationship with therapeutic programs and clinicians together). The potential differences of working alliance in various IBI contexts remained unexamined in most cases (although there are exceptions [13-15]). The conceptual meaning and interpretation of working alliance for patients may be different for unguided interventions versus guided interventions; unless we verify that working alliance has the same conceptual meaning across different IBI contexts, quantitative comparisons of working alliance across IBIs are meaningless. The measurement invariance framework using multigroup confirmatory factor analysis [16] provides a strong methodological tool to examine the conceptual equivalence of working alliance across contexts. Therefore, the first aim of the study was to examine the measurement invariance of working alliance across guided versus unguided IBIs.

In addition to the potential differences in working alliance between guided and unguided interventions, the nature of working alliance may also vary for guided IBIs with different communication modalities. Clinician support can be delivered through video-based messages or text messages, both of which may have unique impacts on the development of working alliance. For example, video-based support is hypothesized to better facilitate the development of working alliance than text-based support because it allows for visual messages with facial expressions that facilitate nonverbal communication and relational bonding [3]. By contrast, text-based support may allow participants to develop more thorough, in-depth responses through words, which can facilitate the establishment of working alliance by enhancing deep emotional processing [3]. Understanding the impact of different communication modalities can help to improve the design of IBIs and maximize the influence of clinician support on the therapeutic process. Therefore, in this study, we aimed to examine the measurement invariance among (1) an unguided IBI (U-IBI), (2) a guided IBI with text-based clinician support (G-IBI-Text), and (3) a guided IBI with video-based clinician support (G-IBI-Video).

The literature on working alliance in face-to-face psychotherapy has consistently suggested that not only the levels but also the trajectories of working alliance matter. For example, studies show that varying trajectories of rupture repair–related patterns of working alliance in face-to-face therapy are differentially related to therapy progress [17], suggesting that the trajectories of working alliance are important for treatment outcomes. However, little is known regarding the trajectories of working alliance in IBIs. The study by Jasper et al [18] examined working alliance in guided IBIs and found that working alliance seemed to be low in the initial weeks of treatment and gradually increased during and at the end of treatment. This suggests that working alliance may generally increase over time for guided IBIs [18]. More studies are needed to examine the trajectories of working alliance developed in both guided and unguided IBIs to understand the development of working alliance and its potential impact on treatment outcomes. Therefore, this study’s second aim was to examine the trajectories of working alliance over the course of treatment in both guided and unguided IBIs.

Many studies have examined the associations between working alliance and treatment outcomes, but the field has not yet reached a consensus regarding this relationship in the context of IBIs. Several systematic reviews suggest mixed relationships between working alliance and treatment outcomes assessed at the end of treatment [3,9,11] and emphasize that the heterogeneity in clinician involvement and support modalities may contribute to the inconsistent results. Nonetheless, a recent meta-analysis [19] that summarized associations between working alliance and treatment outcomes in IBIs across 20 studies found an average weighted effect size of r=0.20 (95% CI 0.14-0.26) for the associations. It also noted that there was no difference between clinician communication modalities (ie, written formats such as email or text compared with oral formats such as telephone or video) or between interventions with or without self-help components (ie, interventions with no self-help components versus interventions that incorporated clinician support and self-help programs). However, no comparisons of working alliance and treatment outcome associations between unguided and guided programs were included in the study. Furthermore, the meta-analysis found significant higher associations between working alliance and treatment outcomes when working alliance was measured at the end of treatment rather than during the early phase of treatment, which indicated that the working alliance trajectories may influence the associations between working alliance and treatment outcomes. In light of these results, our third study aim was to examine the associations between treatment outcomes and trajectories of working alliance in both unguided and guided IBIs.

In summary, there were 3 key aims of this study. First, we examined the measurement invariance of working alliance across 3 conditions of CBT-based IBIs (U-IBI, G-IBI-Text, and G-IBI-Video). We hypothesized that working alliance would be conceptually equivalent across the 3 conditions. Second, we examined the trajectories of working alliance over the course of the brief treatments in the 3 conditions. We expected to see increases in working alliance over time for all conditions. Finally, we examined the associations between working alliance trajectories and treatment outcomes (eg, mental health symptoms and functional impairment) across the 3 conditions. We hypothesized that higher working alliance would predict better treatment outcomes in all 3 conditions.

We conducted secondary data analysis of a randomized controlled trial of a 7-week internet-based psychological intervention. The original study was a 3-arm randomized controlled trial that was designed to approximate treatment situations for treatment-seeking adults in stressful occupations and with no resources for real-time communications (eg, astronauts). Condition 1 was the U-IBI condition in which participants used an unguided, self-help IBI called MyCompass without additional clinician support. Condition 2 was the G-IBI-Text condition in which participants used the same self-help IBI (MyCompass) and received additional asynchronous text-based support from a clinician. Condition 3 was the G-IBI-Video condition in which participants used the same self-help IBI (MyCompass) and received additional asynchronous video-based support from a clinician.

The study design and protocol were approved by the Stony Brook University Institutional Review Board (903034).

The MyCompass program is a self-help IBI designed and shown to improve mild to moderate symptoms of depression, anxiety, and stress [20]. The program offers 14 self-management modules based on CBT principles, each of which comprises 3 sessions lasting 10 minutes each. The MyCompass program also includes homework tasks for each module as well as functions such as mood and symptom tracking, feedback from the program on patient performance, and psychoeducation (refer to Figures S1-S4 in Multimedia Appendix 1 for the interface and examples of MyCompass).

Participants in all 3 conditions were asked to complete at least two modules of their choice on MyCompass during the 7-week treatment period. Participants in the U-IBI condition received automated email reminders to encourage them to use the program, track their symptoms, examine patterns and triggers related to changes in their mood and behaviors, and practice the skills they learned in real-world situations. No clinician support was provided in this condition.

Each participant in the G-IBI-Text condition was assigned to a clinician for additional, asynchronous text-based support. All clinicians (n=10) had master’s or higher-level degrees and were trained and supervised weekly by 2 licensed psychologists (BM and AG). The clinicians initiated 1 weekly message through text at a prescheduled time to provide general support and positive reinforcement for program participation. This text-based contact typically involved encouraging participants to log on to MyCompass or to try a MyCompass module that was relevant to a stressor identified by the participant in a previous message to the clinician. Clinicians were instructed not to introduce skills or concepts not covered by the MyCompass program. Participants could respond to their clinician or initiate contact with them at any time during the 7-week study but were informed that clinicians would only respond to messages during specified business hours. Clinicians were encouraged to use their own words rather than prefabricated responses to communicate with their patients.

The experience of participants in the G-IBI-Video condition was similar to that of participants in the G-IBI-Text condition, except that the clinicians initiated a weekly video message and communicated with participants through asynchronous video messages. Participants could initiate contact with clinicians or respond to clinicians by sending video messages on a communication platform that was specifically designed to receive asynchronous video messages for this study. As in the G-IBI-Text condition, clinicians in the G-IBI-Video condition were instructed not to introduce skills or concepts not covered by the MyCompass program.

Adults with high education attainment who sought treatment for subclinical levels of anxiety, depression, and stress were selected in the original study. The inclusion criteria were as follows: (1) aged ≥18 years; (2) English speaking; (3) enrolled in, or completed, a graduate-level education in science, technology, engineering, or math domains; (4) having a score of ≥5 on the depression subscale, ≥4 on the anxiety subscale, or ≥8 on the stress subscale of the 21-item version of the Depression, Anxiety, and Stress Scale [21], which indicates a moderate or higher level of clinical symptoms; and (5) having a score of ≥5 on any subscales or ≥6 on the global scale of the Sheehan Disability Scale (SDS), which indicates a moderate or higher level of functional impairment [22].

The exclusion criteria were as follows: (1) active suicidal ideation in the past month, (2) any history of suicide attempt within the past 5 years, (3) having a diagnosis of psychotic disorder or bipolar disorder, (4) alcohol or substance dependency in the past 6 months, (5) serious medical problems (eg, seizures or cancer), (6) pregnancy, (7) current participation in psychotherapy, and (8) having recently started a new psychoactive medication (ie, benzodiazepines for <1 month or selective serotonin reuptake inhibitors, tricyclics, or serotonin-norepinephrine reuptake inhibitors for <3 months). The Mini International Neuropsychiatric Interview [23] was used for assessing study eligibility.

Eligible participants completed a baseline assessment consisting of in-person or over-the-telephone clinical assessment and web-based questionnaires administered through Qualtrics. After this baseline assessment, participants were randomly assigned to a condition based on a pregenerated random assignment table—the procedure was weighted to favor assignment to the U-IBI condition. Participants in all 3 conditions received access to the MyCompass program and the study web-based portal.

A total of 300 individuals completed screening forms to indicate interest in the study between May 2018 and September 2018. Of the 300 individuals screened, 155 (51.7%) were excluded for the following reasons: they did not meet the inclusion criteria (n=146, 94.2%), were no longer interested when contacted by the research team (n=4, 2.6%), or met an exclusion criterion (n=5, 3.2%). Subsequently, of the 300 people screened, 145 (48.3%) were enrolled into the trial. These 145 participants were randomized to the U-IBI condition (n=57, 39.3%), the G-IBI-Text condition (n=44, 30.3%), or the G-IBI-Video condition (n=44, 30.3%; refer to Multimedia Appendix 2 for the CONSORT [Consolidated Standards of Reporting Trials] flow diagram).

Most of the 145 participants were women (n=99, 68.3%), of heterosexual orientation (n=125, 86.2%), and identified as White (n=99, 68.3%) and non-Hispanic (n=133, 91.7%). The average age was 30 (SD 8.21) years. In total, 96.6% (140/145) of the participants were college graduates, with 59.3% (86/145) having at least a master’s degree. The proportion of participants identifying as Hispanic was lower in the G-IBI-Text condition than in the other 2 treatment conditions; otherwise, no demographic differences in sex, age, sexual orientation, race, ethnicity, or education attainment were noted across the 3 conditions (refer to Multimedia Appendix 3 for the demographic characteristics of the participants).

The levels of working alliance based on the selected items of ARM-12 for week 1 to week 7 are presented in Figure 1. The treatment outcome variables were moderately correlated concurrently in the range of 0.48 to 0.66 between social functional impairment and depression and in the range of 0.43 to 0.50 between worry and depression and between worry and functional impairment.

The model fit indices for configural, metric, and scalar invariance models for the selected working alliance items at week 1 and week 7 are presented in Table 1. Overall, the scalar invariance model across the 3 treatment conditions reached an excellent model fit at both week 1 and week 7 (chi-square test for model fit; P=.56 and P=.05 in scalar invariance models for week 1 and for week 7, respectively). This suggests that the selected 4 items of the ARM-12 had measurement invariance across treatment groups, indicating that it is appropriate to compare scores across conditions to detect differences on the latent construct of working alliance.

The model fit indices for latent curve models are presented in Table 2. When we constrained the treatment groups to be fully equal (ie, assuming no group differences), the quadratic model was the best model for the entire sample with acceptable fit, except for a slightly elevated index with RMSEA.

This quadratic model for all groups was then used as the baseline model in the multigroup modeling comparison to examine whether there were any group differences in trajectories. We compared this model with alternative models where we allowed 1 parameter to be different at a time. We identified the best-fitting model based on the chi-square difference test, acceptable model fit indices (comparative fit index=0.95, Tucker-Lewis Index=0.97, RMSEA=0.09), and the lowest Akaike information criterion and Bayesian information criterion. The best-fitting model allowed the linear slope to be different in the G-IBI-Text condition but not in the other 2 conditions. In addition, the best-fitting model allowed the residual variance to be different in the G-IBI-Text condition. This indicated that for the best-fitting model, there was a significantly different linear change rate in the G-IBI-Text condition compared with the other 2 conditions.

The estimations for the parameters are presented in Table 3. Specifically, the G-IBI-Text condition had a significant linear slope that was negative (linear slope estimation=−0.44; P=.04) compared with the nonsignificant linear slope for the other 2 conditions (linear slope estimation=0.19; P=.35). This indicates that the working alliance followed a different quadratic pattern in the G-IBI-Text condition compared with the other 2 conditions, in that there was a significant linear decrease only in the G-IBI-Text condition and no significant linear change in working alliance for the U-IBI or G-IBI-Video conditions.

We examined how the intercept (ie, initial level) and the linear slope (ie, the linear change rate) of the working alliance trajectories predicted each treatment outcome (depression, worry, and functional impairment) at the end of treatment and at 1-month follow-up after controlling for each treatment outcome variable at baseline separately. The model fit indices are presented in Table 4. All models reached acceptable fit. The parameter estimations are shown in Table 5.

After controlling for baseline severity, the associations between treatment outcomes and working alliance seemed to vary across conditions. In the G-IBI-Video condition, the intercept of working alliance significantly predicted depression (PHQ-9) and functioning impairment (SDS) negatively at 1-month follow-up (P=.04 for both models) such that a higher initial level of working alliance predicted lower depression and lower functional impairment for the participants at 1-month follow-up after controlling for their baseline levels of depression or functional impairment. In addition, for the G-IBI-Video condition, the linear slope of working alliance negatively predicted worry (PSWQ) at the end of treatment and at 1-month follow-up (P=.03 and P=.009, respectively), which means that a faster increase of working alliance over time would predict less worry at the end of treatment and at 1-month follow-up, after controlling for baseline levels of worry.

By contrast, neither the intercept nor the linear slope of working alliance was associated with any treatment outcomes at either time points for the U-IBI condition or the G-IBI-Text condition, with 1 exception: the linear slope of working alliance was negatively associated with worry (PSWQ) at the end of the treatment (P=.049) in the U-IBI condition. However, this significant effect disappeared at 1-month follow-up.

Internet-based psychological interventions promise to overcome accessibility-related issues associated with face-to-face, synchronous interventions while also embracing a patient-centered and stepped-care approach to mental health services. Despite a growing body of research supporting the efficacy and effectiveness of IBIs, little is known regarding the relevance of psychotherapy relationship factors, such as working alliance, which are known to account for a significant portion of in-person treatment outcomes. Moreover, although some studies have examined the effects of the inclusion of clinician support services in IBIs, to our knowledge, no existing investigation has directly compared the effects of varying degrees of clinician involvement in IBIs on working alliance. This study was designed to address this knowledge gap by examining the measurement invariance and trajectories of working alliance, as well as the associations between working alliance and treatment outcomes across an unguided IBI and guided IBIs with text-based or video-based clinician support. We found that although the conceptual interpretation and the trajectories of working alliance was relatively similar across the 3 conditions, higher working alliance predicted better treatment outcomes only in the video-support (G-IBI-Video) condition.

The results indicated that participants’ ratings of their working alliance with the IBIs were conceptually invariant across the U-IBI condition and the asynchronous G-IBI-Text and G-IBI-Video conditions. This result was consistent with our hypothesis, supporting the equivalence of the underlying construct of working alliance for IBIs and allowed for further between-group comparisons across conditions. A growing number of studies [14,40] have explored the applicability of the tripartite construct of alliance formulated by Bordin [1] to IBIs. This study relied on the administration of a modified version of the ARM-12, which featured selected, reworded items measuring participants’ perceived working alliance with IBIs. The methodological invariance suggests that the varying degrees of clinician involvement did not significantly affect participants’ interpretation of the construct of working alliance with the IBIs. This also indicates that quantitative comparisons of working alliance with the IBIs across the unguided and guided interventions are possible and meaningful. It is worth noting that working alliance with IBIs may be different from working alliance with only clinicians [41]; thus, additional research is needed to further elucidate the nature of patients’ ratings of working alliance with IBIs and their comparability to working alliance ratings for clinicians in face-to-face psychotherapy.

We hypothesized that working alliance would increase over time but found that working alliance followed a quadratic pattern and remained relatively stable in the U-IBI and the G-IBI-Video conditions while displaying significant deterioration in the G-IBI-Text condition. This indicated that working alliance with the IBIs may have been established quickly in the initial weeks. The stable, quadratic pattern of working alliance also corresponds to patterns reported in the literature on face-to-face therapy, suggesting that these patterns may not be unique to IBIs [17].

Nonetheless, the linear decrease in working alliance in the G-IBI-Text condition was surprising. Although deteriorating patterns have been documented in previous studies examining the development of working alliance over the course of face-to-face interventions [42], this trajectory had been rarely detected in working alliance in IBIs. Participants in the G-IBI-Text condition may have established higher expectations for human connection than participants in the U-IBI program; nonetheless, they received less visual and vocal communication with clinicians than participants in the G-IBI-Video condition. The gap between expectation for human connection and the lack of video-based communication may contribute to decreases in feelings connected in the G-IBI-Text condition. Future studies could shed additional light on this finding by examining whether text-based clinician support may interact with changes in treatment expectations, patient role expectation, or other relationship factors to contribute to decreases in working alliance over time.

Our results indicated that working alliance mattered in the G-IBI-Video condition. After controlling for baseline levels of symptoms, higher initial levels of working alliance predicted lower depressive symptoms and less functional impairment at 1-month follow-up. In addition, greater increases in working alliance over time predicted lower worry at the end of treatment and at 1-month follow-up. These results are consistent with the extensive literature on face-to-face therapy [2] as well as the literature on IBIs [11], supporting the robust positive relationship between working alliance and positive treatment outcomes. Our novel design allowed us to show that both the initial levels and the trajectories of working alliance contributed to better treatment outcomes separately for different treatment outcomes. In addition, these prospective associations between working alliance and treatment outcomes were detected at the 1-month follow-up, suggesting at least some level of sustainability for these treatment effects.

By contrast, working alliance was not consistently associated with treatment outcomes in the U-IBI or the G-IBI-Text conditions. These findings were unexpected, although not completely inconsistent with previous studies that did not find significant relationships between CBT-based IBIs and working alliance [43-45]. It is possible that the associations between working alliance and treatment outcomes are only present when clinician support is delivered through video-based modalities, the condition most closely aligned with traditional psychotherapy, which includes visual images, nonverbal facial expressions, and varying voice tones. These findings raise an important issue about the function of working alliance in asynchronous IBIs—although the interpretation and ratings of working alliance were similar across IBIs with or without clinician support, working alliance may only help reduce mental health symptoms when clinician support is present through video-based channels (vs text-based channels or no clinician support). Such findings should be replicated in future studies to compare how different communication modalities may influence the associations between working alliance and treatment outcomes.

The study’s findings should be considered in light of the following limitations. First, our sample was composed of individuals with high educational achievement, which may limit the generalizability of these findings. Future research should attempt to replicate these findings with a sample representative of the general clinical population. Second, working alliance in IBIs may be particularly important for individuals with more moderate to severe levels of psychopathology; hence, future studies should examine whether the associations between working alliance and treatment outcomes among IBIs may vary depending on clinical populations. Third, the MyCompass program delivered in this study is a 7-week intervention. The brief duration of the interventions may have interfered with participants’ ability to display more complex trajectories of working alliance, which may have been observed in longer interventions. Fourth, this study focused on using CBT-based IBIs to treat subclinical levels of depression, anxiety, and stress; therefore, the findings may not be generalizable to other IBIs using different theoretical frameworks or targeting other symptoms. In addition, this study selected 3 treatment outcomes that were moderately to highly correlated to each other. Future studies should select treatment outcome measures to reduce multicollinearity and examine whether the predictions of working alliance on treatment outcomes may differ depending on the outcome variables.

Future studies should also examine the potential mediators through which the initial level or the change rate of working alliance may affect treatment outcomes. For example, it is possible that an initial high level of working alliance would help the patient to use IBIs more frequently or complete the homework more often, which may result in greater symptom reduction. Finally, the study is limited to assessing working alliance with IBIs; future studies are needed to understand similarities and differences between working alliance with only clinicians and working alliance with IBIs.

This study examined the conceptual equivalence, trajectories, and outcome associations of working alliance in a randomized controlled trial with 3 conditions, including unguided IBIs as well as guided IBIs with text-based and video-based support. We found conceptual equivalence of working alliance with the IBIs across the 3 conditions. Our results also revealed a quadratic pattern of working alliance over time in the U-IBI and G-IBI-Video conditions, but a deterioration pattern was revealed in the G-IBI-Text condition. Higher initial-level and faster increases of working alliance in the G-IBI-Video condition predicted lower mental health symptoms and functional impairment at the end of treatment and 1-month follow-up compared with the other 2 conditions. Working alliance was also not consistently associated with treatment outcomes in the U-IBI or G-IBI-Text conditions. Our results suggested that despite similar conceptual interpretation and trajectories, the function of working alliance may differ among IBIs with varying degrees and types of clinician support for high-functioning populations with subclinical levels of distress.