This paper is in the following e-collection/theme issue:

Methods and New Tools in Mental Health Research (424) e-Mental Health and Cyberpsychology (1391) Telehealth and Telemonitoring (1557) Web-based and Mobile Health Interventions (3100) Depression and Mood Disorders; Suicide Prevention (1275) Anxiety and Stress Disorders (1003)

Published on in Vol 9, No 6 (2022): June

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/35496, first published .
Conceptual Invariance, Trajectories, and Outcome Associations of Working Alliance in Unguided and Guided Internet-Based Psychological Interventions: Secondary Analysis of a Randomized Controlled Trial

Conceptual Invariance, Trajectories, and Outcome Associations of Working Alliance in Unguided and Guided Internet-Based Psychological Interventions: Secondary Analysis of a Randomized Controlled Trial

Conceptual Invariance, Trajectories, and Outcome Associations of Working Alliance in Unguided and Guided Internet-Based Psychological Interventions: Secondary Analysis of a Randomized Controlled Trial

Authors of this article:

Xiaochen Luo1 Author Orcid Image ;   Matteo Bugatti2 Author Orcid Image ;   Lucero Molina3 Author Orcid Image ;   Jacqueline L Tilley4 Author Orcid Image ;   Brittain Mahaffey3 Author Orcid Image ;   Adam Gonzalez3 Author Orcid Image
Article Authors Cited by (2) Tweetations (3) Metrics

    Original Paper

    1Department of Counseling Psychology, Santa Clara University, Santa Clara, CA, United States

    2Morgridge College of Education, University of Denver, Denver, CO, United States

    3Department of Psychiatry, Stony Brook University, Stony Brook, NY, United States

    4Psychological and Child & Human Development Area Group, National Institute of Education, Nanyang Technological University, Singapore, Singapore

    Corresponding Author:

    Xiaochen Luo, PhD

    Department of Counseling Psychology

    Santa Clara University

    500 El Camino Real

    Santa Clara, CA, 95050

    United States

    Phone: 1 408 554 4000

    Email: xluo@scu.edu


    Background: The role of working alliance remains unclear for many forms of internet-based interventions (IBIs), a set of effective psychotherapy alternatives that do not require synchronous interactions between patients and therapists.

    Objective: This study examined the conceptual invariance, trajectories, and outcome associations of working alliance across an unguided IBI and guided IBIs that incorporated clinician support through asynchronous text messaging or video messaging.

    Methods: Adults with high education attainment (n=145) with subclinical levels of anxiety, stress, or depressive symptoms were randomized to 1 of 3 treatment conditions for 7 weeks. All participants received treatments from MyCompass, an unguided IBI using cognitive behavior therapy. Participants in condition 2 and 3 received supplemental, asynchronous clinician support through text and video, respectively. Working alliance with the IBIs was measured weekly using select items from the 12-item version of the Agnew Relationship Measure. Symptom and functional outcomes were assessed at baseline, at the end of treatment, and 1-month follow-up.

    Results: Working alliance with the IBIs was conceptually invariant across the 3 conditions. Working alliance followed a quadratic pattern of change over time for all conditions and declined significantly only in the text-support condition. After controlling for baseline symptoms, higher baseline levels of working alliance predicted less depression and less functional impairment at follow-up, whereas faster increases in working alliance predicted less worry at the end of treatment and at follow-up, all of which only occurred in the video-support condition.

    Conclusions: Working alliance with the IBIs was generally established in the initial sessions. Although working alliance is conceptually invariant across IBIs with or without clinician support, the associations between working alliance and treatment outcomes among IBIs may differ depending on clinician involvement and the modalities of support.

    Trial Registration: ClinicalTrials.gov NCT05122429; https://clinicaltrials.gov/ct2/show/NCT05122429

    JMIR Ment Health 2022;9(6):e35496

    doi:10.2196/35496

    Keywords

    working alliance; internet-based psychological interventions; video support; text support; trajectory; MyCompass 


    Background

    Working alliance is often conceptualized as a tripartite construct comprising agreement on therapeutic tasks and goals, as well as the bond between patients and therapists [1]. It has been identified as one of the most robust factors contributing to therapeutic change, with higher working alliance often associated with better treatment outcomes [2]. These effects have been consistently present in in-person psychotherapy as well as synchronous teletherapy, both of which feature direct, face-to-face interactions between clinicians and patients in real time [3]. However, the accessibility of in-person psychotherapy or synchronous teletherapy is limited by the shortage of clinicians, scheduling issues, difficulties with finding therapeutic space for both patients and therapists, transportation challenges (for in-person therapy), instability of internet connection (for synchronous teletherapy), perceived stigma of psychotherapy, and the financial cost of treatment. These barriers to in-person or synchronous teletherapy are especially salient during times of public health crisis, such as the COVID-19 pandemic, when the need for mental health services has surged despite limited supply.

    Technology innovations have offered alternative options such as internet-based interventions (IBIs), which provide accessible mental health services that do not require synchronous communication. An unguided IBI, which is characterized by the delivery of a web-based therapeutic program with no support from clinicians, is an example of such interventions. Many unguided IBIs, also known as self-help programs, are based on cognitive behavior therapy (CBT) principles and involve components of psychoeducation, behavioral and cognitive practice, homework, and tracking-related activities for important variables (eg, mood and behaviors) [4,5]. In contrast to the absence of clinician involvement in unguided programs, guided IBIs often include a self-help program and asynchronous support from clinicians. Such integrated interventions are thought to improve treatment outcomes by leveraging the benefits of the therapeutic relationship between patients and clinicians. Clinicians in guided IBIs often provide low-intensity clinical guidance to facilitate the patient’s independent work with the self-help program [6]. Empirical studies have shown effectiveness for both types of IBIs in treating issues such as anxiety, depression, and traumatic stress for a wide range of populations [3,7,8], supporting their flexible use in situations where in-person communication or real-time telecommunication may be limited. However, what remains unclear is the nature and function of working alliance in guided and unguided IBIs.

    Conceptualization of Working Alliance and the Measurement Invariance Across IBIs

    Previous studies have focused on understanding the level, trajectories, and outcome associations of working alliance in IBIs. A high level of working alliance has been reported by patients across guided IBIs with varying degrees of clinician involvement and different communication modalities [3,9], as well as across unguided programs [10]. Nonetheless, very few studies have directly compared the levels of working alliance between guided and unguided IBIs [11]. Such direct comparisons are needed to clarify (1) the nature of the relationships that patients have established with unguided intervention programs and (2) whether guided IBIs are able to improve outcomes by leveraging the benefits of working alliance with additional clinician support compared with unguided programs [12]. Information regarding these questions will help to determine the contexts in which additional clinician support is needed to improve the outcomes and delivery of IBIs.

    However, a conceptual question arises regarding whether we can quantitatively compare working alliance between guided interventions and unguided interventions. Most previous studies have directly taken measures of working alliance from studies of face-to-face therapy with minor adaptations to IBI contexts (eg, replacing the word therapist with the word program in items to refer to the relationship with therapeutic programs and clinicians together). The potential differences of working alliance in various IBI contexts remained unexamined in most cases (although there are exceptions [13-15]). The conceptual meaning and interpretation of working alliance for patients may be different for unguided interventions versus guided interventions; unless we verify that working alliance has the same conceptual meaning across different IBI contexts, quantitative comparisons of working alliance across IBIs are meaningless. The measurement invariance framework using multigroup confirmatory factor analysis [16] provides a strong methodological tool to examine the conceptual equivalence of working alliance across contexts. Therefore, the first aim of the study was to examine the measurement invariance of working alliance across guided versus unguided IBIs.

    In addition to the potential differences in working alliance between guided and unguided interventions, the nature of working alliance may also vary for guided IBIs with different communication modalities. Clinician support can be delivered through video-based messages or text messages, both of which may have unique impacts on the development of working alliance. For example, video-based support is hypothesized to better facilitate the development of working alliance than text-based support because it allows for visual messages with facial expressions that facilitate nonverbal communication and relational bonding [3]. By contrast, text-based support may allow participants to develop more thorough, in-depth responses through words, which can facilitate the establishment of working alliance by enhancing deep emotional processing [3]. Understanding the impact of different communication modalities can help to improve the design of IBIs and maximize the influence of clinician support on the therapeutic process. Therefore, in this study, we aimed to examine the measurement invariance among (1) an unguided IBI (U-IBI), (2) a guided IBI with text-based clinician support (G-IBI-Text), and (3) a guided IBI with video-based clinician support (G-IBI-Video).

    Trajectories of Working Alliance in IBIs

    The literature on working alliance in face-to-face psychotherapy has consistently suggested that not only the levels but also the trajectories of working alliance matter. For example, studies show that varying trajectories of rupture repair–related patterns of working alliance in face-to-face therapy are differentially related to therapy progress [17], suggesting that the trajectories of working alliance are important for treatment outcomes. However, little is known regarding the trajectories of working alliance in IBIs. The study by Jasper et al [18] examined working alliance in guided IBIs and found that working alliance seemed to be low in the initial weeks of treatment and gradually increased during and at the end of treatment. This suggests that working alliance may generally increase over time for guided IBIs [18]. More studies are needed to examine the trajectories of working alliance developed in both guided and unguided IBIs to understand the development of working alliance and its potential impact on treatment outcomes. Therefore, this study’s second aim was to examine the trajectories of working alliance over the course of treatment in both guided and unguided IBIs.

    Associations Between Working Alliance and Treatment Outcomes in IBIs

    Many studies have examined the associations between working alliance and treatment outcomes, but the field has not yet reached a consensus regarding this relationship in the context of IBIs. Several systematic reviews suggest mixed relationships between working alliance and treatment outcomes assessed at the end of treatment [3,9,11] and emphasize that the heterogeneity in clinician involvement and support modalities may contribute to the inconsistent results. Nonetheless, a recent meta-analysis [19] that summarized associations between working alliance and treatment outcomes in IBIs across 20 studies found an average weighted effect size of r=0.20 (95% CI 0.14-0.26) for the associations. It also noted that there was no difference between clinician communication modalities (ie, written formats such as email or text compared with oral formats such as telephone or video) or between interventions with or without self-help components (ie, interventions with no self-help components versus interventions that incorporated clinician support and self-help programs). However, no comparisons of working alliance and treatment outcome associations between unguided and guided programs were included in the study. Furthermore, the meta-analysis found significant higher associations between working alliance and treatment outcomes when working alliance was measured at the end of treatment rather than during the early phase of treatment, which indicated that the working alliance trajectories may influence the associations between working alliance and treatment outcomes. In light of these results, our third study aim was to examine the associations between treatment outcomes and trajectories of working alliance in both unguided and guided IBIs.

    Summary and Aims of This Study

    In summary, there were 3 key aims of this study. First, we examined the measurement invariance of working alliance across 3 conditions of CBT-based IBIs (U-IBI, G-IBI-Text, and G-IBI-Video). We hypothesized that working alliance would be conceptually equivalent across the 3 conditions. Second, we examined the trajectories of working alliance over the course of the brief treatments in the 3 conditions. We expected to see increases in working alliance over time for all conditions. Finally, we examined the associations between working alliance trajectories and treatment outcomes (eg, mental health symptoms and functional impairment) across the 3 conditions. We hypothesized that higher working alliance would predict better treatment outcomes in all 3 conditions.


    Study Design

    We conducted secondary data analysis of a randomized controlled trial of a 7-week internet-based psychological intervention. The original study was a 3-arm randomized controlled trial that was designed to approximate treatment situations for treatment-seeking adults in stressful occupations and with no resources for real-time communications (eg, astronauts). Condition 1 was the U-IBI condition in which participants used an unguided, self-help IBI called MyCompass without additional clinician support. Condition 2 was the G-IBI-Text condition in which participants used the same self-help IBI (MyCompass) and received additional asynchronous text-based support from a clinician. Condition 3 was the G-IBI-Video condition in which participants used the same self-help IBI (MyCompass) and received additional asynchronous video-based support from a clinician.

    Ethics Approval

    The study design and protocol were approved by the Stony Brook University Institutional Review Board (903034).

    Interventions and Clinicians

    The MyCompass program is a self-help IBI designed and shown to improve mild to moderate symptoms of depression, anxiety, and stress [20]. The program offers 14 self-management modules based on CBT principles, each of which comprises 3 sessions lasting 10 minutes each. The MyCompass program also includes homework tasks for each module as well as functions such as mood and symptom tracking, feedback from the program on patient performance, and psychoeducation (refer to Figures S1-S4 in Multimedia Appendix 1 for the interface and examples of MyCompass).

    Participants in all 3 conditions were asked to complete at least two modules of their choice on MyCompass during the 7-week treatment period. Participants in the U-IBI condition received automated email reminders to encourage them to use the program, track their symptoms, examine patterns and triggers related to changes in their mood and behaviors, and practice the skills they learned in real-world situations. No clinician support was provided in this condition.

    Each participant in the G-IBI-Text condition was assigned to a clinician for additional, asynchronous text-based support. All clinicians (n=10) had master’s or higher-level degrees and were trained and supervised weekly by 2 licensed psychologists (BM and AG). The clinicians initiated 1 weekly message through text at a prescheduled time to provide general support and positive reinforcement for program participation. This text-based contact typically involved encouraging participants to log on to MyCompass or to try a MyCompass module that was relevant to a stressor identified by the participant in a previous message to the clinician. Clinicians were instructed not to introduce skills or concepts not covered by the MyCompass program. Participants could respond to their clinician or initiate contact with them at any time during the 7-week study but were informed that clinicians would only respond to messages during specified business hours. Clinicians were encouraged to use their own words rather than prefabricated responses to communicate with their patients.

    The experience of participants in the G-IBI-Video condition was similar to that of participants in the G-IBI-Text condition, except that the clinicians initiated a weekly video message and communicated with participants through asynchronous video messages. Participants could initiate contact with clinicians or respond to clinicians by sending video messages on a communication platform that was specifically designed to receive asynchronous video messages for this study. As in the G-IBI-Text condition, clinicians in the G-IBI-Video condition were instructed not to introduce skills or concepts not covered by the MyCompass program.

    Participants

    Adults with high education attainment who sought treatment for subclinical levels of anxiety, depression, and stress were selected in the original study. The inclusion criteria were as follows: (1) aged ≥18 years; (2) English speaking; (3) enrolled in, or completed, a graduate-level education in science, technology, engineering, or math domains; (4) having a score of ≥5 on the depression subscale, ≥4 on the anxiety subscale, or ≥8 on the stress subscale of the 21-item version of the Depression, Anxiety, and Stress Scale [21], which indicates a moderate or higher level of clinical symptoms; and (5) having a score of ≥5 on any subscales or ≥6 on the global scale of the Sheehan Disability Scale (SDS), which indicates a moderate or higher level of functional impairment [22].

    The exclusion criteria were as follows: (1) active suicidal ideation in the past month, (2) any history of suicide attempt within the past 5 years, (3) having a diagnosis of psychotic disorder or bipolar disorder, (4) alcohol or substance dependency in the past 6 months, (5) serious medical problems (eg, seizures or cancer), (6) pregnancy, (7) current participation in psychotherapy, and (8) having recently started a new psychoactive medication (ie, benzodiazepines for <1 month or selective serotonin reuptake inhibitors, tricyclics, or serotonin-norepinephrine reuptake inhibitors for <3 months). The Mini International Neuropsychiatric Interview [23] was used for assessing study eligibility.

    Eligible participants completed a baseline assessment consisting of in-person or over-the-telephone clinical assessment and web-based questionnaires administered through Qualtrics. After this baseline assessment, participants were randomly assigned to a condition based on a pregenerated random assignment table—the procedure was weighted to favor assignment to the U-IBI condition. Participants in all 3 conditions received access to the MyCompass program and the study web-based portal.

    A total of 300 individuals completed screening forms to indicate interest in the study between May 2018 and September 2018. Of the 300 individuals screened, 155 (51.7%) were excluded for the following reasons: they did not meet the inclusion criteria (n=146, 94.2%), were no longer interested when contacted by the research team (n=4, 2.6%), or met an exclusion criterion (n=5, 3.2%). Subsequently, of the 300 people screened, 145 (48.3%) were enrolled into the trial. These 145 participants were randomized to the U-IBI condition (n=57, 39.3%), the G-IBI-Text condition (n=44, 30.3%), or the G-IBI-Video condition (n=44, 30.3%; refer to Multimedia Appendix 2 for the CONSORT [Consolidated Standards of Reporting Trials] flow diagram).

    Most of the 145 participants were women (n=99, 68.3%), of heterosexual orientation (n=125, 86.2%), and identified as White (n=99, 68.3%) and non-Hispanic (n=133, 91.7%). The average age was 30 (SD 8.21) years. In total, 96.6% (140/145) of the participants were college graduates, with 59.3% (86/145) having at least a master’s degree. The proportion of participants identifying as Hispanic was lower in the G-IBI-Text condition than in the other 2 treatment conditions; otherwise, no demographic differences in sex, age, sexual orientation, race, ethnicity, or education attainment were noted across the 3 conditions (refer to Multimedia Appendix 3 for the demographic characteristics of the participants).

    Measures

    Working Alliance

    In total, 4 items that were adapted from the Agnew Relationship Measure, 12-item version (ARM-12) were used to examine working alliance across the treatment conditions [24]. The ARM-12 has been widely used to examine working alliance in face-to-face therapy and has shown a strong reliability and good criterion validity with other working alliance measures [25,26]. The ARM-12 is one of the most commonly used questionnaires to assess working alliance in internet-based mental health interventions because of its conciseness and its full representation of the relevant concepts [27-30]. However, most studies have adapted the ARM-12 for IBIs by simply changing the term clinician to program or app in items [13]. Such alteration of wording may create issues with content validity (ie, an original item such as “the clinician seems bored or impatient with me” is modified to “the program seems bored or impatient with me”). Therefore, to enhance the measure’s content validity across the treatment conditions, this study only included items that were assessed by experts’ and users’ consensus in previous qualitative studies as relevant for IBIs [13]. The 4 included items were as follows: “I feel friendly toward the program,” “I have confidence in the program and its techniques,” “I feel I can openly express my thoughts and feelings to the program,” and “The program is supportive.” Each item was rated using a 7-point Likert scale (from 1=strongly disagree to 7=strongly agree). The ARM-12 items were administered after each week of the intervention, starting from the first week and ending in the seventh week.

    The Cronbach α values were .78 and .91 for the 4 items assessed at week 1 and at the end of treatment, respectively. We also examined factorial validity in confirmatory factor analysis for a single common factor of these 4 items at baseline and at the end of treatment, given the previous finding of a core working alliance factor for short versions of the ARM [24]. The single common factor model fit perfectly for the 4 items assessed at week 1 and at the end of treatment (χ22=1.2 and χ22=1.3, respectively; comparative fit index=1.00, Tucker-Lewis Index=1.00, and root mean squared error of approximation (RMSEA)=0.00 in models at week 1 and at the end of treatment), suggesting that a single common factor of the working alliance underlay the 4 items.

    Treatment Outcomes

    The Patient Health Questionnaire-9 (PHQ-9) [31], Penn State Worry Questionnaire (PSWQ) [32], and SDS [22] were used as treatment outcome measures to assess depression, anxiety, and social functioning impairment, respectively, at baseline, at the end of treatment, and 1-month follow-up. The PHQ-9 is a self-report measure for general depression symptoms, with higher scores indicating more depressive symptoms. The Cronbach α values for the internal consistency were .79, .87, and .88 in our sample at baseline, at the end of treatment, and 1-month posttreatment follow-up, respectively. The PSWQ is a 21-item measure for worry symptoms, with higher scores indicating higher levels of worry. The Cronbach α values for the internal consistency were .76, .82, and .79 in our sample at baseline, at the end of treatment, and 1-month posttreatment follow-up, respectively. The SDS is a 3-item measure for social functioning impairment. The Cronbach α values for the internal consistency were .73, .89, and .89 in our sample at baseline, at the end of treatment, and 1-month posttreatment follow-up, respectively.

    Data-Analytic Strategy

    Overview

    Multigroup confirmatory factor analysis and multigroup longitudinal structural equation modeling were used to assess the conceptual invariance, trajectories, and treatment outcome associations across treatment conditions. Data were modeled with Mplus (version 8.2; Muthén & Muthén) [33]. Full information maximum likelihood estimation [34] was used to handle missing data. We evaluated and compared the model fit for all models based on 6 model indices: chi-square [35], comparative fit index [36] (values >0.90 indicate acceptable fit), Tucker-Lewis Index [36] (values >0.90 indicate acceptable fit), RMSEA [37] (values <0.08 indicate acceptable fit), Akaike information criterion (lower values indicate better fit), and Bayesian information criterion (lower values indicate better fit). We compared nested models by calculating a chi-square difference test such that a nonsignificant chi-square difference indicates a preference for the nested, more parsimonious model.

    Conceptual Invariance of Working Alliance

    Confirmatory factor analysis was used to evaluate measurement invariance and determine whether working alliance was conceptually comparable across the 3 IBI conditions. In the multigroup invariance analyses, baseline models with no constraints requiring equality among the groups were compared with various invariance models to determine the best modeling fit to the data [38]. We used a single-factor baseline model with no restraints, requiring equality among the groups as the baseline model, and compared this model with three types of alternative, invariance models: (1) a configural invariance model (testing a single-factor model in all treatment groups without constraining the factor loadings or intercepts), which indicates the same conceptual factor structure across treatment groups; (2) a metric invariance model (constraining the factor loadings to be equivalent across groups), which indicates that in addition to the same factor structure of the single-factor model, these groups have the same factor loadings on the single factor; and (3) a scalar invariance model (constraining both factor loadings and intercepts to be equivalent), which indicates that the 3 conditions have the same factor structure as the single-factor model, the same factor loadings, and the same true values on the latent factor. If the scalar invariance model is the best-fitting model, it indicates that the working alliance is conceptually invariant and that the latent values can be compared across treatment conditions [39]. We examined measurement invariance separately for working alliance assessed at the first week of treatment and at the end of treatment.

    Trajectories of Working Alliance

    Once we determined that working alliance was conceptually invariant (refer to the Results section), we fit a series of univariate latent growth curve models to identify the appropriate change pattern of working alliance for the entire sample. Data were modeled with multiple types of trajectories, including (1) a no-change, intercept-only model where we only estimated means and variance for all measurement points without a slope; this model indicates no change in working alliance over time; (2) a linear change model where we estimated both intercept and slope for the trajectories; this model indicates that working alliance changes in a linear fashion over time; (3) a latent basis model, where an intercept and a slope were estimated but the loading on the slope was not based on the temporal time and is freely estimated; this model indicates that working alliance may change at a nonlinear rate with time; and (4) a quadratic model where we estimated the intercept, a linear slope, and a quadratic slope; this model indicates that working alliance may change in a quadratic pattern. We used the model fit indices to determine the best-fitting model that depicted the trajectories of working alliance among the aforementioned models.

    After identifying the best-fitting model in which everything was set as equal across treatment conditions (ie, the fully constrained model), we created alternative models in which the 3 conditions may not be equal (by gradually loosening the constraints of the parameters) and compared the model fit between alternative models and the fully constrained model. The following parameters (if they existed in the best-fitting model) were loosened to be uniquely estimated in each group one at a time: mean of intercept, mean of linear slope, quadratic slope, mean of autoregressive coefficient (if it existed), variance of intercept, variance of linear slope, variance of quadratic slope, and residual variance. In case of model misspecifications, the cause of misspecification was examined through modification indices. Model modifications were used with caution and applied only if supported by possible theoretical explanations. If any of the alternative models yielded better model fit than the fully constrained model, it indicated differences in the trajectories of working alliance across treatment conditions.

    Associations Between Working Alliance and Treatment Outcomes

    We examined whether the trajectories of working alliance contributed to treatment outcomes in each condition by examining whether the intercept (the initial level) or the change rates (linear or quadratic slope, if identified in previous steps) of the working alliance would predict treatment outcomes on depression (PHQ-9), worry (PSWQ), and social functioning impairment (SDS). We did so by including each outcome and the associations between each outcome and the intercept and slope of working alliance in the best-fitting multigroup structural equation modeling models that were identified from the previous step. We ran separate models for each outcome and controlled for the baseline level of each treatment outcome measure in each model.


    Overview

    The levels of working alliance based on the selected items of ARM-12 for week 1 to week 7 are presented in Figure 1. The treatment outcome variables were moderately correlated concurrently in the range of 0.48 to 0.66 between social functional impairment and depression and in the range of 0.43 to 0.50 between worry and depression and between worry and functional impairment.

    ‎
    Figure 1. The weekly working alliance ratings across treatment for each treatment condition. U-IBI: unguided internet-based intervention; G-IBI-Text: guided internet-based intervention with text-based clinician support; G-IBI-Video: guided internet-based intervention with video-based clinician support.
    View this figure

    Conceptual Invariance of Working Alliance

    The model fit indices for configural, metric, and scalar invariance models for the selected working alliance items at week 1 and week 7 are presented in Table 1. Overall, the scalar invariance model across the 3 treatment conditions reached an excellent model fit at both week 1 and week 7 (chi-square test for model fit; P=.56 and P=.05 in scalar invariance models for week 1 and for week 7, respectively). This suggests that the selected 4 items of the ARM-12 had measurement invariance across treatment groups, indicating that it is appropriate to compare scores across conditions to detect differences on the latent construct of working alliance.

    Table 1. The model fit indices for configural, metric, and scalar invariance of working alliance (selected items from the Agnew Relationship Measure, 12-item version) across the 3 conditions at week 1 and week 7.
    Assessment week and modelFree parameters, nChi-square (df)P valueRMSEAaCFIbTLIc
    Week 1

    		Configural invariance2518.9 (17).340.050.990.99

    		Metric invariance1922.0 (23).520.001.001.00

    		Scalar invarianced1129.2 (31).560.001.001.00
    Week 7 (at the end of treatment)

    		Configural invariance2529.5 (17).030.130.960.96

    		Metric invariance1935.8 (23).040.110.960.97

    		Scalar invarianced1144.8 (31).050.100.960.98

    aRMSEA: root mean squared error of approximation.

    bCFI: comparative fit index.

    cTLI: Tucker-Lewis Index.

    dText in italics indicates the best-fitting model selected.

    Trajectories of Working Alliance

    The model fit indices for latent curve models are presented in Table 2. When we constrained the treatment groups to be fully equal (ie, assuming no group differences), the quadratic model was the best model for the entire sample with acceptable fit, except for a slightly elevated index with RMSEA.

    This quadratic model for all groups was then used as the baseline model in the multigroup modeling comparison to examine whether there were any group differences in trajectories. We compared this model with alternative models where we allowed 1 parameter to be different at a time. We identified the best-fitting model based on the chi-square difference test, acceptable model fit indices (comparative fit index=0.95, Tucker-Lewis Index=0.97, RMSEA=0.09), and the lowest Akaike information criterion and Bayesian information criterion. The best-fitting model allowed the linear slope to be different in the G-IBI-Text condition but not in the other 2 conditions. In addition, the best-fitting model allowed the residual variance to be different in the G-IBI-Text condition. This indicated that for the best-fitting model, there was a significantly different linear change rate in the G-IBI-Text condition compared with the other 2 conditions.

    The estimations for the parameters are presented in Table 3. Specifically, the G-IBI-Text condition had a significant linear slope that was negative (linear slope estimation=−0.44; P=.04) compared with the nonsignificant linear slope for the other 2 conditions (linear slope estimation=0.19; P=.35). This indicates that the working alliance followed a different quadratic pattern in the G-IBI-Text condition compared with the other 2 conditions, in that there was a significant linear decrease only in the G-IBI-Text condition and no significant linear change in working alliance for the U-IBI or G-IBI-Video conditions.

    Table 2. Model fit indices for multigroup latent curve modeling of working alliance (selected items from the Agnew Relationship Measure, 12-item version).
    ModelFree parameters, nAICaBICbChi-square (df)RMSEAcCFIdTLIeΔChi-squaref (Δdf)P value for Δchi-square (Δdf)
    Fully constrained models for the entire sample as 1 group

    		Intercept only33879.213888.07183.5 (32)0.180.790.86N/AgN/A

    		Linear63797.863815.5996.1 (29)0.130.910.9387.4 (3)<.001

    		Latent basis113793.243825.7681.5 (24)0.130.920.93102.0 (8)<.001

    		Quadratich103762.953792.5153.2 (25)0.090.960.97130.3 (7)<.001
    Multigroup modeling allowing for group differences

    		Baseline, fully constrained quadratic model103762.953792.51143.0 (95)0.100.940.96N/AN/A

    		Free intercept123765.323800.79141.3 (93)0.110.940.961.4 (2).50

    		Free linear slope123762.083797.55138.1 (93)0.100.940.964.9 (2).09

    		Free linear slope and quadratic slope143765.393806.78137.4 (91)0.100.940.965.6 (4).23

    		Free quadratic slope123762.913798.37138.9 (93)0.100.940.964.0 (2).14

    		Free variance and covariance223777.703842.73133.7 (83)0.110.930.959.3 (12).68

    		Free residual variance123753.003788.62129.2 (93)0.090.950.9713.8 (2).001

    		Free residual variance and linear slope143752.423793.80124.0 (91)0.090.960.9719.0 (4)<.001

    		Free residual and linear slope only for the text grouph123751.223786.69127.2 (93)0.090.950.9715.7 (2)<.001

    		Free linear slope only for the text group113760.923793.43138.9 (94)0.100.940.964.0 (1).05

    		Free residual variance only for the text group113753.083785.59131.1 (94)0.090.950.9711.9 (1)<.001

    aAIC: Akaike information criterion.

    bBIC: Bayesian information criterion.

    cRMSEA: root mean squared error of approximation.

    dCFI: comparative fit index.

    eTLI: Tucker-Lewis Index.

    fΔChi-square: chi-square difference test.

    gN/A: not applicable (the chi-square difference test is not applicable to the baseline models).

    hThe models presented in italics indicated the best-fitting models in each category. We first fit models for the entire sample and identified the quadratic model as the best-fitting model. Next, we fit the quadratic model to the 3 conditions in multigroup structural equation modeling, constraining the parameters to be the same for each group. We then gradually loosened the constraints to examine alternative models. The best-fitting model for multigroup modeling indicated a model in which the residual variance and linear slope were set to be different for the guided internet-based intervention with text-based clinician support condition only.

    Table 3. Parameter estimation in the best-fitting model.
    ParametersU-IBIa

    		G-IBI-TextbG-IBI-Videoc

    		Estimate (SE)P valueEstimate (SE)P valueEstimate (SE)P value
    Level factor means20.65 (0.30)<.00120.65 (0.30)<.00120.65 (0.30)<.001
    Linear slope factor means−0.19 (0.20).35−0.44d (0.22).04−0.19 (0.20).35
    Quadratic slope factor means0.01 (0.03).720.01 (0.03).720.01 (0.03).72
    Level factor variance9.42 (1.52)<.0019.42 (1.52)<.0019.42 (1.52)<.001
    Linear slope factor variance2.68 (0.63)<.0012.68 (0.63)<.0012.68 (0.63)<.001
    Quadratic slope factor variance0.06 (0.01)<.0010.06 (0.01)<.0010.06 (0.01)<.001
    Residual variance3.05 (0.27)<.0014.98 (0.61)<.0013.05 (0.27)<.001
    Covariance between level factor and linear slope factor0.74 (0.69).290.74 (0.69).290.74 (0.69).29
    Covariance between level factor and quadratic slope factor−0.11 (0.10).28−0.11 (0.10).28−0.11 (0.10).28
    Covariance between linear slope factor and quadratic slope factor−0.37 (0.09)<.001−0.37 (0.09)<.001−0.37 (0.09)<.001

    aU-IBI: unguided internet-based intervention.

    bG-IBI-Text: guided internet-based intervention with text-based clinician support.

    cG-IBI-Video: guided internet-based intervention with video-based clinician support.

    dThe parameters in the 3 conditions were fixed to be the same, except for the ones in italics, which were estimated separately for the guided internet-based intervention with text-based clinician support condition.

    Associations Between Working Alliance and Treatment Outcomes

    We examined how the intercept (ie, initial level) and the linear slope (ie, the linear change rate) of the working alliance trajectories predicted each treatment outcome (depression, worry, and functional impairment) at the end of treatment and at 1-month follow-up after controlling for each treatment outcome variable at baseline separately. The model fit indices are presented in Table 4. All models reached acceptable fit. The parameter estimations are shown in Table 5.

    Table 4. Model fit indices for multigroup models with outcomesa at the end of treatment and 1-month follow-up.
    ModelOutcomeAssessmentFree parameters, nChi-square (df)RMSEAbCFIcTLId
    Model 1PHQ-9ethe end of treatment27168.4 (129)0.080.950.96
    Model 2PHQ-91-month follow-up27176.4 (129)0.090.940.95
    Model 3PSWQfthe end of treatment27172.6 (129)0.090.950.96
    Model 4PSWQ1-month follow-up27171.1 (129)0.080.950.96
    Model 5SDSgthe end of treatment27182.4 (129)0.090.930.94
    Model 6SDS1-month follow-up27163.8 (129)0.080.950.96

    aThe parameters indicated a good fit for all the models incorporating outcome measures.

    bRMSEA: root mean squared error of approximation.

    cCFI: comparative fit index.

    dTLI: Tucker-Lewis Index.

    ePHQ-9: Patient Health Questionnaire-9.

    fPSWQ: Penn State Worry Questionnaire.

    gSDS: Sheehan Disability Scale.

    Table 5. The parameter estimations for alliance-outcome associations.
    SEMa model and predictorsOutcomesU-IBIbG-IBI-TextcG-IBI-Videod


    		EstimateP valueEstimateP valueEstimateP value
    PHQ-9e

    		Model 1


    		Intercept of alliancePHQ-9 at the end of treatment−0.06.77−0.1.71−0.02.91


    		Linear slope of alliancePHQ-9 at the end of treatment−0.44.350.28.65−0.8.09

    		Model 2


    		Intercept of alliancePHQ-9 at 1-month follow-up0.05.860.004.99−0.62f.04


    		Linear slope of alliancePHQ-9 at 1-month follow-up−0.82.140.43.420.04.95
    PSWQg

    		Model 3


    		Intercept of alliancePSWQ at the end of treatment0.59.140.01.990.83.18


    		Linear slope of alliancePSWQ at the end of treatment−1.8.050.82.46−2.85.03

    		Model 4


    		Intercept of alliancePSWQ at 1-month follow-up0.18.62−0.95.080.67.29


    		Linear slope of alliancePSWQ at 1-month follow-up−0.67.402.17.08−3.61.01
    SDSh

    		Model 5


    		Intercept of allianceSDS at the end of treatment−0.42.16−0.47.14−0.55.17


    		Linear slope of allianceSDS at the end of treatment0.59.360.48.51−0.34.72

    		Model 6


    		Intercept of allianceSDS at 1-month follow-up0.07.82−0.62.14−0.76.04
     
    		Linear slope of allianceSDS at 1-month follow-up−0.004.990.85.390.64.43

    aSEM: structural equation modeling.

    bU-IBI: unguided internet-based intervention.

    cG-IBI-Text: guided internet-based intervention with text-based clinician support.

    dG-IBI-Video: guided internet-based intervention with video-based clinician support.

    ePHQ-9: Patient Health Questionnaire-9.

    fItalicized items indicate significance at P<.05 level.

    gPSWQ: Penn State Worry Questionnaire.

    hSDS: Sheehan Disability Scale.

    After controlling for baseline severity, the associations between treatment outcomes and working alliance seemed to vary across conditions. In the G-IBI-Video condition, the intercept of working alliance significantly predicted depression (PHQ-9) and functioning impairment (SDS) negatively at 1-month follow-up (P=.04 for both models) such that a higher initial level of working alliance predicted lower depression and lower functional impairment for the participants at 1-month follow-up after controlling for their baseline levels of depression or functional impairment. In addition, for the G-IBI-Video condition, the linear slope of working alliance negatively predicted worry (PSWQ) at the end of treatment and at 1-month follow-up (P=.03 and P=.009, respectively), which means that a faster increase of working alliance over time would predict less worry at the end of treatment and at 1-month follow-up, after controlling for baseline levels of worry.

    By contrast, neither the intercept nor the linear slope of working alliance was associated with any treatment outcomes at either time points for the U-IBI condition or the G-IBI-Text condition, with 1 exception: the linear slope of working alliance was negatively associated with worry (PSWQ) at the end of the treatment (P=.049) in the U-IBI condition. However, this significant effect disappeared at 1-month follow-up.


    Summary

    Internet-based psychological interventions promise to overcome accessibility-related issues associated with face-to-face, synchronous interventions while also embracing a patient-centered and stepped-care approach to mental health services. Despite a growing body of research supporting the efficacy and effectiveness of IBIs, little is known regarding the relevance of psychotherapy relationship factors, such as working alliance, which are known to account for a significant portion of in-person treatment outcomes. Moreover, although some studies have examined the effects of the inclusion of clinician support services in IBIs, to our knowledge, no existing investigation has directly compared the effects of varying degrees of clinician involvement in IBIs on working alliance. This study was designed to address this knowledge gap by examining the measurement invariance and trajectories of working alliance, as well as the associations between working alliance and treatment outcomes across an unguided IBI and guided IBIs with text-based or video-based clinician support. We found that although the conceptual interpretation and the trajectories of working alliance was relatively similar across the 3 conditions, higher working alliance predicted better treatment outcomes only in the video-support (G-IBI-Video) condition.

    Conceptual Invariance of Working Alliance

    The results indicated that participants’ ratings of their working alliance with the IBIs were conceptually invariant across the U-IBI condition and the asynchronous G-IBI-Text and G-IBI-Video conditions. This result was consistent with our hypothesis, supporting the equivalence of the underlying construct of working alliance for IBIs and allowed for further between-group comparisons across conditions. A growing number of studies [14,40] have explored the applicability of the tripartite construct of alliance formulated by Bordin [1] to IBIs. This study relied on the administration of a modified version of the ARM-12, which featured selected, reworded items measuring participants’ perceived working alliance with IBIs. The methodological invariance suggests that the varying degrees of clinician involvement did not significantly affect participants’ interpretation of the construct of working alliance with the IBIs. This also indicates that quantitative comparisons of working alliance with the IBIs across the unguided and guided interventions are possible and meaningful. It is worth noting that working alliance with IBIs may be different from working alliance with only clinicians [41]; thus, additional research is needed to further elucidate the nature of patients’ ratings of working alliance with IBIs and their comparability to working alliance ratings for clinicians in face-to-face psychotherapy.

    Trajectories of Working Alliance

    We hypothesized that working alliance would increase over time but found that working alliance followed a quadratic pattern and remained relatively stable in the U-IBI and the G-IBI-Video conditions while displaying significant deterioration in the G-IBI-Text condition. This indicated that working alliance with the IBIs may have been established quickly in the initial weeks. The stable, quadratic pattern of working alliance also corresponds to patterns reported in the literature on face-to-face therapy, suggesting that these patterns may not be unique to IBIs [17].

    Nonetheless, the linear decrease in working alliance in the G-IBI-Text condition was surprising. Although deteriorating patterns have been documented in previous studies examining the development of working alliance over the course of face-to-face interventions [42], this trajectory had been rarely detected in working alliance in IBIs. Participants in the G-IBI-Text condition may have established higher expectations for human connection than participants in the U-IBI program; nonetheless, they received less visual and vocal communication with clinicians than participants in the G-IBI-Video condition. The gap between expectation for human connection and the lack of video-based communication may contribute to decreases in feelings connected in the G-IBI-Text condition. Future studies could shed additional light on this finding by examining whether text-based clinician support may interact with changes in treatment expectations, patient role expectation, or other relationship factors to contribute to decreases in working alliance over time.

    Associations Between Working Alliance and Treatment Outcomes

    Our results indicated that working alliance mattered in the G-IBI-Video condition. After controlling for baseline levels of symptoms, higher initial levels of working alliance predicted lower depressive symptoms and less functional impairment at 1-month follow-up. In addition, greater increases in working alliance over time predicted lower worry at the end of treatment and at 1-month follow-up. These results are consistent with the extensive literature on face-to-face therapy [2] as well as the literature on IBIs [11], supporting the robust positive relationship between working alliance and positive treatment outcomes. Our novel design allowed us to show that both the initial levels and the trajectories of working alliance contributed to better treatment outcomes separately for different treatment outcomes. In addition, these prospective associations between working alliance and treatment outcomes were detected at the 1-month follow-up, suggesting at least some level of sustainability for these treatment effects.

    By contrast, working alliance was not consistently associated with treatment outcomes in the U-IBI or the G-IBI-Text conditions. These findings were unexpected, although not completely inconsistent with previous studies that did not find significant relationships between CBT-based IBIs and working alliance [43-45]. It is possible that the associations between working alliance and treatment outcomes are only present when clinician support is delivered through video-based modalities, the condition most closely aligned with traditional psychotherapy, which includes visual images, nonverbal facial expressions, and varying voice tones. These findings raise an important issue about the function of working alliance in asynchronous IBIs—although the interpretation and ratings of working alliance were similar across IBIs with or without clinician support, working alliance may only help reduce mental health symptoms when clinician support is present through video-based channels (vs text-based channels or no clinician support). Such findings should be replicated in future studies to compare how different communication modalities may influence the associations between working alliance and treatment outcomes.

    Limitations and Future Directions

    The study’s findings should be considered in light of the following limitations. First, our sample was composed of individuals with high educational achievement, which may limit the generalizability of these findings. Future research should attempt to replicate these findings with a sample representative of the general clinical population. Second, working alliance in IBIs may be particularly important for individuals with more moderate to severe levels of psychopathology; hence, future studies should examine whether the associations between working alliance and treatment outcomes among IBIs may vary depending on clinical populations. Third, the MyCompass program delivered in this study is a 7-week intervention. The brief duration of the interventions may have interfered with participants’ ability to display more complex trajectories of working alliance, which may have been observed in longer interventions. Fourth, this study focused on using CBT-based IBIs to treat subclinical levels of depression, anxiety, and stress; therefore, the findings may not be generalizable to other IBIs using different theoretical frameworks or targeting other symptoms. In addition, this study selected 3 treatment outcomes that were moderately to highly correlated to each other. Future studies should select treatment outcome measures to reduce multicollinearity and examine whether the predictions of working alliance on treatment outcomes may differ depending on the outcome variables.

    Future studies should also examine the potential mediators through which the initial level or the change rate of working alliance may affect treatment outcomes. For example, it is possible that an initial high level of working alliance would help the patient to use IBIs more frequently or complete the homework more often, which may result in greater symptom reduction. Finally, the study is limited to assessing working alliance with IBIs; future studies are needed to understand similarities and differences between working alliance with only clinicians and working alliance with IBIs.

    Conclusions

    This study examined the conceptual equivalence, trajectories, and outcome associations of working alliance in a randomized controlled trial with 3 conditions, including unguided IBIs as well as guided IBIs with text-based and video-based support. We found conceptual equivalence of working alliance with the IBIs across the 3 conditions. Our results also revealed a quadratic pattern of working alliance over time in the U-IBI and G-IBI-Video conditions, but a deterioration pattern was revealed in the G-IBI-Text condition. Higher initial-level and faster increases of working alliance in the G-IBI-Video condition predicted lower mental health symptoms and functional impairment at the end of treatment and 1-month follow-up compared with the other 2 conditions. Working alliance was also not consistently associated with treatment outcomes in the U-IBI or G-IBI-Text conditions. Our results suggested that despite similar conceptual interpretation and trajectories, the function of working alliance may differ among IBIs with varying degrees and types of clinician support for high-functioning populations with subclinical levels of distress.

    Acknowledgments

    This study is funded by the Human Research Program of the National Aeronautics and Space Administration as well as the National Space Biomedical Research Institute (NNX15A64G).

    Conflicts of Interest

    None declared.

    ‎
    Multimedia Appendix 1

    MyCompass interface examples.

    DOCX File , 1476 KB
    ‎
    Multimedia Appendix 2

    CONSORT(Consolidated Standards of Reporting Trials) flow chart of participants.

    DOCX File , 108 KB
    ‎
    Multimedia Appendix 3

    Participants’ demographic information.

    DOCX File , 17 KB
    1. Bordin ES. The generalizability of the psychoanalytic concept of the working alliance. Psychother Theory Res Pract 1979;16(3):252-260 [FREE Full text] [CrossRef]
    2. Flückiger C, Del Re AC, Wampold BE, Horvath AO. The alliance in adult psychotherapy: a meta-analytic synthesis. Psychotherapy (Chic) 2018 Dec;55(4):316-340 [FREE Full text] [CrossRef] [Medline]
    3. Berger T. The therapeutic alliance in internet interventions: a narrative review and suggestions for future research. Psychother Res 2017 Sep;27(5):511-524 [FREE Full text] [CrossRef] [Medline]
    4. Andrews G, Cuijpers P, Craske MG, McEvoy P, Titov N. Computer therapy for the anxiety and depressive disorders is effective, acceptable and practical health care: a meta-analysis. PLoS One 2010 Oct 13;5(10):e13196 [FREE Full text] [CrossRef] [Medline]
    5. Cuijpers P, van Straten A, Andersson G. Internet-administered cognitive behavior therapy for health problems: a systematic review. J Behav Med 2008 Apr;31(2):169-177 [FREE Full text] [CrossRef] [Medline]
    6. Titov N. Internet-delivered psychotherapy for depression in adults. Curr Opin Psychiatry 2011 Jan;24(1):18-23 [FREE Full text] [CrossRef] [Medline]
    7. Amstadter AB, Broman-Fulks J, Zinzow H, Ruggiero KJ, Cercone J. Internet-based interventions for traumatic stress-related mental health problems: a review and suggestion for future research. Clin Psychol Rev 2009 Jul;29(5):410-420 [FREE Full text] [CrossRef] [Medline]
    8. Ye X, Bapuji SB, Winters SE, Struthers A, Raynard M, Metge C, et al. Effectiveness of internet-based interventions for children, youth, and young adults with anxiety and/or depression: a systematic review and meta-analysis. BMC Health Serv Res 2014 Jul 18;14:313 [FREE Full text] [CrossRef] [Medline]
    9. Sucala M, Schnur JB, Constantino MJ, Miller SJ, Brackman EH, Montgomery GH. The therapeutic relationship in e-therapy for mental health: a systematic review. J Med Internet Res 2012 Aug 02;14(4):e110 [FREE Full text] [CrossRef] [Medline]
    10. Meyer B, Berger T, Caspar F, Beevers CG, Andersson G, Weiss M. Effectiveness of a novel integrative online treatment for depression (Deprexis): randomized controlled trial. J Med Internet Res 2009 May 11;11(2):e15 [FREE Full text] [CrossRef] [Medline]
    11. Pihlaja S, Stenberg JH, Joutsenniemi K, Mehik H, Ritola V, Joffe G. Therapeutic alliance in guided internet therapy programs for depression and anxiety disorders - A systematic review. Internet Interv 2018 Mar;11:1-10 [FREE Full text] [CrossRef] [Medline]
    12. Baumeister H, Reichler L, Munzinger M, Lin J. The impact of guidance on internet-based mental health interventions — A systematic review. Internet Interv 2014 Oct;1(4):205-215 [FREE Full text] [CrossRef]
    13. Berry K, Salter A, Morris R, James S, Bucci S. Assessing therapeutic alliance in the context of mHealth interventions for mental health problems: development of the mobile Agnew relationship measure (mARM) questionnaire. J Med Internet Res 2018 Apr 19;20(4):e90 [FREE Full text] [CrossRef] [Medline]
    14. Gómez Penedo JM, Berger T, Grosse Holtforth M, Krieger T, Schröder J, Hohagen F, et al. The Working Alliance Inventory for guided Internet interventions (WAI-I). J Clin Psychol 2020 Jun;76(6):973-986 [FREE Full text] [CrossRef] [Medline]
    15. Herrero R, Vara MD, Miragall M, Botella C, García-Palacios A, Riper H, et al. Working Alliance Inventory for online interventions-Short Form (WAI-TECH-SF): the role of the therapeutic alliance between patient and online program in therapeutic outcomes. Int J Environ Res Public Health 2020 Aug 25;17(17):6169 [FREE Full text] [CrossRef] [Medline]
    16. Jöreskog KG. Simultaneous factor analysis in several populations*. ETS Res Bulletin Series 2014 Aug 08;1970(2):i-31 [FREE Full text] [CrossRef]
    17. Stiles WB, Glick M, Osatuke K, Hardy G, Shapiro D, Agnew-Davies R, et al. Patterns of alliance development and the rupture-repair hypothesis: are productive relationships U-shaped or V-shaped? J Counsel Psychol 2004;51(1):81-92 [FREE Full text] [CrossRef]
    18. Jasper K, Weise C, Conrad I, Andersson G, Hiller W, Kleinstäuber M. Internet-based guided self-help versus group cognitive behavioral therapy for chronic tinnitus: a randomized controlled trial. Psychother Psychosom 2014;83(4):234-246 [FREE Full text] [CrossRef] [Medline]
    19. Kaiser J, Hanschmidt F, Kersting A. The association between therapeutic alliance and outcome in internet-based psychological interventions: a meta-analysis. Comput Human Behav 2021 Jan;114:106512 [FREE Full text] [CrossRef]
    20. Solomon D, Proudfoot J, Clarke J, Christensen H. E-CBT (myCompass), antidepressant medication, and face-to-face psychological treatment for depression in Australia: a cost-effectiveness comparison. J Med Internet Res 2015 Nov 11;17(11):e255 [FREE Full text] [CrossRef] [Medline]
    21. Lovibond P, Lovibond S. The structure of negative emotional states: comparison of the Depression Anxiety Stress Scales (DASS) with the Beck Depression and Anxiety Inventories. Behav Res Ther 1995 Mar;33(3):335-343 [FREE Full text] [CrossRef]
    22. Sheehan KH, Sheehan DV. Assessing treatment effects in clinical trials with the Discan metric of the Sheehan Disability Scale. Int Clin Psychopharmacol 2008;23(2):70-83 [FREE Full text] [CrossRef]
    23. Lecrubier Y, Sheehan D, Weiller E, Amorim P, Bonora I, Sheehan K, et al. The Mini International Neuropsychiatric Interview (MINI). A short diagnostic structured interview: reliability and validity according to the CIDI. Eur Psychiatr 2020 Apr 16;12(5):224-231 [FREE Full text] [CrossRef]
    24. Cahill J, Stiles WB, Barkham M, Hardy GE, Stone G, Agnew-Davies R, et al. Two short forms of the Agnew Relationship Measure: the ARM-5 and ARM-12. Psychother Res 2012;22(3):241-255 [FREE Full text] [CrossRef] [Medline]
    25. Stiles WB, Agnew-Davies R, Barkham M, Culverwell A, Goldfried MR, Halstead J, et al. Convergent validy of the Agnew Relationship Measure and the Working Alliance Inventory. Psychol Assessment 2002 Jun;14(2):209-220 [FREE Full text] [CrossRef]
    26. Agnew-Davies R, Stiles WB, Hardy GE, Barkham M, Shapiro DA. Alliance structure assessed by the Agnew Relationship Measure (ARM). Br J Clin Psychol 1998 May;37(2):155-172 [FREE Full text] [CrossRef] [Medline]
    27. Kay-Lambkin FJ, Baker AL, Kelly B, Lewin TJ. Clinician-assisted computerised versus therapist-delivered treatment for depressive and addictive disorders: a randomised controlled trial. Med J Aust 2011 Aug 01;195(3):S44-S50 [FREE Full text] [CrossRef] [Medline]
    28. Ormrod JA, Kennedy L, Scott J, Cavanagh K. Computerised cognitive behavioural therapy in an adult mental health service: a pilot study of outcomes and alliance. Cogn Behav Ther 2010;39(3):188-192 [FREE Full text] [CrossRef] [Medline]
    29. Reynolds D, Stiles WB. Online data collection for psychotherapy process research. Cyberpsychol Behav 2007 Feb;10(1):92-99 [FREE Full text] [CrossRef] [Medline]
    30. Reynolds D, Stiles W, Grohol J. An investigation of session impact and alliance in internet based psychotherapy: preliminary results. Counselling Psychother Res 2006 Sep;6(3):164-168 [FREE Full text] [CrossRef]
    31. Kroenke K, Spitzer RL, Williams JB. The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med 2001 Sep;16(9):606-613 [FREE Full text] [CrossRef] [Medline]
    32. Meyer T, Miller M, Metzger R, Borkovec TD. Development and validation of the penn state worry questionnaire. Behav Res Ther 1990;28(6):487-495 [FREE Full text] [CrossRef]
    33. Muthén LK, Muthén BO. Mplus User’s Guide. Eighth Edition. Los Angeles, CA: Muthén & Muthén; 2017.
    34. Arbuckle J, Marcoulides G, Schumacker R. Full information estimation in the presence of incomplete data. In: Advanced Structural Equation Modeling Issues and Techniques. New York: Psychology Press; 1996:243-277.
    35. Bollen KA. A new incremental fit index for general structural equation models. Sociol Methods Res 2016 Jun 30;17(3):303-316 [FREE Full text] [CrossRef]
    36. Bentler PM, Bonett DG. Significance tests and goodness of fit in the analysis of covariance structures. Psychol Bulletin 1980;88(3):588-606 [FREE Full text] [CrossRef]
    37. Hu L, Bentler PM. Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives. Structural Equation Model Multidisciplin J 1999 Jan;6(1):1-55 [FREE Full text] [CrossRef]
    38. Byrne B. Structural Equation Modeling with EQS and EQS/Windows: Basic Concepts, Applications, and Programming. Thousand Oaks, California: SAGE; 1994.
    39. Milfont T, Fischer R. Testing measurement invariance across groups: applications in cross-cultural research. Int J Psychol Res 2010 Jun 30;3(1):111-130 [FREE Full text] [CrossRef]
    40. Heim E, Rötger A, Lorenz N, Maercker A. Working alliance with an avatar: how far can we go with internet interventions? Internet Interv 2018 Mar;11:41-46 [FREE Full text] [CrossRef] [Medline]
    41. Kiluk BD, Serafini K, Frankforter T, Nich C, Carroll KM. Only connect: the working alliance in computer-based cognitive behavioral therapy. Behav Res Ther 2014 Dec;63:139-146 [FREE Full text] [CrossRef] [Medline]
    42. Hersoug AG, Høglend P, Havik OE, Monsen JT. Development of working alliance over the course of psychotherapy. Psychol Psychother 2010 Jun;83(Pt 2):145-159 [FREE Full text] [CrossRef] [Medline]
    43. Andersson G, Paxling B, Wiwe M, Vernmark K, Felix CB, Lundborg L, et al. Therapeutic alliance in guided internet-delivered cognitive behavioural treatment of depression, generalized anxiety disorder and social anxiety disorder. Behav Res Ther 2012 Sep;50(9):544-550 [FREE Full text] [CrossRef] [Medline]
    44. Hadjistavropoulos H, Nugent MM, Alberts NM, Staples L, Dear BF, Titov N. Transdiagnostic Internet-delivered cognitive behaviour therapy in Canada: an open trial comparing results of a specialized online clinic and nonspecialized community clinics. J Anxiety Disord 2016 Aug;42:19-29 [FREE Full text] [CrossRef] [Medline]
    45. Preschl B, Maercker A, Wagner B. The working alliance in a randomized controlled trial comparing online with face-to-face cognitive-behavioral therapy for depression. BMC Psychiatry 2011 Dec 06;11:189 [FREE Full text] [CrossRef]

    ‎
    ARM-12: Agnew Relationship Measure, 12-item version
    CBT: cognitive behavioral therapy
    CONSORT: Consolidated Standards of Reporting Trials
    G-IBI-Text: guided internet-based intervention with text-based clinician support
    G-IBI-Video: guided internet-based intervention with video-based clinician support
    IBI: internet-based intervention
    PHQ-9: Patient Health Questionnaire-9
    PSWQ: Penn State Worry Questionnaire
    RMSEA: root mean squared error of approximation
    SDS: Sheehan Disability Scale
    U-IBI: unguided internet-based intervention

    Edited by J Torous; submitted 07.12.21; peer-reviewed by N Khalili-Mahani, M Wu; comments to author 09.04.22; revised version received 03.05.22; accepted 06.05.22; published 21.06.22

    Copyright

    ©Xiaochen Luo, Matteo Bugatti, Lucero Molina, Jacqueline L Tilley, Brittain Mahaffey, Adam Gonzalez. Originally published in JMIR Mental Health (https://mental.jmir.org), 21.06.2022.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Mental Health, is properly cited. The complete bibliographic information, a link to the original publication on https://mental.jmir.org/, as well as this copyright and license information must be included.