The GET READY study was a pre-post study for remitted patients with an anxiety or depressive disorder [15]. This paper presents the results pertaining to the usage intensity of the GET READY program, the course of symptoms (at baseline and after 3, 6, and 9 months), and the association between usage intensity and the course of symptoms.

This study was conducted in 50 primary care practice settings across the Netherlands. In the Netherlands, most primary care physicians (PCPs) employ an MHP (ie, nurse, psychologist, or social worker) who provides support and treatment for patients with mild mental health problems. These MHPs were involved in the GET READY program. Alternatively, for those patients whose MHP was not participating in the study, the program was offered via an ambulatory mental health care center. Patients began with an FTF meeting with an MHP, whereby they started composing a personalized relapse prevention plan. Next, patients could access web-based modules and a weekly diary via their computer, tablet, or smartphone. They were able to send messages to their MHP, ask for feedback on completed modules, and schedule FTF meetings with their MHP. All MHPs received a 4-hour training course, in which background information on relapse prevention, strategies for relapse prevention, and practical advice on using the program was provided [15]. PCPs did not play an active role in the study, although some MHPs regularly discussed patients with the PCP in their primary care practice.

Patients were eligible to participate if they had received treatment in specialized mental health care centers for anxiety or depressive disorder in the previous 2 years. After receiving acute phase treatment, they were referred to primary care services. They had to be in full or partial remission according to their MHP or clinician (clinical judgment), have scored 50 or higher on the Global Assessment of Functioning scale [16], be at least 18 years old, and be sufficiently fluent in Dutch. Patients were excluded if they were participating in another structured psychological intervention, had no access to the internet, or still received specialized treatment for a comorbid psychiatric disorder. Maintenance antidepressant use was allowed.

We sought to recruit 50 MHPs and 126 patients for this study. Sample size calculations have been described elsewhere [15]. MHPs and patients were recruited from April 2017 to November 2018. Fifty-four MHPs working in primary care practices throughout the Netherlands were recruited via telephone, letters, advertisements on MHP websites, and through the researchers’ professional networks. Informed consent was obtained from MHPs at the start of the training course. PCPs had to agree that MHPs participated in the GET READY study. Patients were recruited either by their MHP or by their clinician at the end of their treatment (N=113), who provided brief information about the study. If patients were interested in participating, then the MHP or clinician asked consent from the patient to provide their contact details to the researchers. Next, consenting patients were contacted by the researchers and received additional information. Informed consent was obtained prior to administering the baseline questionnaire. This questionnaire assessed whether patients met the inclusion criteria pertaining to remission by administering the Inventory of Depressive Symptomatology Self-Report (IDS-SR) and the Beck Anxiety Inventory (BAI). Remission was defined as a score of <39 on the IDS-SR and a score of <30 on the BAI. Scores above these cutoff points indicate severe symptoms that require additional treatment to relapse prevention [17,18]. Therefore, patients with a score of ≥39 on the IDS-SR or ≥30 on the BAI were excluded from the study.

The Medical Ethics Committee of the Vrije Universiteit University Medical Center Amsterdam deemed that ethical approval was not required according to Dutch legislation (registration 2016.280) and thus gave their permission to conduct the study.

The central aim of the program was to prevent relapse via the promotion of self-management skills. In the field of mental health, strengthening self-management skills is increasingly important, insofar as it allows patients to self-manage their own mental health [19]. More information regarding the content of the GET READY program has been published previously [15]. The program comprised several components. The program offered both FTF and web-based contact with an MHP. Every patient had at least 1 FTF engagement at the start of the study, and patients and MHPs were encouraged by the researchers to have FTF contact every 3 months. In addition, patients were encouraged to contact MHPs if their symptoms increased. In the FTF contact between patients and MHPs, usage of the program was discussed, and patients were encouraged to use the relapse prevention plan and to complete the diary and web-based modules. Patients were able to request feedback from their MHP when using the web-based modules. Besides the feedback on specific modules, patients and MHPs could also send and receive messages via the web-based platform. MHPs had access to their patients’ data and could check whether they had logged in or if they had completed modules and the weekly diary. In the event that a patient did not complete a module within a week, they were sent an automatic reminder.

The web-based modules were divided into 2 core modules “relapse psychoeducation” and “relapse prevention plan” (see Multimedia Appendix 1) and 12 optional modules, which included 3 psychoeducation modules with information on depression, anxiety, and medication. The other 9 optional modules contained information on specific topics such as exposure, negative thoughts, and sleep (see Figure 1). These modules also contained exercises, videos, and examples of fictive patients. Some modules had overlapping themes, and patients could easily open these linked modules from the other module (see the dotted lines in Figure 1). Finally, the GET READY program included a “mood and anxiety diary,” which allowed patients to monitor their symptoms. Patients received weekly reminders to complete the diary. When patients logged in for the first time, the core components “relapse psychoeducation,” “relapse prevention plan,” and the “mood and anxiety diary” were available. If patients completed the “relapse psychoeducation” module, the “depression/anxiety/medication psychoeducation” modules were automatically set up. Likewise, if patients completed the “relapse prevention plan,” they could choose which optional modules they wish to complete based on their preferences and goals.

Patients were invited to complete questionnaires at baseline (T0) and after 3 (T1), 6 (T2), and 9 months (T3). Completion of the questionnaires took 20-30 minutes. If necessary, patients received an email reminder after 1 week. As part of the treatment protocol, patients were also prompted to complete the mood and anxiety diary once a week for a period of 9 months (39 times). MHPs were requested to complete a case registration form after each FTF contact, in which the clinical status of patients and the duration and content of the FTF contacts were described. In order to assess the usage intensity of the program, log data from the web-based platform were collected.

Descriptive statistics were used to describe the characteristics of the participants to illustrate the extent to which patients used the program (over time) and to explore the course of symptoms. Explorative analyses were conducted to study the association between usage intensity and the course of symptoms. The course of symptoms was determined for both regular and low users in accordance with the “minimal usage intensity” measure as well as for the separate usage intensity measures. Differences in anxiety and depressive symptoms between regular and low users were tested using the Mann-Whitney U test (as these were nonnormally distributed), while Bonferroni corrections were applied to correct for multiple testing [26]. Generalized estimating equations (GEE) analyses were carried out to examine the longitudinal association between the different usage intensity variables and the course of symptoms. The usage intensity variables indicated usage of the program between baseline and T1, T1 and T2, and T2 and T3. In this way, the association between usage intensity and the course of symptoms at the point of each follow-up questionnaire was assessed by taking into account the usage intensity in the period immediately prior to the follow-up questionnaire. A time-lag model was used, in which an adjustment was made for the outcome at time point t – 1, as it assumed that the outcome at time t was predicted by the outcome at time t – 1 (see Figure 2). All data analyses were performed using SPSS 26 (IBM Corp). Further details regarding the methods employed in this study can be found in the study protocol [15].

The demographic and clinical characteristics of the participants (N=113) are reported in Table 1. The mean age at baseline was 43 (SD 12.9) years. More than half of the participants were females (65/113, 57.5%), while more than half of the participants attended higher professional education or university (64/113, 56.6%). Overall, 36.3% (41/113) of the participants reported being treated for a depressive disorder, 23.9% (27/113) stated they had been treated for an anxiety disorder, and 39.8% (45/113) stated they had been treated for both.

The use of the program is described in 3 subcategories: (1) contact with MHP, (2) completed modules, and (3) diary entries.

In the overall sample, anxiety and depressive symptoms decreased slightly over time. For all 113 participants, the mean BAI score at baseline was 10.2 (SD 6.6). After 9 months, the mean BAI score of the remaining 79 participants was 9.3 (SD 8.2). The differences over time were not significant, as indicated by the overlapping error bars in Figure S1 in Multimedia Appendix 2. A completer analysis (which included only those patients who completed T3) produced similar results. The mean IDS-SR score of all 113 participants decreased from 20.6 (SD 9.5) at baseline to 17.3 (SD 11.8) at 9 months (T3). The differences over time were not significant, as indicated by the overlapping error bars in Figure S2 in Multimedia Appendix 2. A completer analysis produced similar results. Regarding changes in symptomatology (stable/deteriorated/improved), for anxiety symptoms, it was found that the majority of the 79 patients who completed T3 remained stable over time (52/79, 66%), 12 patients (15%) experienced a deterioration of at least 1 SD (defined as a relapse), and 15 patients (19%) saw their anxiety symptoms improve. The numbers were comparable for depression symptoms: 53 patients (67%) remained stable, 8 patients (10%) deteriorated/relapsed, and 18 patients’ (23%) depressive symptoms improved. Seven patients (9%) experienced a deterioration in both their anxiety and depressive symptoms. For all 3 categories (stable/deteriorated/improved), antidepressant medication use remained largely stable. For depressive symptoms, most patients in the stable and deterioration group used antidepressant medication: 57% (30/53) and 75% (6/8) respectively. In the improved group, 50% (9/18) used antidepressant medication. For anxiety symptoms, the majority of patients in the stable and deterioration group used antidepressant medication: 64% (33/52) and 58% (7/12) respectively. In the improved group, 33% (5/15) used antidepressant medication.

This study has shown the usage intensity of the GET READY relapse prevention program, explored the course of symptoms of participants across the duration of the study, and examined the association between usage intensity and the course of symptoms. The core modules were used by ≥70% of the patients, while optional modules were regarded as elective and used as such (<40% of the patients). Of the 113 patients, 27 (23.9%) were defined as regular users according to the minimal usage intensity measure. Usage of the self-management components of the program (the web-based modules and web-based mood and anxiety diary) decreased quickly over time. Although no causal effect of the GET READY intervention on the severity of psychopathology could be established owing to its pre-post design, it appeared that most patients remained stable or experienced symptom improvement while they engaged with the GET READY program. Having more FTF contact with MHPs was significantly associated with an increase in anxiety and depressive symptoms. The other usage intensity variables were not significantly associated with the course of symptoms. Overall, the participants were highly educated and employed. These results are consistent with those reported in other studies on web-based or eHealth interventions [27,28], which have shown that this group is more likely to use web-based interventions. Similarly to Kontos et al [28], we also found it difficult to access patients with lower educational levels, which is problematic given that this program might also be beneficial for this group. Therefore, future relapse prevention studies should attempt to access participants with lower educational levels by seeking input from this group during the developmental phase of interventions.

The core components of the program were used fairly well, as the 2 core modules “relapse psychoeducation” and “relapse prevention plan” were completed by 74.3% (84/113) and 69.9% (79/113) of patients, respectively. As expected, optional modules were used less frequently than the core modules, with less than 40% of patients completing them. This result is consistent with data from Hollandäre et al [29], who showed that their basic modules were used more often than optional modules. Nonetheless, the usage of the optional modules in this study was relatively low in comparison to that in other studies [30-33]. The average usage intensity in these other programs was higher, with around 50% of the participants completing all available modules. However, these studies varied in terms of the number of modules (n=3-12), not to mention that the web-based programs were not focused on relapse prevention but rather on treating anxiety and depressive disorders. In this study, patients had already finished treatment for anxiety or depressive disorders, were in remission, and therefore may not have felt the need to actively engage in a relapse prevention program. One explanation for the relatively low usage could be simply that the patients experienced treatment fatigue [34]. Another explanation might be that since all the patients had received treatment prior to this study, the lessons learned from treatment were very much at the forefront of their mind and, as such, they felt no desire to repeat these lessons in the relapse prevention program. Indeed, it appeared that patients selected the modules that applied to their situation. To conclude, the optional modules in our relapse prevention program were regarded as elective and used as such.

Usage of the program decreased rapidly over time, as most patients used the program for a median of 1 month after registering on the web-based platform. Although this finding has been reported in previous studies on (web-based) guided self-help programs [10,11], it was contrary to the aim of the intervention, which was to provide a flexible program that can be used over a longer period of time depending on patients’ symptom levels. Prior studies demonstrated that both the absence of symptoms and an increase of symptoms might hinder patients’ capacity to actively use a program. Potentially, patients with fewer symptoms may not need to engage with the entire program to feel well again and cease using the program after obtaining the benefits [35,36]. At the same time, a qualitative study on the GET READY program indicated that increased symptom levels might also limit patients from further using the program [37]. An increase in depressive or anxiety symptoms may result in avoidance behavior, which may also lead to avoiding actively working on the web-based modules. This underlines the importance of the proactive role to be played by MHPs in terms of stimulating patients who are vulnerable to relapsing to continue using the program. Another important way of keeping patients engaged might be the further personalization of the intervention content, for example, by increasing the depth of tailored feedback, providing real-time feedback, and customizing the content based on current symptoms [9,37].

Although all of the patients were in remission, they nevertheless appeared vulnerable to relapse: patients had already received an average of 3.5 treatments in specialized care, 53.1% (60/113) had a family history of anxiety/depression, and 39.8% (45/113) had received treatment for both an anxiety and a depressive disorder, while their baseline mean symptom levels showed mild residual anxiety and depressive symptoms. In the overall sample, anxiety and depressive symptoms decreased slightly over time. Most participants remained stable, while 19%-23% of patients experienced symptom improvement. Only 10%-15% of the patients experienced a relapse. In comparison to other studies, our results show lower relapse rates [38,39]. Hardeveld et al [38] found that after 10 months, 20%-30% of patients experienced a relapse of their depressive symptoms. Similarly, Taylor et al [39] found that around 30% of patients experienced a relapse of their anxiety symptoms within a year. Although no causal pathway could be established in this pre-post study, these results nevertheless indicate that the GET READY program could potentially protect patients from relapse. However, as the definition of relapse differs across studies, comparing the results can be difficult. Therefore, efforts should be made in the field to reach a consensus regarding the definition and assessment of relapse in depression and anxiety disorders. Moreover, the effectiveness of the GET READY program in preventing relapse should be tested in a randomized controlled trial (RCT).

Patients who experienced a deterioration in symptoms more often used antidepressant medication than patients whose symptoms improved. However, this result should be interpreted with caution, as no causal pathway can be established. This study design is not feasible to investigate the influence of medication on the course of symptoms. Patients who had more FTF contact with MHPs had significantly higher anxiety and depressive scores than patients who had less FTF contact with MHPs. It is questionable whether the support they received by their MHP was sufficient to engender a subsequent decrease in symptoms. At the same time, this result might indicate that the web-based program in itself does not provide enough support to patients who experience a deterioration of symptoms. As aforesaid, this is a pre-post study; therefore, no causal pathway could be established [40]. An alternative explanation for this significant association might be that patients adequately responded to early symptoms of relapse by reaching out to their MHP. This explanation is in line with an earlier study, which showed that patients with more severe symptoms were more likely to receive help [41]. Furthermore, no evidence was found for an association between the number of completed modules and diary entries on the one hand and the course of symptoms on the other hand. Although no comparison with other relapse prevention programs is currently possible, other treatment studies have found that more completed web-based modules are associated with better anxiety and depression outcomes [31,42]. The same applies to the number of diary entries [43]. When comparing this study to these studies, it becomes apparent that the sample size of these studies was larger. Therefore, as well as the difference in population (remitted vs present disorder) and aims of the program (relapse prevention vs treatment), these studies may also have had more scope to detect an association.

There are several potential limitations in this study. First, attrition from the study was relatively high, with only 79 (69.9%) of the 113 participants completing the last follow-up questionnaire. Despite this, the statistical methods applied in this study, especially GEE analyses, are expedient for handling missing data. Second, self-selection bias and the fact that the patients were highly educated might restrict the generalizability of the results. However, Donkin et al’s [44] study showed no indication that these factors actually limit the generalizability, as they found no evidence that these factors were related to study outcomes. Third, this study had a limited follow-up period of 9 months. A longer follow-up period of 2 years would have provided greater insight into the course of anxiety and depressive symptoms over a longer period of time, which, in turn would have facilitated better comparison with other studies [38,45]. However, for pragmatic reasons, it was not feasible to extend the follow-up period. Fourth, owing to methodological considerations, no time-to-event analysis could be performed, as the assumptions for this analysis could not be met. Finally, the definitions of “regular use” that were used in this study should be interpreted with caution. As the median of several usage intensity measures was relatively low (ie, FTF contact median=1), regular use could still indicate relatively low usage when compared to the intended amount of usage (ie, FTF contact once every 3 months=3 FTF meetings). However, to enhance readability we opted to use the terms “regular use” and “low use.”

This study highlights the importance of providing personalized and guided relapse prevention to remitted patients with anxiety and depressive disorders. Usage of the program decreased quickly over time, possibly indicating a rapid decrease in the motivation of patients. As aforesaid, this decrease in motivation can be explained by different causal factors. Therefore, MHPs have the important task of monitoring and motivating patients via personalized intervention strategies, thus ensuring that patients receive guidance when they need it the most. Further research in an RCT with a longer follow-up duration is necessary to establish the effectiveness of blended relapse prevention programs. Within the design of an RCT, greater insight can also be obtained into the association between usage intensity and the course of symptoms.

When relapse prevention was offered, most patients used the core modules, while optional modules were completed by a smaller sample. As indicated in an earlier study [14], the patients showed that they preferred a low level of time investment for relapse prevention programs. Despite the relatively low usage and low time investment, most patients remained stable while participating in the GET READY study. Patients who had more FTF contact with their MHP experienced more anxiety and depressive symptoms. Owing to the pre-post design of this study, no causal pathway could be established. An RCT is needed to provide insight into the effectiveness of the GET READY program and to further explore the causal relationship between usage intensity and the course of symptoms.