This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Mental Health, is properly cited. The complete bibliographic information, a link to the original publication on http://mental.jmir.org/, as well as this copyright and license information must be included.
Background
University students with depression and anxiety do not easily receive or seek treatment; therefore, internet-based interventions have been suggested to be a promising way to improve treatment accessibility and availability. However, it has not been examined whether a guided, culturally adapted, transdiagnostic, internet-based intervention is effective for treating symptoms of depression, anxiety, or both among university students in Indonesia.
Objective
This study aims to investigate the feasibility (acceptability and satisfaction, usability, and uptake) of a guided, culturally adapted, transdiagnostic, internet-based intervention among university students with symptoms of depression, anxiety, or both in Indonesia.
Methods
Students from Universitas Gadjah Mada, Yogyakarta, Indonesia, were screened for symptoms of depression, anxiety, or both, and filled online informed consent, demographic questionnaires, and a quality of life measure at pretreatment assessment (T0). Subsequently, the participants started the intervention. Seven weeks after T0, the primary outcomes of this feasibility study were analyzed at posttreatment assessment (T1) using the 8-item Client Satisfaction Questionnaire (CSQ-8) and the System Usability Scale (SUS). Mean and SDs for the CSQ-8 and SUS were calculated to examine feasibility. Within-group secondary outcomes (depression, anxiety, and quality of life) were inspected for outliers and normal distribution. Paired-sample t tests were used to investigate differences between time points of secondary outcomes. A mixed-method approach of quantitative and qualitative analyses was adopted. Both the primary and secondary outcomes were additionally explored with an individual semistructured interview and synthesized descriptively.
Results
A total of 50 participants completed the intervention. We found a moderate to high level of satisfaction and acceptability, a slightly below-average level of desirable usability (≥70), and an adherence rate of 52% which was higher than expected given the novelty of the intervention. Results for the secondary outcomes indicated a decrease in depression and anxiety. For depression, the overall mean difference between the 2 time points for depression was 3.92 (95% CI 2.75-5.1; Hedges g 1.15; P<.001). For anxiety, the overall mean difference between the 2 time points was 3.34 (95% CI 2.06-4.61; Hedges g 1.02; P<.001). Further, a moderate effect in improving quality of life was found (g=0.50). Overall, participants were positive about the online intervention and ECoaches (online guidance), and they found the intervention to be culturally appropriate.
Conclusions
A culturally adapted, transdiagnostic, internet-based intervention appears to be acceptable and feasible for reducing symptoms of depression, anxiety, or both, and increasing quality of life in university students in Indonesia. Future studies should include a randomized controlled trial to assess the effectiveness of such interventions as they may supplement existing counseling services in universities, reduce the treatment costs, and maximize treatment accessibility in low-resourced settings.
International Registered Report Identifier (IRRID)
University students may experience common mental health disorders such as depression and anxiety [1-3]. Epidemiological studies [4] have shown that depression and anxiety are prevalent in college students (31%). This percentage is significantly larger when compared with nonstudents (21.4%) in the same age group [5]. The prevalence of anxiety and depression among university students may be influenced by the complexity of the transition from late adolescence to young adulthood [6]. This common mental health risk has been associated with academic demands [7], social adjustment [8], and financial challenges that students may face [9]. Depression and anxiety can be effectively treated with either pharmacotherapy or psychotherapy [10]. This is in line with the biopsychosocial approach suggested by the World Health Organization (WHO) for managing common mental disorders. The biopsychosocial approach promotes a thorough understanding that mental health issues are an outcome of biological, psychological, and social factors [11]. Effective treatment is imperative as it may prevent a series of negative cascades, such as poor academic adjustment, study dropout, and chronicity [12,13]. Many universities have counseling services that play an important role in improving mental health care by providing effective, accessible, and confidential ways to detect, prevent, and treat common mental disorders in students. The effectiveness of university counseling services for students with high levels of distress has been reported [14]. However, most university students do not receive or seek treatment at counseling services for their psychological problems [6] due to stigma, lack of time, lack of motivation, or they have a preference for a self-management approach [15]. In addition, the low university mental health service availability and high underutilization are very common in non-Western countries [16,17]. In Indonesia, university counseling services are unevenly distributed across universities, as they are mostly dependent on the presence and service of faculties of psychology in the universities.
Therefore, novel therapeutic delivery formats for university students with depression and anxiety may be a feasible way to improve treatment accessibility. Digital technologies offer an excellent opportunity for improving treatment availability and accessibility, especially in low- and middle-income countries, given the widespread acceptance of internet and mobile phone use among university students. Studies on interventions delivered via the internet have shown promising outcomes for students in Western countries due to their anonymity, accessibility, and adaptability to student needs [18,19]. Another known characteristic of internet-based interventions is that they can also be delivered with some form of human guidance (eg, by an ECoach) or can be purely self-guided [20]. Guided internet-based interventions were reported to be more effective compared to self-guided interventions, possibly because participants receive feedback and motivation, and this encourages them to proceed with the sessions and understand the content of the intervention better [21].
For Indonesian university students, internet-based interventions may have a high level of feasibility for a number of reasons. First, internet use is widespread among Indonesian university students (74.23% use the internet), compared to the general Indonesian population, with 94% of them using the internet on their mobile phones [22]. Second, guided internet-based interventions may be more attractive than face-to-face treatment in and outside the university, as the latter involves a high cost that might not be affordable for Indonesian students. Thus, internet-based interventions may be more favorable because they include minimal asynchronous support at a much lower cost than face-to-face interventions. Besides, such interventions can be accessed anywhere [20,23], at any time, and are thus more flexible for students who have very demanding daily schedules at the university.
Third, online interventions can reduce the fear of being exposed to others because students can access the online interventions anonymously [23]. By contrast, students may not utilize internet interventions for a number of reasons including time constraints due to their study duties, lack of nonverbal communication with the ECoach, and the high demand of self-discipline associated with self-help interventions [24]. Fourth, in Indonesia, internet-based interventions for depression and anxiety are currently not available for either university students or for the general population. The first randomized control trial on internet-based behavioral activation for adult depression in Indonesia [25,26] showed promising results with lower symptoms at posttest assessment in favor of the intervention with minimum support by a lay counselor who did not have professional education and qualifications in mental health care. However, the intervention was tested in the general population, and thus, it remains unclear whether these interventions are also effective in university students with elevated symptoms of depression and anxiety. Additionally, given that depression and anxiety are highly comorbid conditions [27,28], a transdiagnostic approach could be beneficial as it targets both symptoms of depression and anxiety simultaneously. Findings on transdiagnostic interventions for university students in Western countries with common mental disorders have been encouraging. More specifically, such interventions have shown to reduce symptoms of common mental disorders with moderate to large effect sizes in the range of 0.42-0.80 for outcomes such as depression and anxiety [23,29-31].
To the best of our knowledge, it has not yet been examined whether guided, culturally adapted, transdiagnostic, internet-based interventions are feasible in treating symptoms of depression or anxiety or both among university students in non-Western countries, such as Indonesia. Therefore, in this pilot study, we aimed to investigate the feasibility (defined as acceptability, satisfaction, usability, and uptake) of a guided, culturally adapted, transdiagnostic internet-based intervention among university students with symptoms of depression, anxiety, or both in Indonesia. This paper reports the results of stage 3 from a theory-based cultural adaption framework, namely, preliminary adaptation test [32]. The first 2 stages of the framework, namely, information gathering and preliminary adaptation design, are reported in the protocol paper of this study [33].
MethodsParticipants
Participants were recruited via our study website [34], which was disseminated on posters, social media platforms, and business cards containing brief information about the study. As we described in our protocol [33], there is no golden standard for calculating the sample size of a pilot study [35]. For our study, we performed a post hoc power calculation to describe statistically the power of our chosen sample (n=50 participants). Further details can be found in our study protocol [33]. Participants were eligible if they (1) were students at Universitas Gadjah Mada Yogyakarta with access to broadband internet, (2) were 18 years of age or older, (3) could speak and read Bahasa Indonesia fluently, and (4) experienced mild to moderate depression, anxiety, or both (9-item Patient Health Questionnaire [PHQ-9] score >4, 7-item Generalized Anxiety Disorder [GAD-7] score >4, or both). Exclusion criteria were (1) moderately severe depression, anxiety, or both (PHQ-9 score >14, GAD-7 score >14, or both) and (2) currently receiving psychological treatment for depression, anxiety, or both.
Procedures
Students who were interested in participating in the pilot study filled in their demographic details on the digital registration form on our study website. Subsequently, the students received an email with a link, which directed them to the pretreatment assessment (T0) consisting of the study’s screening questionnaires: the Indonesian version of the PHQ-9 [36,37] and the GAD-7 scale [36,37]. After completing the PHQ-9 and GAD-7 at the screening, participants were asked to provide an informed consent for using their responses to these questionnaires in our analyses (retrospectively). For pragmatic reasons, these eligibility screening scores were also used as a baseline assessment for included participants. Subsequently, an information sheet was provided to them through an emailed link containing an explanation of the study and an online informed consent form that needed to be signed before being able to participate. Those who consented to participate received additional questions about their demographics and quality of life using the Indonesian version of the Euro Quality of Life 5 Dimension-5 Level Scale (EQ5D5L) [38]. The EQ5D5L was used because performance and changes in mental health are captured through one of its dimensions (depression/anxiety) and it is a widely used generic quality of life scale that is also usable for health economic calculations. After completing these additional questionnaires, participants were assigned online to an ECoach (a trained Clinical Psychology Master’s student from the Faculty of Psychology at Universitas Gadjah Mada or a licensed Psychologist) and were given secure login details. Subsequently, the I-AiMentalWELLness (Saya menuju mental sehat) online intervention was activated for them through Mind District [39], an eHealth platform providing digital therapy.
The posttreatment assessment (T1) was given 7 weeks after the pretreatment assessment (T0) and consisted of the primary outcomes of this feasibility study, namely, participants’ reported acceptability and satisfaction, usability, and uptake.
Further details about the procedures of this pilot study can be found in our protocol paper [33]. Ethical approval was obtained from the Medical and Health Research Committee of the Medical Faculty in Universitas Gadjah Mada/DR Sardjito General Hospital (reference number: KE/FK/0098/EC/2018).
MeasuresPrimary Outcomes
The participants’ reported acceptability and satisfaction, as well as the usability and uptake of the I-AiMentalWELLness (Saya menuju mental sehat) intervention were the primary outcomes of this feasibility study. Acceptability and satisfaction were measured using the Indonesian version of the 8-item Client Satisfaction Questionnaire (CSQ-8) [40]. Moreover, we used the Indonesian version of the System Usability Scale (SUS) [41] as a usability outcome. While the psychometric properties for the Indonesian version of the CSQ-8 and the SUS are currently not available, both had good psychometric properties in studies conducted in Western countries (CSQ-8 and SUS Cronbach =.93 and .90, respectively) [42-44]. The internal consistency for CSQ-8 and SUS in this study was good with Cronbach α=.87 and .83, respectively. Moreover, the CSQ-8 has been adapted to internet-based interventions (CSQ-I) and has demonstrated overall good psychometric properties in 2 studies [45].
In the study protocol, we reported that intervention uptake would be measured by participants’ adherence to the intervention. Adherence was defined as the number of log-ons, time spent on site, and number of sessions attempted. However, due to the technical limitations of the platform used, we could not measure adherence using the numbers of log-ons and time spent online as we originally intended to do [33]. Thus, in this paper, we have described adherence as the number of participants who completed the sessions in the online intervention divided by the number of participants who started the intervention [46]. We also calculated the initial uptake rates by dividing the number of participants who responded to the invitation link from the number of participants who initially expressed interest in participating in the study. We did so to give an overall impression about how many students remained interested in doing the intervention when they had the option to do so.
Secondary Outcomes
Secondary outcome measures included depression, anxiety, and quality of life, which were measured using PHQ-9, GAD-7, and EQ5D5L, respectively. The psychometric properties for the Indonesian version of PHQ-9 and GAD-7 are currently not available. Nonetheless, both PHQ-9 and GAD-7 have demonstrated good psychometric properties in studies conducted in Western countries. The PHQ-9 had high internal consistency (α=.89) and a reliability value of 0.84 [47]. Meanwhile, GAD-7 indicated good internal consistency (α=.92) and test–retest reliability of 0.83 [48]. The internal consistency for PHQ-9 was on the edge of being satisfactory (Cronbach α=.70) and that for GAD-7 was good (Cronbach α=.83). The test–retest reliability of the Indonesian version of EQ5D5L was assessed with sequential measurements using the weighted kappa. Results indicated fair agreements of 0.35, 0.30, 0.37, and 0.39 for the dimensions of mobility, self-care, usual activities, and anxiety/depression, respectively [49]. Meanwhile, the intraclass correlation coefficient for Visual Analog Scale was reported as 0.32, indicating a moderate value [49].
Semistructured Interviews
Both the primary and secondary outcomes of this feasibility study were explored additionally with an individual semistructured interview to obtain a broader perspective and realistic understanding of acceptability, satisfaction, and usability concerning the culturally adapted online intervention for our student population. The semistructured interviews were conducted after participants completed the interventions, by videoconferences between the first author (MR) in The Netherlands and 10 participants in Indonesia. We took account of heterogeneous factors by ensuring equal representation of age, gender, educational, and ethnic background to gain a better understanding of the primary outcome. The demographic characteristics of the participants included in the semistructured interview are reported in Table 1.
Demographic characteristic of participants included in the semistructured interview (N=10).
Characteristics
Value
Age (years), mean (SD); range
24.5 (6.73); 19-41
Gender, n (%)
Female
6 (60)
Male
4 (40)
Ethnicity, n (%)
Java
8 (80)
Other
2 (20)
Level of education, n (%)
Bachelor
6 (60)
Doctorate
1 (10)
Master
3 (30)
Study program, n (%)
Psychology
3 (30)
Medical
2 (20)
Other
5 (50)
The interview questions were modified from the study of Devi et al [50] to fit the purpose of this study (Multimedia Appendix 1). The participants’ responses were audio-recorded, and additional note taking was also performed during data collection by the first author and 1 team member (local supervisor) in Indonesia. The interview time range was approximately 30 minutes and was conducted in Bahasa Indonesia.
The secondary outcomes were further explored to understand (1) each participant’s evaluation of his/her ECoach, and (2) the cultural appropriateness of the adapted internet-based intervention. These interview questions were modified [51] to fit the purpose of the study. The interview questions can be found in Multimedia Appendix 1.
The I-AiMentalWELLness Intervention
This intervention targets common cognitive and behavioral processes of both anxiety and depression across all sessions; therefore, it is considered a transdiagnostic intervention. Moreover, this intervention was based on cognitive behavioral therapy principles [51,52]. It was originally developed for the general population in Germany and Switzerland [53] and then adapted to meet the needs of domestic and international university students in The Netherlands [54,55]. For the purpose of this study, the current version was culturally adapted from English to Bahasa Indonesia for the student population based on the heuristic theoretical framework [32] which entails (1) information gathering, (2) preliminary adaptation design, (3) preliminary testing, (4) adaptation refinement, and (5) cultural adaptation trial. As reported in the study protocol [33], the first and second phase involved end users (students) as part of the developmental process of the current intervention. Further, the cultural adaptation process concerned all elements of the intervention including language, images, testimonials, and other examples that might not be applicable to the Indonesian culture. Core components of the intervention (eg, behavioral activation, problem solving, cognitive restructuring) remained unchanged. More specifically, after the focus group discussions, we omitted all parts related to unmarried cohabitation, sexual activities, and terms related to alcoholic beverages due to the majority of university students being Muslim. These topics are considered inappropriate in the Islamic religion. We have also changed all parts related to medication such as antidepressants and sleep medication, winter-related sports, and membership in sports and music school. These culturally adapted changes did not influence the main therapeutic components of the internet-based cognitive behavioral therapy intervention (eg, cognitive restructuring) but they solely reflected the local context. An image of one of the sessions in the I-AiMentalWELLness (Saya menuju mental sehat) intervention is provided in Multimedia Appendix 2.
The online intervention consisted of 7 sessions and an additional booster session, which took place 4 weeks after the seventh session of the intervention was completed. The purpose of the booster session was to refresh the memory of the participants about what they learned in the previous 7 sessions of the intervention. It aimed at reinforcing the progress that has been made throughout the interventions and to prepare in case the participants encounter a new episode of, for example, depressive mood. The transdiagnostic approach for anxiety or depression is found particularly in sessions 5 and 6. For further detailed description of the intervention’s content, the reader is referred to Table 2.
The content of the intervention.
Module
Content description
Session 1: Identifying my needs
Goal setting and behavioral activation
Session 2: Taking action
Problem solving
Session 3: Worth knowing
Psychoeducation on depression and anxiety
Session 4: Thoughts pattern
Cognitive restructuring
Session 5: Dealing with challenges
Solve problems concerning depressive symptoms or exposure to anxiety-provoking situations
Session 6: In daily practice
Solve problems concerning depressive symptoms or exposure to anxiety-provoking situations
Session 7: Future plan
Planning for the future
Session 8: Strong going forward
Booster session
OMa 1: Sleep
Information related to sleep or how to sleep better
OM 2: Perfectionism
Identify high standards and understanding the vicious cycle of perfectionism
OM 3: Gratitude and appreciating good things
Express gratitude and appreciate goodness in life
OM 4: Self-worth
Understanding low self-worth and how to increase it
OM 5: Relaxation
Progressive muscle relaxation
OM 6: Acceptance
Learning to accept unfulfilled needs
OM 7: What is brooding and when is it excessive?
Information on rumination and learning how to overcome it
aOM: optional modules.
Each session consisted of text, exercises, testimonials, and audio recordings with a duration of approximately 60 minutes to complete. A monetary incentive of Indonesian Rupiah (IDR) 125.000 (equivalent to €7/US $8.5) was given to participants who completed sessions 1-4 and another IDR 125.000 (equivalent to €7/US $8.5) was given to participants who completed sessions 5-8. This monetary incentive was meant to compensate the prepaid cards that participants used to access internet via their mobile phones. Participants were only allowed to complete a maximum of 2 sessions per week to have enough time to integrate the skills acquired from the intervention into their daily life. After the completion of each session, participants received individualized asynchronous feedback from their ECoach through the messaging system of the platform. According to our protocol, each ECoach was advised to spend a maximum of 30 minutes to give feedback per participant at the end of each session. The feedback was brief and of motivational nature to encourage participants to continue with the intervention. Participants were able to move to the next session only after reading the feedback of their ECoach. Furthermore, participants were allowed to contact their ECoach whenever they wanted via the messaging system in the platform. In such case, the ECoaches were advised to respond within 48 hours. When the feedback had been read by the participant, subsequently, the ECoach unlocked the next intervention session. The feedback was of a motivational nature, meaning it consisted of positive reinforcement and encouragement, which reflected on participants’ responses to the online exercises and homework assignments.
Statistical and Descriptive Analysis of Qualitative Data
Quantitative analyses were conducted using the IBM SPSS version 25. Because the primary outcomes regarding acceptability, satisfaction, and usability were only assessed after completion of the interventions, these were analyzed only at posttreatment assessment. We calculated means and SD for the CSQ-8 and SUS to display feasibility. Descriptive statistics were used to summarize (1) participants’ demographic information (age, gender, employment, ethnicity, marital status, study program, and level of education) and (2) participants’ uptake.
For the quantitative analyses of within-group secondary outcomes, data were inspected for outliers and normal distribution. In order to investigate differences between pre- and post-assessment of secondary outcomes, paired-sample t tests were used. To investigate baseline differences between completers and dropouts, an independent sample t test or a Mann–Whitney U test was used. Effect sizes were measured using Hedges g, and interpreted as small=0.2, medium=0.5, and high effect=0.8 [56].
In the protocol, we suggested performing thematic analysis according to the guidelines [57] to analyze the transcripts of the interviews with the participants. However, the data were not suitable for this approach because the participants did not elaborate in depth on the questions asked. Thus, we did not analyze the results of the interview but instead we descriptively synthesized the qualitative data for both primary (acceptability, satisfaction, and usability regarding the internet-based intervention) and secondary outcomes (ECoach evaluation and cultural appropriateness of the adapted internet-based intervention).
ResultsParticipants
Of the 240 participants who expressed an interest in participating in the study, 118 did not further respond to the invitation link to participate in the screening (initial uptake=50.8%, 122/240). Of those (n=122) screened for eligibility using the PHQ-9 and GAD-7, 96 were eligible to participate in this study and 26 were excluded after obtaining scores indicating moderately severe to severe or no depression or anxiety. Participants with severe depression or anxiety (n=11) were referred to student counseling services, while those who scored below the mild threshold for depression and anxiety (n=15) were excluded and thanked for their participation and interest in our study. The remaining 96 participants were subsequently invited to provide online informed consent, and they completed the additional quality of life measurement (EQ5D5L) and started the first session (intervention uptake=40.0%, 96/240). Fifty participants completed all 7 sessions and the posttreatment assessment. Almost all participants retained at postassessment also followed the booster session (n=46) 4 weeks after the seventh session.
A total of 46 participants discontinued the intervention during sessions 1-4. On the Minddistrict platform, the ECoaches were able to view their participants’ platform, enabling them to notice inactivity such as unread feedback or no further progress on a session. If such inactivity was observed, the ECoach made further attempts to contact the participants through email and text messages (eg, SMS text messages or WhatsApp). If the ECoach received no response from inactive participants, they were considered to have discontinued the intervention. In the end, the participants who discontinued (n=46) were not contacted again for the posttreatment assessment because all previous attempts at communication had failed. Last, 50 participants dropped out from the intervention (sessions 1-7 and booster session 8) and 46 participants dropped out from the study. The flow chart of study participants is presented in Figure 1.
The flowchart of study participants.
The demographic characteristics of the participants and baseline differences between completers and dropouts are presented in Table 3. There were no significant differences in baseline assessment between completers and dropouts (see also Table 3).
Participants’ characteristics and baseline differences between completers and dropouts.
Characteristics
Completers (N=50)
Dropout (N=46)
t (df)/ꭓ2 (df)
P value
95% CI
Age (years), mean (SD)
22 (3.74)
22 (5.0)
0.11 (94)a
.91
Gender,n (%)
0.003 (1)b
.96
Female
40 (80)
37 (80)
Male
10 (20)
9 (20)
Ethnicity, n (%)
0.85 (1)b
.36
Java
44 (88)
43 (93)
Other
6 (12)
3 (7)
Marital status, n (%)
1.62 (2)b
.44
Divorced
1 (2)
0 (0)
Married
3 (6)
5 (11)
Single
46 (92)
41 (89)
Occupational status,n (%)
0.53 (1)b
.47
Employed
7 (14)
9 (20)
Unemployed
43 (86)
37 (80)
Level of education,n (%)
0.50 (2)b
.78
Bachelor
40 (80)
35 (76)
Doctorate
1 (2)
2 (4)
Master
9 (18)
9 (20)
Studyprogram,n (%)
2.68 (1)b
.10
Psychology
21 (42)
12 (26)
Other
29 (58)
34 (74)
PHQ-9c, mean (SD)
9.46 (3.59)
9.33 (3.41)
0.00 (94)a
.99
–0.06 to 0.06
GAD-7d, mean (SD)
7.54 (3.8)
7.37 (3.46)
–0.22 (94)a
.82
–1.65 to 1.30
EQ5D5Le, mean (SD)
0.83 (0.10)
0.86 (0.12)
916.5f
.08
aIndependent sample t test.
bChi-square test.
cPHQ-9: 9-item Patient Health Questionnaire.
dGAD-7: 7-item Generalized Anxiety Disorder.
eEQ5D5L: Euro Quality of Life 5 Dimension-5 Level Scale.
fMann–Whitney U test.
Primary Outcomes
The participants who completed the intervention (N=50) had a mean score of 25.8 (SD 3.40) on CSQ-8, indicating moderate to high level of satisfaction and acceptability regarding the online intervention. They had a mean score of 65.1 (SD 13.37) on SUS, indicating a slightly below average of the desirable usability of 70 or more [58]. Of the 96 participants who started, only 50 completed the intervention sessions, resulting in a 52% adherence rate.
Secondary Outcomes
There was a significant decrease in the PHQ-9 and GAD-7 scores. For depression, the overall mean differences in PHQ-9 between the 2 time points was 3.92 (95% CI 2.75-5.1; Hedges g 1.15; P<.001). For anxiety, the overall mean differences between the 2 time points in GAD-7 was 3.34 (95% CI 2.06-4.61; Hedges g 1.02; P<.001), as reported in Table 4. Regarding the EQ5D5L scores, there appeared to be an outlier. However, excluding this outlier did not affect the results, therefore the participant was not excluded from the analysis. At posttreatment assessment, there was a significant increase (P=.008) in the EQ5D5L scores, indicating improved quality of life (Table 4).
Pretreatment and posttreatment assessment for PHQ-9, GAD-7, and EQ5D5L.
Variable
T0, mean (SD)
T1, mean (SD)
t (df)/Z
P value
Effect size (g)
95% CI
PHQ-9a
9.46 (3.59)
5.54 (3.21)
6.75 (49)b
<.001
1.15
2.75-5.1
GAD-7c
7.54 (3.8)
4.2 (2.65)
–4.22d
<.001
1.02
2.06-4.61
EQ5D5Le
0.83 (0.10)
0.88 (0.10)
–2.66d
.008
0.50
aPHQ-9: 9-item Patient Health Questionnaire.
bPaired sample test.
cGAD-7: 7-item Generalized Anxiety Disorder.
dWilcoxon signed-rank test.
eEQ5D5L: Euro Quality of Life 5 Dimension-5 Level Scale.
Descriptive Analysis of the Qualitative Data
Overall, participants (n=9) reported to be satisfied when using the online intervention. It was reported to be informative, challenging, and triggered them to further recognize the symptoms of depression and anxiety as a student. The optional module was rated as being the most enjoyable part of the intervention as it provided many personal insights. The recorded relaxation was also in favor as it delivered a calm feeling. Participants further acknowledged the online intervention to be more practical than face-to-face interventions as it overcame geographical and stigma barriers. Only one participant reported being confused for not receiving an immediate response to questions that might arise when working through the sessions.
Good user experience with the intervention was reported, indicating clarity in the instructions and workflow in a session. However, difficulties were also reported as 1 participant felt unfamiliar with reading online content, whereas 2 others had problems with the stability of the internet connection when working on the intervention in a remote area.
Regarding the semistructured interviews, 6 out of 10 participants responded positively on being supported through the internet by an ECoach as they received sufficient support, encouragement, reminders, and had their work progress monitored. Nonetheless, 4 out of 10 participants were less positive about having an ECoach because of the unavailability of synchronous feedback and having difficulty with writing what is on their mind. Moreover, they reported that the likelihood of being misinterpreted was high because of absent nonverbal communication signals. Finally, feedback was written in a generic manner and did not always cover specific tasks, as these participants would have expected the feedback to be more elaborative.
As much as 7 out of 10 participants indicated that the overall content of the intervention was culturally appropriate and relevant to university students in Indonesia. Nonetheless, 3 out of 10 participants indicated that some case examples of the intervention were less relevant to the Indonesian context. For instance, it is uncommon for Indonesian university students to have a part-time job during bachelor studies or to be enrolled in a sport and music academy.
Some talk about working. Not all students work, especially bachelor students.
There were some story examples that seem less relevant to the Indonesian culture, such as joining a music or a sport club.
DiscussionPrincipal Findings
This study has demonstrated promising outcomes for the feasibility of an internet-based intervention for university students in Indonesia, as indicated by participants’ reported acceptability, satisfaction, usability, and uptake regarding the intervention. In addition, a decrease in symptoms of depression and anxiety, and improved quality of life were reported.
Participants reported to have a moderate to a high level of satisfaction and acceptability regarding the online intervention. Results from the interviews also indicated that participants were satisfied with the online intervention as a new approach that acknowledged personal development, and overcame stigma and geographical barriers. The usability/functionality of the internet-based intervention as it applies to the end users was also assessed. One component of usability explained [41] is learnability, which assesses first-time experiences with a new system for end users. A transdiagnostic approach is characterized by a focus on common characteristics that may underpin a number of different psychological disorders such as in case of depression and anxiety disorders. Advantages of such a treatment approach include treating simultaneously these underlying common factors in case of comorbid symptoms [59]. In light of our positive findings regarding the reduction in anxiety and depressive symptoms, the transdiagnostic approach appears promising in the given context. Further research is needed to investigate the effectiveness of such approach in a randomized control trial. However, a transdiagnostic approach delivered online to treat symptoms of depression and anxiety under a single protocol is relatively new. The findings for the mean usability rate for the internet intervention are slightly below average (65.1) for the desired usability ≥70 [58] which is not surprising. Thus, further development and continued improvement are needed to provide individually tailored transdiagnostic intervention, such as by minimizing the use of formal language in the intervention.
With respect to the secondary outcomes of this study, we found a significant decrease (P<.001) in symptoms of depression and anxiety at the posttreatment assessment, and a significant increase (P=.008) in quality of life. Online interventions may offer a promising treatment solution, given the high prevalence of common mental disorders among university students and the under-resourced counseling services [1,2,10]. Many studies have indicated the effectiveness of an online intervention to treat depression and anxiety among university students [10,19]. Understanding quality of life among university students is also important, as it contributes to their physical, psychological, social, and environmental well-being [60]. As the quality of life encompasses various dimensions closely related to the life of a university student, it is important that symptoms of depression and anxiety are treated properly.
It must be noted, of course, that our study did not have a control group. Therefore, the improvements might be attributed to other factors than the intervention, such as spontaneous recovery or regression toward mean.
All participants of this feasibility study were guided by an ECoach who monitored their progress throughout the intervention. The likelihood of completion rate may be associated with therapeutic guidance [42,58,60]. However, we have not tested this hypothesis, and this remains to be confirmed by future studies in low-resource settings.
The secondary outcome of this intervention was further supported by the interview results, which indicated participants’ overall positive response to being treated through the internet by an ECoach, the extent of support received, the average time with regard to receiving feedback, and how comprehensively the feedback was delivered.
A randomized controlled trial will follow this promising feasibility study to further investigate the effectiveness of the existing internet-based intervention to treat depression and anxiety for students within university setting in Indonesia. This is a necessary step before this intervention is made publicly available in Indonesia.
Strengths and Limitations
To the best of our knowledge, this study was one of the first to investigate the feasibility of a guided culturally adapted internet-based intervention to treat symptoms of depression, anxiety, or both among university students in Indonesia. Originally, in the study protocol, we defined appropriate uptake as an adherence rate of 35% due to the novelty of the intervention in Indonesia. However, participants’ adherence rates reached 52% (50/96), which is higher than expected and underlines the feasibility of online interventions for Indonesian university students. It should be noted that we provided monetary incentives to participants who completed the sessions. Thus, it is possible that these incentives have enhanced participation rates. Nevertheless, the amount of incentives was very small and thus it is unlikely that participants completed the study only because of this reason. This feasibility study also has several limitations that must be taken into account. First, the majority of the participants were female, bachelor students, Javanese, and were studying psychology. Future research could benefit from investigating more diverse groups from other faculties and universities to improve the generalizability of these preliminary findings. Second, 46 participants could not be invited to undergo the posttreatment assessment, because they could not be contacted by the ECoach by any means. The research team were unable to locate these participants and thus it limits the understanding of reasons for dropout. Third, data regarding participants’ uptake (log-on time, time spent on site, and number of sessions attempted) of the online intervention could not be provided in this study as participants had previously been instructed to record the time that they spent online. They did so to avoid confusion because Minddistrict (a Dutch company) displayed the Central European time zone. However, this self-reported approach was considered subjective. Other attempts were made by contacting the platform provider to obtain the participants’ recorded time of log-ons but such information was not monitored by the platform. Fourth, the findings regarding the secondary outcome within 2 different time points need to be interpreted with caution as this study was a small pre–post assessment and not a randomized controlled trial. Fifth, another limitation of our study is that our qualitative data were limited in depth, and thus not well suited to thematic analysis. This might have been the result of the way we phrased these questions. In particular, some questions may have been formulated in a too closed manner, thereby preventing the respondents from giving more elaborate answers and convey possible nuances that qualitative research requires. However, the aim of this part of our study was to enrich our quantitative findings. Thus, future qualitative research in this field should aim at more open-ended formulated questions if the aim is to gain in-depth insights into the topic.
Conclusion
Internet-based interventions appear to be acceptable and feasible for reducing symptoms of anxiety and depression and improving quality of life among university students in Indonesia. Future studies should further investigate the (cost-) effectiveness of such interventions as they may be used to supplement existing counseling services in universities, reduce the treatment costs, and maximize treatment accessibility in low-resourced settings.
Semi-structured interview questions.
Screenshot example of intervention session 3 in Bahasa Indonesia.
AbbreviationsCSQ 8
Client Satisfaction Questionnaire-8
EQ5D5L
Euro Quality of Life 5 Dimension-5 Level Scale
GAD 7
Generalized Anxiety Disorder-7
IDR
Indonesian Rupiah
PHQ 9
Patient Health Questionnaire-9
SUS
System Usability Scale
This work was supported by the Indonesia Endowment Fund for Education (Lembaga Pengelola Dana Pendidikan), Ministry of Finance, Republic of Indonesia which provided a PhD scholarship and research funding for the first author (MR). The first author wishes to thank the research assistants Aril, Achi, Icha, Erna, Rara, Olaff, Isti, Deksi, Nana, Dila, Berta, and the study participants of the I-AiMentalWELLness intervention.
HR contributed to the design of the study. MR drafted the manuscript under the supervision of EK and MS. All authors contributed to revising the manuscript. All authors read and approved the final manuscript.
DE has served as a consultant to/on the scientific advisory boards of Sanofi, Novartis, Minddistrict, Lantern, Schoen Kliniken, Ideamed and German health insurance companies (BARMER, Techniker Krankenkasse) and a number of federal chambers for psychotherapy. He is also stakeholder of the Institute for health training online (formerly GET.ON/ now HelloBetter), which aims to implement scientific findings related to digital health interventions into routine care. The remaining authors declare no conflict of interest.
LunKWChanCIpPKMaSYTsaiWWongCWongCHWongTYanDDepression and anxiety among university students in Hong KongHong Kong Med J201892424546647210.12809/hkmj17691530245480FawzyMHamedSAPrevalence of psychological stress, depression and anxiety among medical students in EgyptPsychiatry Res20170925518619410.1016/j.psychres.2017.05.02728575777S0165-1781(16)31300-2SaidDKypriKBowmanJRisk factors for mental disorder among university students in Australia: findings from a web-based cross-sectional surveySoc Psychiatry Psychiatr Epidemiol2013064869354410.1007/s00127-012-0574-x22945366AuerbachRPMortierPBruffaertsRAlonsoJBenjetCCuijpersPDemyttenaereKEbertDDGreenJGHaskingPMurrayENockMKPinder-AmakerSSampsonNASteinDJVilagutGZaslavskyAMKesslerRCWHO WMH-ICS CollaboratorsWHO World Mental Health Surveys International College Student Project: Prevalence and distribution of mental disordersJ Abnorm Psychol201810127762363810.1037/abn0000362302115762018-44951-001PMC6193834AuerbachRPAlonsoJAxinnWGCuijpersPEbertDDGreenJGHwangIKesslerRCLiuHMortierPNockMKPinder-AmakerSSampsonNAAguilar-GaxiolaSAl-HamzawiAAndradeLHBenjetCCaldas-de-AlmeidaJMDemyttenaereKFlorescuSdeGGGurejeOHaroJMKaramEGKiejnaAKovess-MasfetyVLeeSMcGrathJJO'NeillSPennellBScottKTenHMTorresYZaslavskyAMZarkovZBruffaertsRMental disorders among college students in the World Health Organization World Mental Health SurveysPsychol Med20161046142955297010.1017/S003329171600166527484622S0033291716001665PMC5129654PedrelliPNyerMYeungAZulaufCWilensTCollege Students: Mental Health Problems and Treatment ConsiderationsAcad Psychiatry201510213955031110.1007/s40596-014-0205-92514225010.1007/s40596-014-0205-9PMC4527955NamSKChuHJLeeMKLeeJHKimNLeeSMA meta-analysis of gender differences in attitudes toward seeking professional psychological helpJ Am Coll Health201059211011610.1080/07448481.2010.48371420864437NPM094233K815H3XPerryCLauschDWWeatherfordJGoekenRAlmendaresMInternational students' perceptions of university lifeCollege Student Journal2017512279290ul HaqMADarISAslamMMahmoodQKPsychometric study of depression, anxiety and stress among university studentsJ Public Health2017102426221121710.1007/s10389-017-0856-6CuijpersPSijbrandijMKooleSLAnderssonGBeekmanATReynoldsCFThe efficacy of psychotherapy and pharmacotherapy in treating depressive and anxiety disorders: a meta-analysis of direct comparisonsWorld Psychiatry2013061221374810.1002/wps.2003823737423PMC3683266BabalolaENoelPWhiteRThe biopsychosocial approach and global mental health: Synergies and opportunitiesIndian J Soc Psychiatry201733429110.4103/ijsp.ijsp_13_17CuijpersPAuerbachRPBenjetCBruffaertsREbertDKaryotakiEKesslerRCThe World Health Organization World Mental Health International College Student initiative: An overviewInt J Methods Psychiatr Res20190106282e176110.1002/mpr.176130614123BolinskiFBoumparisNKleiboerACuijpersPEbertDRiperHThe effect of e-mental health interventions on academic performance in university and college students: A meta-analysis of randomized controlled trialsInternet Interventions2020042010032110.1016/j.invent.2020.100321MurrayALMcKenzieKMurrayKRRichelieuMAn analysis of the effectiveness of university counselling servicesBritish Journal of Guidance & Counselling2015061244113013910.1080/03069885.2015.1043621YorgasonJBLinvilleDZitzmanBMental Health Among College Students: Do Those Who Need Services Know About and Use Them?Journal of American College Health2010080657217318210.3200/jach.57.2.173-18218809534RobertsTMiguel EspondaGKrupchankaDShidhayeRPatelVRathodSFactors associated with health service utilisation for common mental disorders: a systematic reviewBMC Psychiatry201882218110.1186/s12888-018-1837-130134869Lukito SetiawanJWillingness to seek counselling, and factors that facilitate and inhibit the seeking of counselling in Indonesian undergraduate studentsBritish Journal of Guidance & Counselling20060834340341910.1080/03069880600769654DaviesEBMorrissRGlazebrookCComputer-delivered and web-based interventions to improve depression, anxiety, and psychological well-being of university students: a systematic review and meta-analysisJ Med Internet Res20140516165e13010.2196/jmir.314224836465v16i5e130PMC4051748HarrerMAdamSHFleischmannRJBaumeisterHAuerbachRBruffaertsRCuijpersPKesslerRCBerkingMLehrDEbertDDEffectiveness of an Internet- and App-Based Intervention for College Students With Elevated Stress: Randomized Controlled TrialJ Med Internet Res20180423204e13610.2196/jmir.929329685870v20i4e136PMC5938594AnderssonGTitovNAdvantages and limitations of Internet-based interventions for common mental disordersWorld Psychiatry20140213141110.1002/wps.2008324497236PMC3918007RichardsDMurphyTViganóNTimulakLDohertyGSharryJHayesCAcceptability, satisfaction and perceived efficacy of “Space from Depression” an internet-delivered treatment for depressionInternet Interventions2016095122210.1016/j.invent.2016.06.007Asosiasi Penyelenggara Jasa Internet-Indonesia20172018-11-14https:///www.apjii.or.id/EbertDDZarskiAChristensenHStikkelbroekYCuijpersPBerkingMRiperHInternet and computer-based cognitive behavioral therapy for anxiety and depression in youth: a meta-analysis of randomized controlled outcome trialsPLoS One20153103e011989510.1371/journal.pone.011989525786025PONE-D-14-33759PMC4364968WallinEEKMattssonSOlssonEMGThe Preference for Internet-Based Psychological Interventions by Individuals Without Past or Current Use of Mental Health Treatment Delivered Online: A Survey Study With Mixed-Methods AnalysisJMIR Ment Health201632e2510.2196/mental.532427302200v3i2e25PMC4925931ArjadiRNautaMHBocktingCLHAcceptability of internet-based interventions for depression in IndonesiaInternet Interventions20181381510.1016/j.invent.2018.04.004ArjadiRNautaMHScholteWFHollonSDChowdharyNSuryaniAOUiterwaalCSPMBocktingCLHInternet-based behavioural activation with lay counsellor support versus online minimal psychoeducation without support for treatment of depression: a randomised controlled trial in IndonesiaThe Lancet Psychiatry2018095970771610.1016/s2215-0366(18)30223-230006262WuZFangYComorbidity of depressive and anxiety disorders: challenges in diagnosis and assessmentShanghai Arch Psychiatry2014082642273110.3969/j.issn.1002-0829.2014.04.00625317009sap-26-04-227PMC4194005GarberJWeersingVRComorbidity of Anxiety and Depression in Youth: Implications for Treatment and PreventionClin Psychol (New York)20101217429330610.1111/j.1468-2850.2010.01221.x21499544PMC3074295MusiatPConrodPTreasureJTyleeAWilliamsCSchmidtUTargeted prevention of common mental health disorders in university students: randomised controlled trial of a transdiagnostic trait-focused web-based interventionPLoS One2014494e9362110.1371/journal.pone.009362124736388PONE-D-13-45644PMC3988039MullinADearBFKarinEWoottonBMStaplesLGJohnstonLGandyMFogliatiVTitovNThe UniWellbeing course: A randomised controlled trial of a transdiagnostic internet-delivered cognitive behavioural therapy (CBT) programme for university students with symptoms of anxiety and depressionInternet Interventions2015052212813610.1016/j.invent.2015.02.002DayVMcGrathPJWojtowiczMInternet-based guided self-help for university students with anxiety, depression and stress: A randomized controlled clinical trialBehaviour Research and Therapy20130751734435110.1016/J.BRAT.2013.03.00323639300BarreraMCastroFGStryckerLAToobertDJCultural adaptations of behavioral health interventions: a progress reportJ Consult Clin Psychol20130481219620510.1037/a002708522289132PMC3965302RahmadianaMKaryotakiEPasschierJCuijpersPvan BallegooijenWWimbartiSRiperHGuided internet-based transdiagnostic intervention for Indonesian university students with symptoms of anxiety and depression: A pilot study protocolInternet Interventions20190315283410.1016/j.invent.2018.11.002I-AiMentalWELLness20172017-12-21https://i-aimentalwellness.comHertzogMAConsiderations in determining sample size for pilot studiesRes. Nurs. Health20080431218019110.1002/nur.2024718183564SmarrKLKeeferALMeasures of depression and depressive symptoms: Beck Depression Inventory-II (BDI-II), Center for Epidemiologic Studies Depression Scale (CES-D), Geriatric Depression Scale (GDS), Hospital Anxiety and Depression Scale (HADS), and Patient Health QuestionnaArthritis Care Res2011110763S11S454S46610.1002/acr.2055622588766Patient Health Questionnaire (PHQ) Screener20182018-01-09https://www.phqscreeners.comPurbaFDHunfeldJAMIskandarsyahAFitrianaTSSadarjoenSSRamos-GoñiJMPasschierJBusschbachJJVThe Indonesian EQ-5D-5L Value SetPharmacoEconomics201771035111153116510.1007/s40273-017-0538-9Minddistrict20172017-08-24https://www.minddistrict.com/UtoyoDBLubisDUJayaESArjadiRHanumLAstriKPutriMDDPreliminary study on the effectiveness of short group cognitive behavioral therapy (GCBT) on Indonesian older adultsPLoS One201382e5719810.1371/journal.pone.005719823437339PONE-D-12-33612PMC3578808SharfinaZSantosoHBAn Indonesian adaptation of the System Usability Scale (SUS)20162016 International Conference on Advanced Computer Science and Information Systems, ICACSIS 201615-16 October 2016Malang, IndonesiaMalang, IndonesiaInstitute of Electrical and Electronics Engineers Inc14514810.1109/icacsis.2016.7872776LarsenDLAttkissonCHargreavesWANguyenTDAssessment of client/patient satisfaction: Development of a general scaleEvaluation and Program Planning197912319720710.1016/0149-7189(79)90094-610245370AttkissonCZwickRThe client satisfaction questionnaireEvaluation and Program Planning198215323323710.1016/0149-7189(82)90074-XLewisJRBrownJMayesDKPsychometric Evaluation of the EMO and the SUS in the Context of a Large-Sample Unmoderated Usability StudyInternational Journal of Human-Computer Interaction2015062531854555310.1080/10447318.2015.1064665BoßLLehrDReisDVisCRiperHBerkingMEbertDDReliability and Validity of Assessing User Satisfaction With Web-Based Health InterventionsJ Med Internet Res20160831188e23410.2196/jmir.595227582341v18i8e234PMC5023944vanBWCuijpersPvanSAKaryotakiEAnderssonGSmitJHRiperHAdherence to Internet-based and face-to-face cognitive behavioural therapy for depression: a meta-analysisPLoS One2014797e10067410.1371/journal.pone.010067425029507PONE-D-13-40111PMC4100736KroenkeKSpitzerRLWilliamsJBWThe PHQ-9J Gen Intern Med2001916960661310.1046/j.1525-1497.2001.016009606.x11556941SpitzerRLKroenkeKWilliamsJBWLöweBA brief measure for assessing generalized anxiety disorder: the GAD-7Arch Intern Med20060522166101092710.1001/archinte.166.10.109216717171166/10/1092ArifinBPurbaFDHermanHAdamJMFAtthobariJSchuiling-VeningaCCMKrabbePFMPostmaMJComparing the EQ-5D-3 L and EQ-5D-5 L: studying measurement and scores in Indonesian type 2 diabetes mellitus patientsHealth Qual Life Outcomes20202718110.1186/s12955-020-1282-yDeviRCarpenterCPowellJSinghSExploring the experience of using a web-based cardiac rehabilitation programme in a primary care angina population: a qualitative studyInternational Journal of Therapy and Rehabilitation2014090221943444010.12968/ijtr.2014.21.9.434KnaevelsrudCMaerckerAInternet-based treatment for PTSD reduces distress and facilitates the development of a strong therapeutic alliance: a randomized controlled clinical trialBMC Psychiatry20070419713131010.1186/1471-244X-7-13174421251471-244X-7-13PMC1885249JohnstonLTitovNAndrewsGDearBFSpenceJComorbidity and internet-delivered transdiagnostic cognitive behavioural therapy for anxiety disordersCogn Behav Ther201342318019210.1080/16506073.2012.75310823458319WeiselKKZarskiABergerTKriegerTSchaubMPMoserCTBerkingMDeyMBotellaCBañosRHerreroREtchemendyERiperHCuijpersPBolinskiFKleiboerAGörlichDBeechamJJacobiCEbertDDEfficacy and cost-effectiveness of guided and unguided internet- and mobile-based indicated transdiagnostic prevention of depression and anxiety (ICare Prevent): A three-armed randomized controlled trial in four European countriesInternet Interventions2018410.1016/j.invent.2018.04.002BolinskiFKleiboerAKaryotakiEBosmansJEZarskiAWeiselKKEbertDDJacobiCCuijpersPRiperHEffectiveness of a transdiagnostic individually tailored Internet-based and mobile-supported intervention for the indicated prevention of depression and anxiety (ICare Prevent) in Dutch college students: study protocol for a randomised controlled trialTrials2018032019111810.1186/s13063-018-2477-y2945840710.1186/s13063-018-2477-yPMC5819200KaryotakiEKleinAMRiperHWitLDKrijnenLBolEBolinskiFBurgerSEbertDDAuerbachRPKesslerRCBruffaertsRBatelaanNvan der HeijdeCMVonkPKleiboerAWiersRWCuijpersPExamining the effectiveness of a web-based intervention for symptoms of depression and anxiety in college students: study protocol of a randomised controlled trialBMJ Open2019051495e02873910.1136/bmjopen-2018-02873931092668bmjopen-2018-028739PMC6530301CohenJStatistical power analysis for the behavioral sciences, 2nd ed1988New York, NYLawrence Erlbaum AssociatesMaguireMDelahuntBDoing a Thematic Analysis: A Practical, Step-by-Step Guide for Learning and Teaching ScholarsAll Ireland Journal of Teaching and Learning in Higher Education (AISHE-J)201783335133514BangorAKortumPTMillerJTAn Empirical Evaluation of the System Usability ScaleInternational Journal of Human-Computer Interaction2008073024657459410.1080/10447310802205776BitsikaVSharpleyCFComorbidity of anxiety-depression among Australian university students: implications for student counsellorsBritish Journal of Guidance & Counselling20120840438539410.1080/03069885.2012.701271Al NaggarRAOsmanMTMusaRQuality of Life among University Students in a Single Malaysian InstitutePensee Journal20137510