This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Mental Health, is properly cited. The complete bibliographic information, a link to the original publication on https://mental.jmir.org/, as well as this copyright and license information must be included.
Low engagement rates with digital mental health interventions are a major challenge in the field. Multicomponent digital interventions aim to improve engagement by adding components such as social networks. Although social networks may be engaging, they may not be sufficient to improve clinical outcomes or lead users to engage with key therapeutic components. Therefore, we need to understand what components drive engagement with digital mental health interventions overall and what drives engagement with key therapeutic components.
Horyzons was an 18-month digital mental health intervention for young people recovering from first-episode psychosis, incorporating therapeutic content and a private social network. However, it is unclear whether use of the social network leads to subsequent use of therapeutic content or vice versa. This study aimed to determine the causal relationship between the social networking and therapeutic components of Horyzons.
Participants comprised 82 young people (16-27 years) recovering from first-episode psychosis. Multiple convergent cross mapping was used to test causality, as a secondary analysis of the Horyzons intervention. Multiple convergent cross mapping tested the direction of the relationship between each pair of social and therapeutic system usage variables on Horyzons, using longitudinal usage data.
Results indicated that the social networking aspects of Horyzons were most engaging. Posting on the social network drove engagement with all therapeutic components (
The online social network was a key driver of long-term engagement with the Horyzons intervention and fostered engagement with key therapeutic components and ingredients of the intervention. Online social networks can be further leveraged to engage young people with therapeutic content to ensure treatment effects are maintained and to create virtuous cycles between all intervention components to maintain engagement.
Australian New Zealand Clinical Trials Registry: ACTRN12614000009617; https://www.australianclinicaltrials.gov.au/anzctr/trial/ACTRN12614000009617
Poor engagement with digital mental health interventions is a major challenge in the field [
A recent development in the field is the personalization of digital mental health interventions to enhance the user experience and engagement [
To examine the relationship between engagement and effectiveness, our recent study used a novel method to analyze the relationship between patterns of use of a multicomponent digital platform and treatment effects [
The field of human computer interaction (HCI) utilizes empirical dynamic modeling (EDM) as a nonlinear analysis method to investigate human use of technology from time series data, where interactions with technology change over time and may be bidirectional, and associations can be inferred from the data rather than predetermined hypotheses [
The field of HCI has started to investigate causal and potentially bidirectional relationships between users and technology using MCCM, to improve and design better interactive technology [
Horyzons is an example of a unique long-term multicomponent digital intervention aiming to maintain long-term treatment effects and engagement. Our previous research found a relationship between improved outcomes and combined use of the therapy and social networking components of Horyzons [
Horyzons was a single-blind, 18-month, randomized controlled trial (RCT) conducted with young people recovering from FEP, following 2 years of specialist early intervention treatment. Participants were randomly allocated to either treatment as usual (TAU) or TAU as well as 18 months of access to the Horyzons intervention [
Ethics approval was obtained from the Melbourne Health Research Ethics Committee (2013.146).
Participants included 86 young people allocated to the Horyzons intervention. These participants were recruited after receiving 2 years of specialized care from the Early Psychosis Prevention and Intervention Centre (EPPIC) at Orygen, in Melbourne Australia between October 2013 and January 2017. EPPIC is a specialist FEP service that provides 18 months to 24 months of specialist early intervention for FEP to young people aged 15 years to 24 years [
Of the participants allocated to the intervention arm of the RCT, 5% (4/86) did not engage with the Horyzons platform. As there was no valid usage data, these participants were excluded from the analysis. The remaining 82 intervention participants were aged between 16 years and 27 years at randomization (mean 21, SD 2.88 years). As participants were recruited following their completion of early intervention treatment, this accounts for the age range extending to 27 years old. According to the Horyzons RCT eligibility criteria [
System usage metrics were extracted from the Horyzons online platform for each user for each day of their trial involvement (range: 282-528 days per user), resulting in a total of 47,060 cases of user interaction. See
System usage variables extracted from the Horyzons platform.
Variable type | Variables |
Therapy-related variables (number of) |
Stepsa started Pathwaysb started Actionsc done Visits to suggested contentd Visits to messagese Visits to therapyf |
Social networking–related variables (number of) |
News feedg posts News feed comments Likes made Reactionsh made |
aSteps refer to the intervention modules.
bPathways refer to a collection of intervention modules related to a topic (eg, anxiety).
cActions refer to behavioral activities aiming to translate learning into behavior.
dSuggested content refers to therapeutic content recommended by clinical moderators.
eMessages refer to a private message section, where moderators could contact participants directly.
fVisiting therapy refers to visiting the home page of the therapy component of the intervention.
gThe news feed refers to the social network.
hReactions refer to short support messages in response to a post (eg, thinking of you).
We categorized
The social network was moderated and led by peer workers. MOST peer workers were young people who identified as having a lived experience of mental ill-health and who had been employed and trained to offer support and guidance to others on the MOST platform. The social network was designed for participants to communicate and foster a sense of social support. Participants were able to post comments
MCCM was used to determine the causal and potentially bidirectional relationship between the aforementioned system usage metrics. For example, we tested whether usage of certain aspects of the social network leads to usage of certain therapy components or vice versa. This is a new methodology that extends beyond existing CCM methods and was adapted from that of Van Berkel and colleagues [
CCM is a core component of the EDM approach, which is a set of methods designed to characterize and test causality in complex dynamic systems, such as users interacting with technology over time [
The optimum embedding dimension (E) was identified between each pair of system usage variables tested between the social and therapy components of the intervention. This was done using simplex projection, which is the most direct projection technique and recommended for evaluating embedding dimensions for EDM [
As CCM is a nonlinear approach, it was also important to test whether system usage evolves in a nonlinear way. The rEMD package uses S-maps to characterize the degree of nonlinearity in the time series, by using the E chosen from the previous step of simplex projection and then estimating a linear map that uses the E-dimensional points on a manifold’s surface (ie, dimension of Euclidean space) to predict the future [
CCM was then used to identify potential causal links between pairs of system usage variables for each user. Using this approach, pairs of variables are mapped to each other using the nearest neighbors of each point on the E-dimensional manifolds (ie, the causal effects of
The previous step analyzed the causal relationship between variables for each individual user, which works well when trying to understand the dynamics of an individual ecosystem (ie, 1 user). This step extends upon that approach using a geometric approach and summarizes the results from multiple CCM analyses. This is called MCCM and enables us to obtain insights into the behavior of the entire study population [
The causal and potentially bidirectional relationship between usage of social networking and therapeutic aspects of Horyzons were investigated, to determine whether social networking usage leads to engagement with therapy content or vice versa.
A visual representation of results for each pair of variables for all combinations of variables described in
Visual representation of the direction of the causal relationship between completing therapeutic steps and posting on the social network.
The direction and magnitude of effects are represented by the position of the dots on the x axis. If the dot is to the left of the y axis, it indicates that the effect is in a certain direction (ie, completing therapy steps drives posting on the social network). If the dot is to the right, it indicates that the effect is in the other direction (ie, posting on the social network drives completing therapy steps; see
The distance from the x axis indicates how strong the forecasting ability is compared with a plain correlation, and the dashed vertical line indicates the mean effect size of causality across all participants. Any red dots that are present on graphs should be ignored as they represent invalid participant data, due to CCM not being able to provide power beyond that of a simple correlation [
MCCM was used to determine the causal relationship between social networking and therapy engagement.
Effect sizes (and standard deviations) for causality between therapy usage and social network usage.
Therapy usage | Social network usage | |||
|
Posts made, |
Comments made, |
Likes made, |
Reactions made, |
Steps started | 0.354 (0.359)a | 0.161 (0.326)a | 0.166 (0.315)a | 0.653 (0.230)a |
Pathways started | 0.154 (0.467)a | –0.047 (0.418)b | –0.058 (0.397)b | 0.484 (0.446)a |
Actions completed | 0.055 (0.437)a | –0.142 (0.393)b | –0.148 (0.397)b | 0.390 (0.422)a |
Suggested content visited | 0.295 (0.373)a | 0.120 (0.315)a | 0.110 (0.327)a | 0.582 (0.316)a |
Messages visited | 0.284 (0.359)a | 0.112 (0.306)a | 0.087 (0.338)a | 0.589 (0.288)a |
Therapy visited | 0.360 (0.350)a | 0.177 (0.307)a | 0.167 (0.296)a | 0.649 (0.230)a |
aSocial network usage drives therapy usage.
bTherapy usage drives social network usage.
On the other hand, comments made on the social network and therapy engagement demonstrated a bidirectional relationship, whereby some aspects of therapy engagement such as completing pathways (
Similarly, likes made on the social network and therapy engagement have a bidirectional relationship, whereby some aspects of therapy engagement such as completing pathways (
Finally, our MCCM analysis showed that reacting to posts made on the social network drove all aspects of therapy engagement, including completing steps (
This was the first study to use a novel modeling technique like MCCM to determine the causal relationship between different aspects of use of a long-term multicomponent digital intervention (Horyzons) to improve social functioning, improve vocational recovery, and prevent relapse in FEP. We found that posting on the social network and reacting to posts on the social network led to engaging with all aspects of therapy (including completing steps, pathways, and actions and visiting suggested content, messages, and therapy). We also found that commenting on the social network and liking posts made on the social network led to most aspects of therapy engagement (including completing steps and visiting suggested content, messages, and therapy). In other words, young people’s use of the social network was found to increase therapy use on Horyzons. Conversely, we found a bidirectional relationship between therapeutic and social components, whereby completing actions and pathways led to commenting on posts and liking posts on the social network, even though these aspects of social network usage led to engagement with all other measured aspects of therapy.
Overall, we found that usage of social networking aspects of Horyzons drove engagement with the therapeutic aspects in most cases. Therefore, we need to further understand what drives engagement with the social network [
It is then essential to engineer social networks in digital mental health interventions in ways that promote engagement with key therapeutic ingredients. This is important, as recent research has found that those who maintain use of both social networking and therapy components of an intervention display improved outcomes, but use of social networks alone may not lead to improvements [
In particular, the largest effect sizes were observed for reacting to posts on the social network, which led to usage of all assessed aspects of therapy (
However, it should be noted that this study also indicated that some aspects of therapy led to social network use, potentially generating synergistic effects between social networking and therapy usage once the transition to therapy is made. This, in turn, could promote engagement with both aspects of the intervention where usage of the social network and key therapeutic ingredients reinforce each other, leading to sustained engagement. For example, once the transition is made to engage with therapy, there may be a synergistic effect whereby social networking drives therapy engagement but also some aspects of therapy usage that drive social networking (eg, in this study, we found that the number of actions completed drives commenting on the social network). Learning from others and sharing experiences may lead to increased motivation to engage with the platform and with therapy more specifically and, in turn, lead to people sharing their therapy journeys with each other [
This study also suggests we need to establish how different components of a platform interact to promote engagement and drive positive outcomes, and the MCCM method has enormous potential to optimize platforms to be more engaging and more therapeutic. Future research should focus on how to optimize the platform to make the social network more therapeutic, promote more engagement with therapy, and generate back and forth virtuous cycles between both intervention components. Predictors of engagement could also be explored (eg, using prompts) and applied to aspects of the intervention that drive engagement with therapeutic content or key active ingredients (eg, a notification asking a participant if they would like to react to a new post on the social network) [
However, a number of limitations must also be noted. The sample size for this study was small, comprising 82 young people, so findings should be interpreted with caution. Although this was the first causal analysis exploring how the use of certain intervention components leads to subsequent use of other components, this study only tested the direction of the relationship between therapy and social networking components of a digital mental health intervention. It is possible that other multicomponent digital mental health interventions may include additional components, and some may be more engaging than a social network. It is also possible that multicomponent interventions not focused on mental health may have different engaging components, and future research should consider this.
To date, digital interventions have shown significant limitations with long-term participant engagement, a necessary ingredient toward bringing about long-term improvements in mental health. Multicomponent digital interventions that incorporate social networks have been proposed as a means to promote long-term engagement and tackle elusive long-term targets such as relapse prevention, which are rarely targeted by digital interventions. In a previous study, we identified that sustained use of both therapy and social networking components of a digital mental health intervention led to improved outcomes for young people recovering from FEP. This study adds to this knowledge by showing that the social network is possibly a necessary ingredient for long-term engagement and positive outcomes but not sufficient when used alone. Our study highlights the possible value of the social network in promoting engagement with key therapeutic ingredients of the intervention. Future research should aim to confirm these findings with a larger sample size and, if confirmed, could focus on how to optimize the social network to be more therapeutic, to more widely promote engagement with key therapeutic ingredients of the intervention, and to create virtuous cycles between use of the social network and therapeutic content.
Horyzons step: how to flourish.
Horyzons social network.
Social network posts drive therapy engagement.
Comments made and therapy engagement have a bidirectional relationship.
Likes made and therapy engagement have a bidirectional relationship.
Reactions drive therapy engagement.
convergent cross mapping
empirical dynamic modeling
Early Psychosis Prevention and Intervention Centre
first-episode psychosis
human computer interaction
multiple convergent cross mapping
moderated online social therapy
randomized controlled trial
self-determination theory
treatment as usual
The authors would like to thank the young people who participated in the Horyzons study and agreed to share their data for research purposes. The study was funded by the Mental Illness Research Fund (Victorian Government, Australia), the Australian National Health and Medical Research Council (NHMRC), and the Telstra Foundation. MAJ was supported by an Investigator Grant (APP1177235) from the NHMRC and a Dame Kate Campbell Fellowship from The University of Melbourne. LS was supported by an NHMRC Investigator Grant (2017962). SB was supported by the University of Melbourne Ronald Philip Griffiths Fellowship. BN was supported by an NHMRC senior research fellowship (1137687).
None declared.