Early Attrition Prediction for Web-Based Interpretation Bias Modification to Reduce Anxious Thinking: A Machine Learning Study

Introduction

Background

Approximately half of the US population experience a mental illness during their lifetime [1,2]. During the early stage of the COVID-19 pandemic, researchers estimated an increase of 25.6% in new cases of anxiety disorders per 100,000 people globally [3]. Mental illness is associated with impaired daily functioning, more frequent use of health care resources, and increased risk of suicide [1]. However, more than two-thirds of individuals with a mental illness do not receive treatment [4]. A multitude of barriers impede the initiation and sustained use of face-to-face (ie, traditionally delivered) treatment, including stigma; cost; lack of insurance coverage; and limited availability of support services, especially trained clinicians [2,5-7]. Given these challenges, there is an urgent need to help people manage their mental health in new ways [1,5].

Digital mental health interventions (DMHIs), which harness digital technologies to promote behavior change and maintain health [8], provide an appealing alternative for much-needed treatment outside a clinician’s office [9]. DMHIs may help individuals overcome obstacles to treatment, such as geographic or financial constraints, and may thus reduce the treatment gap among the broader population. Given the limited resources for health care service delivery, low-cost mobile health and eHealth interventions could be key to supporting symptom monitoring and long-term self-management of patients with mental disorders [10]. With an increasing demand for mental health care amid a shortage of mental health professionals, the use of eHealth and mobile health apps is expanding [11-13]. While these solutions have the potential to play an important role in increasing access to mental health services, especially for underserved communities, the clinical community is still determining how to best leverage these solutions [14].

Poor adherence and substantial dropout are common challenges in DMHIs [15]. Adherence, the extent to which users complete a DMHI’s tasks as intended [16,17], is likely to be associated with better treatment outcomes [1]. Although these tasks can vary widely (given the varied designs of DMHIs [16]), it is through engaging with such tasks that DMHIs are thought to achieve their outcomes [17]. However, sustained engagement with these platforms remains a significant issue [10,18-22]. Digital health interventions suffer from rates of dropout ranging from 30% to as high as 90% [1,5,15,19,23,24]. Dropout occurs when a participant prematurely discontinues an intervention (due to various potential reasons, such as technical issues, lack of time or energy, and lack of perceived benefit [18]). Even a modest dropout rate can limit the generalizability of digital intervention findings to only those who completed the study; thus, effective evaluation of treatments becomes a challenge [10,15,21,25-27]. This likely contributes to the uncertainties among clinicians and patients regarding the efficacy, usability, and quality of DMHIs [10]. There are many reasons clinicians tend not to integrate DMHIs into their clinical practice (eg, insufficient knowledge about DMHIs and lack of training about how to integrate them [28,29]). An additional reason is that if patients’ sustained engagement with DMHIs is low and they stop participating in the intervention before achieving meaningful gains, then clinicians have little incentive to view DMHIs as a helpful tool to increase the efficiency and impact of care.

One approach to reducing attrition in DMHIs is to identify participants at a high risk of dropping out at the early stages of the intervention, which would permit the intervention to be adapted to these users’ needs [30]. For example, more support (eg, minimal human contact with a telecoach) could be offered specifically to such users (thereby maintaining scalability [31]). Although increasing attention has recently been dedicated to attrition in various eHealth interventions [32-34], relatively few advances within DMHIs have predicted dropout through streamlined quantitative approaches considering both passive and self-reported data. Testing the effectiveness of interventions on treatment outcomes [35] often takes priority rather than identifying and predicting users at high risk of attrition. Consequently, methodological advancements in attrition prediction have largely taken place outside clinically relevant settings, such as in the eCommerce and social gaming industries [36-38]. This paper develops a data-driven algorithm that includes both passive indicators of user behavior and self-reported measures to identify individuals at a high risk of early attrition in 3 DMHIs; as such, it provides a framework that helps in the personalization of DMHIs to suit individual users based on each individual’s attrition risk.

To predict attrition in DMHIs, there are 2 main considerations [18]. First, we need to define the prediction horizon; that is, researchers should determine the point in an intervention’s timeline at which it would be beneficial to predict which participants are at a high risk of dropping out. This decision may be influenced by an analysis of when in the timeline most participants are actually dropping out; such an analysis may allow the identification and strengthening of weak parts of an intervention. Given that low engagement has been consistently cited as the construct underlying attrition, this decision may also be informed by considering typical patterns of engagement [10,36,37,39-43]. However, engagement is a very broad construct with many components [17], and empirical evidence suggests that engagement fluctuates with time [30]. Thus, carefully defining the feature space and predicting participants who are at a high risk of attrition at meaningful time points in a program can provide valuable information. For example, participants may initially stay in the intervention out of curiosity, which relates to the novelty effect—the human tendency to engage with a novel phenomenon [35], but then lose interest. If a researcher wants to mitigate the impact of the novelty effect, then understanding early-stage dropout (ie, early in the program but after it is no longer brand new and unknown) is critical.

Second, we must consider which factors cause users to drop out of a given DMHI. Answering this question can help researchers and designers tailor the intervention to particular user groups. Demographic variables such as gender, age, income, and educational background have been related to higher attrition rates in digital health interventions [10,44-47]. With respect to participants’ mental health (eg, lifetime symptoms assessed at baseline or current symptoms assessed during the course of the intervention), the presence of mental health symptoms may increase interest toward the use of a digital intervention in an effort to reduce such symptoms [26]. However, certain symptoms (eg, hopelessness) may reduce the participants’ motivation or ability to sustain engagement with an intervention [10,15,20,22,33,48]. In addition to these baseline user characteristics, user clinical functioning (ie, current symptoms and psychological processes that lead to the maintenance of these symptoms), self-reported user context and reactions to interventions (eg, perceived credibility of DMHIs, which is associated with increased engagement and reduced dropout [10]), and passively detected user behavior influence attrition rates in digital platforms [15,31]. This behavior includes time spent using an intervention [38,49,50], the passively detected context (eg, time of the day and day of the week) [49], and type of technology (eg, web, smartphone, computer based, or wearable) [20,51].

Prior studies, mainly in psychology, have predicted attrition primarily with statistical techniques such as ANOVA and regression [46,47,52-54]. In addition, other research has used macrolevel approaches, such as contrasting one intervention’s attrition rate against another’s [39] and examining participant and psychotherapy trial factors that predict dropout rates [55]. Researchers in computer and data science and the mobile gaming industry more commonly leverage passively collected behavioral data from users and have found success in predicting attrition (“churn”) using more advanced techniques, such as linear mixed modeling [37], survival analysis [38], and probabilistic latent variable modeling [36]. More recently, advanced machine learning models, such as deep neural networks, have also been useful for modeling and predicting attrition in mobile gaming [38,50,56,57] and in digital health care applications [20,58]. Our approach builds on work predicting attrition in DMHIs [37,45,47,54,58-60] and incorporates both passively collected behavioral data and self-reported data [1,17,31,60-63].

An attractive DMHI for anxiety is cognitive bias modification for interpretation (CBM-I [64,65]), a web-based program with potential to reach large, geographically diverse samples of adults with anxiety symptoms. CBM-I aims to shift threat-focused interpretation biases in which people with anxiety symptoms tend to assign a negative or catastrophic meaning to situations that are ambiguous. Cognitive models of anxiety suggest that training people with anxiety symptoms to consider benign interpretations of ambiguous situations, as opposed to only rigidly negative interpretations, may reduce anxiety [66-68]. To shift interpretation biases, CBM-I training sessions prompt users to imagine themselves in ambiguous, threat-relevant scenarios (presented in a set of short sentences) and to practice disambiguating each scenario by filling in its final word (typically presented as a word fragment) [65]. Active CBM-I conditions encourage more positive and flexible interpretation of scenarios by providing a final word that assigns a benign or a positive meaning to the ambiguous situation (consider this example: “As you are walking down a crowded street, you see your neighbor on the other side. You call out, but she does not answer you. Standing there in the street, you think that this must be because she was distracted.”). By presenting benign or positive endings for most scenarios (eg, 90%), positive CBM-I conditions train a positive contingency in which users learn to expect that ambiguous potentially threatening situations usually work out fine.

The greatest degree of improvement is expected in positive conditions relative to other active conditions (eg, 50% positive and 50% negative conditions that present positive and negative endings in equal proportions, thereby training flexible interpretation but no contingency) and to control conditions (eg, no training or a neutral condition with emotionally unambiguous scenarios and neutral endings). Thus, this paper focuses on attrition in positive conditions. Despite some mixed results [62,69], a number of studies have shown the effectiveness of positive CBM-I conditions in shifting interpretation biases and reducing anxiety symptoms [19,44,64,70-72]. To benefit from CBM-I programs, people must be able to use them effectively during a sustained period. However, similar to many DMHIs, web-based CBM-I programs face substantial attrition rates [19,73].

Objective

This paper has 3 aims. The first aim is to determine a practical attrition prediction horizon (ie, to determine the session at which it would be beneficial to identify individuals at a high risk of dropping out). The second aim is to identify participants at a high risk of dropping out by leveraging baseline user characteristics, self-reported user context and reactions to the program, passively detected user behavior, and clinical functioning of users within our analysis. The third aim is to explore which of these feature sets are most important for the identification of participants at high risk. To achieve these aims, we propose a multistage pipeline to identify participants who are at a high risk of dropout from the early stages of 3 different DMHI studies. These interventions use web-based CBM-I [64,65] to help individuals change their thinking in response to situations that make them feel anxious or upset [19,44,74]. Note that our proposed pipeline is expected to apply broadly to DMHIs; however, in this paper, we focus on CBM-I programs as a useful starting point and look for important features of attrition in such programs.

Methods

Data Source and Interventions

MindTrails [75] is a multisession, internet-delivered CBM-I training program. To date, >6000 people across >80 countries have enrolled in MindTrails, pointing to participant interest in accessing a technology-delivered, highly scalable intervention that can shift anxious thinking in a targeted and efficient way.

In this paper, we focus on 3 MindTrails studies: Managing Anxiety, Future Thinking, and Calm Thinking. We provide a brief overview of these studies, which were approved by the University of Virginia Institutional Review Board (IRB). We analyzed data from 1277 participants who took part in these studies. Details of the studies are provided in Table 1.

Participants and Procedure

Study 1: Managing Anxiety

The Managing Anxiety study focused on the development of an infrastructure to assess the feasibility, target engagement, and outcomes of a free, multisession, web-based CBM-I program for anxiety symptoms. A large sample of community adults with at least moderate trait anxiety based on an anxiety screener (Anxiety Scale of the 21-item Depression Anxiety Stress Scales, DASS-21 [76]) was randomly assigned to (1) positive CBM-I training (90% positive and 10% negative), (2) 50% positive and 50% negative CBM-I training, or (3) a no-training control condition. Toward the start of CBM-I training, participants also underwent an imagery prime manipulation, an imagination exercise designed to activate the participants’ anxious thinking about a situation in their life. After consenting and enrolling, the participants completed a battery of baseline measures, including demographic information, mental health history, and treatment history. For details about the Managing Anxiety study protocol, including the aims and the outcome measures of the study, refer to the main outcomes paper by Ji et al [19].

The program involved up to 8 web-based training sessions, delivered at least 48 hours apart, with assessments immediately after each session and a follow-up assessment 2 months after the last session. During each session, CBM-I training was provided. This training involved 40 training scenarios, which were designed to take approximately 15 minutes to complete. Study contact, in the form of automated reminder emails sent to all participants, was equivalent in content and schedule regardless of training condition. If participants completed only part of an assessment task, they continued the assessment the next time they returned. If they completed only part of a training task, they restarted the task upon returning. Participants received no monetary compensation. A total of 3960 participants completed the eligibility screener, out of which 807 (20.38%) eligible participants enrolled and completed the baseline assessment. In this paper, only data from the positive intervention arm (ie, positive CBM-I condition) were used (n=252, 31.23% of participants who enrolled and completed the baseline assessment), given our interest in testing predictors of attrition in positive CBM-I across all 3 studies.

Study 2: Future Thinking

The Future Thinking study, a hybrid efficacy-effectiveness trial, focused on testing a multisession, scalable, web-based adaptation of CBM-I to encourage healthier, more positive future thinking in community adults with negative expectations about the future based on the Expectancy Bias Task (shortened from the version used by Namaky et al [77]). After completing the screener, eligible participants provided consent; were enrolled; and were randomly assigned to (1) positive conditions with ambiguous future scenarios that ended positively, (2) 50-50 conditions that ended positively or negatively, or (3) a control condition with neutral scenarios. For details about the aims and outcome measures of the Future Thinking study, refer to the main outcomes paper by Eberle et al [44].

The participants were asked to complete 4 training sessions (40 scenarios each). Assessments were given at baseline, immediately after each session, and during the follow-up assessment 1 month after the last session. Participants had to wait for 2 days before starting the next training session; they had to wait for 30 days before starting the follow-up assessment. Participants had the option of receiving an email or SMS text message reminder when the next session or follow-up assessment was due. If they completed only part of a training or assessment task, they continued the task the next time they returned. The participants received no monetary compensation. A total of 4751 participants completed the eligibility screener, out of which 1221 (25.70%) were eligible and were enrolled. In this paper, only data from the positive CBM-I intervention arm (ie, the positive condition and the positive + negation condition) were used (n=326, 26.70% of enrolled participants).

Study 3: Calm Thinking

The Calm Thinking study, a sequential, multiple assignment, randomized trial, tested the effectiveness of positive CBM-I relative to a psychoeducation comparison condition (randomly assigned at Stage 1). It also tested the addition of minimal human contact (ie, supplemental telecoaching randomly assigned at Stage 2 [78]) for CBM-I participants classified as having a higher risk of dropout early in the study. Additional details can be found in the main outcomes paper by Eberle et al [74].

After completing the anxiety screener (DASS-21-Anxiety Scale), eligible participants provided consent and were enrolled. The participants were asked to complete a baseline assessment and 1 training session per week for 5 weeks (5 sessions total, 40 scenarios each in CBM-I), with an assessment immediately after each session and a follow-up assessment 2 months after the last session. If the participants completed only part of a training or assessment task, they continued the task the next time they returned. They were compensated via e-gift cards (refer to Table 1 for details). A total of 5267 participants completed the eligibility screener, out of which 1748 (33.19%) were eligible and were enrolled. To allow a clean analysis of attrition during positive CBM-I, data [79] from the CBM-I–only intervention arm (n=699, 39.99% of enrolled participants; ie, CBM-I condition excluding participants at high risk who were randomized to receive supplemental coaching) were used in this paper.

In total, 252 Managing Anxiety participants, 326 Future Thinking participants, and 699 Calm Thinking participants were in the positive CBM-I intervention arm of these studies.

Definition of Attrition

In this paper, we predict attrition in multisession DMHIs. A paper by Eysenbach [18] defined two types of attrition: (1) nonuse attrition, which refers to participants who stopped using the intervention (ie, who did not complete the training sessions), and (2) dropout attrition, which refers to participants who were lost to follow-up because they stopped completing research assessments (eg, who did not complete follow-up assessment). In MindTrails studies, training and assessment tasks are intermixed and must be completed in series. For example, the participants cannot complete Session 1 assessment until they complete Session 1 training, they cannot complete Session 2 training until they complete Session 1 assessment, and so on. Due to this sequential design, nonuse and dropout attrition are conflated in our studies. As it is impossible to skip any training or assessment tasks, we simply use the term attrition in this paper.

Ethical Considerations

All 3 studies were reviewed and approved by the IRB of the University of Virginia (Managing Anxiety: IRB #2703; Future Thinking: IRB #2690; and Calm Thinking: IRB #2220). After screening, the eligible participants provided informed consent for “a new internet-based program.” Data were stored in accordance with University of Virginia Information Security policies, and deidentified data were analyzed. In the Calm Thinking study, the participants were compensated with e-gift cards worth up to US $25: US $5 for each assessment at pretreatment and after Sessions 3 and 5, and US $10 for the follow-up assessment. Compensation is detailed by study in Table 1.

Attrition Prediction Pipeline

DMHIs are often divided into multiple phases, sometimes called modules. In this paper, we refer to modules as sessions to mirror the language used by mental health specialists for in-person treatment (eg, holding sessions with a client). We proposed a pipeline that is built to handle multisession DMHI datasets with a diverse set of features. As our focus is on multisession studies, we assumed that the study contained ≥1 assessment or training session to achieve the study goals. Therefore, we required at least 1 observation from each participant for the selected features.

Predicting early-stage dropout in DMHIs is challenging and requires several key tasks. We first determined the prediction horizon of the selected CBM-I interventions (Aim 1). We then organized the remaining tasks into four main steps from the data science and engineering literature: (1) data preprocessing, (2) feature generation, (3) predictive modeling, and (4) feature importance. We outline these steps in the context of attrition prediction in DMHI in Figure 1 and describe each step below.

‎

Prediction Horizon

To analyze when users stopped using the intervention (Aim 1), the proportions of participants who completed each training session (out of the number of participants who started Session 1 training) were visualized (Figure 2). In this figure, each session was considered complete if participants completed the last questionnaire (ie, had an entry in the Task Log for the questionnaire) in the assessment that immediately followed a given training session. For the following reasons, we decided to focus on participants who had started Session 1 training and to predict which of these participants were at a high risk of dropping out before starting Session 2 training (Aim 2). First, our goal is to make inferences about user dropout during DMHIs (and not to simply use baseline assessments to predict which users will fail to even start the program). We restricted the sample to participants who had started Session 1 training because we consider these participants as part of the intent-to-treat sample. Second, the highest rate of attrition was observed between the start of the first training session and the end of the second session’s assessment, with most dropout occurring between the sessions (vs during Session 1 or Session 2). Therefore, we wanted to predict participants at a high risk of dropping out before starting Session 2 training. Notably, the identification of participants who are at a high risk of dropout early in the program might decrease the attrition rate at the end of the intervention. This is because detecting participants at high risk sooner rather than later permits targeted supports to be added to increase retention at pivotal times.

‎

Data Preprocessing

Overview

All data must be preprocessed before analysis, especially data collected outside a controlled laboratory environment. In the following paragraphs, we describe our methods for addressing issues such as invalid participant data, outliers, and missingness during preprocessing.

Invalid Participants

One of the main challenges in web-based digital mental health studies is to distinguish spam and bot-generated responses from real responses [19,79]. Malicious actors often use bots to complete questionnaires when they learn of an appealing incentive, such as monetary compensation for participating in a study. To increase the validity of the input data, we removed suspicious responses such as those that were submitted quickly (eg, <5 s for half of all questions in a given measure) or contained submissions that violated the required wait time (eg, 48 h) between sessions.

Outliers

To reduce the likelihood of identifying coincidental events, we first normalized the data using the z score metric. We then identified and removed outliers; as we did not expect to have very large or small data values [80], we excluded outliers at least 3 SDs from the mean value [81] for numerical variables. For categorical variables, we excluded outliers based on visual inspection of a frequency distribution (a histogram with the Freedman-Diaconis rule to determine the bin width).

Missing Values

Real-world data collection is often messy; technical issues, dropout, and loss of network connection are all common issues that arise and can lead to missing values for some or all items of a given questionnaire. In addition, participants in DMHIs are often given the option to decline to answer items when responding to a self-reported questionnaire. This may be done either implicitly (in which the question is not required) or explicitly (in which the participant is given a set of options, where one of the options is “prefer not to answer” or a similar response). The challenge associated with empty or “prefer not to answer” values is that they both function as missing values.

Missing values are a fundamental issue in digital health interventions for several reasons [82]. Most machine learning techniques are not well prepared to deal with missing data and require that the data be modified through imputation or deletion of the missing records. In addition, missing data may significantly impact the predictive analysis as well as descriptive and inferential statistics [82]. To address these issues, we used several imputation approaches to handle the challenge of missing data in some or all items in the required features and time points for different types of variables. Without imputation, these missing data could lead to more bias, decreased statistical power, and lack of generalizability.

We handled missing data for all features, for each unique time point, using the following methods: out of the initial set of features (221 for Managing Anxiety, 109 for Future Thinking, and 241 for Calm Thinking), we first removed features or variables at a given time point that have missing values in >80% of all valid participants. The percentages of features removed for this reason in Managing Anxiety, Future Thinking, and Calm Thinking studies were 33.94% (75/221), 20.18% (22/109), and 47.30% (114/241), respectively, yielding a final set of 146, 87, and 127 features, respectively. Next, we imputed categorical variables at a given time point with the most frequent answers at that time point of participants with the same demographics. To do so, we grouped participants based on 2 of the demographic characteristics (ie, education and gender, which were the most complete). To impute the numerical individual item variables at a given time point, we used the k-nearest neighbors method [83] to replace the missing values in the same demographic group with the mean value at that time point from the 5 nearest neighbors found in the training set. We used a Euclidean distance metric [84] to impute the missing values.

Unexpected Multiple Observations

Unexpected multiple observations may be present within a DMHI dataset for several reasons. Participants might complete the eligibility screener multiple times to gain access to the intervention if they were previously screened or to achieve a more desirable score. Technical issues can also cause duplicate values. For example, a brief server error may cause a questionnaire to be submitted more than once. We used one of the following two strategies to handle unexpected multiple observations: (1) calculate the average values of each item across the observations or (2) keep the latest observation. We selected one of the abovementioned strategies based on the temporal latency between unexpected multiple observations. If the temporal latency between unexpected multiple observations was less than the mean latency across all participants, we applied the first strategy. Otherwise, the second strategy was selected.

Feature Generation

Baseline User Characteristics

(Note: Measures without citations in this section and the sections below were developed by the MindTrails research team.) At the baseline assessment of the 3 studies, the following demographic variables were assessed: age, gender, race, ethnicity, education, employment status, marital status, income, and country. History of mental health disorders and treatment were also assessed. In the Managing Anxiety and Calm Thinking studies, participants were also asked about the situations that make them anxious; these situations are called anxiety triggers. We included these measures in our baseline user characteristics feature set (Table 2).

Self-Reported User Context and Reactions to Program

The importance of reducing anxiety or changing thinking (Importance Ruler, modified from Case Western Reserve University [63]) and confidence in the intervention (modified from Borkovec and Nau [85]) were assessed at the baseline assessment of every study. In addition, after completing a given session’s assessment, participants were asked for the date they would return for the next session. State anxiety (in Managing Anxiety and Calm Thinking; Subjective Units of Distress, SUDS, modified from Wolpe [87]) or current positive and negative feelings (in Future Thinking) were assessed before and after participants completed each session’s training. The Managing Anxiety and Calm Thinking studies also assessed participants’ peak anxiety when imagining an anxiety-provoking situation in their lives as part of the anxious imagery prime completed toward the start of training. At the end of each session in the Calm Thinking study, the participant’s location, level of distraction, and ease of use of the program were assessed. All of these measures were included in the self-reported user context and reactions to the program feature set (see details in Table 2).

Passive Detection of User Behavior

To further understand participants’ context and behavior when interacting with the platform, the following variables were calculated: time spent on a page, time of day, day of the week, and latency of completing assessments. The type of device (ie, desktop, tablet, smartphone) was also included as a feature given that multiple devices could be used to access the program, each with different characteristics (eg, screen size, input methods, and mobility) that could influence user behavior. In most cases, these variables were extracted for each assessment and training task for each session. For details about which features were extracted for which studies, see Table 2.

User Clinical Functioning

Primary and secondary outcome measures used to evaluate the effectiveness of the intervention were included in the clinical functioning feature set. These measures assessed interpretation bias (Recognition Ratings, RR, modified from Matthews and Mackintosh [65]; and Brief Body Sensations Interpretation Questionnaire, BBSIQ, modified from Clark et al [88]), expectancy bias (Expectancy Bias Task, modified from Namaky et al [77]), anxiety symptoms (Overall Anxiety Severity and Impairment Scale, OASIS, adapted from Norman et al [89]; DASS-21-Anxiety Scale; and Generalized Anxiety Disorder-2 scale, GAD-2, modified from Kroenke et al [90]), comorbid depression symptoms (DASS-21-Depression Scale; and Patient Health Questionnaire-2, PHQ-2, modified from Kroenke et al [61]), and alcohol use (Daily Drinking Questionnaire, DDQ [91]). They also assessed the centrality of anxiety to identity (Anxiety and Identity Circles, modified from Ersner-Hershfield et al [92]) and other cognitive mechanisms, including cognitive flexibility (Cognitive Flexibility Inventory, CFI, adapted from Dennis and Vander Wal [93]), experiential avoidance (Comprehensive Assessment of Acceptance and Commitment Therapy Processes, CompACT, modified from Francis et al [94]), cognitive reappraisal (Emotion Regulation Questionnaire, ERQ, modified from Gross and John [95]), and intolerance of uncertainty (Intolerance of Uncertainty Scale-Short Form, IUS-12, modified from Carleton et al [96]). Finally, they assessed self-efficacy (New General Self-Efficacy Scale, NGSES, modified from Chen et al [97]), growth mindset (Personal Beliefs Survey, PBS, modified from Dweck [98]), optimism (Life Orientation Test-Revised, LOT-R, modified from Scheier et al [99]), and life satisfaction ([86]; Quality of Life Scale, QOL [100]). For details about which features were extracted for which studies, see Table 2.

Predictive Modeling

Overview

For each study, predictors of attrition were investigated after participants started Session 1 training, imputing any missing values for features collected during Session 1 training or assessment.

To identify participants at high risk of dropping out before starting the second training session, the following predictors of attrition were investigated: baseline user characteristics (at the pretest assessment), self-reported user context and reactions to the program, passively detected user behavior, and clinical functioning of users. We used data from the pretest, the first training session, and the assessment following the first training session.

Dropout Label

For each participant, we calculated a binary ground truth label for their actual dropout status before starting the second training session, where 0 indicates the participant started training for the second session and 1 indicates the participant did not start training for the second session (ie, dropped out). A participant was deemed as having started a given session’s training if they had an entry in the Task Log for the Affect task, which was administered immediately before the first page of training materials for each session.

Class Imbalance

Class imbalance is a common problem for supervised learning tasks such as attrition prediction. Such datasets have 1 or more classes (eg, “did not dropout” in the case of Calm Thinking) that have a greater number of observations than other classes (eg, “dropped out” in Calm Thinking). Class imbalance can worsen the performance of machine learning models by biasing them toward learning the more commonly occurring classes. We used the synthetic minority oversampling technique [101] to help rectify the class imbalance.

The synthetic minority oversampling technique resolves this challenge by generating synthetic samples for the minority class, with the aim of balancing the distribution of samples between the 2 classes. The technique operates by selecting 2 or more samples from the minority class and computing the difference between their features. This difference is then added to the feature values of one of the selected samples to create a new synthetic sample. This process is repeated to generate a sufficient number of synthetic samples, which are then added to the original dataset to achieve an optimal balance between the majority and minority classes. It has proven to be very effective in dealing with class imbalance problems for tabular datasets [102] (Figure 2).

Classification

Binary classification is a well-studied problem in the machine learning literature [103,104], and a plethora of models and approaches exist for predicting attrition. We selected leading machine learning models, beginning with simpler, more interpretable models and progressing to more expressive models for identifying the best predictors of early-stage dropout. We trained and validated a range of models, described in detail below and listed in Table 3. Models that learn a linear decision boundary are typically the first approach for binary classification problems. These models separate participants into 2 classes defined by the estimated decision boundary, in our case participants who drop out and those who remain. The logistic regression model estimates this decision boundary by minimizing the mean squared error of predictions in the training set [105]. Similarly, the support vector machine (SVM) estimates this boundary by maximizing the distance from the edge of each class. Some nonlinearity is also introduced into the SVM by projecting its feature space with the radial basis function (RBF) kernel [106].

Other models estimate a nonlinear decision boundary. A decision tree model estimates a continuous piecewise boundary, with each piece indicating a different set of conditions that leads to a particular leaf node of the tree [107]. We further evaluated several tree-based ensemble models. In ensemble models, multiple submodels are composed to form a prediction. The random forest model uses decision trees as its submodel, creating a “forest” (set) of such trees. The random forest estimates the best feature subset to give to each tree while maximizing the average prediction accuracy over all trees [107]. Similarly, AdaBoost comprises multiple shallow decision trees, giving a weighting to each tree according to the overall prediction accuracy [107].

Finally, gradient boosting algorithms (and the related extreme gradient boosting [XGBoost] method [108]) were used to train ensembles of decision trees. Gradient boosting minimizes an objective function that is differentiable with respect to all submodel parameters, and the submodel parameters are adjusted via gradient descent. XGBoost [108] is based on the same concept, but also includes parameter regularization to prevent overfitting and second-order derivatives to control gradient descent. The regularized greedy forest (RGF) model was also evaluated. RGF not only includes tree-structured regularization learning, but also employs a fully corrective regularized greedy algorithm [109]. Finally, a multilayer perceptron model was used. This neural network model implements a feed-forward architecture that backpropagates error with stochastic gradient descent [110].

We employed 10-fold cross-validation stratified by dropout label (ie, dropout vs not dropout) across 100 iterations. Hyperparameter tuning was performed using group 5-fold cross-validation on the training set. Hyperopt [111] was used to optimize hyperparameters including the number of estimators, learning rate, maximum tree depths, C parameter, and γ. We evaluated up to 30 combinations of these parameters to maximize the model’s average macro–F₁-score across 5 folds. The set of hyperparameters that achieved the highest average macro–F₁-score across the 5 folds was chosen to train the model on the entire training set during the outer split.

Model Optimization

To enhance model performance and efficiency, optimization techniques were applied. For instance, in the SVM model, we selected the RBF kernel with γ determined as 1/(number of features × X.var ()) to control the influence of training examples. In decision tree models, all features were considered for finding the best splits, while feature subsampling was employed to reduce model correlation and variance.

Our selected criterion for the decision model is entropy, which measures the degree of disorder of the features in relation to the target. The optimum split is chosen by the feature with the lowest entropy. It gets its maximum value when the probability of the classes is the same. A node is pure when the entropy has its minimum value, which is zero. For the random forest model, we take all the features that make sense in every tree.

In the XGBoost model, we set the subsample ratio of columns for each level equal to 0.4. Sampling occurs once for every new tree. The γ parameter in XGBoost is used as a threshold for creating new splits in the tree; it represents the minimum loss reduction required to make a further partition on a leaf node of the tree. We set γ=8. To control the balance of positive and negative weights in a binary classification problem, we set the parameter scale_pos_weight = sum(negative instances) / sum(positive instances). This parameter allows adjustment of the relative weight of positive instances in the cost function, by setting it to the ratio of negative to positive instances. This can help to handle imbalanced datasets where one class is underrepresented, as in our case. The eta parameter, learning rate, controls the step size shrinkage used in updating the weights to prevent overfitting. We tuned eta for our models and dataset and got the value 0.01. After each boosting step in XGBoost, we can directly get the weights of newly added features, and eta shrinks the feature weights and the weights of all the features in the model to make the boosting process more conservative. The α=.3 parameter in XGBoost is used as a regularization term on the weights; it represents the L1 regularization term, which is used to add a penalty term to the cost function that is proportional to the absolute value of the weights. This helps to prevent overfitting by shrinking the weights toward zero. The λ=0.4 parameter in XGBoost is also used as a regularization term on the weights; it represents the L2 regularization term, which is used to add a penalty term to the cost function that is proportional to the square of the weights. This helps to prevent overfitting by shrinking the weights toward zero.

For RGF, we used the min-penalty regularization with sum-to-zero sibling constraints to improve the interpretability of the model. For logistic regression, we set the regularization to elasticnet and the regularization strength to 1, C=1. For a multilayer perceptron, the activation function is set to the rectified linear unit function, represented as f(x)=max(0,x). The initial learning rate for the Adam algorithm is also set to 0.001. It is worth noting that we kept the other hyperparameters of the models at their default values to avoid overfitting and to ensure the stability of the models.

Evaluation Metrics

We used 3 standard metrics to evaluate attrition prediction: macro–F₁-score, area under the curve (AUC; ie, area under the receiver operating characteristic [ROC] curve), and accuracy. For macro–F₁-score, an F₁-score is first computed for each class. The F₁-score is the harmonic mean of precision (proportion of positive predictions that are correct) and recall (proportion of positive classes that are correctly predicted; true positive rate), and it rewards true positives and penalizes false positives and false negatives. F₁-scores range from 0 (when no positive predictions are correct) to 1 (when all positive predictions are correct, and no incorrect negative predictions are made). Macro–F₁-score is the arithmetic mean of F₁-scores across classes and is widely used when classes are imbalanced because it avoids bias toward the majority class by weighting each class’s F₁-score equally.

AUC, a widely adopted performance metric, measures the trade-off between the true positive rate and the false positive rate (proportion of negative classes that are incorrectly predicted as positive) by plotting these rates against one another for various classification thresholds (ie, probabilities above which a positive prediction is made) and quantifying the area under the resulting ROC curve; this area provides an aggregate measure of performance across all possible thresholds. AUC ranges from 0 (when no positive classes are correctly predicted and all negative classes are incorrectly predicted) to 1 (when all positive classes are correctly predicted and no negative classes are incorrectly predicted), indicating the model’s ability to differentiate between positive and negative classes (a value of .5 reflects random prediction).

Accuracy, in turn, is the proportion of all predictions (positive and negative) that are correct and ranges from 0 (no predictions are correct) to 1 (all predictions are correct), providing a straightforward assessment of the model’s overall performance, although it can be misleading in isolation when classes are imbalanced. For macro–F₁-score and AUC, values above .5 are generally considered to reflect reasonable performance, while for accuracy, a value above .7 is considered reasonable.

Feature Importance

Aim 3 of this paper is to explore which feature sets are most important for the identification of participants at high risk. To analyze this, the effect of each feature set on the prediction models was calculated (ie, Gini importance [112]). We report the mean Gini importance score across 2 iterations. Gini importance scores reflect the importance of a feature set relative to others (not absolute importance) and can range from 0 to 1, with higher scores reflecting greater importance.

Results

Model Performance

The results demonstrate that with these predictors (number of features for the Managing Anxiety, Future Thinking, and Calm Thinking studies: 146, 86, and 127, respectively), we were able to identify participants with a high risk of dropping out before starting the second training session of each study (macro–F₁-score for XGBoost in the Managing Anxiety, Future Thinking, and Calm Thinking studies: .832, .770, and .917, respectively; Table 3). These results show the effectiveness of different feature sets in predicting attrition in the early stages of the DMHIs. Moreover, these results show the superiority of the XGBoost and the random forest models in predicting attrition (see Table 3). XGBoost always places more importance on functional space when reducing the cost of a model, while random forest tries to place more importance on hyperparameters to optimize the model.

Sensitivity to Imputation

To assess the impact of imputation on our prediction models, we conducted an ablation experiment (ie, systematic removal of a component of the model to test its effect) that eliminated the imputation step from our pipeline. We used the XGBoost classification model in this experiment, as it demonstrated superior performance throughout our analyses. The results, presented in Table 4, reveal a substantial decrease in performance when imputation using k-nearest neighbors is removed from the pipeline, highlighting the importance of imputation in our prediction models.

Feature Importance

To investigate how the different feature sets affect the performance of attrition prediction, we calculated the average importance score (ie, weight) for the important features from the selected high-performing classifier after 100 iterations. Overall, a few trends emerged in identifying individuals at high risk of dropout: the passively detected user behavior feature set, and then the self-reported user context and reaction to the program feature set, are consistently more important than the user baseline characteristics and user clinical functioning feature sets for predicting early-stage attrition in a multisession CBM-I intervention (Figure 3). More specifically, we found that features involving passive detection of user behavior, such as time spent on a web page of CBM-I training exercises, time of day, latency in completing assessments, and the type of device used, were the most informative predictors of attrition, with mean Gini importance scores across 2 iterations of .033 (95% CI .019-.047), .029 (95% CI .023-.035), and .045 (95% CI .039-.051) for the Managing Anxiety, Future Thinking, and Calm Thinking studies, respectively (Figure 3). However, it should be noted that these observed patterns were not statistically tested for significance.

‎

Discussion

Principal Findings

In this research, we investigated the potential of predicting early attrition from 3 studies of multisession, web-based, positive CBM-I training programs using a combination of features derived from training and assessment data, including baseline user characteristics, self-reported user context and reactions to the program, passive detection of user behavior, and user clinical functioning. Our proposed pipeline was able to identify participants who were at a high risk of dropping out early in these studies. Our pipeline provides a framework (ie, data preprocessing, feature generation, predictive modeling, and feature importance) for predicting attrition in DMHIs broadly, although the particulars (eg, features) will vary with each application. Our results also show that passive features describing user behavior when interacting with a DMHI can be a valuable feature for identifying individuals at high risk of dropping out. In our analyses, interestingly, passive features of user behavior were more informative to this prediction than other features, including user clinical functioning, emphasizing the utility of considering users’ real-time behavior in predicting early attrition.

While these findings need to be validated in future studies, they highlight the value of considering the collection and use of such features in algorithms for predicting attrition in future DMHI designs. Key next steps include the need to make these data-driven approaches transferable to real-world care settings (ie, beyond research settings). Clinicians tend not to integrate DMHIs into their clinical practice, in part due to lack of training and understanding about how DMHIs work, which DMHIs to choose, and how to integrate them [28]. Helping clinicians determine which of their patients is likely to stick with a DMHI (and benefit from it) may help address some of these clinician concerns, and further personalization of the approaches may be useful. Along these lines, more longitudinal features capturing user interaction with DMHIs could enable a level of personalization and customization that has historically been challenging to achieve with only baseline characteristics. It will also be important to address the challenges raised by the complexities of interpreting these algorithms (ie, determining which factors were key to predicting attrition). When the algorithms seem impenetrable, it may increase clinicians’ discomfort with applying them in their practice.

The findings also highlight the value of using both passive user behavioral data collected during the DMHI and the users’ self-report data. Predicting clinical outcomes from single indicators has routinely not been successful. Speaking to the historical challenges in predicting response to depression treatments, van Bronswijk et al [113] noted that “no single moderator is likely to be robust enough, on its own, to reliably guide treatment selection..., and indeed none have been identified.” This has led many researchers to recognize the value of novel methods, such as machine learning, that allow for multivariate prediction. This paper extends this approach further by integrating multiple sources of information, beyond only self-report features. This has several advantages, including reducing user burden by not relying solely on self-reported measures; it allows for prediction to be carried out based on meaningful data about users that they may not have introspective access, or comfort, to report effectively.

Model Performance and Feature Importance

Features extracted from the early stages of a given study (ie, baseline assessment and Session 1 training or assessment; Table 2) were highly predictive of attrition before starting Session 2 training (Table 3). Particularly important was the feature set involving passive detection of user behavior (Figure 3), which consisted of time spent on page, time of the day, day of the week, time spent on tasks, latency between tasks, and device type. Although it is unclear which passive features were most informative (a useful future direction), it may be that certain passive features (eg, time on a web page) contain real-time information about engagement, motivation, or ability to use the program not captured by other measures (eg, self-reports of the importance of reducing anxiety or confidence in the program at baseline or self-reports of ease of using the program at the end of Session 1). However, the feature importance level varied by classifier and study, highlighting the complexity of identifying individual predictors of attrition. Nevertheless, future studies may benefit from including similar feature sets, especially behavioral features.

Furthermore, our analyses revealed that predicting attrition in DMHI studies is not an easy problem; otherwise, simpler models such as the logistic regression and SVM models may have provided sufficient predictive power. The more complex models that leverage ensembles (random forest, gradient boosting, XGBoost, etc) performed substantially better without overfitting to the data by making use of cross-validation and parameter tuning. These models are also inherently interpretable, making it easier to explain results to various audiences, including clinicians and other stakeholders. Overall, these results suggest that ensemble and forest models may provide a strong baseline when predicting attrition in CBM-I studies.

Transfer of Knowledge

Given the sparsity of the original dataset, we expected that models would perform better when given informative priors from similar studies. For example, we can use data from the Managing Anxiety study to provide informative priors to the prediction model that is then trained to predict attrition in the Calm Thinking study. We found that, despite their common use of the MindTrails web infrastructure and use of CBM-I interventions, the 3 studies (Managing Anxiety, Future Thinking, and Calm Thinking) had substantially different data distributions (ie, attrition rate and raw values for given features). The studies also had different model performance, not only when each study used all of its own features (Table 3), but also when the studies used only the features they shared (Table 5). Thus, although our findings provide insights into next steps for this research, their generalizability to other CBM-I studies and DMHIs more broadly is somewhat limited.

This wide variation in data distributions and model performance points to the larger challenge of generalizability in eHealth studies. To address this issue in future work on eHealth attrition prediction using machine learning, we recommend researchers to (1) consider what aspects of our proposed pipeline may be relevant to their specific context and (2) incorporate more advanced transfer learning techniques. Transfer learning is a machine learning method that leverages knowledge learned from one problem and applies it to a related but different problem. Advanced transfer learning techniques can enhance DMHIs by using existing knowledge, addressing class imbalance and feature extraction, and incorporating insights from large datasets to drive actionable solutions for reducing attrition and increasing engagement in DMHIs.

Applied Example

Low engagement in a DMHI may manifest as low initial uptake, substantial early dropout, or failure to adhere long term to the intervention techniques intended to change behavior. Predicting attrition is complicated by the many reasons for which a person may drop out (eg, the program is not meeting their needs or has already met their needs). Still, identifying participants at a high risk for dropout at an early stage may enable allocation of further support specifically to users who may need it, thus improving engagement while retaining scalability [31]. For example, we implemented a probability prediction algorithm in the Calm Thinking study (instead of the binary classification algorithm used in this paper) to predict each participant’s probability of not completing the second session. This probability, the user’s attrition risk score, was then compared with a threshold τ set by the project coordinator (based on a goal to have roughly equal cell sizes after the second randomization point in the study’s sequential, multiple assignment, randomized trial design). Participants (n=547) whose attrition risk score was ≥τ were deemed to have a higher risk of dropping out and were then randomized to receive supplemental telecoaching (n=282) or not (n=265). Those in the coaching condition received an email connecting them with their coach, who proposed a phone call to discuss study goals, reinforce use, and address any technical issues or other study questions. (We excluded higher-risk participants randomized to supplemental telecoaching [n=282] from analyses for this paper.) For more details about this implementation, refer to the main outcomes paper on the Calm Thinking study by Eberle et al [74].

Limitations

One limitation of our analyses is that we focused on participants who started Session 1 training and excluded many participants who dropped out before that point. Another limitation is that we had to use the existing features of the studies, which narrowed our options for feature extraction. It is possible that the model would be further improved with more detailed features (eg, user continuous location [GPS]; passive detection of more finely grained user behavior at the level of individual items vs at the level of scale scores or the entire training or assessment task). In addition, the feature importance results should be interpreted cautiously; readers should refrain from inferring a causal relationship between these features and early attrition. Further research is needed to establish the extent to which such features cause or are a consequence of risk for attrition; it might also be informative to evaluate different imputation and modeling strategies. Furthermore, we used imputation strategies for all missing numeric values, even in cases where dropout meant the meaning of a given measure no longer applied (eg, for Return Intention, imputing number of days expected to return for Session 2 even when the participant did not complete Session 1; for Impact of Anxious Imagery Prime, imputing peak anxiety during the prime even when the participant started training but never completed the prime). Future studies should consider (1) removing features containing missing values that cannot be meaningfully imputed or (2) restricting the sample to participants who completed all features that cannot be meaningfully imputed. Finally, future work should seek to identify and, if needed, mitigate potential algorithmic biases. For example, the studies in this paper required participants to have internet access and were optimized for computer delivery, which may lead to underrepresentation in the training data for demographic groups that lack internet access or are dependent on smartphones [13,114]. While some studies have shown that including demographic features (eg, gender and race) in early dropout prediction has minimal impact on algorithmic fairness [115], it is prudent to perform a sensitivity analysis excluding these features, to compare model performance by demographic group, and to use bias-aware model calibration techniques when possible [116].

Conclusions

This paper aimed to identify participants at a high risk of dropout during the early stage of 3 multisession, web-based CBM-I studies using a combination of self-reported and passively detected measures. Our findings suggest that features involving passive detection of user behavior, such as time spent on a web page of CBM-I training exercises, time of the day, latency in completing assessments, and the type of device used, were the most informative predictors of attrition. In addition, our results showed that using all features extracted from a given study led to the best predictive performance, highlighting the importance of using a combination of feature types when predicting attrition. Consequently, using passive indicators of user behavior, in conjunction with self-reported measures, can increase the accuracy of predicting dropout in web-based CBM-I studies. Although our pipeline provides a framework to consider while predicting attrition in DMHIs, many interesting, open questions remain about how extensively our findings generalize to other CBM-I studies (eg, in populations with diagnosed anxiety [vs trait anxiety], in mobile app–based [vs web-based] CBM-I studies, in CBM-Is embedded in managed care settings [vs on a public website]) and to DMHIs more broadly (eg, unguided web-based cognitive behavioral therapy [117]). Our analyses highlight the challenge of generalizability in DMHI studies and the need for more personalized attrition prevention strategies. Overall, our results emphasize the potential value of understanding user behavior in early stages of the program and using it as a predictor of dropout, which may guide development of more effective and efficient DMHIs.