This paper is in the following e-collection/theme issue:

Anxiety and Stress Disorders (1013) Cost-Effectiveness and Economics (234) Web-based and Mobile Health Interventions (3128)
Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/10866, first published .
Economic Evaluation of an Internet-Based Stress Management Intervention Alongside a Randomized Controlled Trial

Economic Evaluation of an Internet-Based Stress Management Intervention Alongside a Randomized Controlled Trial

Economic Evaluation of an Internet-Based Stress Management Intervention Alongside a Randomized Controlled Trial

Authors of this article:

Fanny Kählke1 Author Orcid Image ;   Claudia Buntrock1 Author Orcid Image ;   Filip Smit2, 3, 4 Author Orcid Image ;   Matthias Berking1 Author Orcid Image ;   Dirk Lehr5 Author Orcid Image ;   Elena Heber6 Author Orcid Image ;   Burkhardt Funk7 Author Orcid Image ;   Heleen Riper2, 8 Author Orcid Image ;   David Daniel Ebert1, 2 Author Orcid Image
Article Authors Cited by (10) Tweetations (1) Metrics

    Original Paper

    1Institute for Psychology, Department of Clinical Psychology and Psychotherapy, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany

    2Amsterdam Public Health, Department of Clinical, Neuro- and Developmental Psychology, Vrije Universiteit, Amsterdam, Netherlands

    3Amsterdam Public Health Research Institute, Department of Epidemiology and Biostatistics, VU University Medical Centre, Amsterdam, Netherlands

    4Netherlands Institute of Public Mental Health, Centre of Health-Economic Evaluation, Trimbos Institute, Utrecht, Netherlands

    5Institute for Psychology, Department of Health Psychology and Applied Biological Psychology, Leuphana University, Lüneburg, Germany

    6GET.ON Institute for Online Health Trainings, Hamburg, Germany

    7Institute of Information Systems, Leuphana University, Lüneburg, Germany

    8Telepsychiatric Centre, University of Southern Denmark, Odense, Denmark

    Corresponding Author:

    Fanny Kählke, MPH, MEd

    Institute for Psychology

    Department of Clinical Psychology and Psychotherapy

    Friedrich-Alexander-Universität Erlangen-Nürnberg

    2nd Floor

    Nägelsbachstrasse 25a

    Erlangen, 91052

    Germany

    Phone: 49 9131 85 ext 67568

    Fax:49 9131 85 67576

    Email: fanny.kaehlke@fau.de


    Background: Work-related stress is widespread among employees and associated with high costs for German society. Internet-based stress management interventions (iSMIs) are effective in reducing such stress. However, evidence for their cost-effectiveness is scant.

    Objective: The aim of this study was to assess the cost-effectiveness of a guided iSMI for employees.

    Methods: A sample of 264 employees with elevated symptoms of perceived stress (Perceived Stress Scale≥22) was assigned to either the iSMI or a waitlist control condition (WLC) with unrestricted access to treatment as usual. Participants were recruited in Germany in 2013 and followed through 2014, and data were analyzed in 2017. The iSMI consisted of 7 sessions plus 1 booster session. It was based on problem-solving therapy and emotion regulation techniques. Costs were measured from the societal perspective, including all direct and indirect medical costs. We performed a cost-effectiveness analysis and a cost-utility analysis relating costs to a symptom-free person and quality-adjusted life years (QALYs) gained, respectively. Sampling uncertainty was handled using nonparametric bootstrapping (N=5000).

    Results: When the society is not willing to pay anything to get an additional symptom-free person (eg, willingness-to-pay [WTP]=€0), there was a 70% probability that the intervention is more cost-effective than WLC. This probability rose to 85% and 93% when the society is willing to pay €1000 and €2000, respectively, for achieving an additional symptom-free person. The cost-utility analysis yielded a 76% probability that the intervention is more cost-effective than WLC at a conservative WTP threshold of €20,000 (US $25,800) per QALY gained.

    Conclusions: Offering an iSMI to stressed employees has an acceptable likelihood of being cost-effective compared with WLC.

    Trial Registration: German Clinical Trials Register DRKS00004749; https://www.drks.de/DRKS00004749

    International Registered Report Identifier (IRRID): RR2-10.1186/1471-2458-13-655

    JMIR Ment Health 2019;6(5):e10866

    doi:10.2196/10866

    Keywords

    work; occupational stress; economic evaluation; internet; quality of life; clinical trials, randomized 


    Background

    Up to 27% of the workforce in Europe suffers from elevated stress levels [1]. According to the effort-reward imbalance model [2] and the job demand-control model [3] situations characterized by an imbalance between high effort (eg, workload) and low reward (eg, job insecurity) or high demand and low job decision latitude lead to high levels of strain. This strain is known to be a risk factor for psychological and physiological health consequences such as sleeping problems [4], mental health problems [5], cardiovascular disease [6], and chronic pain [7]. Consequently, the resulting economic burden due to productivity losses (eg, sick leave) [8] and higher health care consumption and out-of-pocket payments is substantial [9-11]. The estimated costs of work-related stress range from US $221.13 million to US $187 billion and therefore impose a tremendous burden on society [12]. Psychological interventions can be effective in reducing stress [13], but the availability of face-to-face treatments is limited [14]. Web-based and mobile-based interventions have been proposed to overcome the limitations of traditional face-to-face interventions. Such interventions are low-threshold interventions, are available 24/7, and are associated with low costs [15].

    In a recent meta-analysis, it has been shown that internet-based stress management interventions (iSMIs) are effective with an effect size of d=0.43 (95% CI 0.31-0.51) on perceived stress [16] and a small effect on depression and anxiety, but lack evidence regarding cost-effectiveness. Internet-based interventions are often argued to be cost-effective, yet there exists little evidence. Donker et al [17] found that internet-based interventions for common mental health disorders have a considerable probability of being more cost-effective when compared with control groups. Most health economic outcome studies evaluated internet-based interventions for alcohol consumption [18], smoking cessation [19], anxiety [20], and depression [17,21]. Hedman et al compared an iSMI with an internet-based cognitive behavior therapy (iCBT) for treatment of health anxiety, where the iSMI resulted in lower costs [22].

    Objectives

    To our knowledge, there exist no cost-effectiveness and cost-utility analyses of iSMIs from the societal perspective. Thus our aim was to establish the cost-effectiveness and cost-utility of this iSMI for employees.


    Design

    This study is a health-economic evaluation with a 6-month time horizon from a societal perspective alongside a 2-arm randomized controlled trial (RCT) in Germany to establish the cost-effectiveness and cost-utility of an iSMI for employees with elevated work-related stress in combination with usual care compared with a waitlist control condition (WLC) with access to treatment as usual [23]. The present health-economic evaluation followed guidelines from the International Society for Pharmacoeconomics and Outcomes Research RCT-cost-effectiveness analysis Task Force report and the recommendations of the Consolidated Health Economic Evaluation Reporting Standard [24,25]. The trial included 264 participants who were randomly allocated in a 1:1 ratio with a block size of 2 to either iSMI or WLC. An independent researcher not otherwise involved in the study performed the randomization using randomization software (Randlist, Datinf GmbH) [26]. Participants were included in the study if they were 18 years or older, currently employed, and scored 22 or above on the Perceived Stress Scale (PSS-10). One SD (SD 6.2) above the mean (PSS-10=15.3) in a large working population [27] was chosen as a cut-off value to select participants with an elevated level of stress. The exclusion criteria were to be at risk of suicide or dissociative symptoms or having been diagnosed with a psychosis. The Ethics Committee of the Philipps-University of Marburg, Germany, approved the study. The trial was registered (DRKS00004749) in the German Clinical Trials Register.

    Intervention

    The most popular models to explain work-related stress are the effort-reward imbalance and the job demand-control model. According to the effort-reward imbalance model [2], work-related stress is generated by high effort (eg, pace of work and workload) and low reward received in return (eg, inadequate salary, promotion prospects, and job security). The job demand-control model [3] identifies high demand (eg, high workload) and low job decision latitude (eg, autonomy and control over the job) as factors that lead to high levels of job strain. This strain is known to be a risk factor for adverse health consequences, such as mental health problems [5], chronic pain [7], and cardiovascular disease [6]. Ideally, job strain should be reduced by changing adverse working conditions such as small rooms and bad equipment. As changing these may be difficult, stressors on an individual level such as inadequate coping strategies can also be addressed. Interventions based on Lazarus’s transactional model aim to empower the individual to reduce or modify problems at work (ie, high effort, low rewards, or low decision latitudes). This model identifies 2 strategies of coping with stressors: problem-oriented coping, to actively change or adapt stressors, and emotion-oriented coping, to cope with negative emotions due to stressors at the workplace. Thus, the iSMI is based on 2 main components: problem solving and emotion regulation. Problem solving is an evidence-based method for dealing with such problems and has been proven to be successful in improving mental health [28]. However, employees are frequently faced with unsolvable problems, which are associated with strong negative affective reactions and require effective regulation strategies. Improvement of emotion regulation skills has been shown to be both promising for reducing psychopathological symptoms [29] and a mechanism of change in previous studies using this iSMI [30]. Deficits in emotion regulation may also be an important factor for the development and persistence of mental health symptoms [31]. Yet, emotion-focused coping is regarded as the forgotten component, whereas problem-focused coping by means of problem-solving techniques is a well-established component of most cognitive-behavioral stress management trainings.

    The iSMI is based on Lazarus’s transactional model of stress and includes problem solving and emotion regulation. The intervention consists of 8 sessions composed of modules for psycho-education (session 1), problem solving (sessions 2 and 3), emotion regulation (sessions 4-6), planning for the future (session 7), and a booster session (session 8). In addition, participants could choose optional modules covering different topics, for example, time management, rumination and worrying, psychological detachment from work, and sleep hygiene. Each module takes approximately 45 to 60 min to complete. Participants were advised to complete 1 to 2 modules per week. Transfer tasks such as homework assignments were integrated into the intervention to help participants integrate learned skills into daily life. Participants received nontherapeutic feedback by an e-Coach after each completed module. E-Coaches had a degree in psychology, and feedbacks were based on a standardized manual on feedback writing. Participants could also opt in for an additional text message coach along the iSMI (eg, short relaxation exercises). A detailed description of the iSMI can be found elsewhere [32]. The clinical effectiveness of the iSMI has been positively evaluated in a series of RCTs [23,30,31,33,34].

    Outcome Measures

    Self-reported measures of stress and social functioning (PSS-10 and Short-Form Six-Dimension; SF-6D) were collected at baseline (T1), post treatment (T2; 7 weeks after randomization), and 6-month follow-up (T3) using a secured Web-based assessment system (AES, 256-bit encrypted).

    Clinical Outcome

    The level of perceived stress was measured by the PSS-10 [27]. Cronbach alphas indicated that the internal consistency ranged from .70 to .91 over different measurement points in this study [30]. Symptom-free status was operationalized as scoring 2 SDs below the PSS-10 sample mean at T1 (mean 25.52, SD 3.91) [23,35].

    Quality-Adjusted Life Years

    Quality-Adjusted Life Years (QALYs) were used as the primary outcome in the cost-utility analysis. QALYs were computed using the SF-6D [36]. A QALY gain of 0.5 indicates full health throughout the 6-month trial period. The SF-6D is more sensitive to change in mild conditions than the more commonly used EQ-5D and was used for the main analysis [37].

    Resource Use and Costing

    We assessed direct and indirect costs which occurred over the previous 3 months at baseline, and at 6-month follow-up. All costs were calculated in Euros for the reference year 2013 (index factor 1.04 based on the year 2010), referring to the German consumer price index [38]. Costs were converted to US dollar using the purchasing power parities reported by the Organization for Economic Cooperation and Development. For the reference year 2013, €1 was equated to US $1.29.

    The Trimbos Institute and Institute of Medical Technology Questionnaire for Costs Associated with Psychiatric Illness (TiC-P) adapted to the German health care system was used [39]. This is a widely used and reliable instrument for collecting self-reported data on health care utilization and productivity losses in patients with mild to moderate mental health conditions [40-46]. The German version has been used in a number of health economic evaluations alongside randomized trials [21,41,42,44]. The standard unit cost prices were multiplied by the units of resource use for each participant. Multimedia Appendix 1 presents direct medical and direct nonmedical costs by health service type. Cumulated costs of the trial were estimated using the area under curve method to linearly interpolate 3 months costs as measured at each measurement point to cover the full follow-up period of 6 months [47].

    Health Care Costs

    Health care costs were calculated according to the guidelines of Kraut and Bock et al [48,49]. We included unit costs for a physician; a medical specialist; psychological services such as a psychiatrist and psychotherapist; and allied health services such as physiotherapy, massage, occupational therapy, as well as inpatient care and rehabilitation.

    Medication

    Unit costs of prescription drugs were calculated using the German register for pharmaceutical drugs Rote Liste [50]. The basis for calculating costs of prescribed medication is the pharmacy retail price accounting for a specific pharmacy and manufacture’s discount. The discount rates vary between private and statutory health insurances [48]. Therefore, we weighted the mean costs of the 3 largest packages with the same agent based on the daily defined dose by the statutory population share (88,80% of the German population are statutorily insured).

    Intervention Costs

    The provider (GET.ON Institute GmbH) of the iSMI intervention GET.ON Stress estimated the current market price of the intervention at €299 (US $386) per participant. This flat tariff covers all costs for developing and hosting the intervention plus coaching of the participants. In general, it was assumed that every participant owned a computer, had access to the internet, and used the iSMI in their leisure time after working hours. Hence, these costs were not included.

    Patient and Family Costs

    Participants self-reported the cost of their out-of-pocket expenses (eg, for over-the-counter drugs). Direct nonmedical travel costs were calculated based on self-reported data that included the used method of transportation (ie, bus, taxi, or car) and round-trip distance to reach health care services. Each kilometer by car was valued at €0.30 [51]. Time spent by participants completing the intervention and/or receiving or waiting for treatment by a physician was considered part of their leisure time. The opportunity cost of leisure time, defined as the cost associated with the next best alternative use of a particular resource, was valued at €23.10 per hour. This was based on Bock et al’s recommendations [48], which estimated these costs based on the average net wage of German employees plus their average pension and unemployment insurance contributions.

    Costs incurred from a domestic help (help with daily chores) or production losses resulting from unpaid work such as informal care by friends and family were calculated using the substitution method. These costs were based on the average gross hourly wage earned by a domestic worker, as suggested by Bock et al [48]. This time was valued at €18.33 per hour.

    Costs of Productivity Losses

    Absenteeism costs were calculated by applying the human capital approach [52]. In doing so, the number of work loss days was multiplied by the participant’s average gross daily wage based on their reported monthly salary. In addition, participants reported the number of workdays for which they reported lesser efficiency. On the basis of the Osterhaus method [53], these days were multiplied by an inefficiency score, which resulted in lost-workday equivalents due to presenteeism. Subsequently, based on self-reported monthly salary, their gross wages per day were calculated and used to calculate the costs that occurred due to presenteeism.

    Statistical Analysis

    This study was powered to detect a mean difference of d=0.35 in the primary outcome (PSS) between the groups at post measurement. Cost data are usually heavily skewed to the right, with large variance requiring very large sample sizes to test the statistical significance of cost differences. Instead, we adopted a probabilistic decision-making approach for our economic analyses [54]. This procedure takes the stochastic uncertainty of the trial data into account [55] and informs the decision makers on probabilities rather than statistical significance. Due to the 6-month follow-up period, no discounting was applied.

    All analyses were conducted in accordance with the intention-to-treat (ITT) principle. Missing clinical outcome data were imputed using a Markov Chain Monte Carlo multivariate imputation algorithm with 10 estimations per missing value.

    Missing cost data were imputed using the regression imputation procedure implemented in Stata to obtain the required predicted values. Predictors of outcome and dropout were identified via (logistic) regression. Differences in PSS score and symptom-free status between groups were assessed at follow-up using the Chi-square test. At baseline, mean SF-6D utility values were similar in both groups (WLC: mean 0.65, SD 0.08 and iSMI: mean 0.65, SD 0.11). Therefore, no baseline adjustments were made when calculating QALYs. Differences in QALYs between iSMI and WLC were assessed using independent samples t tests.

    Analysis of Cost-Effectiveness and Cost-Utility

    For the cost-effectiveness analyses, the incremental cost-effectiveness ratio (ICER) was calculated as incremental costs per unit of effect (QALY and symptom-free status). Symptom-free status is meaningful for decision makers and was used as the preferred effect measure as there was no difference between beta coefficients from an OLS regression on the binary outcome compared with beta coefficients from a linear probability model in GLM (GLM: beta=.36, P<.001 and OLS: beta=.36, P<.001).

    The ICER was calculated as ICER=(costsiSMI−costsWLC) /(effectsiSMI−effectsWLC), where costs are the cumulated costs over the 6-month period and effect are QALY gains or symptom-free status.

    Stochastic uncertainty in the ICER was handled using nonparametric bootstrapping, which is a resampling technique applied to the trial data, which generates 5000 simulations of the ICER. The incremental costs and incremental effects were obtained under a bootstrapped seemingly unrelated regression equations model and allowed for correlated residuals of the cost and effect equations [56]. The 5000 bootstrap replications of costs and effects were also used to obtain 95% CIs based on the percentile method.

    In a next step, the simulated ICERs were plotted in a cost-effectiveness plane. On the plane, incremental effects are depicted on the horizontal x-axis and the incremental costs on the vertical y-axis. Each dot in the cost-effectiveness plane represents 1 bootstrapped ICER.

    The willingness-to-pay (WTP) threshold reflects the maximum amount the society would be willing to pay for a health benefit (eg, a symptom-free person or a QALY gained). As the WTP ceiling for gaining 1 unit of health (eg, gaining 1 QALY or obtaining symptomatic remission in 1 person) is an unknown quantity, a cost-effectiveness acceptability curve was presented, which displays the probability of the intervention being cost-effective for 1 additional unit of health gained at varying WTP ceilings. All analyses were performed using Stata version 13 [57].

    Sensitivity Analyses

    The robustness of the outcomes was assessed using several sensitivity analyses. First, we used the EQ-5D-3L (European Quality of Life 5 Dimensions 3 Level) instrument [58] for the calculation of QALYs. Second, there is uncertainty regarding the cost of the intervention due to changing demand. Therefore, we conducted sensitivity analyses assuming higher and lower interventions costs (±€100). Third, inpatient costs tend to be very high, but they were only reported by a few participants (n=9, 3.4%). Such outliers may lead to distorted outcomes results, so they were removed in the final sensitivity analysis.


    Sample

    Multimedia Appendix 2 presents the baseline characteristics. Interested participants were recruited from the general working population via mass media (eg, newspaper articles and television) and with the aid of a health insurance company (BARMER) within their occupational health management program. An open-access website [59] was used to sign-up for study participation. The sample predominately consisted of full-time employed middle-aged women living with a partner. A comprehensive description of the study sample and the participant flow can be found elsewhere [23]. We did not observe any clinically relevant baseline differences between study conditions.

    Study Dropouts

    The study attrition was low: 10.6% (28/264) of participants did not complete the 6-month follow-up assessment. The dropout rates between the groups, with 12.8% (17/132) in the iSMI condition and 8.33% (11/132) in the WLC condition, did not differ significantly (χ21=1.4 P=.23).

    Outcome Measures

    The iSMI improved by 9.75 (SD 6) PSS-10 stress units between pre and 6-month follow-up, whereas the WLC improved by 3.0 units (SD 6) PSS. Differences regarding symptom-free status based on the PSS-10 between groups were assessed at follow-up (iSMI: 79/132, 59.8%; WLC: 31/132, 23.5%; χ21=35.9; P<.001; NNT (Number needed to treat) =2.75, 95% CI 2.11-3.95) [23]. However, the intervention and the WLC did not differ significantly in terms of SF-6D QALY gains (iSMI=0.35, SD 0.04 vs WLC=0.35, SD 0.35; t262=−1.625; P=.10).

    Costs

    At baseline, mean total costs were €3239 (US $4178) in the iSMI and €3183 (US $4178) in the WLC, which is only a small difference of €56 (US $72), indicating that randomization had resulted in a well-balanced trial. Table 1 presents the average 6-month accumulated per-participant costs by study condition. The costs are clustered into health care costs, patient and family costs, and costs stemming from productivity losses. After 6 months, total incremental costs were €380 (US $490); thus, the iSMI group had less costs than WLC (iSMI: €5258 and WLC: €5642). Health care costs were, on average, higher in the iSMI group compared with WLC. Hospital admissions were a major cost driver. Regarding the patient and family costs, the iSMI had less costs than WLC. Informal care was decreased by €241 for the iSMI. Finally, productivity losses produced the highest cost differences of €487, exceeding the intervention costs, meaning that the iSMI produced less cost than WLC.

    Cost-Effectiveness

    Table 2 shows the incremental costs, effects, and cost-effectiveness ratios based on 5000 bootstrapped simulations. The bootstrapped ICER for symptom-free status on the PSS-10 was dominant. The cost-effectiveness plane is shown in Figure 1. The majority (70%) of the bootstrapped ICERs fell in the south-east quadrant, indicating a 70% probability that the intervention produces greater health at lower costs than WLC. Hence, the iSMI intervention dominates the WLC condition from a societal perspective. The remaining 30% of ICERs fell in the north-east quadrant, indicating a 30% probability that the intervention produces greater health at greater costs than WLC. Figure 2 presents the cost-effectiveness acceptability curve. If the decision maker is willing to pay €1000 and €3000 for gaining a symptom-free person, the intervention’s probability of being more cost-effective than WLC rises to 85% and 97%, respectively.

    Table 1. Average costs per participant (in €) by condition at 6-months follow-up (area under the curve, intention-to-treat-sample, N=264).
    Cost categoryInternet-based stress management intervention (n=132), mean (SD)Waitlist control condition (n=132), mean (SD)Incremental costs, difference
    Health care costs (€)

    		Intervention299 (Reflects a fixed price)0 (Reflects a fixed price)299

    		Physician services132 (139)147 (175)−15

    		Psychological services111 (291)209 (468)−98

    		Hospital in-patient342 (2222)188 (1237)154

    		Hospital semiresidential234 (1444)77 (798)157

    		Rehabilitation8 (41)89 (658)−81

    		Nonphysician services167 (293)174 (314)−7

    		Prescription drugs50 (97)56 (105)−6
    Patient and family costs (€)

    		Over the counter drugs48 (88)48 (78)0

    		Opportunity costs485 (754)526 (892)−42

    		Travel expenses27 (48)49 (94)−21

    		Domestic help or informal care424 (1213)665 (1327)−241
    Productivity losses (€)

    		Absenteeism1346 (2184)1655 (3436)−309

    		Presenteeism1578 (1471)1756 (1849)−178
    Total costs (€)a5258 (5493)5642 (6000)−384

    aDue to rounding, numbers presented may not add up precisely to the totals provided.

    Table 2. Results of the main and sensitivity analysis based on 5000 bootstrap simulations. Costs are expressed in 2013 Euros.
    Analysis and outcomeIncremental costs, € (95% CI)Incremental effects, points (95% CI)Incremental cost-effectiveness ratio, €/points (95% CI)aDistribution over the cost-effectiveness plane, %
    North-east quadrantbSouth-east quadrantcSouth-west quadrantdNorth-west quadrante
    Main analysis

    		Perceived stress (range 0-40)−386 (−1794 to 1006)6.27 (4.9 to 7.7)fDominant (dominant to 171)3070—g—

    		Symptom-free status (0/1)−386 (−1794 to 1006)0.362 (0.25 to 0.47)fDominant (dominant to 3360)3070——

    		QALYsh (range: 0-1)−386 (−1794 to 1006)0.0074 (−0015 to 0.016)Dominanti266923
    Sensitivity analysis 1j

    		Perceived stress (range 0-40)−616 (−1731 to 485)6.27 (4.9 to 7.7)fDominant (dominant to 81)1387——

    		Symptom-free status ( 0/1)−616 (−1731 to 485)0.362 (0.25 to 0.47)fDominant (dominant to 1415)1387——

    		QALYs (range: 0-1)−616 (−1731 to 485)0.0074 (−0015 to 0.016)Dominanti128323
    Sensitivity analysis 2k, €+100 added to intervention costs

    		Perceived stress (range 0-40)−286 (−1694 to 1106)6.27 (4.9 to 7.7)fDominant (dominant to 187)3466——

    		Symptom-free status (0/1)−286 (−1694 to 1106)0.362 (0.25 to 0.47)fDominant (dominant to 3419)3466——

    		QALYs (range: 0-1)−286 (−1694 to 1106)0.0075 (−0015 to 0.016)Dominanti316423
    Sensitivity analysis 2k, €-100 added to intervention costs

    		Perceived stress (range 0-40)−486 (−1894 to 906)6.27 (4.9 to 7.7)fDominant (dominant to 155)2476——

    		Symptom-free status (0/1)−486 (−1894 to 906)0.362 (0.25 to 0.47)fDominant (dominant to 2764)2476——

    		QALYs (range: 0-1)−486 (−1894 to 906)0.0075 (−0015 to 0.016)Dominanti227323
    Sensitivity analysis 3l

    		QALYs (range: 0-1)−386 (−1794 to 1006)0.00186 (−0.010 to 0.014)Dominanti49142216

    aIn line with the best practice ISPOR guidelines on ‘Model Parameter Estimation and Uncertainty’ we did not report negative incremental cost-effectiveness ratios (ICERs) as they are meaningless. Instead we used the term dominant which implies that the intervention has a higher effect and less cost compared with the WLC.

    bThe north-east quadrant of the CE plane, indicating that intervention is more effective and more costly.

    cThe south-east quadrant of the CE plane, indicating that intervention is more effective and less costly.

    dThe south-west quadrant of the CE plane, indicating that intervention is less effective and less costly.

    eThe north-west quadrant of the CE plane, indicating that intervention is less effective and more costly.

    fP<.05.

    gThe distribution of the ICERs (N=5000) sums to 100%. If the distribution only falls into 2 quadrants, there will not be any ICER in the other 2 quadrants (= 0%).

    hQALYs: quality-adjusted life years.

    iA dependably accurate 95% confidence interval for this distribution cannot be defined because there is no line through the origin that excludes alpha/2 of the distribution.

    jSensitivity analysis 1 analyses not including in-patient care.

    kSensitivity analysis 2 analyses adding €±100 of intervention costs.

    lSensitivity analysis 3 analyses for EQ5D quality-adjusted life years.

    ‎
    Figure 1. Scatterplot of 5000 replicates of the incremental cost-effectiveness ratio (mean differences in costs and symptom-free status) on the cost-effectiveness plane: internet-based stress-management intervention versus waitlist control condition.
    ‎
    Figure 2. Cost-effectiveness acceptability curve showing the probability of the internet-based stress-management intervention being cost-effective at varying willingness-to-pay ceilings (based on 5000 replicates of the incremental cost-effectiveness ratio using mean differences in costs and symptom-free status).

    Cost-Utility

    The ICER based on QALY gains showed a small health benefit (approximately 0.001 QALYs gained) for lower mean costs (€386; US $498). Of the simulated ICERs, 69% (as seen in Figure 3) fell in the south-east quadrant, reflecting the intervention’s probability of dominating WLC, whereas 26% fell in the north-east quadrant , indicating higher costs and health gains, and 2% fell in the south-west quadrant and 3% in north-west quadrant. Assuming a WTP of €10,000 and €20,000 for gaining 1 QALY, the probability rose to 73% and 76%, respectively (Figure 4).

    ‎
    Figure 3. Scatterplot of 5000 replicates of the incremental cost-effectiveness ratio (mean differences in costs and quality-adjusted life years based on the Short-Form Six-Dimension) on the cost-effectiveness plane: internet-based stress-management intervention versus waitlist control condition. QALY: quality-adjusted life years; SF-6D: Short-Form Six-Dimension.
    ‎
    Figure 4. Cost-effectiveness acceptability curve showing the probability of the internet-based stress-management intervention being cost-effective at varying willingness-to-pay ceilings (based on 5000 replicates of the incremental cost-effectiveness ratio using mean differences in costs and quality-adjusted life years based on the Short-Form Six-Dimension). QALY: quality-adjusted life years; SF-6D: Short-Form Six-Dimension.

    Sensitivity Analyses

    Using the EQ-5D-3L resulted in a smaller incremental QALY gain in favor of the intervention group (0.28 QALY, SD 0.05) compared with WLC (0.28 QALY, SD 0.05), which was not statistically significant (t262=−0.296; P=.77). This is in line with available evidence that the EQ-5D-3L suffers from ceiling effects in milder conditions [37]. Nevertheless, at a WTP of €20,000 for gaining a QALY, the probability of being cost-effective was 71%.

    As inpatient costs were reported from only a few participants but were associated with high costs, these costs might have distorted the results. Excluding these costs led to higher ICERs for both outcomes (eg, symptom-free status and QALYs). The probability of being cost-effective rose to 86% and 96% at a WTP of €0 and €1000 with regard to symptom-free status, and 86% and 90% for gaining a QALY, respectively.

    Increasing and subsequently reducing the intervention costs by €100 led to a 66% and 76% probability that the intervention produces a greater health gain at lower costs than WLC with regard to symptom-free status and 1-point improvement.


    Principal Findings

    This study evaluated the cost-effectiveness and cost-utility of a Web-based guided self-help intervention for employees with elevated stress levels aimed at reducing perceived stress compared with WLC from the societal perspective. The intervention had a significant and favorable effect on perceived stress after 6 months and a high probability of being cost-effective compared with the control condition. The overall conclusion of this study does not change when using any of the assumptions, as explored in the sensitivity analyses.

    Strengths and Limitations

    First, we had missing data, which were handled using imputation techniques to perform an ITT analysis of both effects and costs [60]. As dropout rate was very low (12.8% for the iSMI and 8.33% for the WLC at 6 months), it is unlikely that this has biased the results substantially. Second, the costs and effects were only evaluated over a 6 months period. Hence, we cannot draw any conclusions about long-term effects. Third, self-reported costs and effects might have led to social desirability and/or recall bias. Nonetheless it seems unlikely that this bias differed systematically between groups due to absent baseline differences. Fourth, approaches used for cost estimation of lost productivity are based on the participants’ wages which do not reflect the average wages in the general population. Fifth, a waitlist control group design with unrestricted access to treatment as usual was chosen, which causes participants to be less motivated to initiate health-related behavior changes and thus over-accentuates effects [61]. Sixth, the majority of the sample was female, which is a common feature of mental health internet-based interventions [62]. The gender imbalance might limit the generalizability of study findings. Finally, the use of behavioral interventions does not result in improved working conditions that could cause less job strain. However, the potential of workplace-related interventions is often not fully utilized, and hence, such interventions are not systematically implemented. Thus, we recommend a combined implementation to design healthy working conditions.

    Comparison With Findings From Other Studies

    The results of this study with an effect size of d=0.83 [23] on perceived stress are in line with the meta-analytic evidence (pooled effect size of d=0.43, 95% CI: 0.31-0.54) [16].

    In addition, some evidence exists for the economic benefits of stress management and internet-based interventions to reduce depressive symptoms in employees. However, to the best of our knowledge, this study is the first study to evaluate the cost-effectiveness of a Web-based guided self-help intervention for employees with elevated stress levels.

    Jacobsen et al evaluated the costs of a self- and professional-administered stress-management intervention not delivered over the internet in patients undergoing chemotherapy compared with usual care [63]. Lower costs and statistically higher quality of life outcomes were found in the intervention group. Hedman et al compared behavioral stress management with iCBT for treatment of severe health anxiety. The iSMI resulted in lower costs but was not considered cost-effective [22].

    In a Web-based intervention by Geraedts [64], the probabilities of cost-effectiveness were 0.62 (societal perspective) and 0.55 (employer’s perspective) compared with WLC in employees with depressive symptoms. The intervention was not judged cost-effective. Besides that, the reduction of depressive symptoms was rather small (d=0.16) [65] compared with our study (d=0.64) [23] at post measurement. However, Buntrock et al reported an effect size of d=0.69 for a Web-based intervention for the prevention of depression. This intervention has an acceptable likelihood of being more cost-effective than enhanced usual care [21]. Focusing on perceived stress rather than on depressive symptoms in employees seems to be a cost-effective strategy to reduce the mental burden.

    Clinical Implications

    The results of this study support the idea that iSMIs could be a promising cost-effective strategy in reducing adverse effects of persistent stress in the workplace. Intervention costs were mainly driven by psychologists who acted as e-Coaches. Yet, studies showed that iSMIs are also effective when delivered in a less costly adherence-focused guidance and pure self-help format [33]. However, meta-analytic evidence shows that guidance yields higher effect sizes [66]. Therefore, the cost-effectiveness of guided versus unguided iSMI needs to be evaluated.

    Long-term costs caused by persistent stress, such as staff turnover or mental health disorder onsets, were not taken into account. Future studies should investigate the long-term economic effects of iSMIs. The sample consisted predominately of middle-aged women. Future research should focus on the general German working population regarding recruitment, implementation, and dissemination.

    Conclusions

    This study demonstrated that this iSMI has a high probability of being cost-effective in reducing stress levels when compared with WLC. Given the increasing stress in the workplace and the small number of people who are reached via available health care services [67], it would be worthwhile to integrate such iSMIs into routine occupational health care, which conventionally only consists of face-to-face therapy by occupational health physicians.

    Acknowledgments

    The authors gratefully acknowledge the contribution of all participants, research assistants, therapists, and all others who contributed to this study. Funding was received from the European Union (project number: EFRE: CCI 2007DE161PR001) and the BARMER (German statutory health insurance company). The funders did not have a role in study design, data collection, analyses or interpretation of the results, or the decision to publicize the study results.

    Authors' Contributions

    MB obtained the funding for this trial and DDE obtained the funding for the analysis. EH was responsible for the study administration. FK analyzed the data and drafted the manuscript on the health-economic evaluation, and DDE supervised this process. All authors contributed to the further writing of the manuscript and approved the final version of the manuscript.

    Conflicts of Interest

    DDE, DL, EH, and MB are stakeholders of the “Institute for Online Health Trainings,” which aims to transfer scientific knowledge related to this research into routine health care.

    ‎

    Multimedia Appendix 1

    Unit costs for the type of health service utilized by the participants.

    PDF File (Adobe PDF File), 14KB
    ‎

    Multimedia Appendix 2

    Demographic characteristics: means/counts, standard deviations/percentages at baseline.

    PDF File (Adobe PDF File), 14KB
    1. Eurofound. Developments in Working Life in EuropeurWORK Annual Review. Luxembourg. Publications Office of the European Union; 2016.
    2. Siegrist J. Adverse health effects of high-effort/low-reward conditions. J Occup Health Psychol. Jan 1996;1(1):27-41. [CrossRef] [Medline]
    3. Karasek Jr RA. Job demands, job decision latitude, and mental strain: implications for job redesign. Adm Sci Q. Jun 1979;24(2):285-308. [CrossRef]
    4. Åkerstedt T, Orsini N, Petersen H, Axelsson J, Lekander M, Kecklund G. Predicting sleep quality from stress and prior sleep--a study of day-to-day covariation across six weeks. Sleep Med. Jun 2012;13(6):674-679. [CrossRef] [Medline]
    5. Theorell T, Hammarström A, Aronsson G, Träskman BL, Grape T, Hogstedt C, et al. A systematic review including meta-analysis of work environment and depressive symptoms. BMC Public Health. Aug 1, 2015;15:738. [FREE Full text] [CrossRef] [Medline]
    6. Dragano N, Siegrist J, Nyberg ST, Lunau T, Fransson EI, Alfredsson L, et al. IPD-Work Consortium. Effort–reward imbalance at work and incident coronary heart disease: a multicohort study of 90,164 individuals. Epidemiology. Dec 2017;28(4):619-626. [FREE Full text] [CrossRef] [Medline]
    7. Sihawong R, Sitthipornvorakul E, Paksaichol A, Janwantanakul P. Predictors for chronic neck and low back pain in office workers: a 1-year prospective cohort study. J Occup Health. 2016;58(1):16-24. [FREE Full text] [CrossRef] [Medline]
    8. Niedhammer I, Chastang J, Sultan-Taïeb H, Vermeylen G, Parent-Thirion A. Psychosocial work factors and sickness absence in 31 countries in Europe. Eur J Public Health. Aug 2013;23(4):622-629. [CrossRef] [Medline]
    9. Uegaki K, Bakker I, de Bruijne M, van der Beek A, Terluin B, van Marwijk H, et al. Cost-effectiveness of a minimal intervention for stress-related sick leave in general practice: results of an economic evaluation alongside a pragmatic randomised control trial. J Affect Disord. Jan 2010;120(1-3):177-187. [CrossRef] [Medline]
    10. Rebergen DS, Bruinvels DJ, van Tulder MW, van der Beek AJ, van Mechelen W. Cost-effectiveness of guideline-based care for workers with mental health problems. J Occup Environ Med. Mar 2009;51(3):313-322. [CrossRef] [Medline]
    11. Kalia M. Assessing the economic impact of stress--the modern day hidden epidemic. Metabolism. Jun 2002;51(6 Suppl 1):49-53. [CrossRef] [Medline]
    12. Hassard J, Teoh KR, Visockaite G, Dewe P, Cox T. The cost of work-related stress to society: a systematic review. J Occup Health Psychol. Jan 2018;23(1):1-17. [CrossRef] [Medline]
    13. Richardson KM, Rothstein HR. Effects of occupational stress management intervention programs: a meta-analysis. J Occup Health Psychol. Jan 2008;13(1):69-93. [CrossRef] [Medline]
    14. Hilton MF, Whiteford HA, Sheridan JS, Cleary CM, Chant DC, Wang PS, et al. The prevalence of psychological distress in employees and associated occupational risk factors. J Occup Environ Med. Jul 2008;50(7):746-757. [CrossRef] [Medline]
    15. Griffiths F, Lindenmeyer A, Powell J, Lowe P, Thorogood M. Why are health care interventions delivered over the internet? A systematic review of the published literature. J Med Internet Res. 2006;8(2):e10. [FREE Full text] [CrossRef] [Medline]
    16. Heber E, Ebert DD, Lehr D, Cuijpers P, Berking M, Nobis S, et al. The benefit of web-and computer-based interventions for stress: a systematic review and meta-analysis. J Med Internet Res. Feb 17, 2017;19(2):e32. [FREE Full text] [CrossRef] [Medline]
    17. Donker T, Blankers M, Hedman E, Ljótsson B, Petrie K, Christensen H. Economic evaluations of internet interventions for mental health: a systematic review. Psychol Med. Dec 2015;45(16):3357-3376. [CrossRef] [Medline]
    18. Blankers M, Nabitz U, Smit F, Koeter MW, Schippers GM. Economic evaluation of internet-based interventions for harmful alcohol use alongside a pragmatic randomized controlled trial. J Med Internet Res. 2012;14(5):e134. [FREE Full text] [CrossRef] [Medline]
    19. Graham AL, Chang Y, Fang Y, Cobb NK, Tinkelman DS, Niaura RS, et al. Cost-effectiveness of internet and telephone treatment for smoking cessation: an economic evaluation of The iQUITT Study. Tob Control. Nov 2013;22(6):e11. [FREE Full text] [CrossRef] [Medline]
    20. Nordgren LB, Hedman E, Etienne J, Bodin J, Kadowaki A, Eriksson S, et al. Effectiveness and cost-effectiveness of individually tailored internet-delivered cognitive behavior therapy for anxiety disorders in a primary care population: a randomized controlled trial. Behav Res Ther. Aug 2014;59:1-11. [FREE Full text] [CrossRef] [Medline]
    21. Buntrock C, Berking M, Smit F, Lehr D, Nobis S, Riper H, et al. Preventing depression in adults with subthreshold depression: health-economic evaluation alongside a pragmatic randomized controlled trial of a web-based intervention. J Med Internet Res. Jan 4, 2017;19(1):e5. [FREE Full text] [CrossRef] [Medline]
    22. Hedman E, Andersson E, Ljótsson B, Axelsson E, Lekander M. Cost effectiveness of internet-based cognitive behaviour therapy and behavioural stress management for severe health anxiety. BMJ Open. 2016;6(4):e009327. [FREE Full text] [CrossRef] [Medline]
    23. Heber E, Lehr D, Ebert DD, Berking M, Riper H. Web-based and mobile stress management intervention for employees: a randomized controlled trial. J Med Internet Res. 2016;18(1):e21. [FREE Full text] [CrossRef] [Medline]
    24. Husereau D, Drummond M, Petrou S, Carswell C, Moher D, Greenberg D, et al. CHEERS Task Force. Consolidated Health Economic Evaluation Reporting Standards (CHEERS) statement. Br Med J. Mar 25, 2013;346:f1049. [FREE Full text] [CrossRef] [Medline]
    25. Ramsey S, Willke R, Briggs A, Brown R, Buxton M, Chawla A, et al. Good research practices for cost-effectiveness analysis alongside clinical trials: the ISPOR RCT-CEA Task Force report. Value Health. 2005;8(5):521-533. [FREE Full text] [CrossRef] [Medline]
    26. Datinf Randlist - Software to create Randomisation Lists. URL: http://randomisation.eu/ [accessed 2019-04-12] [WebCite Cache]
    27. Lesage F, Berjot S, Deschamps F. Psychometric properties of the French versions of the perceived stress scale. Int J Occup Med Environ Health. Jun 2012;25(2):178-184. [FREE Full text] [CrossRef] [Medline]
    28. Malouff JM, Thorsteinsson EB, Schutte NS. The efficacy of problem solving therapy in reducing mental and physical health problems: a meta-analysis. Clin Psychol Rev. Jan 2007;27(1):46-57. [CrossRef] [Medline]
    29. Berking M, Ebert D, Cuijpers P, Hofmann SG. Emotion regulation skills training enhances the efficacy of inpatient cognitive behavioral therapy for major depressive disorder: a randomized controlled trial. Psychother Psychosom. 2013;82(4):234-245. [CrossRef] [Medline]
    30. Ebert D, Lehr D, Heber E, Riper H, Cuijpers P, Berking M. Internet- and mobile-based stress management for employees with adherence-focused guidance: efficacy and mechanism of change. Scand J Work Environ Health. Dec 1, 2016;42(5):382-394. [FREE Full text] [CrossRef] [Medline]
    31. Ebert DD, Lehr D, Smit F, Zarski A, Riper H, Heber E, et al. Efficacy and cost-effectiveness of minimal guided and unguided internet-based mobile supported stress-management in employees with occupational stress: a three-armed randomised controlled trial. BMC Public Health. 2014;14:807. [FREE Full text] [CrossRef] [Medline]
    32. Heber E, Ebert DD, Lehr D, Nobis S, Berking M, Riper H. Efficacy and cost-effectiveness of a web-based and mobile stress-management intervention for employees: design of a randomized controlled trial. BMC Public Health. 2013;13:655. [FREE Full text] [CrossRef] [Medline]
    33. Ebert DD, Heber E, Berking M, Riper H, Cuijpers P, Funk B, et al. Self-guided internet-based and mobile-based stress management for employees: results of a randomised controlled trial. Occup Environ Med. May 2016;73(5):315-323. [CrossRef] [Medline]
    34. Zarski A, Lehr D, Berking M, Riper H, Cuijpers P, Ebert DD. Adherence to internet-based mobile-supported stress management: a pooled analysis of individual participant data from three randomized controlled trials. J Med Internet Res. 2016;18(6):e146. [FREE Full text] [CrossRef] [Medline]
    35. Jacobson NS, Truax P. Clinical significance: a statistical approach to defining meaningful change in psychotherapy research. J Consult Clin Psychol. Feb 1991;59(1):12-19. [CrossRef] [Medline]
    36. Ware J, Kosinski M, Keller SD. A 12-Item Short-Form Health Survey: construction of scales and preliminary tests of reliability and validity. Med Care. Mar 1996;34(3):220-233. [CrossRef] [Medline]
    37. Brazier J, Roberts J, Tsuchiya A, Busschbach J. A comparison of the EQ-5D and SF-6D across seven patient groups. Health Econ. Sep 2004;13(9):873-884. [CrossRef] [Medline]
    38. Federal Statistical Office of Germany. 2015. URL: https:/​/www.​destatis.de/​EN/​FactsFigures/​NationalEconomyEnvironment/​Prices/​ConsumerPriceIndices/​ConsumerPriceIndices.​html, [accessed 2016-08-30] [WebCite Cache]
    39. Van Hakkaart RL, Van Straten A, Donker M, Tiemens B. Manual Trimbos/iMTA Questionnaire for Costs Associated with Psychiatric Illness (TIC-P). Rotterdam. Institute for Medical Technology Assessment; Jan 2002.
    40. Bouwmans C, de Jong K, Timman R, Zijlstra-Vlasveld M, van der Feltz-Cornelis C, Tan SS, et al. Feasibility, reliability and validity of a questionnaire on healthcare consumption and productivity loss in patients with a psychiatric disorder (TiC-P). BMC Health Serv Res. 2013;13:217. [FREE Full text] [CrossRef] [Medline]
    41. Nobis S, Lehr D, Ebert DD, Berking M, Heber E, Baumeister H, et al. Efficacy and cost-effectiveness of a web-based intervention with mobile phone support to treat depressive symptoms in adults with diabetes mellitus type 1 and type 2: design of a randomised controlled trial. BMC Psychiatry. 2013;13:306. [FREE Full text] [CrossRef] [Medline]
    42. Lin J, Lüking M, Ebert D, Buhrman M, Andersson G, Baumeister H. Effectiveness and cost-effectiveness of a guided and unguided internet-based Acceptance and Commitment Therapy for chronic pain: study protocol for a three-armed randomised controlled trial. Internet Interv. Mar 2015;2(1):7-16. [FREE Full text] [CrossRef]
    43. Buntrock C, Ebert DD, Lehr D, Cuijpers P, Riper H, Smit F, et al. Evaluating the efficacy and cost-effectiveness of web-based indicated prevention of major depression: design of a randomised controlled trial. BMC Psychiatry. 2014;14:25. [FREE Full text] [CrossRef] [Medline]
    44. Thiart H, Ebert DD, Lehr D, Nobis S, Buntrock C, Berking M, et al. Internet-based cognitive behavioral therapy for insomnia: a health economic evaluation. Sleep. Oct 01, 2016;39(10):1769-1778. [FREE Full text] [CrossRef] [Medline]
    45. Boß L, Lehr D, Berking M, Riper H, Schaub MP, Ebert DD. Evaluating the (cost-)effectiveness of guided and unguided Internet-based self-help for problematic alcohol use in employees--a three arm randomized controlled trial. BMC Public Health. Oct 12, 2015;15:1043. [FREE Full text] [CrossRef] [Medline]
    46. Sander L, Paganini S, Lin J, Schlicker S, Ebert DD, Buntrock C, et al. Effectiveness and cost-effectiveness of a guided internet- and mobile-based intervention for the indicated prevention of major depression in patients with chronic back pain-study protocol of the PROD-BP multicenter pragmatic RCT. BMC Psychiatry. Jan 21, 2017;17(1):36. [FREE Full text] [CrossRef] [Medline]
    47. Matthews JN, Altman DG, Campbell MJ, Royston P. Analysis of serial measurements in medical research. Br Med J. Jan 27, 1990;300(6719):230-235. [FREE Full text] [CrossRef] [Medline]
    48. Bock J, Brettschneider C, Seidl H, Bowles D, Holle R, Greiner W, et al. [Calculation of standardised unit costs from a societal perspective for health economic evaluation]. Gesundheitswesen. Jan 2015;77(1):53-61. [CrossRef] [Medline]
    49. Krauth C. [Methods of cost determination in the health economic evaluation]. Gesundh ökon Qual manag. Oct 27, 2010;15(05):251-259. [CrossRef]
    50. Rote Liste. [Red List® 2013 Book Edition - Single Issue: Drug List for Germany (Including EU Approvals and Certain Medical Devices)]. Frankfurt/Main. Rote Liste Service GmbH; 2013.
    51. [Income Tax Code]. Berlin. Bundesministerium der Justiz (Federal Office of Justice); 2008.
    52. Drummond MF, Sculpher MJ, Claxton K, Stoddart GL, Torrance GW. Methods for the Economic Evaluation of Health Care Programmes, Fourth Edition. Oxford. Oxford University press; 2015.
    53. Osterhaus JT, Gutterman DL, Plachetka JR. Healthcare resource and lost labour costs of migraine headache in the US. Pharmacoeconomics. Jul 1992;2(1):67-76. [CrossRef] [Medline]
    54. van Hout BA, Al MJ, Gordon GS, Rutten FF. Costs, effects and C/E-ratios alongside a clinical trial. Health Econ. Oct 1994;3(5):309-319. [CrossRef] [Medline]
    55. Briggs A, Gray A. Handling uncertainty when performing economic evaluation of healthcare interventions. Health Technol Assess. 1999;3(2):1-134. [FREE Full text] [Medline]
    56. Willan AR, Briggs AH, Hoch JS. Regression methods for covariate adjustment and subgroup analysis for non-censored cost-effectiveness data. Health Econ. May 2004;13(5):461-475. [CrossRef] [Medline]
    57. Stata: Release 13. Statistical Software.College Station. TX. StataCorp LP; 2013.
    58. EuroQol Group. EuroQol—A new facility for the measurement of health-related quality of life. Health Policy. Dec 1990;16(3):199-208. [CrossRef] [Medline]
    59. GesundheitsTraining.Online. URL: http://www.geton-training.de/ [accessed 2019-04-12] [WebCite Cache]
    60. Buchberger B, Heymann R, Huppertz H, Friepörtner K, Pomorin N, Wasem J. The effectiveness of interventions in workplace health promotion as to maintain the working capacity of health care personal. GMS Health Technol Assess. 2011;7:Doc06. [FREE Full text] [CrossRef] [Medline]
    61. Furukawa TA, Noma H, Caldwell DM, Honyashiki M, Shinohara K, Imai H, et al. Waiting list may be a nocebo condition in psychotherapy trials: a contribution from network meta-analysis. Acta Psychiatr Scand. Sep 2014;130(3):181-192. [CrossRef] [Medline]
    62. Crisp DA, Griffiths KM. Participating in online mental health interventions: who is most likely to sign up and why? Depress Res Treat. 2014;2014:790457. [FREE Full text] [CrossRef] [Medline]
    63. Jacobsen PB, Meade CD, Stein KD, Chirikos TN, Small BJ, Ruckdeschel JC. Efficacy and costs of two forms of stress management training for cancer patients undergoing chemotherapy. J Clin Oncol. Jun 15, 2002;20(12):2851-2862. [CrossRef] [Medline]
    64. Geraedts AS, van Dongen JM, Kleiboer AM, Wiezer NM, van Mechelen W, Cuijpers P, et al. Economic evaluation of a web-based guided self-help intervention for employees with depressive symptoms: results of a randomized controlled trial. J Occup Environ Med. Jun 2015;57(6):666-675. [CrossRef] [Medline]
    65. Geraedts AS, Kleiboer AM, Wiezer NM, van Mechelen W, Cuijpers P. Short-term effects of a web-based guided self-help intervention for employees with depressive symptoms: randomized controlled trial. J Med Internet Res. May 2014;16(5):e121. [FREE Full text] [CrossRef] [Medline]
    66. Richards D, Richardson T. Computer-based psychological treatments for depression: a systematic review and meta-analysis. Clin Psychol Rev. Jun 2012;32(4):329-342. [CrossRef] [Medline]
    67. Levi L. Bridging the science–policy and policy–implementation gaps. In: Cooper CL, Campbell Quick J, editors. The Handbook of Stress and Health: A Guide to Research and Practice. Chichester. John Wiley & Sons; Apr 19, 2017:5-23.

    ‎
    iCBT: internet-based cognitive behavior therapy
    ICER: incremental cost-effectiveness ratio
    iSMI: internet-based stress management interventions
    ITT: intention-to-treat
    PSS-10: Perceived Stress Scale
    QALYs: quality-adjusted life years
    RCT: randomized controlled trial
    WLC: waitlist control condition
    WTP: willingness-to-pay

    Edited by G Eysenbach; submitted 04.05.18; peer-reviewed by T Donker, Y Lee, P Jimenez; comments to author 30.08.18; revised version received 02.11.18; accepted 30.12.18; published 15.05.19.

    Copyright

    ©Fanny Kählke, Claudia Buntrock, Filip Smit, Matthias Berking, Dirk Lehr, Elena Heber, Burkhardt Funk, Heleen Riper, David Daniel Ebert. Originally published in JMIR Mental Health (http://mental.jmir.org), 15.05.2019.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Mental Health, is properly cited. The complete bibliographic information, a link to the original publication on http://mental.jmir.org/, as well as this copyright and license information must be included.