This paper is in the following e-collection/theme issue:

Anxiety and Stress Disorders (1006) Research Instruments, Questionnaires, and Tools (542) Diagnostic Tools in Mental Health (222)

Published on in Vol 4, No 4 (2017): Oct-Dec

The 12-item Self-Report World Health Organization Disability Assessment Schedule (WHODAS) 2.0 Administered Via the Internet to Individuals With Anxiety and Stress Disorders: A Psychometric Investigation Based on Data From Two Clinical Trials

The 12-item Self-Report World Health Organization Disability Assessment Schedule (WHODAS) 2.0 Administered Via the Internet to Individuals With Anxiety and Stress Disorders: A Psychometric Investigation Based on Data From Two Clinical Trials

The 12-item Self-Report World Health Organization Disability Assessment Schedule (WHODAS) 2.0 Administered Via the Internet to Individuals With Anxiety and Stress Disorders: A Psychometric Investigation Based on Data From Two Clinical Trials

Authors of this article:

Erland Axelsson1, 2 Author Orcid Image ;   Elin Lindsäter1, 2 Author Orcid Image ;   Brjánn Ljótsson1, 3 Author Orcid Image ;   Erik Andersson1 Author Orcid Image ;   Erik Hedman-Lagerlöf1, 2, 4 Author Orcid Image
Article Authors Cited by (36) Tweetations (7) Metrics

    Original Paper

    1Division of Psychology, Department of Clinical Neuroscience, Karolinska Institutet, Stockholm, Sweden

    2Gustavsberg Primary Care Clinic, Gustavsberg, Sweden

    3Centre for Psychiatry Research, Department of Clinical Neuroscience, Karolinska Institutet, Stockholm, Sweden

    4Osher Center for Integrative Medicine, Department of Clinical Neuroscience, Karolinska Institutet, Stockholm, Sweden

    Corresponding Author:

    Erland Axelsson, MSc

    Division of Psychology

    Department of Clinical Neuroscience

    Karolinska Institutet

    Nobels Väg 9

    Stockholm, 171 65

    Sweden

    Phone: 46 70 617 10 39

    Email: erland.axelsson@ki.se


    Background: The World Health Organization Disability Assessment Schedule 2.0 (WHODAS 2.0) is a widespread measure of disability and functional impairment, which is bundled with the Diagnostic and Statistical Manual of Mental Disorders (Fifth Edition) for use in psychiatry. Administering psychometric scales via the Internet is an effective way to reach respondents and allow for convenient handling of data.

    Objective: The aim was to study the psychometric properties of the 12-item self-report WHODAS 2.0 when administered online to individuals with anxiety and stress disorders. The WHODAS 2.0 was hypothesized to exhibit high internal consistency and be unidimensional. We also expected the WHODAS 2.0 to show high 2-week test-retest reliability, convergent validity (correlations approximately .50 to .90 with other self-report measures of functional impairment), that it would differentiate between patients with and without exhaustion disorder, and that it would respond to change in primary symptom domain.

    Methods: We administered the 12-item self-report WHODAS 2.0 online to patients with anxiety and stress disorders (N=160) enrolled in clinical trials of cognitive behavior therapy, and analyzed psychometric properties within a classical test theory framework. Scores were compared with well-established symptom and disability measures, and sensitivity to change was studied from pretreatment to posttreatment assessment.

    Results: The 12-item self-report WHODAS 2.0 showed high internal consistency (Cronbach alpha=.83-.92), high 2-week test-retest reliability (intraclass correlation coefficient=.83), adequate construct validity, and was sensitive to change. We found preliminary evidence for a three-factorial structure, but one strong factor accounted for a clear majority of the variance.

    Conclusions: We conclude that the 12-item self-report WHODAS 2.0 is a psychometrically sound instrument when administered online to individuals with anxiety and stress disorders, but that it is probably fruitful to also report the three subfactors to facilitate comparisons between studies.

    Trial Registration: Clinicaltrials.gov NCT02540317; https://clinicaltrials.gov/ct2/show/NCT02540317 (Archived by WebCite at http://www.webcitation.org/6vQEdYAem); Clinicaltrials.gov NCT02314065; https://clinicaltrials.gov/ct2/show/NCT02314065 (Archived by WebCite at http://www.webcitation.org/6vQEjlUU8)

    JMIR Ment Health 2017;4(4):e58

    doi:10.2196/mental.7497

    Keywords

    disability; Internet; psychometrics; questionnaire; validity; WHODAS 


    The World Health Organization Disability Assessment Schedule 2.0 (WHODAS 2.0) [1,2] is an assessment tool developed by the World Health Organization (WHO) to measure disability and functional impairment in accordance with the International Classification of Functioning, Disability and Health [3]. The WHODAS 2.0 comes bundled with the Diagnostic and Statistical Manual of Mental Disorders (Fifth Edition; DSM-5), and is endorsed as a new and useful measure of functional impairment in psychiatric disorders [4]. The WHODAS 2.0 measures average functioning in everyday situations for the last 30 days, and surveys six domains of functioning: (1) cognition (understanding and communicating), (2) mobility (ability to move and get around), (3) self-care (eg, with regard to hygiene, dressing, and eating) (4) getting along with others, (5) life activities (ability to attend to everyday responsibilities), and (6) participation in society [1]. The most widespread and evaluated form of the WHODAS 2.0 is the 36-item structured interview version, which takes approximately 20 minutes to complete and has excellent psychometric properties [2]. This study, however, concerns the shorter 12-item self-assessment questionnaire version of the WHODAS 2.0. In the WHODAS 2.0 field trials, the reduced 12-item scale, despite only taking approximately 5 minutes to complete, explained 81% of the variance in the 36-item scale [2]. Several large-scale studies have suggested that the 12-item WHODAS 2.0 is a reliable and valid instrument when administered as an interview [5,6] or in a pencil-and-paper format [7,8]. As to dimensionality, both a one-factor structure [6,8] and a second-order model that specifies the six WHODAS 2.0 domains of functioning as subfactors to an overarching disability variable [5,6] have been suggested.

    Administering psychometric questionnaires via the Internet is rapidly becoming more common in both research and routine mental care. Compared with conventional pencil-and-paper administration, there are many advantages of this online approach. Respondents can complete the necessary questionnaires wherever an Internet connection is available, and for the clinician or researcher data are quickly and easily stored, scored, analyzed, and interpreted with less risk for human error. Questionnaires are easily integrated with routine care software for evaluation and record keeping, as well as digital monitoring systems and Web-based psychological treatments. Loss of individual item scores may be prevented entirely, time of measurement may be registered and determined by prespecified time schedules, and respondents may be readily contacted via automatic email or text-message reminders. Although it has often been found that well-established scales do well regardless of administration format, online adaptions of validated scales should preferably undergo separate validation [9,10]. There is thus a need for separate validation of the self-rated 12-item WHODAS 2.0 when administered via the Internet and, to the knowledge of the authors, no study has yet investigated the convergent or discriminant validity, responsiveness, test-retest reliability, or factor structure of the self-rated 12-item WHODAS 2.0 when administered online to individuals with common mental disorders.

    Based on data from two clinical trials of cognitive behavior therapy (CBT) for anxiety and stress disorders, we aimed to present estimates of test-retest reliability and thoroughly investigate item score distributions, convergent and discriminant validity, as well as the factor structure of the 12-item online WHODAS 2.0 when administered to individuals with anxiety and stress disorders. We expected the scale to be unidimensional, possibly with the six domains of functioning as subfactors (see previous), and with high internal consistency (Cronbach alpha>.80) as seen in previous studies. We expected strong baseline Pearson correlations (approximately .50 to .90) between the WHODAS 2.0 and other measure of disability or functional impairment, as well as substantial, yet slightly weaker, baseline Pearson correlations (approximately .30 to .70) with the measures of depression and general anxiety. We expected the WHODAS 2.0 to discriminate well between chronic stress patients with and without International Classification of Diseases, Tenth Revision (ICD-10) exhaustion disorder, a disorder characterized by burnout-like symptoms including fatigue and cognitive weariness, and typically regarded as highly disabling [11]. We also hypothesized that the WHODAS 2.0 would be sensitive to within-group change in primary psychiatric symptom domain.


    Design

    This was a psychometric study of the WHODAS 2.0 administered online to patients with anxiety and stress disorders. Data were collected from clinical trials of CBT for severe health anxiety (n=60) and stress disorders (n=100) conducted at Karolinska Institutet and Gustavsberg primary care clinic, Stockholm, Sweden. Both trials were approved by the Stockholm regional ethics review board (2015/415-31/5, 2014/1530-31/2), registered at ClinicalTrials.gov (NCT02540317, NCT02314065), and participants provided informed consent. Data used for this study were collected between September 2015 and July 2016.

    Recruitment

    Both clinical trials employed patient self-referral via the Internet, and advertised in newspapers as well as on online social media networks. Study applicants completed a series of online screening symptom questionnaires before a diagnostic interview with a licensed psychologist. This interview primarily served to survey eligibility criteria and lead up to a decision regarding inclusion or exclusion (ie, this decision was based on the psychiatric interview), but also served to collect important clinical data (eg, comorbid diagnoses). After the pretreatment assessment, which was conducted online, patients underwent randomization and subsequent treatment. All included patients were at least 18 years of age. The severe health anxiety sample had a principal diagnosis of DSM-5 somatic symptom disorder or illness anxiety disorder, whereas the stress disorders sample had a principal diagnosis of DSM-5 adjustment disorder or ICD-10 exhaustion disorder (for a brief introduction to this disorder, see [11]). We intend to provide an in-depth description of the methods of the clinical trials, including the recruitment process, in the primary publications.

    Procedure

    All questionnaires were completed through a simple Web-based interface with white background, radio buttons, and checkboxes. All 60 patients in the severe health anxiety sample and 50 patients in the stress disorders trial received CBT (12 weeks, disorder-specific) for their principal disorder. The WHODAS 2.0 was administered before and after CBT. In addition, patients with severe health anxiety completed the WHODAS 2.0 at screening, thus allowing for estimates of test-retest reliability. Other measures used to validate the WHODAS 2.0 were also administered before and after treatment.

    Clinical Instruments for Diagnostic Assessment

    Both DSM-5 somatic symptom disorder and illness anxiety disorder were assessed with the Health Preoccupation Diagnostic Interview, which exhibits excellent interrater reliability [12]. Both ICD-10 exhaustion disorder and DSM-5 adjustment disorder were assessed with a clinical interview developed specifically for the stress disorders trial that closely followed the diagnostic criteria of ICD-10 and DSM-5. Comorbid psychiatric disorders were surveyed with the Mini-International Neuropsychiatric Interview, which is a reliable and valid instrument for assessing psychiatric disorders [13].

    Self-Rated Measures of Functional Impairment

    The self-report 12-item WHODAS 2.0 [2] instructs the respondent to determine his or her difficulty in engaging in particular activities (eg, “taking care of [...] household responsibilities” and “maintaining a friendship”), as rated on a scale from “none” (no difficulty) to “extreme or cannot do” and corresponding to six domains of functioning (see Introduction). We employed the WHO simple scoring method [1] that gives a 12-item WHODAS 2.0 score range from 12 to 60, where higher scores indicate higher disability or loss of function. This type of straightforward additive scoring has been shown to correlate strongly (r>.98, ρ=.999) with more complex scoring methods incorporating weights based on item response patterns [5,14]. Just like the pencil-and-paper version, the WHODAS 2.0 was presented over two pages with items 1 to 5 on the first page, and items 6 to 12 on the second page.

    The Sheehan Disability Scale (SDS) is a well-established three-item measure of psychiatric symptom-related functional impairment with a sum score range from 0 to 30, with a higher score indicating a higher degree of functional impairment [15,16]. The Work Ability Index (WAI) measures work ability with a sum score range from 7 to 49, and higher score indicating higher work ability [17-19]. The SDS and WAI were used as indexes of functional impairment.

    Self-Rated Measures of Primary Psychiatric Symptoms

    The Health Anxiety Inventory (HAI) is a 64-item questionnaire that measures health anxiety on a scale from 0 to 192, with a higher score indicating more health anxiety [20-22]. The Perceived Stress Scale (PSS) is a 14-item measure of perceived stress with a range from 0 to 56, with a higher score indicating more stress [23]. The HAI and PSS were used to describe the samples in terms of primary symptom domains (ie, health anxiety and stress).

    Self-Rated Measures of General Psychiatric Symptoms

    The self-reported Montgomery-Åsberg Depression Rating Scale (MADRS-S) is a widely used nine-item questionnaire that measures depressive symptoms on a scale from 0 to 54, with higher scores indicating more symptoms of depression [24-26]. The seven-item Generalized Anxiety Disorder scale (GAD-7) measures general anxiety from 0 to 21, with higher scores indicating more general anxiety [27]. The MADRS-S and GAD-7 were used to assess the common symptom domains of depression and general anxiety and facilitate comparison of the two samples.

    Statistical Analysis

    Analyses were done in SPSS version 23.0.0.2 (IBM Corp, Armonk, NY, USA) and R 3.3.2 [28] with lavaan 0.5-22 [29]. We pooled the severe health anxiety and stress disorders samples (total N=160), dropped three multivariate outliers, and employed structural equation modeling to assess the validity of the simple one-factor model of disability endorsed by the WHO [1], as well as the second-order model fitted in two previous studies [5,6]. The latter had latent variables corresponding both to the six WHODAS 2.0 dimensions of functioning and an overarching latent disability variable. We employed weighted least squares means and variance adjusted estimation, which is adequate for categorical data and nonnormal manifest variables [30]. Based on the recommendations of Byrne [31], we established a priori criteria for acceptable model fit in terms of a comparative fit index (CFI) greater than 0.90, a Tucker-Lewis index (TLI) greater than 0.90, and a root mean square error of approximation (RMSEA) of 0.08 or lower. Post hoc exploratory factor analysis was based on principal axis factoring with promax rotation (ie, factors were assumed to be correlated).

    Internal consistency was investigated in terms of Cronbach alpha, complemented by adjusted item-total correlations (ITCs), which are not as strongly affected by the number of scale items. For instruments of typical length, Cronbach alpha≥.9 is usually regarded as excellent, ≥8 as good, and ≥7 as acceptable. Test-retest reliability was estimated based on a two-way mixed-effects model absolute agreement intraclass correlation coefficient (ICC) and data from a subsample (n=25) from the severe health anxiety trial that had completed the screening and pretreatment assessments within 14 days (mean 6.8, SD 3.2, range 1-13).

    We used an independent samples t test to assess if patients with and without a clinical diagnosis of exhaustion disorder differed with regard to WHODAS 2.0 score, and then performed a receiver operating characteristic analysis to assess the ability (area under the curve [AUC]) of the WHODAS 2.0 to identify cases of exhaustion disorder. Pearson correlations were used to investigate baseline associations between the WHODAS 2.0 and other self-rated measures.

    To evaluate sensitivity to change, we compared pretreatment and posttreatment mean scores using paired samples t test and calculated effect sizes as the difference in means at t1 and t2, divided by the t1 standard deviation. Based on the Jacobson and Truax reliable change index [32], we classified patients as either reliably improved (reduction in the HAI>29.9 or reduction in the PSS>7.1) or not reliably improved. As recommended in the assessment of responsiveness, we also further differentiated between reliably improved patients based on whether they improved more than a minimal important difference in their primary symptom domain [33]. In other words, for both samples we estimated change in functional impairment in three strata: (1) patients who did not made a reliable improvement on the HAI or PSS, (2) patients who reliably improved but only to a slight to moderate degree, and (3) patients who reliably improved to a large degree. However, it was necessary to set the threshold for changing more than a minimal important difference relatively high (1.5 SD) due to the precision of the instruments requiring a relatively large score change (1.21 SD in the case of HAI, 1 SD in the case of PSS) for patients to have experienced true change with 95% certainty, as assessed with the reliable change index [32]. Item-level change was assessed with the Wilcoxon signed-rank test.

    The WHODAS 2.0 response rate was 100% for the screening and pretreatment assessments. Because there was a small proportion of missing posttreatment data (for the WHODAS 2.0: 3%, 3/100 in the stress sample; 5%, 3/60 in the severe health anxiety sample), and we did not aim to investigate treatment effects but rather the responsiveness of the scale, data were used on a complete case (not intention-to-treat) basis.


    Sample Characteristics

    Sample characteristics for the severe health anxiety sample, the stress disorders sample, and the pooled sample are presented in Table 1.

    Factor Structure

    Neither the one-factor model endorsed by the WHO (χ254=1699.1, P<.001; CFI=0.29, TLI=0.14, RMSEA=0.44, standardized root mean square residual [SRMR]=0.52) nor the second-order model presented by previous studies achieved acceptable fit (χ248=132.7, P<.001; CFI=0.96, TLI=0.95, RMSEA=0.11, SRMR=0.09), and we did not see implementing any of the modification indexes as theoretically justifiable.

    Table 1. Sample characteristics.
    MeasureSevere health anxiety (n=60)Stress disorders (n=100)Total (N=160)
    Demographics



    		Age (years), mean (SDa), range36.4 (11.9), 18-7846.2 (8.8), 26-6542.5 (11.1), 18-78

    		Gender (female), n (%)40 (67)85 (85)125 (78)
    Psychiatric symptomsb



    		HAI, mean (SD), range105.6 (24.7), 51-164——

    		PSS, mean (SD), range—36.8 (7.1), 17-52—

    		MADRS-S, mean (SD), range13.9 (7.4), 1-3419.7 (7.5), 3-4017.5 (7.9), 1-40

    		GAD-7, mean (SD), range12.0 (5.3), 2-21c10.8 (4.8), 2-2111.2 (5.0), 2-21c

    		MDD, n (%)11 (18)13 (13)24 (15)

    		≥1 anxiety disorder/OCDd, n (%)34 (57)19 (19)53 (33)
    Functional impairmente



    		WHODAS 2.0, mean (SD), range21.1 (6.3), 12-3624.7 (8.5), 12-5123.4 (7.9), 12-51

    		SDS, mean (SD), range10.4 (6.7), 0-26——

    		WAI, mean (SD), range—33.0 (8.4), 13.0-47.0—

    		On sick leave, n (%)3 (5)14 (14)17 (11)

    aSD: standard deviation.

    bHAI: Health Anxiety Inventory; PSS: Perceived Stress Scale; MADRS-S: Montgomery-Åsberg Depression Rating Scale self-report version; GAD-7: Generalized Anxiety Disorder 7-item scale; MDD: major depressive disorder; OCD: obsessive compulsive disorder;.

    cGAD-7 data only available from a subsample of the severe health anxiety sample (n=43).

    dAt least one anxiety or obsessive compulsive disorder that is not severe health anxiety.

    eWHODAS 2.0: World Health Organization Disability Assessment Schedule 2.0; SDS: Sheehan Disability Scale; WAI: Work Ability Index.

    Table 2. Factor loadings of the WHODAS 2.0a.

    Item    		Factor 1: psychosocialbFactor 2: self-carebFactor 3: mobilitybA priori dimensionc
    1. Standing long periods–.008–.097.877Mobility
    2. Household responsibilities.540–.027.264Household
    3. Learning new tasks.740–.002.035Cognitive
    4. Joining community activities.848–.046.042Society
    5. Emotionally affected.652–.087.074Society
    6. Concentrating.683–.010–.040Cognitive
    7. Walking long distance–.003.329.597Mobility
    8. Washing whole body–.068.893.132Self-care
    9. Getting dressed.078.909–.126Self-care
    10. Dealing with strangers.704.167–.164Social
    11. Maintaining friendships.618.128–.101Social
    12. Work/school activities.814–.059.037Household

    aWHODAS 2.0: World Health Organization Disability Assessment Schedule 2.0.

    bFactor loadings (regression coefficients) based on principal axis factoring with promax rotation.

    cCognitive: understanding and communicating; household: life activities; mobility: getting around; social: getting along with others; society: participation in society.

    Post hoc exploratory factor analysis (Kaiser-Meyer-Olkin test=0.85, Barlett test<.001) was suggestive of a three-factor solution, with one very strong factor (eigenvalue=5.4, 45.0% of variance explained) and two weak factors (eigenvalues=1.1-1.7, 9.3%-14.2% of variance explained).

    After rotation (Table 2) it was clear that one of the two weak factors was primarily associated with items 1 and 7 (“getting around”), and the other weak factor was primarily associated with items 8 and 9 (“self-care”), whereas the strong factor was associated with the remaining items (“understanding and communicating,” “getting along with others,” “life activities,” and “participation in society”). Interfactor correlations were substantial (r=.40-.46). Dropping items 1, 7, 8, and 9 had little impact on interindividual sum score variance, and the resulting mean was heavily correlated with the conventional one (r=.97). Thus, the factor analysis indicated that a majority of items had high loadings on a general disability factor, but that “mobility” and “self-care” emerged as distinct, but weak, factors.

    Item Score Distributions

    Parameters related to item score distributions are presented in Table 3. Overall, patients scored low on items that concerned functional impairment in self-care, and higher on items that concerned difficulties participating in society and everyday life activities, as well as items that concerned cognitive impairment. Unlike the summary score, several item distributions were skewed and showed high kurtosis. There was also evidence of a floor effect with regard to several items.

    Internal Consistency

    Adjusted baseline ITCs are presented in Table 3, where the mean adjusted ITC was 0.60 (SD 0.11). Cronbach alpha values were good to excellent for both the total sum score and the proposed psychosocial subscale (Table 4).

    Test-Retest Reliability

    The test-retest reliability of the WHODAS 2.0, based on the severe health anxiety sample data (n=25), was estimated at ICC=0.83 (95% CI 0.62-0.92) and the correlation between the measurements was r=.71. The test-retest reliability of the proposed psychosocial subscale was similar (ICC=0.85, 95% CI 0.67-0.94; r=.75).

    Construct Validity: Associations With Other Measures

    In the stress disorders sample, patients with a principal diagnosis of exhaustion disorder had a significantly higher WHODAS 2.0 mean score than those who had a principal diagnosis of adjustment disorder (mean difference 8.84, 95% CI 5.96-11.72; t98=6.10, P<.001), and the WHODAS 2.0 discriminated well between stress patients with and without exhaustion disorder (AUC=0.81, 95% CI 0.73-0.89; P<.001). The proposed psychosocial subscale fared about as well (AUC=0.79, 95% CI 0.70-0.87; P<.001). Baseline correlations between the WHODAS 2.0 as well as the subscale and other self-assessment questionnaires are shown in Table 5.

    Sensitivity to Change

    Effect sizes and tests pertaining to responsiveness are presented in Table 6. Wilcoxon signed-rank tests showed that all but one WHODAS 2.0 item (item 9, related to self-care) changed in the severe health anxiety sample, whereas 9 of 12 items changed (but not items 1, 8, or 9—one mobility and both self-care items) in the stress disorders sample.

    Table 3. Web-based self-report 12-item WHODAS 2.0 item scoresa.

    Item and subscale    		Mean (SDb)
    Median (range)
    Floor
    Ceiling
    Skew
    Kurtosis    		ITCc
    Item







    		1. Standing long periods1.54 (0.89)1 (1-4)66%0%1.581.470.44

    		2. Household responsibilities2.07 (1.02)2 (1-4)38%0%0.47–1.000.64

    		3. Learning new tasks1.87 (1.02)2 (1-5)49%1%0.85–0.370.70

    		4. Joining community activities2.23 (1.19)2 (1-5)38%4%0.55–0.770.78

    		5. Emotionally affected3.21 (1.14)4 (1-5)13%6%–0.69–0.600.59

    		6. Concentrating2.37 (1.05)2 (1-5)26%1%0.17–0.900.61

    		7. Walking long distance1.51 (0.82)1 (1-5)66%1%1.632.290.52

    		8. Washing whole body1.20 (0.55)1 (1-4)86%0%3.1210.220.48

    		9. Getting dressed1.16 (0.47)1 (1-4)88%0%3.4412.870.48

    		10. Dealing with strangers1.83 (1.01)2 (1-5)49%1%1.100.430.66

    		11. Maintaining friendships1.85 (1.07)1 (1-5)53%1%0.97–0.170.59

    		12. Work/school activities2.54 (1.20)3 (1-5)25%6%0.27–0.880.75

    		Total score23.37 (7.91)23 (12-51)8%0%0.680.52—
    Subscale







    		Psychosocial12.60 (4.70)12.76 (5.60-24.84)8%0%0.31–0.49—

    		Self-care2.12 (0.86)1.80 (1.80-7.21)84%0%3.2011.23—

    		Mobility2.25 (1.15)1.47 (1.47-6.49)56%0%1.561.76—

    aWHODAS 2.0: World Health Organization Disability Assessment Schedule 2.0. Based on baseline data from two clinical trials of severe health anxiety and stress disorders (total N=160, items scored from 1 to 5).

    bSD: standard deviation.

    cITC: adjusted item-total correlation.

    Table 4. Web-based self-report 12-item WHODAS 2.0a internal consistency.
    SampleScreeningPretreatmentPosttreatment

    		Cronbach alphanCronbach alphanCronbach alphan
    Total score






    		Severe health anxiety.8360.8660.8756

    		Stress disorder——.90100.9297
    Psychosocial subscaleb






    		Severe health anxiety.8260.8860.8756

    		Stress disorder——.89100.9197

    aWHODAS 2.0: World Health Organization Disability Assessment Schedule 2.0.

    bPsychosocial subscale with regression weights applied.

    Table 5. WHODAS 2.0 associations (bivariate Pearson correlations) with other self-rated questionnairesa.

    		WHODAS 2.0b
    Psychosocial subscale    		MADRS-SbGAD-7bSDSb
    WHODAS 2.0—.96c.60c.58c,d.66c
    Psychosocial subscale.97e—.65c.54c,d.67c
    MADRS-S.65e.64e—.54c,d.59c
    GAD-7.45e.45e.71e—.40c
    WAI–.71e–.71e–.55e–.28e—

    aAll bivariate Pearson correlations significant at α=.05.

    bWHODAS 2.0: World Health Organization Disability Assessment Schedule 2.0; MADRS-S: Montgomery-Åsberg Depression Rating Scale-self-report version; GAD-7: Generalized Anxiety Disorder 7-item scale; SDS: Sheehan Disability Scale; WAI: Work Ability Index.

    cSevere health anxiety sample (n=60).

    dGAD-7 data only available from a subsample of the severe health anxiety sample (n=43).

    eStress disorder sample (n=100).

    Table 6. Responsiveness stratified by change in primary symptom domaina.
    Change in primary symptom domainb
    n (%)    		ESc
    Mean change (95% CI)    		Pd
    Severe health anxiety sample




    		HAI reliably improved >1.5 SDe32 (5)




    		WHODAS 2.0
    		1.217.19 (5.17, 9.22)<.001


    		SDS
    		1.007.28 (4.59, 9.98)<.001

    		HAI reliably improved ≤1.5 SD8 (14)




    		WHODAS 2.0
    		0.522.75 (–0.85, 6.35).11


    		SDS
    		0.531.88 (–1.07, 4.82).18

    		HAI not reliably improved18 (31)




    		WHODAS 2.0
    		0.292.00 (–0.13, 4.13).06


    		SDS
    		0.221.39 (–0.59, 3.37).16
    Stress disorder sample




    		PSS reliably improved >1.5 SD33 (34)




    		WHODAS 2.0
    		0.806.42 (4.27, 8.58)<.001


    		WAI
    		–0.49–4.14 (–5.75, –2.52)<.001

    		PSS reliably improved ≤1.5 SD12 (12)




    		WHODAS 2.0
    		0.392.83 (0.09, 5.58).04


    		WAI
    		–0.25–2.29 (–5.10, 0.51).10

    		PSS not reliably improved52 (54)




    		WHODAS 2.0
    		0.191.75 (-0.10, 3.60).06


    		WAI
    		–0.13–0.96 (–2.66, 0.73).26

    aAll estimates based on data from patients that completed the posttreatment assessment. Status as improved or not improved based on the Jacobson and Truax reliable change index [32].

    bHAI: Health Anxiety Inventory; WHODAS 2.0: World Health Organization Disability Assessment Schedule 2.0; SDS: Sheehan Disability Scale; PSS: Perceived Stress Scale; WAI: Work Ability Index.

    cStandardized effect sizes (ES) are calculated with the stratum-specific pretreatment standard deviation as nominator.

    dP values based on paired samples t test.

    eSD: standard deviation.


    Summary of Principal Results

    To our knowledge, this was the first in-depth analysis of the reliability and validity of the self-rated 12-item WHODAS 2.0 when administered online to patients with anxiety and stress disorders. In line with our hypotheses, the WHODAS 2.0 exhibited high internal consistency, acceptable test-retest reliability, and was demonstrated to identify cases of exhaustion disorder. As expected, the WHODAS 2.0 also showed substantial associations with other measures of functioning, as well as slightly weaker but yet substantial associations with primary symptom measures. The instrument was sensitive to change in the primary psychiatric symptom domain, as illustrated by a convincing gradient in change effect size over the nonimproved versus slightly improved versus much improved strata (Table 6). It might be noted that there was no significant change in WHODAS 2.0 for the minimally changed severe health anxiety group, but this is not surprising given the very small size of this subsample (n=8). Taken together, our findings suggest that the self-rated 12-item online WHODAS 2.0—when administered to individuals with anxiety and stress disorders—is a valid measure of disability and functional impairment. The results are important because online administration facilitates handling of data in a wide range of research and routine care settings. Key strengths of this study are that two psychiatric samples with different primary conditions could be examined, that other scales could be used to validate the WHODAS 2.0 scores in both samples, and that the ability of the WHODAS 2.0 to identify patients with a clinical diagnosis of exhaustion disorder could be investigated in the stress disorders sample.

    Factor Structure

    Regarding factor structure, we could neither confirm the expected one-factor WHO solution nor the second-order model put forward by previous investigators [5,6]. A post hoc model derived from exploratory analysis saw most items loading heavily (>.5) on one strong factor, and four items (8, 9, 1, and 7) loading heavily on two other factors. The results are somewhat similar to those seen with the WHODAS 2.0 administered to individuals with musculoskeletal pain [34], and suggests that items 8 and 9 (“self-care”), as well as items 1 and 7 (“getting around”), largely tap into sources of variance other than that responsible for most of the sum score (see Table 3). Our subjective impression is that these four items seem more related to physical incapacity than the rest of the scale. Because (1) these items showed apparent evidence of a floor effect (their median sum being zero), (2) the sum score primarily relied on another very strong factor, and (3) the interfactor correlations were moderately strong, the impact of multidimensionality is likely to be very small in the study of change in otherwise healthy samples with anxiety or stress disorders. On the other hand, the interfactorial correlations were small enough to complicate direct comparisons of scores between studies. Especially when anxiety or stress disorder samples differ in terms of somatic comorbidity, it is probably informative to compare the “getting around” and “self-care” subdomains separately. That is, based on the results of this study our tentative recommendation would be for studies on anxiety and stress disorders to report both the self-rated 12-item WHODAS 2.0 sum score, as well as the three subfactors, here referred to as “psychosocial,” “self-care,” and “getting around” (Table 2). However, we wish to emphasize that this factor solution was solely based on an explorative analysis, that the second-order model with six subfactors [5,6] was not far off the mark, and that additional confirmatory factor analyses based on data from similar samples are warranted to arrive at more firm conclusions.

    Score Distribution

    On an item level, the most striking aspect of the score distribution was that both items 8 and 9 (ie, the “self-care” items) had a mode score of 1 and appeared to pick up on very little variance in the two samples. This may suggest that individuals with a principal diagnosis of severe health anxiety or a stress disorder are typically not much impaired in terms of ability to carry out everyday self-care tasks such as washing oneself or getting dressed.

    Internal Consistency and Test-Retest Reliability

    Although the internal consistency and test-retest reliability estimates were in line with our expectations, and also fulfilled commonly accepted quality criteria for health status questionnaires [35], it is conceivable that the estimates of test-retest reliability were deflated because a psychiatric interview and the decision to include patients in a clinical trial took place between the screening and pretreatment measurements. However, the means from these two measurement points were not significantly different (t59=1.55, P=.13), which for example speaks against patients having altered their responses so as to be included in one of the trials.

    Convergent Validity and Sensitivity to Change

    In this study, correlations with other measures of psychiatric symptoms and functional impairment corroborate the construct validity of the self-rated 12-item WHODAS 2.0. As this study was based on data from two clinical trials, we do not find it unfit or surprising that the WHODAS 2.0 was highly correlated with measures of anxiety and depression which were likely to be the primary reasons for functional impairment.

    Due to the limited sample size and variability in the size of substrata (nonchanged vs minimally changed vs much changed) we wish to emphasize that the significance tests pertaining to change (Table 6) are of limited value, and emphasize that all P values ought to be interpreted alongside their corresponding effect sizes. Over and above the effect size gradient over the nonchanged versus minimally changed versus much changed strata, a general trend was also that the 12-item WHODAS 2.0 changed slightly more from pretreatment to posttreatment than the WAI. This may suggest that the WHODAS 2.0 covers dimensions of functional impairment that, compared to work ability, are more likely to change in a clinical trial of CBT for chronic stress disorders.

    Limitations

    The primary limitation of this study is that over and above clinical diagnoses we did not have access to “hard” measures of functional impairment, collected by other means than self-assessment (eg, register data on disability status) that could be used to validate the WHODAS 2.0. One consequence of this is that all indexes of change were to some degree susceptible to social desirability bias or the possibility that patients reported change so as to please their therapist rather than as a consequence of real change in symptoms or disability. However, it has been demonstrated that Web-based survey administration is relatively robust to desirability bias [36], and all scales used to validate the WHODAS 2.0 (ie, the HAI, PSS, MADRS-S, GAD-7, SDS, and WAI) have, at least in conventional form, been shown to be associated with non-self-reported validators such as clinical diagnoses, sick leave, and health care consumption (eg, [16,37-41]).

    We also had no control over what equipment the patients used to access the Web-based WHODAS 2.0, meaning we could not determine the significance of filling in the questionnaire via a mobile phone app rather than a conventional browser on a desktop computer, for example. The results showing good psychometric properties of the WHODAS 2.0 also suggest that there was no substantial measurement error related to the type of device used. Another threat to the generalizability of our findings is that the two samples were relatively homogenous due to the eligibility criteria for the two clinical trials, which were not primarily designed to study the psychometric properties of the WHODAS 2.0. Therefore, it is preferable to validate the findings of this study, particularly with regard to factor structure, in anxiety and stress disorder samples recruited through other means.

    Comparison With Prior Work

    The item mean score profile seen in this study—with substantial functional impairment in the “understanding and communicating,” “life activities,” and “getting along with others” domains but very low functional impairment in the “self-care” domain—is highly similar to the item mean score profile seen in studies of interview and pencil-and-paper versions of the WHODAS 2.0 administered to individuals with common mental disorders [8,14]. This suggests that the response pattern does not change much due to the online format.

    Before this study, the 12-item WHODAS 2.0 had also been administered online to individuals with common anxiety, stress, and mood disorders in several clinical trials (Table 7, see Multimedia Appendices 1 and 2 for details).

    Table 7. Previous studies that have administered the 12-item self-report WHODAS 2.0 online to respondents with common mental disordersa.

    Author (year)
    N    		Diagnosisb
    Mean
    Cronbach alpha
    WHODAS
    2.0c change
    over time?
    12-60 scaled0-48 scaled0-100 scaled
    Allen et al (2016) [42]63PD26.014.029.2.89Yes
    Andrews et al (2011) [43]37SAD26.614.630.4—Yes
    Mason & Andrews (2014) [44]173eMixed CMD25.013.027.1—Yes
    Mewton et al (2012) [45]588GAD25.713.728.5.90Yes
    Newby et al (2016a) [46]2109Mixed CMD26.314.329.8>.88Yes
    Newby et al (2016b) [47]16SHA20.38.317.3.83Yes
    Perini et al (2008) [48]13MDD—f—f—f—Yes
    Spence et al (2011) [49]244PTSD32.120.141.9.92—
    Titov et al (2008a) [50]105SAD26.314.329.8—Yes
    Titov et al (2008b) [51]88SAD25.013.027.1—Yes
    Williams et al (2013) [52]69MDD41.229.260.8—Yes
    Williams et al (2014) [53]560SAD39.327.356.9—Yes
    Williams et al (2015) [54]75MDD44.832.868.3.83Yes
    Wootton et al (2011) [55]118OCD30.418.438.3.91—

    aInformation from articles complemented by personal communication via email.

    bPD: panic disorder; SAD: social anxiety disorder; CMD: common mental disorder; GAD: generalized anxiety disorder; SHA: severe health anxiety; MDD: major depressive disorder; PTSD: posttraumatic stress disorder; OCD: obsessive compulsive disorder; . See Multimedia Appendices 1 and 2 for details.

    cWHODAS 2.0: World Health Organization Disability Assessment Schedule 2.0.

    dThe 12-60 scale has items scored 1-5, the 0-48 scale has items scored 0-4, and the 0-100 scale is the 0-48 scale divided by 48 and then multiplied by 100.

    eAnother sample (n=135) in this study completed the WHODAS 2.0 on a computer, but not via the Internet.

    fUnknown; author could not be reached.

    Although the primary aim of these trials was not to study the psychometric properties of the WHODAS 2.0, these studies presented estimates of both internal consistency (Cronbach alpha≥.83) and baseline mean scores that are very much in line with data in this trial, and thus lend further support to the validity and generalizability of our findings.

    Conclusions

    This is, to date, the most extensive investigation into the psychometric properties of the self-rated 12-item version of the WHODAS 2.0 when administered via the Internet to individuals with anxiety and stress disorders. When administered online to individuals with anxiety and stress disorders, the WHODAS 2.0 exhibits high internal consistency, high convergent validity, adequate test-retest reliability, and is sensitive to change. We conclude that the psychometric properties of the self-rated 12-item version of the WHODAS 2.0 are acceptable when the instrument is administered via the Internet to individuals with anxiety and stress disorders, but suggest that the three subfactors found in this study be reported alongside the sum score to facilitate comparisons between studies.

    Acknowledgments

    This work was supported by Karolinska Institutet and Stockholm County Council, which are public institutions that did not influence the study design, execution, or publication process.

    Authors' Contributions

    All authors made significant contributions to the planning, recruitment, execution, analysis, and publication of this study.

    Conflicts of Interest

    None declared.

    ‎

    Multimedia Appendix 1

    Search strategy and data collection for brief review.

    PDF File (Adobe PDF File), 110KB
    ‎

    Multimedia Appendix 2

    Flowchart illustrating study selection process for brief review.

    PNG File, 43KB
    1. Üstün TB, Kostanjsek N, Chatterji S, Rehm J, editors. Measuring Health and Disability: Manual for WHO Disability Assessment Schedule (WHODAS 2.0). Malta. World Health Organization; 2010.
    2. Üstün TB, Chatterji S, Kostanjsek N, Rehm J, Kennedy C, Epping-Jordan J, World Health Organization, et al. National Institutes of Health. Developing the World Health Organization Disability Assessment Schedule 2.0. Bull World Health Organ. Nov 1, 2010;88(11):815-823. [FREE Full text] [CrossRef] [Medline]
    3. World Health Organization. International Classification of Functioning, Disability and Health: ICF. Geneva. World Health Organization; 2001.
    4. American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders. Arlington, VA. American Psychiatric Association; 2013.
    5. Andrews G, Kemp A, Sunderland M, Von KM, Ustun TB. Normative data for the 12 item WHO Disability Assessment Schedule 2.0. PLoS One. Dec 2009;4(12):e8343. [FREE Full text] [CrossRef] [Medline]
    6. Weeks M, Garber BG, Zamorski MA. Disability and mental disorders in the Canadian Armed Forces. Can J Psychiatry. Apr 2016;61(1 Suppl):56S-63S. [FREE Full text] [CrossRef] [Medline]
    7. McEvoy PM, Mahoney AE. Achieving certainty about the structure of intolerance of uncertainty in a treatment-seeking sample with anxiety and depression. J Anxiety Disord. Jan 2011;25(1):112-122. [CrossRef] [Medline]
    8. Luciano JV, Ayuso-Mateos JL, Fernández A, Serrano-Blanco A, Roca M, Haro JM. Psychometric properties of the twelve item World Health Organization Disability Assessment Schedule II (WHO-DAS II) in Spanish primary care patients with a first major depressive episode. J Affect Disord. Feb 2010;121(1-2):52-58. [CrossRef] [Medline]
    9. Buchanan T. Internet-based questionnaire assessment: appropriate use in clinical contexts. Cogn Behav Ther. 2003;32(3):100-109. [CrossRef] [Medline]
    10. Gosling SD, Vazire S, Srivastava S, John OP. Should we trust web-based studies? A comparative analysis of six preconceptions about internet questionnaires. Am Psychol. Mar 2004;59(2):93-104. [CrossRef] [Medline]
    11. Grossi G, Perski A, Osika W, Savic I. Stress-related exhaustion disorder - clinical manifestation of burnout? A review of assessment methods, sleep impairments, cognitive disturbances, and neuro-biological and physiological changes in clinical burnout. Scand J Psychol. Dec 2015;56(6):626-636. [CrossRef] [Medline]
    12. Axelsson E, Andersson E, Ljótsson B, Wallhed Finn D, Hedman E. The health preoccupation diagnostic interview: inter-rater reliability of a structured interview for diagnostic assessment of DSM-5 somatic symptom disorder and illness anxiety disorder. Cogn Behav Ther. Jun 2016;45(4):259-269. [CrossRef] [Medline]
    13. Sheehan DV, Lecrubier Y, Sheehan KH, Amorim P, Janavs J, Weiller E, et al. The Mini-International Neuropsychiatric Interview (M.I.N.I.): the development and validation of a structured diagnostic psychiatric interview for DSM-IV and ICD-10. J Clin Psychiatry. 1998;59 Suppl 20:22-33;quiz 34. [Medline]
    14. Sjonnesen K, Bulloch AG, Williams J, Lavorato D. Characterization of disability in Canadians with mental disorders using an abbreviated version of a DSM-5 emerging measure: the 12-Item WHO Disability Assessment Schedule (WHODAS) 2.0. Can J Psychiatry. Apr 2016;61(4):227-235. [FREE Full text] [CrossRef] [Medline]
    15. Leon AC, Shear MK, Portera L, Klerman GL. Assessing impairment in patients with panic disorder: the Sheehan Disability Scale. Soc Psychiatry Psychiatr Epidemiol. Mar 1992;27(2):78-82. [CrossRef] [Medline]
    16. Leon AC, Olfson M, Portera L, Farber L, Sheehan DV. Assessing psychiatric impairment in primary care with the Sheehan Disability Scale. Int J Psychiatry Med. 1997;27(2):93-105. [CrossRef] [Medline]
    17. De Zwart BC, Frings-Dresen MH, van Duivenbooden JC. Test-retest reliability of the Work Ability Index questionnaire. Occup Med (Lond). Jun 2002;52(4):177-181. [CrossRef] [Medline]
    18. Radkiewicz P, Widerszal-Bazyl M, NEXT-Study Group. Psychometric properties of Work Ability Index in the light of comparative survey study. Elsevier; 2005. Presented at: 2nd International Symposium on Work Ability; Oct 18-20, 2004; Verona, Italy.
    19. Ilmarinen J. The Work Ability Index (WAI). Occup Med. Mar 01, 2007;57(2):160. [CrossRef]
    20. Salkovskis PM, Rimes KA, Warwick HMC, Clark DM. The Health Anxiety Inventory: development and validation of scales for the measurement of health anxiety and hypochondriasis. Psychol Med. Jul 2002;32(5):843-853. [CrossRef] [Medline]
    21. Hedman E, Ljótsson B, Andersson E, Andersson G, Lindefors N, Rück C, et al. Psychometric properties of Internet-administered measures of health anxiety: an investigation of the Health Anxiety Inventory, the Illness Attitude Scales, and the Whiteley Index. J Anxiety Disord. Apr 2015;31:32-37. [CrossRef] [Medline]
    22. Hedman E, Lekander M, Ljótsson B, Lindefors N, Rück C, Andersson G, et al. Optimal cut-off points on the health anxiety inventory, illness attitude scales and whiteley index to identify severe health anxiety. PLoS One. 2015;10(4):e0123412. [FREE Full text] [CrossRef] [Medline]
    23. Cohen S, Kamarck T, Mermelstein R. A global measure of perceived stress. J Health Soc Behav. Dec 1983;24(4):385-396. [CrossRef] [Medline]
    24. Svanborg P, Åsberg M. A new self-rating scale for depression and anxiety states based on the Comprehensive Psychopathological Rating Scale. Acta Psychiatr Scand. Jan 1994;89(1):21-28. [CrossRef] [Medline]
    25. Svanborg P, Åsberg M. A comparison between the Beck Depression Inventory (BDI) and the self-rating version of the Montgomery Asberg Depression Rating Scale (MADRS). J Affect Disord. May 2001;64(2-3):203-216. [CrossRef] [Medline]
    26. Carlbring P, Brunt S, Bohman S, Austin D, Richards J, Öst LG, et al. Internet vs. paper and pencil administration of questionnaires commonly used in panic/agoraphobia research. Comput Hum Behav. May 2007;23(3):1421-1434. [CrossRef]
    27. Spitzer RL, Kroenke K, Williams JB, Löwe B. A brief measure for assessing generalized anxiety disorder: the GAD-7. Arch Intern Med. May 22, 2006;166(10):1092-1097. [CrossRef] [Medline]
    28. R Core Team. R: A Language and Environment for Statistical Computing. Vienna, Austria. R Foundation for Statistical Computing; 2016. URL: https://www.r-project.org/ [accessed 2017-11-23] [WebCite Cache]
    29. Rosseel Y. lavaan: an R package for structural equation modeling. J Stat Softw. 2012;48(2):1-36. [CrossRef]
    30. Li C. Confirmatory factor analysis with ordinal data: comparing robust maximum likelihood and diagonally weighted least squares. Behav Res Methods. Sep 2016;48(3):936-949. [CrossRef] [Medline]
    31. Byrne BM. Structural Equation Modeling with AMOS: Basic Concepts, Applications and Programming. Mahwah, NJ. Lawrence Erlbaum Associates; 2001.
    32. Jacobson NS, Truax P. Clinical significance: a statistical approach to defining meaningful change in psychotherapy research. J Consult Clin Psychol. Feb 1991;59(1):12-19. [CrossRef] [Medline]
    33. Revicki D, Hays RD, Cella D, Sloan J. Recommended methods for determining responsiveness and minimally important differences for patient-reported outcomes. J Clin Epidemiol. Feb 2008;61(2):102-109. [CrossRef] [Medline]
    34. Saltychev M, Mattie R, McCormick Z, Laimi K. Confirmatory factor analysis of 12-Item World Health Organization Disability Assessment Schedule in patients with musculoskeletal pain conditions. Clin Rehabil. May 2017;31(5):702-709. [CrossRef] [Medline]
    35. Terwee CB, Bot SD, de Boer MR, van der Windt DA, Knol DL, Dekker J, et al. Quality criteria were proposed for measurement properties of health status questionnaires. J Clin Epidemiol. Jan 2007;60(1):34-42. [CrossRef] [Medline]
    36. Kreuter F, Presser S, Tourangeau R. Social desirability bias in CATI, IVR, and web surveys: the effects of mode and question sensitivity. Public Opinion Quarterly. Jan 26, 2009;72(5):847-865. [CrossRef]
    37. Eklund M, Bäckström M, Tuvesson H. Psychometric properties and factor structure of the Swedish version of the Perceived Stress Scale. Nord J Psychiatry. Oct 2014;68(7):494-499. [CrossRef] [Medline]
    38. Seivewright H, Salkovskis P, Green J, Mullan N, Behr G, Carlin E, et al. Prevalence and service implications of health anxiety in genitourinary medicine clinics. Int J STD AIDS. Aug 2004;15(8):519-522. [CrossRef] [Medline]
    39. Ericsson M, Poston II WS, Linder J, Taylor JE, Haddock CK, Foreyt JP. Depression predicts disability in long-term chronic pain patients. Disabil Rehabil. Apr 15, 2002;24(6):334-340. [CrossRef] [Medline]
    40. Kroenke K, Spitzer RL, Williams JB, Löwe B. The Patient Health Questionnaire Somatic, Anxiety, and Depressive Symptom Scales: a systematic review. Gen Hosp Psychiatry. 2010;32(4):345-359. [CrossRef] [Medline]
    41. van Den Berg TI, Elders LA, de Zwart BC, Burdorf A. The effects of work-related and individual factors on the Work Ability Index: a systematic review. Occup Environ Med. Apr 2009;66(4):211-220. [CrossRef] [Medline]
    42. Allen AR, Newby JM, Mackenzie A, Smith J, Boulton M, Loughnan SA, et al. Internet cognitive-behavioural treatment for panic disorder: randomised controlled trial and evidence of effectiveness in primary care. BJPsych Open. Mar 2016;2(2):154-162. [FREE Full text] [CrossRef] [Medline]
    43. Andrews G, Davies M, Titov N. Effectiveness randomized controlled trial of face to face versus Internet cognitive behaviour therapy for social phobia. Aust N Z J Psychiatry. Apr 2011;45(4):337-340. [CrossRef] [Medline]
    44. Mason EC, Andrews G. The use of automated assessments in internet-based CBT: the computer will be with you shortly. Internet Interv. Oct 2014;1(4):216-224. [CrossRef]
    45. Mewton L, Wong N, Andrews G. The effectiveness of internet cognitive behavioural therapy for generalized anxiety disorder in clinical practice. Depress Anxiety. Oct 2012;29(10):843-849. [CrossRef] [Medline]
    46. Newby JM, Mewton L, Andrews G. Transdiagnostic versus disorder-specific internet-delivered cognitive behaviour therapy for anxiety and depression in primary care. J Anxiety Disord. Mar 2017;46:25-34. [CrossRef] [Medline]
    47. Newby JM, Mahoney AE, Mason EC, Smith J, Uppal S, Andrews G. Pilot trial of a therapist-supported internet-delivered cognitive behavioural therapy program for health anxiety. Internet Interv. Nov 2016;6:71-79. [CrossRef]
    48. Perini S, Titov N, Andrews G. The Climate Sadness program of Internet-based treatment for depression: a pilot study. EJAP. Dec 23, 2008;4(2):18-24. [CrossRef]
    49. Spence J, Titov N, Solley K, Dear BF, Johnston L, Wootton B, et al. Characteristics and treatment preferences of people with symptoms of posttraumatic stress disorder: an internet survey. PLoS One. 2011;6(7):e21864. [FREE Full text] [CrossRef] [Medline]
    50. Titov N, Andrews G, Schwencke G, Drobny J, Einstein D. Shyness 1: distance treatment of social phobia over the Internet. Aust N Z J Psychiatry. Jul 2008;42(7):585-594. [CrossRef] [Medline]
    51. Titov N, Andrews G, Schwencke G. Shyness 2: treating social phobia online: replication and extension. Aust N Z J Psychiatry. Jul 2008;42(7):595-605. [CrossRef] [Medline]
    52. Williams AD, Blackwell SE, Mackenzie A, Holmes EA, Andrews G. Combining imagination and reason in the treatment of depression: a randomized controlled trial of internet-based cognitive-bias modification and internet-CBT for depression. J Consult Clin Psychol. Oct 2013;81(5):793-799. [FREE Full text] [CrossRef] [Medline]
    53. Williams AD, O'Moore K, Mason E, Andrews G. The effectiveness of internet cognitive behaviour therapy (iCBT) for social anxiety disorder across two routine practice pathways. Internet Interv. Oct 2014;1(4):225-229. [CrossRef]
    54. Williams AD, O'Moore K, Blackwell SE, Smith J, Holmes EA, Andrews G. Positive imagery cognitive bias modification (CBM) and internet-based cognitive behavioral therapy (iCBT): a randomized controlled trial. J Affect Disord. Jun 01, 2015;178:131-141. [FREE Full text] [CrossRef] [Medline]
    55. Wootton BM, Titov N, Dear BF, Spence J, Kemp A. The acceptability of Internet-based treatment and characteristics of an adult sample with obsessive compulsive disorder: an Internet survey. PLoS One. 2011;6(6):e20548. [FREE Full text] [CrossRef] [Medline]

    ‎
    AUC: area under the curve
    CBT: cognitive behavior therapy
    CFI: comparative fit index
    DSM-5: Diagnostic and Statistical Manual of Mental Disorders (Fifth Edition)
    GAD-7: Generalized Anxiety Disorder seven-item scale
    HAI: Health Anxiety Inventory
    ICC: intraclass correlation coefficient
    ICD-10: International Classification of Diseases, Tenth Revision
    ITC: item-total correlations
    MADRS-S: Montgomery-Åsberg Depression Rating Scale-self-report
    PSS: Perceived Stress Scale
    RMSEA: root mean square error of approximation
    SDS: Sheehan Disability Scale
    SRMR: standardized root mean square residual
    TLI: Tucker-Lewis index
    WAI: Work Ability Index
    WHODAS 2.0: World Health Organization Disability Assessment Schedule 2.0

    Edited by G Eysenbach; submitted 13.02.17; peer-reviewed by O Garin, M Ferrer; comments to author 26.03.17; revised version received 19.08.17; accepted 04.10.17; published 08.12.17.

    Copyright

    ©Erland Axelsson, Elin Lindsäter, Brjánn Ljótsson, Erik Andersson, Erik Hedman-Lagerlöf. Originally published in JMIR Mental Health (http://mental.jmir.org), 08.12.2017.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Mental Health, is properly cited. The complete bibliographic information, a link to the original publication on http://mental.jmir.org/, as well as this copyright and license information must be included.