This was a psychometric study of the WHODAS 2.0 administered online to patients with anxiety and stress disorders. Data were collected from clinical trials of CBT for severe health anxiety (n=60) and stress disorders (n=100) conducted at Karolinska Institutet and Gustavsberg primary care clinic, Stockholm, Sweden. Both trials were approved by the Stockholm regional ethics review board (2015/415-31/5, 2014/1530-31/2), registered at ClinicalTrials.gov (NCT02540317, NCT02314065), and participants provided informed consent. Data used for this study were collected between September 2015 and July 2016.

Both clinical trials employed patient self-referral via the Internet, and advertised in newspapers as well as on online social media networks. Study applicants completed a series of online screening symptom questionnaires before a diagnostic interview with a licensed psychologist. This interview primarily served to survey eligibility criteria and lead up to a decision regarding inclusion or exclusion (ie, this decision was based on the psychiatric interview), but also served to collect important clinical data (eg, comorbid diagnoses). After the pretreatment assessment, which was conducted online, patients underwent randomization and subsequent treatment. All included patients were at least 18 years of age. The severe health anxiety sample had a principal diagnosis of DSM-5 somatic symptom disorder or illness anxiety disorder, whereas the stress disorders sample had a principal diagnosis of DSM-5 adjustment disorder or ICD-10 exhaustion disorder (for a brief introduction to this disorder, see [11]). We intend to provide an in-depth description of the methods of the clinical trials, including the recruitment process, in the primary publications.

All questionnaires were completed through a simple Web-based interface with white background, radio buttons, and checkboxes. All 60 patients in the severe health anxiety sample and 50 patients in the stress disorders trial received CBT (12 weeks, disorder-specific) for their principal disorder. The WHODAS 2.0 was administered before and after CBT. In addition, patients with severe health anxiety completed the WHODAS 2.0 at screening, thus allowing for estimates of test-retest reliability. Other measures used to validate the WHODAS 2.0 were also administered before and after treatment.

Both DSM-5 somatic symptom disorder and illness anxiety disorder were assessed with the Health Preoccupation Diagnostic Interview, which exhibits excellent interrater reliability [12]. Both ICD-10 exhaustion disorder and DSM-5 adjustment disorder were assessed with a clinical interview developed specifically for the stress disorders trial that closely followed the diagnostic criteria of ICD-10 and DSM-5. Comorbid psychiatric disorders were surveyed with the Mini-International Neuropsychiatric Interview, which is a reliable and valid instrument for assessing psychiatric disorders [13].

The self-report 12-item WHODAS 2.0 [2] instructs the respondent to determine his or her difficulty in engaging in particular activities (eg, “taking care of [...] household responsibilities” and “maintaining a friendship”), as rated on a scale from “none” (no difficulty) to “extreme or cannot do” and corresponding to six domains of functioning (see Introduction). We employed the WHO simple scoring method [1] that gives a 12-item WHODAS 2.0 score range from 12 to 60, where higher scores indicate higher disability or loss of function. This type of straightforward additive scoring has been shown to correlate strongly (r>.98, ρ=.999) with more complex scoring methods incorporating weights based on item response patterns [5,14]. Just like the pencil-and-paper version, the WHODAS 2.0 was presented over two pages with items 1 to 5 on the first page, and items 6 to 12 on the second page.

The Sheehan Disability Scale (SDS) is a well-established three-item measure of psychiatric symptom-related functional impairment with a sum score range from 0 to 30, with a higher score indicating a higher degree of functional impairment [15,16]. The Work Ability Index (WAI) measures work ability with a sum score range from 7 to 49, and higher score indicating higher work ability [17-19]. The SDS and WAI were used as indexes of functional impairment.

The Health Anxiety Inventory (HAI) is a 64-item questionnaire that measures health anxiety on a scale from 0 to 192, with a higher score indicating more health anxiety [20-22]. The Perceived Stress Scale (PSS) is a 14-item measure of perceived stress with a range from 0 to 56, with a higher score indicating more stress [23]. The HAI and PSS were used to describe the samples in terms of primary symptom domains (ie, health anxiety and stress).

The self-reported Montgomery-Åsberg Depression Rating Scale (MADRS-S) is a widely used nine-item questionnaire that measures depressive symptoms on a scale from 0 to 54, with higher scores indicating more symptoms of depression [24-26]. The seven-item Generalized Anxiety Disorder scale (GAD-7) measures general anxiety from 0 to 21, with higher scores indicating more general anxiety [27]. The MADRS-S and GAD-7 were used to assess the common symptom domains of depression and general anxiety and facilitate comparison of the two samples.

Analyses were done in SPSS version 23.0.0.2 (IBM Corp, Armonk, NY, USA) and R 3.3.2 [28] with lavaan 0.5-22 [29]. We pooled the severe health anxiety and stress disorders samples (total N=160), dropped three multivariate outliers, and employed structural equation modeling to assess the validity of the simple one-factor model of disability endorsed by the WHO [1], as well as the second-order model fitted in two previous studies [5,6]. The latter had latent variables corresponding both to the six WHODAS 2.0 dimensions of functioning and an overarching latent disability variable. We employed weighted least squares means and variance adjusted estimation, which is adequate for categorical data and nonnormal manifest variables [30]. Based on the recommendations of Byrne [31], we established a priori criteria for acceptable model fit in terms of a comparative fit index (CFI) greater than 0.90, a Tucker-Lewis index (TLI) greater than 0.90, and a root mean square error of approximation (RMSEA) of 0.08 or lower. Post hoc exploratory factor analysis was based on principal axis factoring with promax rotation (ie, factors were assumed to be correlated).

Internal consistency was investigated in terms of Cronbach alpha, complemented by adjusted item-total correlations (ITCs), which are not as strongly affected by the number of scale items. For instruments of typical length, Cronbach alpha≥.9 is usually regarded as excellent, ≥8 as good, and ≥7 as acceptable. Test-retest reliability was estimated based on a two-way mixed-effects model absolute agreement intraclass correlation coefficient (ICC) and data from a subsample (n=25) from the severe health anxiety trial that had completed the screening and pretreatment assessments within 14 days (mean 6.8, SD 3.2, range 1-13).

We used an independent samples t test to assess if patients with and without a clinical diagnosis of exhaustion disorder differed with regard to WHODAS 2.0 score, and then performed a receiver operating characteristic analysis to assess the ability (area under the curve [AUC]) of the WHODAS 2.0 to identify cases of exhaustion disorder. Pearson correlations were used to investigate baseline associations between the WHODAS 2.0 and other self-rated measures.

To evaluate sensitivity to change, we compared pretreatment and posttreatment mean scores using paired samples t test and calculated effect sizes as the difference in means at t₁ and t₂, divided by the t₁ standard deviation. Based on the Jacobson and Truax reliable change index [32], we classified patients as either reliably improved (reduction in the HAI>29.9 or reduction in the PSS>7.1) or not reliably improved. As recommended in the assessment of responsiveness, we also further differentiated between reliably improved patients based on whether they improved more than a minimal important difference in their primary symptom domain [33]. In other words, for both samples we estimated change in functional impairment in three strata: (1) patients who did not made a reliable improvement on the HAI or PSS, (2) patients who reliably improved but only to a slight to moderate degree, and (3) patients who reliably improved to a large degree. However, it was necessary to set the threshold for changing more than a minimal important difference relatively high (1.5 SD) due to the precision of the instruments requiring a relatively large score change (1.21 SD in the case of HAI, 1 SD in the case of PSS) for patients to have experienced true change with 95% certainty, as assessed with the reliable change index [32]. Item-level change was assessed with the Wilcoxon signed-rank test.

The WHODAS 2.0 response rate was 100% for the screening and pretreatment assessments. Because there was a small proportion of missing posttreatment data (for the WHODAS 2.0: 3%, 3/100 in the stress sample; 5%, 3/60 in the severe health anxiety sample), and we did not aim to investigate treatment effects but rather the responsiveness of the scale, data were used on a complete case (not intention-to-treat) basis.

Sample characteristics for the severe health anxiety sample, the stress disorders sample, and the pooled sample are presented in Table 1.

Neither the one-factor model endorsed by the WHO (χ²₅₄=1699.1, P<.001; CFI=0.29, TLI=0.14, RMSEA=0.44, standardized root mean square residual [SRMR]=0.52) nor the second-order model presented by previous studies achieved acceptable fit (χ²₄₈=132.7, P<.001; CFI=0.96, TLI=0.95, RMSEA=0.11, SRMR=0.09), and we did not see implementing any of the modification indexes as theoretically justifiable.

Post hoc exploratory factor analysis (Kaiser-Meyer-Olkin test=0.85, Barlett test<.001) was suggestive of a three-factor solution, with one very strong factor (eigenvalue=5.4, 45.0% of variance explained) and two weak factors (eigenvalues=1.1-1.7, 9.3%-14.2% of variance explained).

After rotation (Table 2) it was clear that one of the two weak factors was primarily associated with items 1 and 7 (“getting around”), and the other weak factor was primarily associated with items 8 and 9 (“self-care”), whereas the strong factor was associated with the remaining items (“understanding and communicating,” “getting along with others,” “life activities,” and “participation in society”). Interfactor correlations were substantial (r=.40-.46). Dropping items 1, 7, 8, and 9 had little impact on interindividual sum score variance, and the resulting mean was heavily correlated with the conventional one (r=.97). Thus, the factor analysis indicated that a majority of items had high loadings on a general disability factor, but that “mobility” and “self-care” emerged as distinct, but weak, factors.

Parameters related to item score distributions are presented in Table 3. Overall, patients scored low on items that concerned functional impairment in self-care, and higher on items that concerned difficulties participating in society and everyday life activities, as well as items that concerned cognitive impairment. Unlike the summary score, several item distributions were skewed and showed high kurtosis. There was also evidence of a floor effect with regard to several items.

Adjusted baseline ITCs are presented in Table 3, where the mean adjusted ITC was 0.60 (SD 0.11). Cronbach alpha values were good to excellent for both the total sum score and the proposed psychosocial subscale (Table 4).

The test-retest reliability of the WHODAS 2.0, based on the severe health anxiety sample data (n=25), was estimated at ICC=0.83 (95% CI 0.62-0.92) and the correlation between the measurements was r=.71. The test-retest reliability of the proposed psychosocial subscale was similar (ICC=0.85, 95% CI 0.67-0.94; r=.75).

In the stress disorders sample, patients with a principal diagnosis of exhaustion disorder had a significantly higher WHODAS 2.0 mean score than those who had a principal diagnosis of adjustment disorder (mean difference 8.84, 95% CI 5.96-11.72; t₉₈=6.10, P<.001), and the WHODAS 2.0 discriminated well between stress patients with and without exhaustion disorder (AUC=0.81, 95% CI 0.73-0.89; P<.001). The proposed psychosocial subscale fared about as well (AUC=0.79, 95% CI 0.70-0.87; P<.001). Baseline correlations between the WHODAS 2.0 as well as the subscale and other self-assessment questionnaires are shown in Table 5.

Effect sizes and tests pertaining to responsiveness are presented in Table 6. Wilcoxon signed-rank tests showed that all but one WHODAS 2.0 item (item 9, related to self-care) changed in the severe health anxiety sample, whereas 9 of 12 items changed (but not items 1, 8, or 9—one mobility and both self-care items) in the stress disorders sample.

To our knowledge, this was the first in-depth analysis of the reliability and validity of the self-rated 12-item WHODAS 2.0 when administered online to patients with anxiety and stress disorders. In line with our hypotheses, the WHODAS 2.0 exhibited high internal consistency, acceptable test-retest reliability, and was demonstrated to identify cases of exhaustion disorder. As expected, the WHODAS 2.0 also showed substantial associations with other measures of functioning, as well as slightly weaker but yet substantial associations with primary symptom measures. The instrument was sensitive to change in the primary psychiatric symptom domain, as illustrated by a convincing gradient in change effect size over the nonimproved versus slightly improved versus much improved strata (Table 6). It might be noted that there was no significant change in WHODAS 2.0 for the minimally changed severe health anxiety group, but this is not surprising given the very small size of this subsample (n=8). Taken together, our findings suggest that the self-rated 12-item online WHODAS 2.0—when administered to individuals with anxiety and stress disorders—is a valid measure of disability and functional impairment. The results are important because online administration facilitates handling of data in a wide range of research and routine care settings. Key strengths of this study are that two psychiatric samples with different primary conditions could be examined, that other scales could be used to validate the WHODAS 2.0 scores in both samples, and that the ability of the WHODAS 2.0 to identify patients with a clinical diagnosis of exhaustion disorder could be investigated in the stress disorders sample.

Regarding factor structure, we could neither confirm the expected one-factor WHO solution nor the second-order model put forward by previous investigators [5,6]. A post hoc model derived from exploratory analysis saw most items loading heavily (>.5) on one strong factor, and four items (8, 9, 1, and 7) loading heavily on two other factors. The results are somewhat similar to those seen with the WHODAS 2.0 administered to individuals with musculoskeletal pain [34], and suggests that items 8 and 9 (“self-care”), as well as items 1 and 7 (“getting around”), largely tap into sources of variance other than that responsible for most of the sum score (see Table 3). Our subjective impression is that these four items seem more related to physical incapacity than the rest of the scale. Because (1) these items showed apparent evidence of a floor effect (their median sum being zero), (2) the sum score primarily relied on another very strong factor, and (3) the interfactor correlations were moderately strong, the impact of multidimensionality is likely to be very small in the study of change in otherwise healthy samples with anxiety or stress disorders. On the other hand, the interfactorial correlations were small enough to complicate direct comparisons of scores between studies. Especially when anxiety or stress disorder samples differ in terms of somatic comorbidity, it is probably informative to compare the “getting around” and “self-care” subdomains separately. That is, based on the results of this study our tentative recommendation would be for studies on anxiety and stress disorders to report both the self-rated 12-item WHODAS 2.0 sum score, as well as the three subfactors, here referred to as “psychosocial,” “self-care,” and “getting around” (Table 2). However, we wish to emphasize that this factor solution was solely based on an explorative analysis, that the second-order model with six subfactors [5,6] was not far off the mark, and that additional confirmatory factor analyses based on data from similar samples are warranted to arrive at more firm conclusions.

On an item level, the most striking aspect of the score distribution was that both items 8 and 9 (ie, the “self-care” items) had a mode score of 1 and appeared to pick up on very little variance in the two samples. This may suggest that individuals with a principal diagnosis of severe health anxiety or a stress disorder are typically not much impaired in terms of ability to carry out everyday self-care tasks such as washing oneself or getting dressed.

Although the internal consistency and test-retest reliability estimates were in line with our expectations, and also fulfilled commonly accepted quality criteria for health status questionnaires [35], it is conceivable that the estimates of test-retest reliability were deflated because a psychiatric interview and the decision to include patients in a clinical trial took place between the screening and pretreatment measurements. However, the means from these two measurement points were not significantly different (t₅₉=1.55, P=.13), which for example speaks against patients having altered their responses so as to be included in one of the trials.

In this study, correlations with other measures of psychiatric symptoms and functional impairment corroborate the construct validity of the self-rated 12-item WHODAS 2.0. As this study was based on data from two clinical trials, we do not find it unfit or surprising that the WHODAS 2.0 was highly correlated with measures of anxiety and depression which were likely to be the primary reasons for functional impairment.

Due to the limited sample size and variability in the size of substrata (nonchanged vs minimally changed vs much changed) we wish to emphasize that the significance tests pertaining to change (Table 6) are of limited value, and emphasize that all P values ought to be interpreted alongside their corresponding effect sizes. Over and above the effect size gradient over the nonchanged versus minimally changed versus much changed strata, a general trend was also that the 12-item WHODAS 2.0 changed slightly more from pretreatment to posttreatment than the WAI. This may suggest that the WHODAS 2.0 covers dimensions of functional impairment that, compared to work ability, are more likely to change in a clinical trial of CBT for chronic stress disorders.

The primary limitation of this study is that over and above clinical diagnoses we did not have access to “hard” measures of functional impairment, collected by other means than self-assessment (eg, register data on disability status) that could be used to validate the WHODAS 2.0. One consequence of this is that all indexes of change were to some degree susceptible to social desirability bias or the possibility that patients reported change so as to please their therapist rather than as a consequence of real change in symptoms or disability. However, it has been demonstrated that Web-based survey administration is relatively robust to desirability bias [36], and all scales used to validate the WHODAS 2.0 (ie, the HAI, PSS, MADRS-S, GAD-7, SDS, and WAI) have, at least in conventional form, been shown to be associated with non-self-reported validators such as clinical diagnoses, sick leave, and health care consumption (eg, [16,37-41]).

We also had no control over what equipment the patients used to access the Web-based WHODAS 2.0, meaning we could not determine the significance of filling in the questionnaire via a mobile phone app rather than a conventional browser on a desktop computer, for example. The results showing good psychometric properties of the WHODAS 2.0 also suggest that there was no substantial measurement error related to the type of device used. Another threat to the generalizability of our findings is that the two samples were relatively homogenous due to the eligibility criteria for the two clinical trials, which were not primarily designed to study the psychometric properties of the WHODAS 2.0. Therefore, it is preferable to validate the findings of this study, particularly with regard to factor structure, in anxiety and stress disorder samples recruited through other means.

The item mean score profile seen in this study—with substantial functional impairment in the “understanding and communicating,” “life activities,” and “getting along with others” domains but very low functional impairment in the “self-care” domain—is highly similar to the item mean score profile seen in studies of interview and pencil-and-paper versions of the WHODAS 2.0 administered to individuals with common mental disorders [8,14]. This suggests that the response pattern does not change much due to the online format.

Before this study, the 12-item WHODAS 2.0 had also been administered online to individuals with common anxiety, stress, and mood disorders in several clinical trials (Table 7, see Multimedia Appendices 1 and 2 for details).

Although the primary aim of these trials was not to study the psychometric properties of the WHODAS 2.0, these studies presented estimates of both internal consistency (Cronbach alpha≥.83) and baseline mean scores that are very much in line with data in this trial, and thus lend further support to the validity and generalizability of our findings.

This is, to date, the most extensive investigation into the psychometric properties of the self-rated 12-item version of the WHODAS 2.0 when administered via the Internet to individuals with anxiety and stress disorders. When administered online to individuals with anxiety and stress disorders, the WHODAS 2.0 exhibits high internal consistency, high convergent validity, adequate test-retest reliability, and is sensitive to change. We conclude that the psychometric properties of the self-rated 12-item version of the WHODAS 2.0 are acceptable when the instrument is administered via the Internet to individuals with anxiety and stress disorders, but suggest that the three subfactors found in this study be reported alongside the sum score to facilitate comparisons between studies.