We used data collected in the baseline measurement of a study evaluating triage decisions in general practice. All patients with mental health problems visiting a GP working in a primary care center in the northern part of the Netherlands between January 1 and December 31, 2014 were included in the study (N=408). All included participants provided informed consent. Participants filled in the Dutch paper-and-pen version of the 4DSQ and only patients with complete data were included in the analyses (92.9%, 379/408). As a result, our final sample consisted of 379 participants with a mean age of 44.8 years (SD 16.5, range 16 to 87). Of the participants, 66.8% (253/379) were female. No significant differences in age (P=.715) or sex (P=.205) were found between responders with complete and without complete data.

Since all four of the 4DSQ scales are used and interpreted separately, we performed the psychometric evaluation and our analyses for each of the four scales separately. We followed the five steps described in the analysis plan used for the PROMIS study, which was aimed at improving patient-reported outcome instruments [8].

The descriptive statistics and the estimation of the GRM parameters were done in STATA 14.0. The CFA model was estimated using the lavaan package in R [18,19]. Monotonicity was checked using the R mokken package [20] and DIF with the R lordif package [21]. The CAT simulation was done with the CatIRT package in R [22].

The sample’s mean total score on the 4DSQ distress scale was 18.6 (SE 0.43, range 0-32, median 20), with an overall Cronbach alpha of .92. The mean depression score was 3.4 (SE 0.20, range 0-12, median 2), with a Cronbach alpha of .90. The mean score for anxiety was 5.5 (SE 0.27, range 0-23, median 4), with a Cronbach alpha of .87. Finally, for the somatization scale, the sample scored 11.6 on average (SE 0.35, range 0-32, median 11), with a Cronbach alpha of .85. These results were comparable to other studies [4,7]. The descriptive statistics of the single items on the four scales are shown in Multimedia Appendix 2. Removing any one of the items did not change the internal consistency of any of the four scales.

Regarding the first assumption, unidimensionality, we concluded that the items of the anxiety scale showed a good model fit for all four fit indices of the CFA. The items of the distress and depression scales showed a good fit for three of the four indices, but not for RMSEA, although they nearly did. For good fit, RMSEA should be lower than 0.06, but it was 0.08 (distress) and 0.07 (depression). The items of the somatization scale showed good fit for two out of four indices, but not for RMSEA (0.07 instead of less than 0.06) and TLI (0.94 instead of greater than 0.95).

Regarding the second assumption, out of 321 items pairs within the four scales (equation 1), two item pairs with a residual correlation above 0.2 were observed, indicating local independency. They were items 20 and 39 (sleep-related), and items 47 and 48 (trauma-related), all from the distress scale.

321=(½)(6)(5) + (½)(16)(15) + (½)(12)(11) + (½)(16)(15) (1)

The scalability coefficient of all items was higher than 0.3, indicating that all items met the third assumption of monotonicity.

The parameter estimates of the GRM for all items of the four scales are shown in Multimedia Appendix 3. Item 33 (“would be better off dead”) of the depression scale showed the highest alpha (7.377) and discriminates best between persons with low and high levels of depression. For the three other scales, the highest alphas were observed for item 37 (3.483, distress, “no longer feel like doing anything”), item 27 (5.527, anxiety, “feel frightened”), and item 16 (1.855, somatization, “pain in the chest”). All other items showed an alpha above 1, except for items 47 and 48 (distress), item 50 (anxiety), and items 6 and 8 (somatization).

It was found that 43 items showed CRCs as expected. Five items on the anxiety scale (40, 42, 43, 49, and 50) and two items on the somatization scale (5 and 14) did not show CRCs as expected. For those items, the probability to answer “sometimes” was always lower than the probability for one of the other responses, regardless of the trait level.

As an example, Figure 1 shows the CRCs of the items with the highest (item 33; α=7.377, ß₁=0.688, ß₂=1.349) and lowest (item 35; α=2.457, ß₁=0.119, ß₂=0.828) discrimination parameter (α) of the depression scale. The higher discrimination parameter of item 33 indicates an ability to demarcate fine gradations between persons with similar levels of depression. This can be observed in Figure 1, which shows steep curves for different answer categories for item 33. Item 35 (no escape from situation) is more easily endorsed than item 33 in general (would be better off dead), which is indicated by the location of the curves more on the left side of the graph. Persons with a high depression level are most likely to answer “sometimes” to item 33, and to answer “regularly”, “often,” or “very often or constantly” to item 35.

For the depression, anxiety, and somatization subscales, no items showed DIF. The only item that showed significant and relevant uniform and non-uniform DIF was item 41 (“I quickly get emotional”) from the distress scale for the covariate gender.

The characteristics of the simulated CAT under different levels of measurement precision (allowing the standard error of the estimated underlying trait to gradually increase; stopping rule 1) are shown in Table 1. For each scale, the standard error of theta that was equal to the standard error of the full version scale is indicated. For example, the standard error of the full version scale of distress was 0.4. When allowing the standard error of theta to be maximal 0.4, the mean number of items administered could be decreased from 16 to 6.3. The correlation between the distress level based on 6.3 items and the distress level based on all items was high (0.96). Comparable results were found for the three other scales. With the first stopping rule, we were able to reduce the mean number of items administered to 5 for depression (from 6), to 8.3 for anxiety (from 12), and to 12.9 for somatization (from 16), while correlations between CAT and full test scores remained high. Applying CAT with the first stopping rule to all four scales could reduce the total number of 4DSQ items from 50 to, on average, 34 items.

The results of combining the first stopping rule with the second stopping rule are shown in Table 2. For distress, the average number of items could be further decreased from 6.3 to 5, but the correlation also decreased from 0.96 to 0.79. Therefore, we did not apply the second stopping rule to this scale. For the three other scales, the number of average items could be decreased, while the correlation remained high. Overall, when applying the CAT with both stopping rules (except for distress), the 4DSQ could be reduced from 50 to 22 items.

In summary, when applying CAT to the 4DSQ and applying two stopping rules to the subscales of anxiety, depression, somatization, and one stopping rule to the subscale distress, the total number of items on the 4DSQ could be reduced by 56% on average (from 50 to 22 items), without losing a considerable amount of measurement precision.

Our simulation study showed that CAT may increase the efficiency of the 4DSQ and could reduce responders’ burden by more than 50%. These results were also found in other CAT studies, such as on the Center for Epidemiological Studies-Depression Scale (CES-D), where the total scale of 20 items could be reduced to 7 items [23].

Some CATs to measure anxiety and depression have already been used and evaluated in clinical (specialist) care [24-26]. These CATs appeared to be useful for longitudinal monitoring of symptoms, since they were as reliable over time as traditional questionnaires [27].

A CAT version of the 4DSQ seems especially useful in general practices, for example, as part of a broad online assessment to investigate the severity of psychological problems of patients. As the number of patients visiting their GP with mental health problems is increasing [28], there is a growing need for an efficient screener for mental health problems. Many Dutch GPs already use the 4DSQ. An efficient, shortened 4DSQ could be combined with other mental health questionnaires, while keeping responders’ burden as low as possible. GPs have only a limited time and often have to make important decisions about referring patients with mental health problems. An online severity assessment, ideally preceding the first consultation, could be helpful as a first quick evaluation on which to base further (treatment) decisions. Some GPs use the 4DSQ as an agenda-setting tool to talk about the psychological problems of their patients. An online assessment could fulfill the same agenda-setting function.

However, some obstacles for the successful implementation of a CAT version of the 4DSQ in general practice exist. First, current information and communication technology (ICT) possibilities in general practices are insufficient for the implementation of CAT, which requires sophisticated statistical software. Second, it is not clear to what extent GPs are willing to implement a CAT version of the 4DSQ. GPs may use responses from individual 4DSQ items, such as item 47 or 48 on traumatic events, for a quick clinical evaluation, and this information may be lost when applying CAT. Lastly, it is not clear if CAT is appropriate for all patients. Previous research on CAT after inpatient rehabilitation suggests that it might only be feasible to collect (complete) data for a specific subset of patients [29]. Some patients may prefer a paper-and-pencil version of a questionnaire to an online assessment. Although a CAT version of the 4DSQ might not be immediately available for use in clinical practice, some studies have already shown that CAT versions of traditional questionnaires can be used in a clinical setting [24-26] and are well accepted by patients [25]. Recently developed, free-to-use online CAT platforms [30,31] are likely to enable the development of new CAT questionnaires. Moreover, some Dutch GPs already have been using an online screener to assess mental health problems, so application of a CAT version of the 4DSQ in clinical practice may be within reach.

As this was a simulation study, we used responses to a paper-and-pencil version of the 4DSQ. In reality, responders might behave differently when receiving a computerized adaptive assessment. For example, we do not know if the actual computer administration might influence responses or what effect differences in the item order may have. However, a previous study showed that differences between results from a simulation CAT and a real CAT were small [32]. We used data from a sample from a northern region of the Netherlands, but parameter estimates based on data from different regions and countries might also differ.

Regarding the psychometric evaluation, our data showed some weaknesses. For most items of the four subscales of the 4DSQ, the assumptions for an IRT analysis were met. The assumption of unidimensionality was not met perfectly for all four scales, although it nearly was. Moreover, some items showed other limitations, such as correlations between item pairs or differential item functioning. These items might be left out in future (real-time) CAT versions of the 4DSQ. As in other studies, we found relevant DIF for the item “emotionality” on the distress scale. Women tend to more easily agree with this item compared to men, even when they have a similar underlying level of distress. When looking at the individual responses to the CAT of the distress scale, the item “emotionality” was only administered to participants with a very low level of distress. This indicates that the DIF on this item does not bias the CAT outcomes, as this item is not informative enough to be included in the final CAT. When looking at the distribution and the CRC of some items of the anxiety and somatization scales, participants either endorse option 0 or option 1 to 2. Patients apparently have difficulties differentiating between response categories 1 and 2. This might be solved in future studies by grouping response options 1 and 2 for certain items, making them dichotomous.

Data from this simulation study in general agreed with assumptions needed for CAT. CAT seems useful for improving the efficiency of the 4DSQ by 56%, without losing a considerable amount of measurement precision. Of course, this simulation study is only the first step towards a CAT version of the 4DSQ that could be implemented in clinical practice and it should be followed by a study on a real-time CAT and eventually by an evaluation of the developed CAT version in a clinical setting.