This paper is in the following e-collection/theme issue:

Depression and Mood Disorders; Suicide Prevention (1363) Digital Mental Health Interventions, e-Mental Health and Cyberpsychology (1514) Methods and New Tools in Mental Health Research (454) Mobile Health in Psychiatry (426) Research Instruments, Questionnaires, and Tools (576) Anxiety and Stress Disorders (1053) Electronic/Mobile Data Capture, Internet-based Survey & Research Methodology (832)

Published on in Vol 8, No 10 (2021): October

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/29615, first published .
Validity and Reliability of the Self-administered Psycho-TherApy-SystemS (SELFPASS) Item Pool for the Daily Mood Tracking of Depressive Symptoms: Cross-sectional Web-Based Survey

Validity and Reliability of the Self-administered Psycho-TherApy-SystemS (SELFPASS) Item Pool for the Daily Mood Tracking of Depressive Symptoms: Cross-sectional Web-Based Survey

Validity and Reliability of the Self-administered Psycho-TherApy-SystemS (SELFPASS) Item Pool for the Daily Mood Tracking of Depressive Symptoms: Cross-sectional Web-Based Survey

Authors of this article:

Gwendolyn Mayer1 Author Orcid Image ;   Svenja Hummel1 Author Orcid Image ;   Nadine Gronewold1 Author Orcid Image ;   Neele Oetjen1 Author Orcid Image ;   Thomas Hilbel2 Author Orcid Image ;   Jobst-Hendrik Schultz1 Author Orcid Image
Article Authors Cited by (2) Tweetations (2) Metrics

    Original Paper

    1Department of General Internal Medicine and Psychosomatics, University Hospital Heidelberg, Heidelberg, Germany

    2Electrical Engineering and Applied Sciences, Westphalian University of Applied Sciences, Gelsenkirchen, Germany

    *these authors contributed equally

    Corresponding Author:

    Gwendolyn Mayer, Dr sc hum

    Department of General Internal Medicine and Psychosomatics

    University Hospital Heidelberg

    Im Neuenheimer Feld 130 3

    Heidelberg, 69120

    Germany

    Phone: 49 6221 56 35685

    Email: gwendolyn.mayer@med.uni-heidelberg.de


    Background: e-Mental health apps targeting depression have gained increased attention in mental health care. Daily self-assessment is an essential part of e-mental health apps. The Self-administered Psycho-TherApy-SystemS (SELFPASS) app is a self-management app to manage depressive and comorbid anxiety symptoms of patients with a depression diagnosis. A self-developed item pool with 40 depression items and 12 anxiety items is included to provide symptom-specific suggestions for interventions. However, the psychometric properties of the item pool have not yet been evaluated.

    Objective: The aim of this study is to investigate the validity and reliability of the SELFPASS item pool.

    Methods: A weblink with the SELFPASS item pool and validated mood assessment scales was distributed to healthy subjects and patients who had received a diagnosis of a depressive disorder within the last year. Two scores were derived from the SELFPASS item pool: SELFPASS depression (SP-D) and SELFPASS anxiety (SP-A). Reliability was examined using Cronbach α. Construct validity was assessed through Pearson correlations with the Patient Health Questionnaire-9 (PHQ-9), the General Anxiety Disorder Scale-7 (GAD-7), and the WHO-5-Wellbeing-Scale (WHO-5). Logistic regression analysis was performed as an indicator for concurrent criterion validity of SP-D and SP-A. Factor analysis was performed to provide information about the underlying factor structure of the item pool. Item-scale correlations were calculated in order to determine item quality.

    Results: A total of 284 participants were included, with 192 (67.6%) healthy subjects and 92 (32.4%) patients. Cronbach α was set to .94 for SP-D and α=.88 for SP-A. We found significant positive correlations between SP-D and PHQ-9 scores (r=0.87; P<.001) and between SP-A and GAD-7 scores (r=0.80; P<.001), and negative correlations between SP-D and WHO-5 scores (r=–0.80; P<.001) and between SP-A and WHO-5 scores (r=–0.69; P<.001). Increasing scores of SP-D and SP-A led to increased odds of belonging to the patient group (SP-D: odds ratio 1.03, 95% CI 1.01-1.05; P<.001; SP-A: 1.05, 1.05-1.01; P=.01). The item pool yielded 2 factors: one that consisted of mood-related items and another with somatic-related items.

    Conclusions: The SELFPASS item pool showed good psychometric properties in terms of reliability, construct, and criterion validity. The item pool is an appropriate source for daily mood tracking in future e-mental health apps among patients with depression. Our study provides general recommendations for future developments as well as recommendations within the item pool.

    JMIR Ment Health 2021;8(10):e29615

    doi:10.2196/29615

    Keywords

    self-management; mood tracking; validity; reliability; item pool; questionnaire; depression; anxiety; mood assessment 


    Mood Tracking and Symptom Monitoring in e-Mental Health Apps

    e-Mental health apps targeting depression and anxiety play an increasing role in mental health care ranging from smartphone-based monitoring apps to extensive treatment applications [1-3]. Their evidence is regularly reviewed and shows that especially guided interventions are as successful as treatment-as-usual controls [4,5]. However, some barriers are still prevalent to fully exploit the potential of e-mental health apps in clinical practice. This might be owing to the fact that apps from clinical studies are not necessarily available in app stores, and choosing the right app poses difficulties among potential users [6]. Moreover, web-based interventions are subject to high attrition rates, as users may drop out soon after downloading apps [7,8]. The reasons for this low user engagement were identified beside others in the poor usability, the lack of user-centric design, and in their low ability to identify a crisis [9].

    Regular mood tracking and symptom monitoring is a necessary step to identify sudden fluctuations that may hint at a suicidal crisis. The purpose of mood tracking in e-mental health apps for depression lies in the enhanced opportunities for self-reflection on mood and on their patterns and potential triggers of symptom aggravation [10]. Mood tracking is already implemented in a major share of e-mental health apps for depression. A recent review of Qu et al [11] on the functionality of 29 top-rated depression apps found that 19 (66%) included tools for mood tracking based on regular self-reports. Another review that focused on both, mobile apps and browser-based programs for depression, found mood-tracking functionalities in 86% [1]. Repeated measurement of a person’s mood or behavior in real time is referred to as ecological momentary assessment [12], and recent studies have shown the predictive power of mobile technology for depressive symptoms by capturing step counts and sedentary behavior [13], or vital parameters including sleep quality and heart rate [14]. However, currently available apps and programs hardly exploit the full technological potential of connecting wearable functionalities of the mobile phone or sensor data to regular mood queries [15].

    Regarding anxiety apps, there is only scarce data on the availability and technical realization of mood tracking thus far. The last comprehensive study [2] that contributed to this question was published in 2017 and found 52 apps, 29% of which provided “emotional ratings.” It remains unclear if these ratings target mood tracking or rather initial screenings.

    Digital Mood-Related Self-assessment Tools

    Digital versions of self-report scales regarding psychiatric symptoms show a comparable reliability to paper-pencil versions [16,17]. However, a detailed review of the diagnostic provenance of questions used in e-mental health apps has been missing thus far. The most frequently used questionnaire in depression management apps and chatbots is either the depression module of the Patient Health Questionnaire (PHQ-9) or self-developed questions [11,18].

    The PHQ-9 is a brief depression screening instrument that has been validated across a variety of medical conditions [19,20]. The patients’ task is to self-assess the severity of 9 depressive symptom criteria over a span of the past 2 weeks; cut-off scores then allow for the assessment of depression severity. However, there are several disadvantages of the use of the PHQ-9 in a self-management app for depression. First, mood tracking in common e-mental health apps should be performed at least on a daily basis to provide timely information on the progression or worsening of symptoms and to make suggestions for symptom-specific interventions [1]. In such high-frequency self-assessment, it is important to ensure variety within the daily survey process to avoid the test routine. It has been shown that participants’ compliance to give valid responses to frequent mood assessment declines with high repetition rates [21]. The PHQ-9 is a self-assessment instrument that is not conceptualized for daily use, as the adherence to answering the same 9 questions every day may be low. Moreover, the questions refer to the past 2 weeks, and patients with depression may experience negative memory bias [22]. Finally, the PHQ-9 items screen depression criteria on the basis of DSM-IV [19], which differs slightly from the International Classification of Diseases (ICD) criteria. In general, DSM-IV diagnostic criteria require the presence of fewer symptoms than ICD-10 criteria, which reveals a slightly higher prevalence of depression in countries that rely on DSM rather than ICD [23]. Therefore, depression criteria that target an ICD-based diagnostic health system, similar to European countries, should not only rely on PHQ-9 items, even though it is an accepted tool in the clinical routine.

    Comorbidity of Depression and Anxiety

    Recently since the introduction of the mixed anxiety-depressive disorder (F41.2) category in the ICD-10 in 1992, the complexity of differential diagnostics of anxiety and depression became evident [24]. Data on the prevalence of mental disorders in the United States reveal that 50%-60% of individuals with major depressive disorder (MDD) have also received a diagnosis of an anxiety disorder [25], and comparable data have been reported in Germany [26]. Theories on the relationship between the 2 diseases existed early on. Clark et al [27] described with their “tripartite model of anxiety and depression” that the general negative feelings are common to both syndromes, while the lack of positive feelings distinguishes depression from anxiety. According to them, only scales with symptom-specific content can sufficiently distinguish between the 2 syndromes [27,28]. In clinical practice, screening instruments are used specifically for 1 of the 2 diseases; for example, the PHQ-9 for depression [19] and the Generalized Anxiety Disorder Scale-7 (GAD-7) [29] are both modules of the PHQ [30]. However, some instruments cover both in the same test; for example, the Hospital Anxiety and Depression Scale (HADS) [31]. Developing an item pool for the usage of web-based daily mood assessments, while considering symptoms of comorbid anxiety in patients with depression, is necessary to provide symptom-specific intervention suggestions.

    Somatic Aspects in the Presentation of Depression

    Somatic symptoms play an essential role in depressive symptomatology over the lifespan. Recent results have pointed out that depressive symptoms tend to shift with increasing age from a mood-related profile of symptoms to rather vegetative somatic symptoms including psychomotor agitation, gastrointestinal problems, or sleep disturbances [32]. Few vegetative symptoms are considered part of the diagnostic routine; for example, fatigue, loss of appetite, and sleep [33]. Understanding the individual burden profile has been found to be crucial for targeted treatment, as patients with higher values in self-criticism respond better to psychotherapy than those with a somatic symptom profile [34]. As a result, in developing an item pool for daily mood tracking in e-mental health apps, somatic symptoms should be considered to a certain extent. With regard to comorbid anxiety symptoms, somatic aspects of a panic disorder, which are associated with the characteristics of specific illnesses, have as well been detected early on [35].

    Scale Development Process

    Scale development is a defined procedure that has been described comprehensively [36-38]. In short, 5 major steps are usually followed, beginning with (1) the generation of an item pool that is based on an extensive literature review and definition of the core concepts and the targeted population for the future scale. Item writing should involve simple and straightforward language while avoiding complex, ambiguous items. If a Likert-scale format is chosen, considerations about allowing midrange ratings are highly dependent on targets and core assumptions regarding the response behavior. Hence, no general conclusion regarding which response format is best is feasible [39]. (2) Qualitative analysis of the so far generated items and their content is the second step to (3) prepare a pilot with a small sample size. After that, (4) a larger evaluation study with a selected sample should be carried out by applying psychometric properties for reliability and different types of validity. Only then, norm values with a representative sample would be established [36]. The following investigation includes steps 1 to 4, while step 3 was included in a feasibility study that is currently being prepared for publication.

    Objectives

    The aim of this study is the development of an item pool of mood-related questions for daily self-assessment in an e-mental health app to cover the main aspects of depressive symptoms in accordance with the diagnostic criteria of ICD-10 and comorbid symptoms of anxiety. We investigated the psychometric properties’ reliability, construct validity, criterion validity, and item-scale correlation of the item pool, which may be used for future self-management apps with suggestions for symptom-specific interventions. Thus, we aim to develop recommendations for the integration of mood-tracking items in future e-mental health apps.


    The Self-administered Psycho-Therapy-SystemS App

    The mobile app Self-administered Psycho-TherApy-SystemS (SELFPASS) was developed in a German study that received federal funding. This app was designed to improve the self-management of patients with depression on the basis of an individualized daily mood score. The target group comprises patients diagnosed with depression, who often wait a long time for a face-to-face psychotherapy [40]. The app allows for daily monitoring of depressive symptoms on the one hand and daily interventions to support patients on the other hand. SELFPASS does not claim to replace a face-to-face psychotherapy but rather to help patients during the waiting period, in order to bridge the treatment gap [41].

    An item pool of 52 depression- and anxiety-related questions was developed to cover the main aspects of depressive symptoms and to provide suggestions for a pool of individualized interventions. Out of all items, 40 questions refer to depressive and 12 to anxiety symptoms. All questions are presented in Multimedia Appendix 1 in the original German version and translated to English (Multimedia Appendix 2). The item pool was developed jointly by mental health experts, who were part of an interdisciplinary team of psychologists and physicians in the clinic of psychosomatics. Items were translated by the authors and critically checked by active colleagues from the United States and other English-speaking countries. The main development approach was a rational construction strategy; that is, the process was guided mainly by theoretical considerations on the nature of depressive symptomatology [42]. The content of the depression items followed the major and minor symptoms of major depression in accordance with ICD-10 criteria, of which major symptoms are depressed mood, loss of interest, and loss of energy. Minor symptoms referred to the commonly listed ones including lack of concentration, feelings of worthlessness, guilt, pessimistic future expectations, suicidal ideation, sleep disturbances, and loss of appetite [33]. We used the following standardized diagnostic instruments as an additional source of information regarding the nature of symptom queries; however, all questions were rephrased: depression screening using 2 questions [43], PHQ-9 [19], HADS [31], and the short version of the Beck Depression Inventory (BDI-V) [44]. As these instruments are part of routine diagnostics [45] and are also validated in their German version [46,47], they provided an evidence-based foundation for the development of the SELFPASS items. We considered the following as anxiety symptoms: nervousness, excessive worry, accompanied by the inability to stop them, restlessness and not being able to relax, tendency to panic, and the fear of something awful happening. Anxiety questions were inclined toward clinical instruments (GAD-7), and further psychometric scales [48] including the State-Trait Anxiety Inventory (STAI) [49] and the anxiety subscale of the HADS, all of which have shown good validity in their German version [49,50]. Moreover, we included 2 of our own anxiety-related questions targeting the feeling of tightness in the chest and difficulty breathing, which are common symptoms of a panic attack [35].

    All questions were rephrased. Each symptom was assessed through 4 different items in accordance with recommendations in the literature regarding test construction [51]: 2 of them were formulated in a negative direction and 2 in a positive direction to ensure diversion and to control for response bias (for example, “I have trouble concentrating on something” and “I can stick to one thing and concentrate fully on it”). The items were scored on a 6-point Likert scale ranging from 0=don’t agree to 5=agree. After starting the SELFPASS app, the user is asked to complete a daily self-assessment of at least 6 items from the SELFPASS item pool. Two of the questions relate to major depressive symptoms, 2 of them to minor symptoms, and the remaining 2 to anxiety symptoms. The number of daily questions may increase on the basis of the answers of the previous day, as the algorithm is designed to track individual symptoms. If a value of 3 is exceeded (positively formulated questions were automatically recoded by the algorithm), the symptom will be assessed again with an alternative formulation on the next day. Thus, the app is able to generate an individualized symptom profile based on the patient’s most prevalent current symptoms of the respective last 3 days and suggests 3 potentially appropriate interventions. For example, a result of highly prevalent anxiety symptoms will recommend relaxation interventions, while symptoms of ruminating and self-doubt will lead to an intervention suggestion for behavioral activation and cognitive restructuring.

    Study Design

    We used a cross-sectional, web-based survey design to investigate the validity and reliability of the SELFPASS item pool. The study population consisted of 2 groups. The first group included healthy subjects, who reported not having any affective disorder within in the last 3 years. The second group included patients, who have received a diagnosis of any depressive disorder within the last year. We excluded patients with bipolar disorder, a psychosis, or suicidal ideation. Ethical approval for this study was granted by the Ethics Commission of the Medical Faculty of Heidelberg University (S-031/2020).

    Recruitment

    The survey was made available on the internet via the soscisurvey.de [52] platform with 1 link each for patients and healthy subjects. The 2 versions included the same questionnaires except for demographic details. Thus, potentially psychiatric disorders in healthy subjects, who might take part in the study coincidentally, could be excluded. Exclusion criteria were a diagnosis with a bipolar affective disorder, a psychosis, or another psychiatric disorder within the last 3 years, and suicidality. Potentially suicidal participants were forwarded to an extra page with contact information for support. The group of healthy adults was recruited through social media channels and personal contact networks.

    Patients were recruited within the Heidelberg University Hospital. They were contacted personally, via email or by post, and received a weblink to the study. Thus, the presence of a physicians’ diagnosis could be ensured, which is a prerequisite to assess criterion validity. Exclusion criteria were applied in advance.

    Validation Procedures

    We provided 4 major parts of mood-related questions in random order: the German versions of the PHQ-9, GAD-7, WHO-5-Wellbeing-Scale (WHO-5) [53], as well as the SELFPASS item pool.

    The PHQ-9 is a 9-item tool to assess depressive symptoms. The responses are rated on a 4-point Likert scale ranging from 0=not at all to 3=nearly every day. GAD-7 assesses 7 anxiety items on a scale from 0=not at all to 3=nearly every day. Both the PHQ-9 and GAD-7 consider a cut-off score of 10, with higher values indicating MDD [54] or moderate anxiety symptoms [55].

    WHO-5 consists of 5 items, rated on a scale from 0=no time to 5=all of the time, for subjective well-being of the participant, with a high score indicating higher well-being. It was initially introduced by the World Health Organization (WHO) in 1998 as a first step in a 10-item screening process of depression and is usually followed by a diagnostic interview [56].

    In this study, all psychometric instruments are used for construct validation, which refers to the alignment of the concepts measured by the instrument with their theoretical construct [57]. While the outcomes of the PHQ-9 and GAD-7 relative to SELFPASS scores serve as an indicator for convergent validity, the relationship between WHO-5 scores and SELFPASS scores demonstrate divergent validity. In turn, criterion validity targets the relationship between the results by the tested measures to an external criterion [57]. In our study, we considered the presence or absence of a valid physicians’ depression diagnosis as an external criterion to show the validity of the SELFPASS scores.

    Data Analysis

    Statistical analysis was carried out in multiple steps using SPSS (version 24; IBM Corp) [58].

    We excluded those data sets with a relative speed index of >1.75 as well as above 10% missing data per participant following pragmatic considerations and recommendations of the literature [59,60]. The option “no specification” for the SELFPASS items was treated as missing values for all statistical analyses.

    After descriptive analysis of all participants (including means, SDs, and frequencies) we calculated subscales among the 40 SELFPASS depression (SP-D) and the 12 SELFPASS anxiety (SP-A) items.

    We used Cronbach α to determine internal consistency of the subscales. Cronbach α>.80 was considered a threshold for acceptance [61,62].

    Convergent validity was assessed using Pearson correlation coefficients of SP-D with PHQ-9 and SP-A with GAD-7. Pearson correlation coefficients were also computed for WHO-5 and SP-D and for the sum of SP-A and SP to assess discriminant validity. Following Cohen’s [63] definition, r>0.30 can be interpreted as a moderate and r>0.50 a strong correlation.

    Logistic regression analysis was performed to analyze the power of SP-D and SP-A to distinguish between patients and healthy subjects. The results will serve as criterion validation. We considered the R2 value in accordance with Nagelkerke [64] as good if it fell within a range of R2=0.20 and 0.40.

    We carried out an explorative factor analysis to investigate the underlying factor structure of the items. At first, we applied the analysis to the whole sample, and a more detailed investigation studied the factor structure within the data of only the patients. The Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy was interpreted as acceptable if its value exceeded 0.50 [65]. The Bartlett test [66] of sphericity revealed a significant (P<.001) result, which indicated that the items were appropriate for factor analysis [67]. We chose a principal axis factor analysis approach followed by oblimin rotation, as recommended for factors that might show intercorrelations [67].

    Finally, we correlated the items’ values with the respective scale value to obtain the individual item-scale correlation for the analysis of the discriminatory power of each item. We applied the rule of thumb to remove items with a correlation below r=0.30 [68,69]. The share of missing values of single items and their inter-item correlations was as well interpreted as an indicator for low item quality. We considered items with more than 20 inter-item correlations below r=0.20 as critical in accordance with recommendations from the literature [70]. However, the final decision about deletion of the respective items was made jointly together after the analysis of missing values and discriminatory power. All results on the level of single items are reported in Multimedia Appendix 3.

    A level of P<.05 was considered significant in all statistical tests.


    Participants

    In total, 329 participants responded to the web-based questionnaire from end-March to mid-August 2020. After excluding respondents with a relative speed index of >1.75 (n=31) and more than 10% of missing items (n=4), as well as healthy subjects with a psychiatric diagnosis (n=10), the final sample comprised 284 participants. The sample consisted of 192 (67.6%) healthy subjects and 92 (32.4%) patients. The demographic characteristics of the participants are shown in Table 1.

    Table 1. Demographic characteristics of the study sample (N=284).
    CharacteristicsParticipants

    		Healthy subjects (n=192)Patients (n=92)Total
    Age (years), mean (SD)29.02 (9.70)40.46 (14.91)32.73 (12.8)
    Gender, n (%)

    		Male43 (22.4)38 (41.3)81 (28.5)

    		Female148 (77.1)53 (57.6)201 (70.8)

    		Other1 (0.5)1 (1.1)2 (0.7)
    Family status, n (%)

    		Single162 (84.4)46 (50.0)208 (73.2)

    		Married19 (9.9)31 (33.7)50 (17.6)

    		Divorced7 (3.6)7 (7.6)14 (4.9)

    		Widowed0 (0)2 (2.2)2 (0.7)

    		Separated0 (0)5 (5.4)5 (1.8)

    		Other4 (2.1)1 (1.1)5 (1.8)
    Level of education, n (%)

    		No degree0 (0)3 (3.3)3 (1.1)

    		High school64 (33.3)37 (40.2)101 (35.6)

    		College123 (64.1)47 (51.1)170 (59.9)

    		Dissertation/PhD5 (2.6)2 (2.2)7 (2.5)

    		Other0 (0)3 (3.3)3 (1.1)
    Profession, n (%)

    		Self-employed3 (1.6)1 (1.1)4 (1.4)

    		Worker5 (2.6)9 (9.8)14 (4.9)

    		Civil servant3 (1.6)3 (3.3)6 (2.1)

    		Employee51 (26.6)37 (40.2)88 (31.0)

    		Not working3 (1.6)11 (12.0)14 (4.9)

    		Student/pupil123 (64.1)17 (18.5)140 (49.3)

    		Retired3 (1.6)5 (5.4)8 (2.8)

    		Other1 (0.5)9 (9.8)10 (3.5)

    Psychometric Scores and SELFPASS Subscales

    The mean scores of all psychometric scales and the SELFPASS subscales are presented in Table 2. On average, the healthy subjects had lower scores on all scales than the patients, except for the WHO-5.

    Table 2. Scores of healthy subjects and patients on the psychometric scales.
    ScaleHealthy subjects (n=192), mean (SD)Patients (n=92), mean (SD)
    SELFPASSa for depression53.33 (24.62)87.40 (29.27)
    SELFPASS for anxiety18.63 (9.65)31.37 (11.62)
    Overall Self-administered Psycho-TherApy-SystemS71.96 (32.37)118.77 (38.63)
    Patient Health Questionnaire-95.39 (3.95)10.40 (5.23)
    General Anxiety Disorder Scale-75.03 (3.88)8.76 (4.53)
    WHO-5-Wellbeing-Scale14.46 (4.86)8.57 (5.47)

    a SELFPASS: Self-administered Psycho-TherApy-SystemS.

    Reliability

    The internal consistency of the 40-item SP-D subscale and the 12-item SP-A subscale was assessed from a Cronbach α of .94 for SP-D (n=240) and .88 for SP-A (n=275).

    Construct Validity

    Table 3 shows the results of construct validity analysis. Regarding convergent validity, the SELFPASS depression and anxiety subscale show positive correlations with the scores of PHQ-9 and GAD-7, respectively, as valid depression and anxiety measures. The data reveal significant negative correlations among all 3 SELFPASS scores with WHO-5 scores, showing discriminant validity.

    Table 3. Pearson correlation analysis to determine the correlation between the scores of SELFPASSa subscales with those of the Patient Health Questionnaire-9, General Anxiety Disorder Scale-7, and WHO-5-Wellbeing-Scale.

    		Pearson correlation coefficient

    		SELFPASS for depressionSELFPASS for anxietyOverall SELFPASS
    Convergent validity

    		Patient Health Questionnaire-90.87b0.74b

    		General Anxiety Disorder Scale-70.70b0.80b
    Discriminant validity

    		WHO-5-Wellbeing-Scale–0.80b–0.69b–0.80b

    a SELFPASS: Self-administered Psycho-TherApy-SystemS.

    bP<.01.

    Criterion Validity

    The results of the logistic regression analysis for patients and healthy subjects are presented in Table 4. The overall model was significant (χ22=91.39; P<.001; N=284) as well as the coefficients SP-D and SP-A. Increasing scores for depression and anxiety increase the odds of being part of the patient group. Details are presented in Table 4.

    Table 4. Results of logistic regression analysis indicating the probability of being part of the patient group on the basis of the scores of the SELFPASS depression and anxiety subscalesa.
    Predictorβ (SE)P valueOdds ratio (95% CI)
    Constant–4.15 (0.49)<.0010.02
    SELFPASS for depression0.03 (0.01)<.0011.03 (1.01-1.05)
    SELFPASS for anxiety0.05 (0.02).011.05 (1.01-1.10)

    aSELFPASS: Self-administered Psycho-TherApy-SystemS; Omnibus test: χ22=91.39; P<.001; Hosmer–Lemeshow test: χ28=7.16; P=.52; Nagelkerke R2=0.38.

    Exploratory Factor Analysis and Item Analysis

    A principal axis analysis with the whole sample revealed a 2-factor solution after scree plot analysis. The 2 factors accounted for 36.70% of the variance, and oblique rotation was performed. The intercorrelation of the 2 factors was r=0.40. The KMO measure of sampling adequacy was 0.94. Four items showed inter-item correlations with coefficients less than r=0.20 in more than 20 cases (SP 20, 23, 25, and 39). Item 25 was not answered in 23 of 240 (8.1%) cases. Item-scale correlations were calculated and helped evaluate the following items as critical: SP 20, 23, 24, 25, 33, and 41. After excluding them, another exploratory factor analysis explained 40.95% of the variance. Details on the level of single items as well as the item-scale correlations are presented in Multimedia Appendix 3.

    The patient subgroup showed a KMO measure of 0.70. A principal axis analysis with the patient sample accounted for 32.77% of the variance. After excluding the critical items, 35.90% of the variance was accounted for. The intercorrelation of the 2 factors was r=0.21.

    The items as well as the loadings of the items in the whole sample and the patient subsample are presented in Table 5.

    Table 5. Items, associated symptoms, and results of the principal axis factor analysis of the whole sample after oblique rotation (N=284), and of the patients’ sample (n=92).
    ItemItem formulationSymptomFactor 1 (all)Factor 1 (patients)Factor 2 (all)Factor 2 (patients)
    SELFPASSa for depression

    		SP1I feel depressed, sad or hopeless.CSb10.730.73


    		SP2I easily burst into tears.CS10.400.42


    		SP3I am cheerful and in good spirits.CS10.870.82


    		SP4I feel easy and carefree.CS10.880.77


    		SP5I have much less desire and enjoyment for things I usually like to do.CS20.760.72


    		SP6I have no interest in people around me.CS20.330.37


    		SP7I can laugh at funny moments.CS20.540.54


    		SP8I can enjoy pleasant things and be happy about them.CS20.630.53


    		SP9I feel exhausted and sluggish.CS30.820.68


    		SP10I can\'t force myself to do anything.CS30.610.47


    		SP11Decision making is easy for me.CS30.480.34


    		SP12I am full of drive and energy.CS30.860.83


    		SP13I have problems in concentrating on something.ASc10.740.72


    		SP14My thoughts keep on slipping away.AS10.630.64


    		SP15I can dwell on one thing with my full concentration.AS10.690.59


    		SP16I am not easily distracted.AS10.540.49


    		SP17I am just not good enough.AS20.540.40


    		SP18Others can do things much better than I can.AS20.350.35


    		SP19I am satisfied with myself.AS20.760.60


    		SP20dI take care of my appearance.AS20.20

    		0.32

    		SP21I should have done things much differently in the past.AS30.430.40


    		SP22I have made mistakes. It´s not surprising I feel bad.AS30.470.51


    		SP23dI am not perfect. But who is?AS30.290.24


    		SP24dI don’t deserve to feel bad.AS3-0.03-0.23


    		SP25dIt can only get worse.AS40.190.140.19

    		SP26The future has nothing to offer for me.AS40.530.62


    		SP27I am looking forward to the future.AS40.670.67


    		SP28Time heals all wounds. Everything will be alright.AS40.610.46


    		SP29Sometimes I think it would be better to be dead.AS5

    		0.590.45

    		SP30I think a lot about death.AS5
    		0.330.41

    		SP31I think about putting hands on myself.AS5

    		0.560.31

    		SP32I have already thought about how to kill myself.AS5

    		0.630.46

    		SP33dI sleep too much.AS60.16

    		0.35

    		SP34I have trouble falling asleep and/or wake up constantly.AS60.550.40


    		SP35My sleep was restful and sufficient.AS60.76

    		0.45

    		SP36I slept well.AS60.740.44


    		SP37I feel a constant hunger or appetite for food.AS7
    		-0.060.08

    		SP38I don’t feel like eating anything.AS70.330.30


    		SP39I have a good appetite.AS70.28

    		0.27

    		SP40I eat enough and I follow a balanced diet.AS70.42

    		0.43
    SELFPASS for anxiety

    		SP41dI hope that I don’t get sick.CAe
    		0.180.07-0.36

    		SP42Sometimes I have an oppressive feeling in my stomach.CA0.560.55


    		SP43I am worried that something terrible will happen.CA0.370.59


    		SP44Sometimes I start panicking suddenly.CA0.520.58


    		SP45When I’m worried, I still can keep my control.CA0.560.46


    		SP46Disturbing thoughts run through my mind.CA0.520.66


    		SP47I’m calm.CA0.790.78


    		SP48When I think of my current affairs, I get anxious.CA0.590.61


    		SP49I feel safe and secureCA0.770.69


    		SP50I’m worried about something going wrong soon.CA
    		0.480.43

    		SP51Sometimes I feel tightness in my chest.CA0.660.63


    		SP52Sometimes I can’t breathe properly.CA0.530.59


    		Eigen value

    		16.9514.275.654.10

    aSELFPASS: Self-administered Psycho-TherApy-SystemS.

    bCS: core symptom.

    cAS: additional symptom.

    ditems that should be excluded or reformulated.

    eCA: comorbid anxiety.


    Principal Findings

    This study aimed to investigate the psychometric properties of an item pool of mood-related questions for daily self-assessment, which cover symptoms of depression and comorbid anxiety to make suggestions for symptom-specific interventions. The item pool was developed for the use within the e-mental health app SELFPASS and for future developments. Through a web-based cross-sectional survey design, the instrument emerged as reliable and valid. The psychometric properties are shown in a representative study population of healthy subjects and patients with a diagnosis of depression within the past year. Considering that average PHQ-9 scores of 10.4 indicate a moderate severity of depression [54] and GAD-7 scores of 8.76 indicate mild symptoms of anxiety [55], the patients seemed to be considerably affected and were hence eligible to test the SELFPASS item pool appropriately.

    Both subscales assessing symptoms of depression (SP-D) and anxiety (SP-A) showed high correlations with standardized psychometric instruments (PHQ-9 and GAD-7). This demonstrates a high inherent construct validity. The WHO-5 as an indicator of subjective well-being was negatively correlated with SP-D and SP-A, which in turn reveals good discriminatory construct validity. The negative association of the WHO-5 with depression and anxiety scales has already been shown in other validation studies [71,72]. Moreover, the results of the SELFPASS subscales in the whole sample were able to predict the affiliation to the patient group, which was interpreted as concurrent criterion validity. As expected, increasing scores for depression and anxiety measured through SELFPASS increase the odds of being part of the patient group.

    An exploratory factor analysis indicated an underlying 2-factor structure of the item pool that covered a mood-related factor on the one hand and a somatic factor on the other hand in the whole sample. The factor structure of the patient sample even increased this structure, including more items in the second factor, which were related to paying attention to appearance, suicidal thoughts, sleep, appetite, and the fear of becoming sick. There was 1 item that did not fully seem to fit to this interpretation (“I’m worried about something going wrong soon”). A potential explanation might lie in the timing of investigation, as many patients were concerned with becoming infected with SARS-CoV-2 at that time [73].

    A closer investigation of screening instruments for depression shows that validation studies of the PHQ-9 among different populations; for example, in palliative care, these instruments show a comparable 2-fold factor structure of 1 factor focusing on cognitive and affective aspects and another one relating to somatic symptoms [74]. Although for palliative patients with a high somatic burden, this might be an obvious result; however, similar results were obtained in a psychiatric sample [75]. Therefore, somatic aspects of depression might reveal an underlying structure of depression and anxiety, which is reminiscent of the “Tripartite model of depression and anxiety,” which states that beside the already described “generally negative affect” as a third factor “somatic symptoms” [27,75]. Moreover, suicidal ideation has been identified as particularly crucial among patients with depression with somatic syndrome [74,75].

    Enhancing the quality of e-mental health apps, especially with regard to a successful crisis management in case of symptom exacerbation, has already been identified as a necessary step to reduce dropout rates and increase adherence to digital interventions [9]. The integration of appropriate mood-tracking items, as provided by the presented item pool in future e-mental health apps, is a necessary measure in integrating digital assistance in routine clinical practice.

    Recommendations for the Use of the SELFPASS Item Pool

    The SELFPASS item pool is suitable for daily mood assessment in any kind of web-based intervention or e-mental health app. It is designed for highly frequent repetitive use providing approximately 6 items every day out of the total item pool based on the results of the previous days. Following this purpose, we recommend reformulating items 20, 23, 24, 25, 33, and 41 for appropriate use and future validation studies. For any other use, the items may be dropped as well. Although item 31, 32, 37, and 39 also showed a lack of quality after item analysis, we recommend retaining these items as they assess important depressive symptoms.

    Based on our experience with the development and validation of the SELFPASS item pool, some general recommendations may be provided to ensure optimization of the items (Table 6). Simultaneously, we summarized recommending conclusions within the SELFPASS item pool.

    Table 6. Recommendations for the future use of the SELFPASSa item pool.
    TopicGeneral recommendationRecommendation for use within the SELFPASS item pool
    Item presentationProvide a diversion in item presentation to increase adherence.Ask for 2 main symptoms, 2 additional symptoms and two anxiety symptoms per day. If 1 symptom exceeds a critical score, pursue this symptom with an alternative item.
    Symptom coverageCover somatic, cognitive, and emotional aspects of symptomatology.Provide a random choice of daily items following the rules described above to cover a broad range of symptoms.
    Crisis managementInclude crisis management in case of positive answers to suicidal ideation.Same as the general recommendation.
    Discriminatory powerEvaluate each question with regard to whether it might be able to sufficiently differentiate between healthy subjects and patients. Hence, symptoms including increased appetite and sleep were excluded by the Beck Depression Inventory [76], but we do not follow this approach.Reformulate items 20, 23, 24, 25, 33, and 41. A closer focus on the manifestation of these symptoms in patients with depressive and anxiety symptoms is recommended.
    External validationProvide validation of single assessments, such as BMI, as matching self-assessments of body weight or sleep parameters delivered by a sensor as control for self-assessed sleep quality.Same as the general recommendation.
    VariationProvide positive and negative directions of items.Same as the general recommendation.
    Targeted useBe aware of the target population of the questionnaire.The item pool addresses patients with a pre-existing diagnosis of depression for daily monitoring of their symptoms.

    aSELFPASS: Self-administered Psycho-TherApy-SystemS.

    Directions of Future Research

    With the introduction of the ICD-11, some diagnostic criteria for depressive episodes will change, which should be considered for further use of the SELFPASS item pool. It is expected that the previous 3 main symptoms of depression will be reduced to 2. Fatigue and lack of drive are then considered additional symptoms [76]. Feelings of worthlessness and guilt will be summarized to one, as well as sleep and appetite. Then, psychomotor agitation or retardation, formerly included in the somatic symptoms, will be considered as own symptoms [77]. This could be relevant for apps that provide symptom-specific intervention suggestions as SELFPASS does.

    Moreover, much evidence has been provided that ecological momentary assessments delivered through mobile data have the potential to become behavioral markers for mental health symptoms; for example, movement profiles collected from GPS data and circadian sleep rhythms recorded through phone usage [78]. Unfortunately, mental health apps beyond scientific studies hardly make use of data processing technologies that allow for personalized intervention content based on the activities of the users [15]. Future studies may focus on validation approaches of different data sources in one instrument; for example, daily questions for self-assessment, sensor data, and further behavioral markers delivered through mobile devices. Validation with the help of sensor data, finally, is a powerful approach to verify the validity of the item pool during use over a longer period, to ascertain its test-retest validity. As of this writing, the item pool might serve as an ecological momentary assessment tool based on questions only, which might be asked several times per day.

    Beside the somatic symptoms outlined above and considered in our item pool, it should be noted that depression is often accompanied by several somatic conditions including obesity, cardiovascular diseases, diabetes, pain, or even multimorbidity [79]. However, existing monitoring apps thus far do not target the simultaneous management of mental and somatic conditions, except for medication adherence [80]. Future approaches should build upon existing results and integrate single solutions to transdiagnostic apps that thus far exist within the field of mental health [81] but rather do not pertain to mental and physical conditions. Our item pool might serve as a valuable source and can be complemented by physical conditions.

    Limitations

    There are some limitations to this study, which should be considered. First, the data were collected during the 2020 COVID-19 pandemic, which might have had an impact on the mental health of the participants [82,83]. In addition, the patient and healthy subject group differ in terms of group size, age, and gender. Regarding criterion validity, only a diagnosis of depression was defined as an inclusion criterion for the patient group. Owing to the described comorbidity of both disorders, anxiety was nevertheless also considered an external criterion to determine criterion validity. Another study should focus on comparing patient populations with validated diagnoses of both depression and anxiety separately. Thus, another confirmatory factor analysis should be performed to clarify the factor structure of the item pool. We provide a validation of a whole item pool of 52 questions, although the number of the daily questions is much lower based on the symptom profile of the respective user. A validation of this process of choice would require a substantially high number of patients with different symptomatology, which was not possible to carry out in our study design. Moreover, we did not carry out cognitive interviewing with a small sample size of patients, which is sometimes carried out during test construction [84]. However, as the symptoms of depression are already well studied, we decided to rely on the experience of the existing diagnostic instruments. As a final limitation, delusionary and psychotic symptoms that are relevant in case of a psychotic depression were not considered in the SELFPASS item pool. Psychotic depression is a subtype of depression in ICD-10, and no changes regarding ICD-11 are expected [85]. Thus, if future e-mental health apps focus as well on more severe forms of depressive disorders, further items should be added and validated.

    Conclusions

    The SELFPASS item pool is a valid and reliable source for daily self-assessment in e-mental health apps. It follows diagnostic standards of depression and comorbid anxiety symptoms. The item pool is a valuable source of questions for daily mood tracking in future e-mental health apps. Further developments have to focus on optimizing the wording of single items as well as the adaptations that are expected from ICD-11 in the near future.

    Acknowledgments

    This study was part of the Self-administered Psycho-TherApy-SystemS (SELFPASS) project, which was funded by the Federal Ministry of Education and Research (FKZ 13GW0157B). We thank Ali Zafar, MA, MSc, for revising the English translation of the item pool. We further thank Joshua Barniske for revising the manuscript.

    Conflicts of Interest

    None declared.

    ‎
    Multimedia Appendix 1

    Self-administered Psycho-TherApy-SystemS (SELFPASS) question catalogue (German original).

    DOCX File , 15 KB
    ‎
    Multimedia Appendix 2

    Self-administered Psycho-TherApy-SystemS (SELFPASS) question catalogue (English translation).

    DOCX File , 27 KB
    ‎
    Multimedia Appendix 3

    Results of the item analysis of SELFPASS depression (SP-D) and SELFPASS anxiety (SP-A): Means, SD, Item-scale-correlation, missing values (N, %) and number significant inter-item correlations per item (N). SELFPASS: Self-administered Psycho-TherApy-SystemS.

    DOCX File , 33 KB
    1. Bubolz S, Mayer G, Gronewold N, Hilbel T, Schultz J. Adherence to Established Treatment Guidelines Among Unguided Digital Interventions for Depression: Quality Evaluation of 28 Web-Based Programs and Mobile Apps. J Med Internet Res 2020 Jul 13;22(7):e16136 [FREE Full text] [CrossRef] [Medline]
    2. Sucala M, Cuijpers P, Muench F, Cardoș R, Soflau R, Dobrean A, et al. Anxiety: There is an app for that. A systematic review of anxiety apps. Depress Anxiety 2017 Jun;34(6):518-525. [CrossRef] [Medline]
    3. Van Ameringen M, Turna J, Khalesi Z, Pullia K, Patterson B. There is an app for that! The current state of mobile applications (apps) for DSM-5 obsessive-compulsive disorder, posttraumatic stress disorder, anxiety and mood disorders. Depress Anxiety 2017 Jun;34(6):526-539. [CrossRef] [Medline]
    4. Karyotaki E, Efthimiou O, Miguel C, Bermpohl FMG, Furukawa TA, Cuijpers P, Individual Patient Data Meta-Analyses for Depression (IPDMA-DE) Collaboration, et al. Internet-Based Cognitive Behavioral Therapy for Depression: A Systematic Review and Individual Patient Data Network Meta-analysis. JAMA Psychiatry 2021 Apr 01;78(4):361-371. [CrossRef] [Medline]
    5. Andersson G, Carlbring P, Titov N, Lindefors N. Internet Interventions for Adults with Anxiety and Mood Disorders: A Narrative Umbrella Review of Recent Meta-Analyses. Can J Psychiatry 2019 Jul;64(7):465-470 [FREE Full text] [CrossRef] [Medline]
    6. Van Daele T, Best P, Bernaerts S, Van Assche E, De Witte NA. Dropping the E: The potential for integrating e-mental health in psychotherapy. Curr Opin Psychol 2021 Feb 23;41:46-50. [CrossRef] [Medline]
    7. Fleming T, Bavin L, Lucassen M, Stasiak K, Hopkins S, Merry S. Beyond the Trial: Systematic Review of Real-World Uptake and Engagement With Digital Self-Help Interventions for Depression, Low Mood, or Anxiety. J Med Internet Res 2018 Jun 06;20(6):e199 [FREE Full text] [CrossRef] [Medline]
    8. Baumel A, Muench F, Edan S, Kane JM. Objective User Engagement With Mental Health Apps: Systematic Search and Panel-Based Usage Analysis. J Med Internet Res 2019 Sep 25;21(9):e14567 [FREE Full text] [CrossRef] [Medline]
    9. Torous J, Nicholas J, Larsen ME, Firth J, Christensen H. Clinical review of user engagement with mental health smartphone apps: evidence, theory and improvements. Evid Based Ment Health 2018 Aug;21(3):116-119. [CrossRef] [Medline]
    10. Caldeira C, Chen Y, Chan L, Pham V, Chen Y, Zheng K. Mobile apps for mood tracking: an analysis of features and user reviews. AMIA Annu Symp Proc 2017;2017:495-504 [FREE Full text] [Medline]
    11. Qu C, Sas C, Daudén Roquet C, Doherty G. Functionality of Top-Rated Mobile Apps for Depression: Systematic Search and Evaluation. JMIR Ment Health 2020 Jan 24;7(1):e15321 [FREE Full text] [CrossRef] [Medline]
    12. Shiffman S, Stone AA, Hufford MR. Ecological momentary assessment. Annu Rev Clin Psychol 2008 Apr;4(1):1-32. [CrossRef] [Medline]
    13. Moshe I, Terhorst Y, Opoku Asare K, Sander LB, Ferreira D, Baumeister H, et al. Predicting Symptoms of Depression and Anxiety Using Smartphone and Wearable Data. Front Psychiatry 2021;12:625247 [FREE Full text] [CrossRef] [Medline]
    14. Mohr DC, Zhang M, Schueller SM. Personal Sensing: Understanding Mental Health Using Ubiquitous Sensors and Machine Learning. Annu Rev Clin Psychol 2017 May 08;13:23-47 [FREE Full text] [CrossRef] [Medline]
    15. Burger F, Neerincx MA, Brinkman W. Technological State of the Art of Electronic Mental Health Interventions for Major Depressive Disorder: Systematic Literature Review. J Med Internet Res 2020 Jan 20;22(1):e12599 [FREE Full text] [CrossRef] [Medline]
    16. Alfonsson S, Maathz P, Hursti T. Interformat reliability of digital psychiatric self-report questionnaires: a systematic review. J Med Internet Res 2014 Dec 03;16(12):e268 [FREE Full text] [CrossRef] [Medline]
    17. Vallejo MA, Jordán CM, Díaz MI, Comeche MI, Ortega J. Psychological assessment via the internet: a reliability and validity study of online (vs paper-and-pencil) versions of the General Health Questionnaire-28 (GHQ-28) and the Symptoms Check-List-90-Revised (SCL-90-R). J Med Internet Res 2007 Jan 31;9(1):e2 [FREE Full text] [CrossRef] [Medline]
    18. Martinengo L, Van Galen L, Lum E, Kowalski M, Subramaniam M, Car J. Suicide prevention and depression apps' suicide risk assessment and management: a systematic assessment of adherence to clinical guidelines. BMC Med 2019 Dec 19;17(1):231 [FREE Full text] [CrossRef] [Medline]
    19. Kroenke K, Spitzer RL, Williams JBW. The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med 2001 Sep;16(9):606-613 [FREE Full text] [CrossRef] [Medline]
    20. Kroenke K, Spitzer RL, Williams JB, Löwe B. The Patient Health Questionnaire Somatic, Anxiety, and Depressive Symptom Scales: a systematic review. Gen Hosp Psychiatry 2010;32(4):345-359. [CrossRef] [Medline]
    21. Wilhelm P, Schoebi D. Assessing Mood in Daily Life. Eur J Psychol Assess 2007 Jan;23(4):258-267. [CrossRef]
    22. Urban EJ, Charles ST, Levine LJ, Almeida DM. Depression history and memory bias for specific daily emotions. PLoS One 2018;13(9):e0203574 [FREE Full text] [CrossRef] [Medline]
    23. Sjöberg L, Karlsson B, Atti A, Skoog I, Fratiglioni L, Wang H. Prevalence of depression: Comparisons of different depression definitions in population-based samples of older adults. J Affect Disord 2017 Oct 15;221:123-131. [CrossRef] [Medline]
    24. Barkow K, Heun R, Wittchen H, Bedirhan Ustün T, Gänsicke M, Maier W. Mixed anxiety-depression in a 1 year follow-up study: shift to other diagnoses or remission? J Affect Disord 2004 Apr;79(1-3):235-239. [CrossRef] [Medline]
    25. Kaufman J, Charney DCOMD. Comorbidity of mood and anxiety disorders. Depress Anxiety 2000;12 Suppl 1:69-76. [CrossRef] [Medline]
    26. Steffen A, Nübel J, Jacobi F, Bätzing J, Holstiege J. Mental and somatic comorbidity of depression: a comprehensive cross-sectional analysis of 202 diagnosis groups using German nationwide ambulatory claims data. BMC Psychiatry 2020 Mar 30;20(1):142 [FREE Full text] [CrossRef] [Medline]
    27. Clark LA, Watson D. Tripartite model of anxiety and depression: psychometric evidence and taxonomic implications. J Abnorm Psychol 1991 Aug;100(3):316-336. [CrossRef] [Medline]
    28. Beck AT. Cognitive therapy and the emotional disorders. New York, NY: International Universities Press; 1976.
    29. Spitzer RL, Kroenke K, Williams JBW, Löwe B. A brief measure for assessing generalized anxiety disorder: the GAD-7. Arch Intern Med 2006 May 22;166(10):1092-1097. [CrossRef] [Medline]
    30. Gesundheitsfragebogen für Patienten (PHQ-D).   URL: https:/​/www.​klinikum.uni-heidelberg.de/​fileadmin/​Psychosomatische_Klinik/​pdf_Material/​PHQ_Komplett_Fragebogen1.​pdf [accessed 2021-09-09]
    31. Herrmann-Lingen C, Buss S, Snaith RP, Herrmann CH. HADS-D – Hospital Anxiety and Depression Scale – Deutsche Version: Ein Fragebogen zur Erfassung von Angst und Depressivität in der somatischen Medizin. ScienceOpen. 1995.   URL: https://www.scienceopen.com/document?vid=93f89556-7dbf-4792-9833-0d4822d5a58a [accessed 2021-09-09]
    32. Schaakxs R, Comijs HC, Lamers F, Beekman ATF, Penninx BWJH. Age-related variability in the presentation of symptoms of major depressive disorder. Psychol Med 2017 Feb;47(3):543-552. [CrossRef] [Medline]
    33. ICD-10: International Statistical Classification of Diseases and Related Health Problems, 10th Revision. World Health Organization. 2010.   URL: https://icd.who.int/browse10/Content/statichtml/ICD10Volume2_en_2010.pdf [accessed 2021-09-09]
    34. Høstmælingen A, Ulvenes P, Nissen-Lie HA, Eielsen M, Wampold BE. Trajectories of change in chronic depression: Differences in self-criticism and somatic symptoms between users of antidepressants and nonmedicated patients. J Couns Psychol 2021 Jul 01. [CrossRef] [Medline]
    35. Meuret AE, White KS, Ritz T, Roth WT, Hofmann SG, Brown TA. Panic attack symptom dimensions and their relationship to illness characteristics in panic disorder. J Psychiatr Res 2006 Sep;40(6):520-527. [CrossRef] [Medline]
    36. Moosbrugger H, Kelava A. Test theory and questionnaire construction. Berlin: Springer; 2020.
    37. Morgado FFR, Meireles JFF, Neves CM, Amaral ACS, Ferreira MEC. Scale development: ten main limitations and recommendations to improve future research practices. Psicol Reflex Crit 2017 Jan 25;30(1):3 [FREE Full text] [CrossRef] [Medline]
    38. Kline P. The Handbook of Psychological Testing. New York, NY: Psychology Press; 2000.
    39. Clark LA, Watson D. Constructing validity: Basic issues in objective scale development. Psychological Assessment 1995 Sep;7(3):309-319. [CrossRef]
    40. Kazdin AE. Addressing the treatment gap: A key challenge for extending evidence-based psychosocial interventions. Behav Res Ther 2017 Jan;88:7-18. [CrossRef] [Medline]
    41. Schröder J, Berger T, Westermann S, Klein JP, Moritz S. Internet interventions for depression: new developments. Dialogues Clin Neurosci 2016 Jun;18(2):203-212 [FREE Full text] [Medline]
    42. Moosbrugger H, Brandt H. Itemkonstruktion und Antwortverhalten. In: Testtheorie und Fragebogenkonstruktion. Berlin: Springer; 2020:67-89.
    43. Grulke N, Bailer H, Blaser G, Geyer M, Hinz A, Schmutzer G, et al. [Screening for depression -- two questions practically oriented]. Wien Med Wochenschr 2005 Jul;155(13-14):297-302. [CrossRef] [Medline]
    44. Furlanetto LM, Mendlowicz MV, Romildo Bueno J. The validity of the Beck Depression Inventory-Short Form as a screening and diagnostic instrument for moderate and severe depression in medical inpatients. J Affect Disord 2005 May;86(1):87-91. [CrossRef] [Medline]
    45. Schneider F, Schorr S. S3 guideline / National Care Guideline for Unipolar Depression. Berlin: Springer; 2017.
    46. Schmitt M, Altstötter-Gleich C, Hinz A, Maes J, Brähler E. Normwerte für das Vereinfachte Beck-Depressions-Inventar (BDI-V) in der Allgemeinbevölkerung. Diagnostica 2006 Apr;52(2):51-59. [CrossRef]
    47. Martin A, Rief W, Klaiberg A, Braehler E. Validity of the Brief Patient Health Questionnaire Mood Scale (PHQ-9) in the general population. Gen Hosp Psychiatry 2006;28(1):71-77. [CrossRef] [Medline]
    48. Bandelow B, Lichte T, Rudolf S, Wiltink J, Beutel M. S3-Leitlinie Angststörungen. Berlin: Springer; 2015.
    49. von Lothar Laux P, Glanzmann P, Spielberger SCD. Das State-Trait-Angstinventar (STAI): theoretische Grundlagen und Handanweisung. Weinheim: Beltz; 1981.
    50. Löwe B, Müller S, Brähler E, Kroenke K, Albani C, Decker O. Validierung und Normierung eines kurzen Selbstratinginstrumentes zur Generalisierten Angst (GAD-7) in einer repräsentativen Stichprobe der deutschen Allgemeinbevölkerung. Psychother Psych Med 2007 Feb;57(02). [CrossRef]
    51. Smith GT, Fischer S, Fister SM. Incremental validity principles in test construction. Psychol Assess 2003 Dec;15(4):467-477. [CrossRef] [Medline]
    52. SoSci Survey – the Solution for Professional Online Questionnaires. SoSci Survey GmbH.   URL: https://www.soscisurvey.de [accessed 2021-09-09]
    53. WHO (Fünf) - Fragebogen Zum Wohlbefinden´ (Version 1998). Psychiatric Research Unit. WHO Collaborating Centre in Mental Health.   URL: https://www.psykiatri-regionh.dk/who-5/Documents/WHO5_German.pdf [accessed 2021-09-09]
    54. Moriarty AS, Gilbody S, McMillan D, Manea L. Screening and case finding for major depressive disorder using the Patient Health Questionnaire (PHQ-9): a meta-analysis. Gen Hosp Psychiatry 2015;37(6):567-576. [CrossRef] [Medline]
    55. Kroenke K, Spitzer RL, Williams JB, Monahan PO, Löwe B. Anxiety disorders in primary care: prevalence, impairment, comorbidity, and detection. Ann Intern Med 2007 Mar 06;146(5):317-325. [CrossRef] [Medline]
    56. Topp CW, Østergaard SD, Søndergaard S, Bech P. The WHO-5 Well-Being Index: a systematic review of the literature. Psychother Psychosom 2015;84(3):167-176 [FREE Full text] [CrossRef] [Medline]
    57. Souza A, Alexandre N, Guirardello E. Psychometric properties in instruments evaluation of reliability and validity. Epidemiol Serv Saude 2017;26(3):649-659 [FREE Full text] [CrossRef] [Medline]
    58. IBM SPSS Statistics, version 24. IBM.   URL: https://www.ibm.com/support/pages/downloading-ibm-spss-statistics-24 [accessed 2021-09-09]
    59. Bannon J. Missing data within a quantitative research study: How to assess it, treat it, and why you should care. J Am Assoc Nurse Pract 2015 Apr;27(4):230-232. [CrossRef] [Medline]
    60. Leiner DJ. Too Fast, Too Straight, Too Weird: Post Hoc Identification of Meaningless Data in Internet Surveys. Surv Res Methods 2019;13(3):229-248. [CrossRef]
    61. Bland JM, Altman DG. Cronbach's alpha. BMJ 1997 Feb 22;314(7080):572 [FREE Full text] [CrossRef] [Medline]
    62. Schmitt N. Uses and abuses of coefficient alpha. Psychological Assessment 1996;8(4):350-353. [CrossRef]
    63. Cohen J. Statistical Power Analysis for the Behavioral Sciences. New York, NY: Routledge; 1988.
    64. Best H, Wolf CLR. Logistische Regression. In: Handbuch der sozialwissenschaftlichen Datenanalyse. Berlin: Springer; 2010:827-854.
    65. Kaiser HF, Rice J. Little Jiffy, Mark Iv. Educ Psychol Meas 2016 Jul 02;34(1):111-117. [CrossRef]
    66. Bartlett MS. Tests of significance in factor analysis. Br J Psychol 1950;3(2):77-85. [CrossRef]
    67. Field A. Discovering Statistics Using IBM SPSS Statistics (5th edition). New York, NY: Sage Publications; 2017.
    68. Beck AT, Steer RA. Internal consistencies of the original and revised Beck Depression Inventory. J Clin Psychol 1984 Nov;40(6):1365-1367. [CrossRef] [Medline]
    69. Ebrahimi A, Samouei R, Mousavii SG, Bornamanesh AR. Development and validation of 26-item dysfunctional attitude scale. Asia Pac Psychiatry 2013 Jun;5(2):101-107. [CrossRef] [Medline]
    70. Piedmont RL. Inter-item Correlations. In: Michalos AC, editor. Encyclopedia of Quality of Life and Well-Being Research. Dordrecht: Springer; 2014.
    71. Ghaheri A, Omani-Samani R, Sepidarkish M, Hosseini M, Maroufizadeh S. The Four-item Patient Health Questionnaire for Anxiety and Depression: A Validation Study in Infertile Patients. Int J Fertil Steril 2020 Oct;14(3):234-239 [FREE Full text] [CrossRef] [Medline]
    72. Dadfar M, Momeni Safarabad N, Asgharnejad Farid AA, Nemati Shirzy M, Ghazie Pour Abarghouie F. Reliability, validity, and factorial structure of the World Health Organization-5 Well-Being Index (WHO-5) in Iranian psychiatric outpatients. Trends Psychiatry Psychother 2018;40(2):79-84 [FREE Full text] [CrossRef] [Medline]
    73. Usher K, Durkin J, Bhullar N. The COVID-19 pandemic and mental health impacts. Int J Ment Health Nurs 2020 Jun;29(3):315-318 [FREE Full text] [CrossRef] [Medline]
    74. Chilcot J, Rayner L, Lee W, Price A, Goodwin L, Monroe B, et al. The factor structure of the PHQ-9 in palliative care. J Psychosom Res 2013 Jul;75(1):60-64. [CrossRef] [Medline]
    75. Beard C, Hsu K, Rifkin L, Busch A, Björgvinsson T. Validation of the PHQ-9 in a psychiatric sample. J Affect Disord 2016 Mar 15;193:267-273. [CrossRef] [Medline]
    76. Lieb K. Affektive Störungen. In: Intensivkurs Psychiatrie und Psychotherapie. Munich: Elsevier; 2019:197-231.
    77. 6A70 Single episode depressive disorder. ICD-11 for Mortality and Morbidity Statistics.   URL: https://icd.who.int/browse11/l-m/en#/http%3a%2f%2fid.who.int%2ficd%2fentity%2f578635574 [accessed 2021-09-09]
    78. Mohr DC, Ho J, Duffecy J, Baron KG, Lehman KA, Jin L, et al. Perceived barriers to psychological treatments and their relationship to depression. J Clin Psychol 2010 Apr;66(4):394-409 [FREE Full text] [CrossRef] [Medline]
    79. Read JR, Sharpe L, Modini M, Dear BF. Multimorbidity and depression: A systematic review and meta-analysis. J Affect Disord 2017 Oct 15;221:36-46. [CrossRef] [Medline]
    80. Armitage LC, Kassavou A, Sutton S. Do mobile device apps designed to support medication adherence demonstrate efficacy? A systematic review of randomised controlled trials, with meta-analysis. BMJ Open 2020 Jan 30;10(1):e032045 [FREE Full text] [CrossRef] [Medline]
    81. Pennou A, Lecomte T, Potvin S, Khazaal Y. Mobile Intervention for Individuals With Psychosis, Dual Disorders, and Their Common Comorbidities: A Literature Review. Front Psychiatry 2019;10:302 [FREE Full text] [CrossRef] [Medline]
    82. Du J, Mayer G, Hummel S, Oetjen N, Gronewold N, Zafar A, et al. Mental Health Burden in Different Professions During the Final Stage of the COVID-19 Lockdown in China: Cross-sectional Survey Study. J Med Internet Res 2020 Dec 02;22(12):e24240 [FREE Full text] [CrossRef] [Medline]
    83. Hummel S, Oetjen N, Du J, Posenato E, Resende de Almeida RM, Losada R, et al. Mental Health Among Medical Professionals During the COVID-19 Pandemic in Eight European Countries: Cross-sectional Survey Study. J Med Internet Res 2021 Jan 18;23(1):e24983 [FREE Full text] [CrossRef] [Medline]
    84. Willis GB. Analysis of the Cognitive Interview in Questionnaire Design. Oxford: Oxford University Press; 2015.
    85. Østergaard SD, Meyers BS, Flint AJ, Mulsant BH, Whyte EM, Ulbricht CM, STOP-PD Study Group. Measuring psychotic depression. Acta Psychiatr Scand 2014 Mar;129(3):211-220 [FREE Full text] [CrossRef] [Medline]

    ‎
    BDI-V: Beck Depression Inventory
    GAD-7: General Anxiety Disorder Scale-7
    HADS: Hospital Anxiety and Depression Scale
    ICD: International Classification of Diseases
    KMO: Kaiser-Meyer-Olkin
    PHQ-9: Patient Health Questionnaire-9
    SELFPASS: Self-administered Psycho-TherApy-SystemS
    SP-A: SELFPASS anxiety
    SP-D: SELFPASS depression
    STAI: State-Trait Anxiety Inventory
    WHO: World Health Organization
    WHO-5: WHO-5-Wellbeing-Scale

    Edited by J Torous; submitted 14.04.21; peer-reviewed by N De Witte, C Cheng; comments to author 25.05.21; revised version received 16.07.21; accepted 06.08.21; published 18.10.21

    Copyright

    ©Gwendolyn Mayer, Svenja Hummel, Nadine Gronewold, Neele Oetjen, Thomas Hilbel, Jobst-Hendrik Schultz. Originally published in JMIR Mental Health (https://mental.jmir.org), 18.10.2021.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Mental Health, is properly cited. The complete bibliographic information, a link to the original publication on https://mental.jmir.org/, as well as this copyright and license information must be included.