Published on in Vol 10 (2023)

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/37225, first published .
Digital Phenotyping for Differential Diagnosis of Major Depressive Episode: Narrative Review

Digital Phenotyping for Differential Diagnosis of Major Depressive Episode: Narrative Review

Digital Phenotyping for Differential Diagnosis of Major Depressive Episode: Narrative Review

Review

1Department of Psychiatry and Memory Clinic, University Hospital of Nice, Nice, France

2Research Department Cognitive Assistants, Deutsches Forschungszentrum für Künstliche Intelligenz GmbH, Saarbrücken, Germany

3Department of Psychiatry and Psychotherapy, Saarland University Medical Center, Hombourg, Germany

4Department of Psychiatry, Hopital Pasteur, University Hospital of Nice, Nice, France

5Department of Psychiatry, School of Medicine and Health Sciences, Carl von Ossietzky University of Oldenburg, Bad Zwischenahn, Germany

6Research Department Sémagramme Team, Institut national de recherche en informatique et en automatique, Nancy, France

7Research Department, Cognition-Behaviour-Technology Lab, University Côte d’Azur, Nice, France

8Research Department Stars Team, Institut national de recherche en informatique et en automatique, Sophia Antipolis - Valbonne, France

Corresponding Author:

Eric Ettore, MD

Department of Psychiatry and Memory Clinic

University Hospital of Nice

30 Voie Romaine

Nice, 06000

France

Phone: 33 633306394

Email: ettore.e@chu-nice.fr


Background: Major depressive episode (MDE) is a common clinical syndrome. It can be found in different pathologies such as major depressive disorder (MDD), bipolar disorder (BD), posttraumatic stress disorder (PTSD), or even occur in the context of psychological trauma. However, only 1 syndrome is described in international classifications (Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition [DSM-5]/International Classification of Diseases 11th Revision [ICD-11]), which do not take into account the underlying pathology at the origin of the MDE. Clinical interviews are currently the best source of information to obtain the etiological diagnosis of MDE. Nevertheless, it does not allow an early diagnosis and there are no objective measures of extracted clinical information. To remedy this, the use of digital tools and their correlation with clinical symptomatology could be useful.

Objective: We aimed to review the current application of digital tools for MDE diagnosis while highlighting shortcomings for further research. In addition, our work was focused on digital devices easy to use during clinical interview and mental health issues where depression is common.

Methods: We conducted a narrative review of the use of digital tools during clinical interviews for MDE by searching papers published in PubMed/MEDLINE, Web of Science, and Google Scholar databases since February 2010. The search was conducted from June to September 2021. Potentially relevant papers were then compared against a checklist for relevance and reviewed independently for inclusion, with focus on 4 allocated topics of (1) automated voice analysis, behavior analysis by (2) video and physiological measures, (3) heart rate variability (HRV), and (4) electrodermal activity (EDA). For this purpose, we were interested in 4 frequently found clinical conditions in which MDE can occur: (1) MDD, (2) BD, (3) PTSD, and (4) psychological trauma.

Results: A total of 74 relevant papers on the subject were qualitatively analyzed and the information was synthesized. Thus, a digital phenotype of MDE seems to emerge consisting of modifications in speech features (namely, temporal, prosodic, spectral, source, and formants) and in speech content, modifications in nonverbal behavior (head, hand, body and eyes movement, facial expressivity, and gaze), and a decrease in physiological measurements (HRV and EDA). We not only found similarities but also differences when MDE occurs in MDD, BD, PTSD, or psychological trauma. However, comparative studies were rare in BD or PTSD conditions, which does not allow us to identify clear and distinct digital phenotypes.

Conclusions: Our search identified markers from several modalities that hold promise for helping with a more objective diagnosis of MDE. To validate their potential, further longitudinal and prospective studies are needed.

JMIR Ment Health 2023;10:e37225

doi:10.2196/37225

Keywords



Major depression is a frequent syndrome affecting more than 264 million people worldwide [1]. Major depressive episode (MDE), as defined by the current criteria [2], describes a large heterogeneous clinical syndrome comprising more than 1490 combinations of symptoms [3]. The possibility of fulfilling MDE criteria with opposite symptoms, such as insomnia and hypersomnia, decrease/increase in appetite or agitation, and psychomotor retardation, highlights this heterogeneity. This heterogeneity could explain why studies with large samples of patients with major depression show that only 30% of them remitted with a first-line antidepressant and another 30% did not remit after 4 consecutive treatment trials [4].

These various “profiles” of MDE may require different therapeutic approaches. In fact, according to international classifications [2], MDE symptoms are the same whether in major depressive disorder (MDD) or in bipolar disorder (BD). In clinical practice, hypomanic or manic episodes have been the most recognizable characteristics of BD, while depression seems to be most frequent clinical manifestation [5,6]. Performing differential diagnosis between MDD and BD during an MDE is challenging. Approximately 20% of people with MDE would be misdiagnosed as MDD when the correct diagnosis is BD [7]. Thereby, the delay for a correct BD diagnosis can vary from almost 7 to 10 years after the first mood symptoms [8]. These misdiagnoses can have several damaging consequences, for example, induction of manic, hypomanic, or mixed states; development of treatment resistance; or cycle acceleration [9].

Furthermore, Rytwinski et al [10] showed in their meta-analysis that almost 52% of patients with posttraumatic stress disorder (PTSD) have comorbidity with MDD. In fact, in the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition (DSM-5), several symptoms are shared between MDD and PTSD [11] and a study by Gros et al [12] showed that in patients classified by DSM-5 as having MDD and PTSD, only the presence of trauma (criterion A of DSM-5) could differentiate MDD from PTSD efficiently. Moreover, patients classified as comorbid between MDD and PTSD showed the highest severity of symptoms, more cognitive deterioration, higher suicidality, and worse prognosis compared with noncomorbid patients [12]. One possible explanation could be the time-dependent fluctuation of PTSD symptoms [13]. In a dynamic model, MDD could be a progression or an evolution of psychotraumatism [14]. Although there is a frequent association between MDD and PTSD, the link between them remains unclear. However, it points to the importance of exploring the presence of psychological trauma and PTSD in patients with MDE because it carries the potential to drastically impact further therapeutic care. Misdiagnosing PTSD or the presence of psychological trauma could lead to a worse prognosis due to inappropriate drug treatment or psychotherapy.

In this article, we focus on the use of new digital markers in MDE, given their potential to serve as an additional and objective diagnostic support. For this, we explore studies in MDE, especially when it is presented in the context of either MDD or BD, as well as when it is presented alongside the presence of psychological trauma or PTSD.

Digital phenotyping refers to the moment-to-moment quantification of human behavior in everyday life using data from digital devices [15]. It suggests the possibility of revealing clinically relevant information by a continuous and nonintrusive monitoring of behavioral and mental states. Two categories of data can be captured by digital phenotyping. First, active data, which require the input of the participants being studied. It includes, for example, recordings, responses to surveys, and social media activity. Passive data, however, do not require the individual’s participation to be captured. They include, for example, accelerometer-derived data or GPS coordinates [16]. Digital markers correspond to disease indicators obtained digitally, which can be used to define a digital phenotype [17]. It refers to the possibility of capturing, through computerized measurement tools, certain symptoms or behavior specific to a psychiatric disorder.

There is a specific interest in psychiatry in which symptoms and clinical states are mainly measured using question-based scales and without biological markers. Thus, the interest of digital phenotyping would be to obtain objective and quantifiable measurements of these symptoms or behaviors [16]. Moreover, diagnostic categories do not capture the heterogeneity of symptoms, and variability between patients can lead to misdiagnosis and incorrect treatments [18]. Identifying objective markers of clinical states, including trans-diagnostic symptoms, could improve disease classification and treatment [19].

Thus, digital phenotyping should play a role in routine clinical practice, especially by improving clinical diagnosis and treatment by an early detection of condition onset, by assessing treatment response, or even by detecting relapse [20]. Therefore, we assume that new digital measures may provide indicators for the heterogeneous characteristics of MDE and could help to better distinguish between its potential different clinical profiles. Based on this rationale, we performed a narrative review including studies on several technologies, such as speech and video analysis and physiological measures, namely, heart rate variability (HRV) and electrodermal activity (EDA) for the assessment of MDE.


Search Strategy

We conducted this narrative review from June to September 2021, which mainly concerned the current application of digital tools for MDE diagnosis. The following electronic databases were searched: PubMed/MEDLINE, Web of Science, and Google Scholar. The review was limited to articles in English or French and because we aimed to establish an overview of the most recent advances in these domains, we restricted our search to studies published after January 2010. For this paper, we decided to concentrate on digital tools that are easy to use during a clinical interview, minimally invasive, and least dependent on compliance. Thus, we focused on 4 types of digital markers: (1) automated voice analysis, behavior analysis by (2) video and physiological measures, (3) HRV, and (4) EDA. We then used broad search terms to capture as many studies as possible that are specifically related to these technologies and we associated them with terms related to specific psychiatric issue. We concentrated on MDE and on several psychiatric conditions where it may occur, especially on (1) MDD, (2) BD, (3) PTSD, and (4) psychological trauma. The search strings are included in Multimedia Appendix 1.

Inclusion and Exclusion Criteria

The inclusion criteria were as follows: primary articles or reviews dealing with depression, BD, bipolar depression, PTSD, or psychological trauma using digital tools for speech or video analysis or analysis of physiological parameters such as HRV or skin conductance. The exclusion criteria were as follows: articles related to activity monitoring on social networks or by SMS text messages or phone and articles related to ecological momentary assessment. There is an abundance of literature on these topics, but we wanted to focus here on digital tools that can be used specifically during a social interaction such as an interview with the clinician. Unrelated and redundant articles or studies using technologies for therapeutic purposes were also excluded. Theoretical papers, study protocols, letters, books or book chapters, statistical reviews, and dissertations were also excluded.

Study Selection

Three authors (EE, AK, and PR) independently screened relevant titles and abstracts for this narrative review. Then, all 3 authors screened relevant papers for eligibility. Finally, according to inclusion and exclusion criteria, full texts of eligible articles were obtained (EE).


Overview

We selected 74 articles dealing with the use of the 4 aforesaid digital tools for diagnosis assessment under different psychiatric conditions. A total of 39 articles were selected for MDD, 18 for BD, and 17 for PTSD and psychological trauma. Results are summarized on 3 tables. See Table 1 for MDD, and Multimedia Appendix 2 for BD, PTSD and psychological trauma.

Table 1. Summary of speech analysis, nonverbal behavioral analysis, heart rate variability (HRV), and electrodermal activity studies (EDA) in patients with major depressive disorder (MDD).
StudyParticipantsRecording settingPrincipal findingsPrincipal features
Speech analysis studies

Alghowinem et al [21]
  • 30 with MDD and 30 in HCa
  • Reading tasks with negative and positive meaning
  • Free speech by answering questions, where the patients describe events that had aroused significant emotions
  • MFCCb, jitter, shimmer, energy, and loudness features were robust in getting the general characteristic of depressive speech
  • Prosodic, spectral, and source features

Arevian et al [22]
  • 15 with MDD, 14 with BDc, 14 with schizophrenia, and 14 with affective schizophrenia
  • Free speech (answering questions from an app with phone call)
  • Correlation with providers’ global assessment:
  • ↓ Negative emotional language
  • ↑ Positive emotional language
  • ↑ Complex word use
  • Type of words used

Hönig et al [23]
  • 219 participants with BDI (BD type I) assessment
  • Read speech, spontaneous speech, telling an imagined story
  • Depression severity:
  • ↓ Average and SD MFCC 2
  • ↓ SD MFCC 3
  • ↓ Average pitch
  • ↓ Shimmer
  • ↓ Spectral harmonicity
  • ↓ Speech rate (↑ Average syllables duration)
  • Prosodic, spectral, source, and filter features

Horwitz et al [24]
  • 35 participants with Hamilton Rating Scale for Depression and Quick Inventory of Depressive Symptomatology assessment
  • Free speech, reading, and distinct vowels
  • Depression severity:
  • ↑ Jitter and shimmer
  • ↓ Harmonic-to-noise ratio
  • ↓ First and third formants
  • ↓ Speech rate
  • ↓ (Read) and ⇧ (free speech) pitch-based features
  • Prosodic, spectral, source, and filter features

Mundt et al [25]
  • 105 with MDD
  • Free and with tasks: reciting the alphabet, counting, reading, and sustained vowels
  • Correlation with depression severity:
  • ↑ Total speech time
  • ↑ Total pause time
  • ↑ Variable pause length
  • ↑ Percentage pause time
  • ↓ Speech pause ratio
  • ↓ Speaking rate
  • Temporal and prosodic features

Quatieri et al [26]
  • 35 with MDD
  • Free speech for prosodic measurements and distinct vowels for shimmer, jitter, and aspiration measurements
  • Depression severity and psychomotor retardation:
  • ↑ Shimmer
  • ↑ Aspiration
  • ↓ Harmonic-to-noise ratio
  • Depression severity
  • ↑ Jitter
  • ↓ Pitch variance
  • ↓ Average velocity
  • Prosodic, source, and filter features

Shinohara et al [27]
  • 30 with MDD and 14 in HC
  • Free speech and reading phrases aloud
  • Detection of emotional elements (ie, “anger” and “joy”)
  • 2 indices extracted: “vitality” and “mental activity”
  • MDD correlated with:
  • ↓ Vitality scores (effect size 1.03 and AUCd 0.76, sensitivity 0.93, and specificity 0.55)
  • Temporal and “emotional” prosodic features

Taguchi et al [28]
  • 36 with MDD and 36 in HC
  • 3 tasks: reading task, verbal fluency task, and reading task again
  • Significant discrimination between MDD and HC with MFCC 2 (sensitivity 77.8% and specificity 86.1%)
  • Spectral features

Wang et al [29]
  • 47 with MDD and 57 in HC
  • 4 tasks: video watching, question answering (natural speech and interview), text reading, and picture describing
  • Each task involved 3 emotional materials: positive, negative, and neutral
  • Patients with MDD (no matter which emotion or task was involved, with a large effect size):
  • ↓ Loudness
  • ↓ MFCC 5
  • ↓ MFCC 7
  • ↓ Fundamental frequency (F0) and MFCC 3 (in some scenarios with moderate effect size)
  • Prosodic and spectral features

Xu et al [30]
  • 45 with MDD, 43 with schizophrenia, and 41 in HC
  • Semistructured face-to-face interviews
  • Patients with MDD:
  • ↑ % of words related to the past and sadness
  • ↑ Conversational interruption
  • ↑ Response time
  • Temporal and prosodic features, as well as the type of words used

Yamamoto et al [31]
  • 97 with MDD, 68 with BD, and 76 in HC
  • 10 minutes of free speech
  • Depression severity:
  • ↓ Speech rate
  • ↑ Pause time
  • ↑ Response time
  • Temporal features
Nonverbal behavioral analysis studies

Alghowinem et al [32]
  • Database with depression status (Quick Inventory of Depressive Symptomatology, 16 Items)
  • Free interview
  • Eye activity
  • Head pose
  • Average recalls (mean of sensitivity and specificity) was 70% for detecting depression

Bhatia et al [33]
  • 47 with MDD
  • No task
  • Symptom severity was evaluated at 1, 7, 13, and 21 weeks
  • Head movement
  • Head movement synchrony did not change over the course of treatment with a change in depression severity

Dibeklioglu et al [34]
  • 49 with MDD
  • No task
  • Symptom severity was evaluated at 1, 7, 13, and 21 weeks
  • Facial movement
  • Head movement
  • Vocal prosody
  • For depression recognition: AUC was 67.25% for the fusion of head movement dynamics and vocal prosody
  • AUC was 73.16% for facial movement dynamics and vocal prosody
  • AUC was 77.77% for a combination of facial and head movement dynamics
  • AUC was 78.67% by fusion of all modalities

Fiquer et al [35]
  • 40 with MDD
  • Evaluation at baseline (T0) and after a 2 weeks’ tDCSe treatment (T1)
  • Describing their current mood during a 15-minute interview
  • Face and trunk recording
  • Ethogram (21 questions), 20 nonverbal categories: 10 indicative of high-energy and favorable disposition to social interaction (eye contact, illustrative gestures, symmetric smile, raised eyebrows, yes/no nodding, head up, head to side, verbal backchannel, body posture toward the interlocutor); 10 indicative of low energy, negative feelings, or social disinterest (folded arms, head down, shrug, asymmetric smile, adaptive gestures, crying, frown, tight lips, lips down, silence); 1 verbal category: speaking
  • Clinical improvement:
  • ↓ Head down, lips down, frown, and crying
  • ↑ Yes nodding and eye contact
  • Facial, head, and hand expressive movements were associated with the severity of depression

Fiquer et al [36]
  • 78 patients with MDD: 50 from the Netherlands and 28 from Brazil
  • During the HDRSf interview, face and trunk recording of both the patient and interviewer; scoring by a blinded observator
  • Speaking effort: patients’ behaviors
  • Encouragement: interviewers’ behaviors
  • No association between behavioral variables and baseline severity of depression
  • Patients who did not respond to treatment or did not remit: ⇧ speaking effort from before to after treatment

Fiquer et al [37]
  • 100 with MDD and 83 in HC
  • 41 with MDD treated by sertraline (25-200 mg) at the first hospital
  • 59 with MDD; 23 treated by escitalopram (10-20 mg) and 36 treated by tDCS in a second hospital
  • Semistructured 15-minute interview with general questions, face and trunk recording
  • 2 assessments: before treatment (T0) and after 8 weeks of treatment (T1)
  • Scoring by a blinded observator
  • Ethogram (21 questions) from Fiquer et al [35]
  • Patients with MDD:
  • ↑ Shrug, head, and lips down; adaptive hand gestures; frown, head, and lips down and cry
  • ↓ Asymmetric smile, eye contact, and smile
  • NVBg was not associated with depression severity and did not significantly change after depression treatment
  • Treatment responders at baseline:
  • ↑ Interpersonal proximity; head down; adaptive hand gestures; frown, cry, and folded arms; head to side; no nodding
  • ↓ Eye contact

Girard et al [38]
  • 18 with MDD
  • Recording during the first 3 interview questions (depressed mood, feelings of guilt, and suicidal ideation)
  • Facial expressivity defined by the Facial Action Coding System in terms of individual muscle movements called AUsh
  • With symptom severity:
  • ↓ AU 12 activity (smile and signal affiliative intent)
  • ↑ AU 14 activity (contempt, negative affect, and signal nonaffiliative intent)
  • ↑ Facial expressions associated with contempt

Girard et al [39]
  • 33 with MDD
  • Recording during the first 3 interview questions (depressed mood, feelings of guilt, and suicidal ideation)
  • Facial expressivity defined by the Facial Action Coding System in terms of individual muscle movements called AUs
  • With symptom severity
  • ↓ Affiliative facial expressions (AUs 12 and 15)
  • ↑ Nonaffiliative facial expressions (AU 14)
  • ↓ Head motion (ie, amplitude and velocity)

Guo et al [40]
  • 52 males and 52 females in the depressed group (Patient Health Questionnaire–9 >5)
  • 52 males and 52 females in HC (BDI <5)
  • Stimuli tasks of 3 emotional valences (watching film clips, replying to 9 free-response questions, reading 3 phonetically balanced passages containing affective content, and describing pictures)
  • Facial expressivity
  • Audio
  • For depression recognition:
  • Watching film clips showed the highest recognition rates (AUC up to 0.798 and 0.807)
  • Positive emotional stimuli greater than negative emotional stimuli

Jiang et al [41]
  • 12 with MDD
  • Evaluated before and after deep brain stimulation
  • Free interview
  • Facial expressivity (7 basic emotions)
  • AUC 0.75 detecting response to treatment

Kacem et al [42]
  • 49 with MDD
  • No task
  • Symptom severity was evaluated at 1, 7, 13, and 21 weeks
  • Facial movement
  • Head movement
  • For depression recognition:
    • Facial movement was greater than head movement (AUC 66.19% vs 61.43%)
    • With 2 modalities combined, AUC 70.83%
HRV studies

Adolph et al [43]
  • 85 outpatients with suicidal ideation
  • ECGi:
    • At rest quietly for 3 minutes: “Resting HRV”
    • After watching a sad film: “HRV reactivity”
  • HFj-HRV reactivity but not HF-HRV at rest was predictive of higher scores on suicidal ideation
  • Frequency domain: HF

Fernandes et al [44]
  • 50 with MDD
  • ECG at rest
  • Video for positive and negative NVB
  • HF and RMSSDk were positively correlated with positive NVB
  • Negative NVB was not associated with HRV
  • Frequency domain: HF and LFl
  • Time domain: RMSSD
  • Ethogram (Fiquer et al [35])

Giurgi-Oncu et al [45]
  • 78 with MDD
  • 24-hour Holter ECG
  • Evaluation at 1 and 6 months after therapy (sertraline)
  • ↑ HRV at 1 and 6 months after selective serotonin reuptake inhibitors
  • Frequency domain: HF
  • Time domain: SDNNm and RMSSD

Hartmann et al [46]
  • 62 with MDD and 65 in HC
  • 2 time measures: before and after 2 weeks of antidepressant treatment
  • 15 minutes of resting ECG 1-2 days before treatment
  • At baseline compared with HC: ↓ HRV (HF, LF, SD1n, and RMSSD)
  • After treatment: HRV normalized in MDD for HF, LF, SD2o, and SD1/SD2 ratio
  • Frequency domain: LF, HF, and LF-to-HF ratio
  • Time domain: SDNN, RMSSD, pNN50p
  • Nonlinear: SD1, SD2, SD1-to-SD2 ratio

Kircanski et al [47]
  • 722 with MDD, 309 with anxious depression (according to the HDRS interview), and 413 without
  • Outcomes at 8 weeks
  • 2-minute seated ECG recording, first with eyes open and second with eyes closed
  • In anxious depression, better treatment response if: ↑ HRV in pretreatment
  • In nonanxious depression, better treatment response if: ↓ HRV in pretreatment
  • Time domain: RMSSD
  • Heart rate

Lee et al [48]
  • 34 with MDD
  • Outcome at 12 weeks of antidepressant treatment
  • 5-minute ECG
  • Positive correlation at baseline:
  • HDRS items 14 and 15 with LF-to-HF ratio
  • Positive correlation at the endpoint:
    • HDRS item 5 and LF
    • HDRS items 7 and 13 (fatigue-related item) and LF
    • HDRS item 8 and LF, SDNN, and RMSSD
    • HDRS total, LF, and HF
  • Frequency domain: VLFq, LF, HF, LF-to-HF ratio
  • Time domain: SDNN, RMSSD

Neyer et al [49]
  • 50 with MDD
  • ECG at rest for the HRV measure
  • Measures before and after treatment
  • Depressive symptoms improved without any change in HRV
  • Frequency domain: HF
  • Time domain: RMSSD

Sarlon et al [50]
  • 89 with MDD
  • HRV: blood volume pulse finger clip sensor
  • Skin conductance: Velcro tape with integrated Ag/AgCl electrodes
  • Skin temperature: Nexus temperature
  • Beat-to-beat interval: elastic belt with a breathing sensor
  • 3 conditions: baseline sitting for 5 minutes, with emotion-induced stressors (recall an unpleasant stressful experience), and relaxed state after 300 seconds
  • No association was found between HRV and symptom severity
  • Frequency domain: VLF, LF, HF, LF-to-HF ratio
  • Time domain: SDNN, RMSSD, beat-to-beat interval
  • Skin conductance
  • Skin temperature
EDA studies

Kim et al [51]
  • 30 with MDD and 37 in HC
  • ProComp Infiniti (SA7500, computerized biofeedback system, thought technology)
  • EDA during 5 experimental phases: baseline, mental arithmetic task, recovery from the stress task, relaxation task, recovery from the relaxation task
  • Classifying participants with MDD versus controls: 74% accuracy, 74% sensitivity, 71% specificity
  • Stress and relaxation tasks were the most relevant
  • MSCLr, SDSCLs, SKSCLt, and NSSCRu

Kim et al [52]
  • 30 with MDD and 31 in HC
  • As per Kim et al [51]
  • Classifying participants with MDD versus controls: 70% accuracy, 70% sensitivity, 71% specificity
  • MSCL, SDSCL, SSCLv, MSCRw, NNSCRx, and poststroke depression

Litwińska-Bołtuć et al [53]
  • 97 with MDD
  • Follow-up at 1 year
  • EDOR (Electro Dermal Orienting Reactivity) test (Emotra AB)
  • Hyporeactive patients: relapse or recurrence of depression was nearly 5 times higher
  • MSCR

Pedrelli et al [54]
  • 31 with MDD
  • Follow-up at 8 weeks
  • 2 E4 Empatica wristbands, one on each wrist for 22 hours/day, 7 days/ week
  • Smartphone sensor data: movisensXS phone app (movisens GmbH)
  • Correlations between the models’ estimate of HDRS scores and clinician-rated HDRS: from ORy 0.46 (CI 0.42-0.74) to OR 0.7 (CI 0.66-0.74)
  • Empatica:
    • EDA: MSCR, PSCRz, AASCRaa
    • Peripheral skin temperature
    • Heart rate
    • 3-axis accelerometer and sleep characteristics
  • Smartphone data: mobile-based social interactions (number of calls, SMS text messages), activity patterns (walking), number of apps used

Smith et al [55]
  • 11 with MDD
  • 16 in HC
  • Shimmer3 GSR+ unit with Shimmer Optical Pulse Sensing Probe; at rest with eyes closed for 3 minutes
  • 81% accuracy for detecting depression with HRV
  • No benefit to including skin conductance response to improve accuracy
  • HRV
  • Time domain: SDNN, beat-to-beat interval
  • Nonlinear: SD1, SD2
  • Skin conductance response: MSCR, SDSCRab, PSCR, and AASCR

aHC: healthy control.

bMFCC: mel-frequency cepstral coefficient.

cBD: bipolar disorder.

dAUC: area under the curve.

etDCS: transcranial direct current stimulation.

fHDRS: Hamilton Depression Rating Scale.

gNVB: nonverbal behavior.

hAU: action unit.

iECG: electrocardiogram.

jHF: high frequency

kRMSSD: root-mean-square surface distance.

lLF: low frequency

mSDNN: SD of the NN (R-R) intervals.

nSD1: SD of points perpendicular to the major axis of the Poincaré plot.

oSD2: SD of points along the major axis of the Poincaré plot.

pPNN50: proportion of NN50 divided by the total number of NN (R-R) intervals.

qVLF: very-low frequency.

rMSCL: mean amplitude of the skin conductance level.

sSDSCL: standard deviations of the skin conductance level.

tSKSCL: skewness of the of the skin conductance level.

uNSSCR: nonspecific skin conductance response.

vSSCL: slope of the skin conductance level.

wMSCR: mean amplitude of the skin conductance response.

xNNSCR: number of nonspecific skin conductance responses.

yOR: odds ratio.

zPSCR: peak skin conductance response.

aaAASCR: average amplitude skin conductance response.

abSDSCR: standard deviations of the skin conductance response.

Major Depressive Disorder

Speech Analysis

Prosodic abnormalities in patients with depression are well known and a monotonous speech or reduced prosody can be easily and frequently heard. It is assumed that cognitive and physiological changes in depression could affect speech production, and changes in the automatic and somatic nervous system cause disturbances in muscle tension and respiratory rate. These changes will not only influence vocal folds and vocal tract dynamics, but also constrain articulatory movement [56].

Speech analysis, which consists of automatically extracting vocal and linguistic features from audio signals, offers the possibility to detect and measure these changes and serves as an additional objective assessment of depression. Traditionally, acoustic features can be divided into 5 types: temporal, prosodic, spectral, source, and filter features. Temporal features, defined as speech prosodic timing measures, appear to be a promising measure of depression and its severity [57].

Yamamoto et al [31] used 3 temporal features in a prospective follow-up study: speech rate, pause time, and response time. They found a correlation with scores on the Hamilton Rating Scale for Depression (HAM-D). Mundt et al [25] found 5 prosodic timing measures to be significantly correlated with depression severity: total speech time, total pause time, percentage pause time, speech pause ratio, and speaking rate. Xu et al [30] found that people with depression have a longer response time and interrupt their communication partner more often. To summarize, people with depression used a much larger percentage of words related to the past and sadness emotions. Similarly, Shinohara et al [27] used a “vitality” score based on emotional elements of speech (ie,  joy, anger) and showed a negative correlation between the vitality score and HAM-D scores. Arevian et al [22] analyzed the types of word used and found, among others, more negative emotion and less complex word use in patients with depression.

Finally, Alghowinem et al [21] and Hönig et al [23] found that the average syllable duration could be positively correlated with depression severity.

With prosodic features in people with depression, reduced fundamental frequency (F0) range and average are frequently found [23,24,26,58]. F0 corresponds to the lowest frequency of the speech signal perceived as pitch. It could explain the depressive monotonous speech and could be a consequence of disturbances in laryngeal muscle tension due to psychomotor retardation. The latter may also explain reduction in F0 variability [56,57]. However, some studies also report no significant correlation between F0 variables and depression [25,28].

Several studies on depression showed a shift in spectral energy, and mel-frequency cepstral coefficients (MFCCs) are often used as spectral features. Spectral features characterize the spectrum of speech, which at a given time correspond to the frequency distribution of the speech signal. Taguchi et al [28] showed that the second dimension of MFCC allowed a significant discrimination between patients with depression and controls. Further, Wang et al [29] found that MFCC 5, MFCC 7, and loudness were consistently lower in people with depression. According to the task designed during speech recording, MFCC 3 and F0 were significantly lower than in people with depression than in healthy people.

Finally, source features such as harmonic-to-noise ratio (HNR), jitter, and shimmer also tend to increase with depression severity, supporting the hypothesis of a more breathy phonation in depressed speech [26]. This could be explained by a more open and turbulent glottis, linked to a reduction in laryngeal muscle tension. The study by Hönig et al [23] also supports this hypothesis. Alghowinem et al [21] showed that not only shimmer and jitter but also loudness and MFCC features are part of the general characteristics of depressive speech.

Nonverbal Behavior Analysis

Traditionally, evaluation of clinical depression is based on patients’ verbal information from psychological interviews and rating scales or self-report questionnaires. However, verbal analysis has several limitations. First, patients may underreport or overreport depressive symptoms due to different reasons (eg, social stigma). Second, it can be assumed that most human communication is taking place through nonverbal behavior. Indeed, a large part of nonverbal behavior is mostly outside of conscious control, and these cues and signals may differ from verbal reports. Facial expression, gestures, and body postures are mainly involuntary, and therefore represent a privileged way of expressing feelings and emotions [35,59]. Girard et al [38,39] showed that when the severity of depressive symptoms was high, participants made fewer affiliative facial expression such as smiling, and more nonaffiliative facial expression such as contempt. They also showed that patients exhibited diminished head motion (amplitude and velocity). These results support the social withdrawal hypothesis in depression. In fact, rather than affective valence with an increase or decrease in facial expression, it is the social-communicative value with affiliative expression that seems to be affected in depression. Thus, nonverbal behavior may serve to maintain interpersonal distance while facilitating social withdrawal.

Fiquer et al [35] supported the assumption that depression involves behaviors related to social withdrawal and negative feelings. Crying, asymmetric smiling, and motionless head/eyebrows were associated with higher levels of depression severity, while patients’ illustrative gestures, defined as “the hand and arm movements used to support the speech,” were associated with a lower level of depression severity.

Moreover, decrease of negative facial and head indicators and increase of eye contact and yes nodding accompanied clinical improvement, whereas illustrative gestures did not [35]. In another study, Fiquer et al [37] compared nonverbal behavior in patients with depression versus controls. They showed that patients with depression displayed higher levels of negative nonverbal behaviors (shrug, head and lips down, adaptative hand gesture, frown, and cry) and lower levels of positive nonverbal behaviors (eye contact and smile). The authors raise the important role of these behaviors in social withdrawal [37]. Nevertheless, the aforementioned nonverbal behaviors was not associated with depression severity before treatment, which suggests its independence.

The authors concluded that certain typically found nonverbal behaviors could represent a predisposition to depression possibly influenced by personality features. These social symptoms expressed through nonverbal behavior may be the last ones to vanish [37].

On the contrary, Fiquer et al [36] studied nonverbal variables such as speaking effort, encouragement, attunement, and change in attunement, and found no association between these behaviors and the severity of depression. Finally, several recent studies achieved classifying depression with high accuracy using video analysis with head and facial movements [34,42], only head movement [33], facial expressivity [41], or with head pose and eye activity [32]. Guo et al [40] found that the highest accuracy of depression recognition was when patients were watching film clips of positive and negative emotional stimuli compared with other tasks.

In a recent paper, Alghowinem et al [60] used feature selection methods and found that some features have a high capability for distinguishing between depression severities. For speech analysis, the strongest features were the temporal features, F0, HNR, formants, and MFCC; and for behavioral analysis, it was the left-right eye movement and gaze direction and the yaw head movement. The authors showed that these features outperformed all other features in depression detection.

The studies mentioned so far demonstrate the usefulness of behavioral analysis in understanding the mechanisms that underlie the onset and course of depression. These analyses are less influenced by conscious control compared with verbal communication [59], which could make them a reliable objective measure.

Physiological Measures
Heart Rate Variability

HRV corresponds to the beat-to-beat variations in the heart rate over a given period. HRV appears to be a good reflection of the heart’s ability to modulate its rhythm in response to external and internal stimuli. HRV is regulated by the autonomic nervous system and, as a result, can inform of its functioning. HRV is related to the influence of the parasympathetic nervous system (PNS) and the sympathetic nervous system (SNS) [61]. Three parameters are commonly used: high-frequency HRV (HF-HRV; 0.15-0.40 Hz), low frequency HRV (LF-HRV; 0.04-0.15 Hz), and the LF-to-HF ratio. HF-HRV is mainly under the control of the PNS, whereas LF-HRV is under the influence of both PNS and SNS, but mostly the SNS. The LF-to-HF ratio is considered an index of sympathovagal balance [61].

In addition, time-domain parameters such as the root-mean-square of successive differences (RMSSD) and the SD of normal-to-normal intervals (SDNN) are frequently used [62]. Normal-to-normal is the interval between 2 heartbeats. HRV can be an indicator of the ability to regulate stress and emotions, which is particularly relevant in psychiatric disorders [63]. Furthermore, a good cardiovascular adaptability can be reflected by a high HRV [64], while LF can be a risk factor of cardiovascular diseases [65]. A meta-analysis in depression [66] showed significant reductions in frequency domain parameters such as HF, LF, and very-low frequency (VLF) HRV as well as a significantly higher LF-to-HF ratio compared with healthy controls. Significant reductions in time domain parameters were further found with reduced RMSSD, SDNN, and interbeat intervals.

Some recent studies have focused on different clinical subdimensions of MDD. Lee et al [48] showed a positive correlation between fatigue-related items of the Hamilton Depression Rating Scale (HDRS) and LF-HRV. The latter was positively correlated with the items “midnight insomnia” and “slower thinking.” In the same line, Adolph et al [43] found that HF-HRV in reaction to watching a sad film was predictive of higher scores on suicidal ideation, whereas HF-HRV at rest was not predictive of suicidal ideation. Further studies investigated the relationship between HRV and treatment response. First, Hartmann et al [46] found that patients with MDD normalized their HRV after treatment, especially for HF-HRV and LF-HRV. Further, Giurgi-Oncu et al [45] showed a normalization of HRV parameters (including HF, RMSSD, and SDNN) with the depression treatment by therapy and medication. Kircanski et al [47] predicted treatment success among patients with depression with or without anxiety. The authors found that a higher HF-HRV in pretreatment had better outcomes than lower HF-HRV in anxious depression, whereas lower HF-HRV had better outcomes than higher HF-HRV in nonanxious depression.

By contrast, Sarlon et al [50] found no association between HRV and depressive symptoms severity. Neyer et al [49] showed in a prospective study with HRV measures before and after treatment that depressive symptoms improved without change in HRV, suggesting a more complex relationship.

Finally, Fernandes et al [44] explored the association between HRV and nonverbal behavior using a validated ethogram. Results showed a positive correlation among HRV (HF and RMSSD) and positive nonverbal behaviors, but no association between HRV and negative affect. The authors suggested that HRV and nonverbal behaviors could be regulated by vagal activity, which may be related to the social engagement system.

Electrodermal Activity

The most basic indicators of the state of the autonomic nervous system are heart rate and EDA. EDA can measure electrical conductance of the skin, which depends on the quantity of sweat secreted by glands in the hypodermis and reflects sympathetic nervous activity.

EDA has a tonic and a phasic component. The former is measured in the skin conductance level (SCL) and the latter with the skin conductance response (SCR) [67]. EDA can be used as an indicator for emotional reactivity [68]. In fact, rather than the valence of emotion, it is the intensity that seems related to EDA [69] and negative stimulation induces more extensive reactions than positive [70]. Concerning emotional reactivity, we distinguish labile and stable patients for EDA. EDA labile patients could be described as calm and deliberative, whereas EDA stable patients tend to be more irritable, emotionally expressive, and active. EDA labile patients would be dependent on anxiety traits and better control for potentially threatening stimuli [71].

Several studies show that EDA could be an indication of nonconscious emotional processes [72-74]. A recent systematic review [67] showed lower EDA, especially lower SCL and SCR, in patients with depression compared with healthy controls. Others studies report that, compared with patients with depression also experiencing agitation, those with psychomotor retardation or symptoms of inhibition have lower EDA levels [75-77]. Moreover, patients with “psychotic ” and “endogenous” depression could have lower EDA than “nonendogenous” depression [77-79].

Other recent studies showed moderate to high accuracy in classifying patients with MDD and healthy controls [51,52], such as that by Smith et al [55], but the latter used HRV and SCR for depression detection and found no benefit in including SCR to improve accuracy. In addition, Pedrelli et al [54] showed that EDA and HRV features from wearables as well as activity level and sleep parameters could provide an estimate of changes in severity of depressive symptoms.

In a cohort of patients with MDD at 1-year follow-up, Litwińska-Bołtuć et al [53] concluded that SCR hyporeactivity was associated with recurrent episodes and relapse, and recurrence of depression was almost 5 times higher than that in the reactive patients.

Concluding Remarks

Based on these aforementioned studies, the digital phenotype of MDD could be identified as follows: a reduction in certain speech temporal features (ie, speech rate or speech time), an increase in others (ie, pause time or response time) [25,31], and a change in the type of words used (ie, less complex and more emotionally negative) [22,27]. In addition, prosodic features (ie, F0 range and average) [23,24,26,58], variability [56,57], spectral features [28,29], and source features (ie, HNR, jitter, shimmer [21,26]) are changed.

The behavior of patients with depression could be characterized by a more negative nonverbal behavior (ie, head motion, facial expressivity, and hand or body gestures) [35,38,39]. But more than the valence, it seems that it is the affiliative behaviors that are impacted, supporting a social withdrawal [38,39]. In fact, certain patterns of behavior when interacting with others may predict the severity of symptoms or treatment response [36,59,80-83]. Finally, physiological measures are mainly lowered for both HRV (ie, HF, LF, VLF, and time domains) [66] and EDA (ie, SCR and SCL) [67].

Bipolar Disorder

Speech Analysis

Several studies managed to classify the course of mood episodes or relapses with high confidence. For instance, speech pause duration [84] and number of longer pauses [85] are significantly higher in depressive states than in a hypomanic or euthymic state. Concerning prosodic and source features, Guidi et al [85-87] repeatedly found an increase of F0 in hypomanic states compared with euthymic or depressive states. Similar results were also obtained by Vanello et al [88]. Likewise, filter features seem to increase with manic mood [89], especially F1 and F2 formants, which correspond to the first and the second peak in the spectrum that results from a resonance of the vocal tract.

Finally, many studies extracted several speech features (ie, the openSMILE toolkit), achieving good classification accuracies [90,91]. Classification of depressive states could be obtained with an area under the curve (AUC) value of 0.78 and manic and mixed states with an AUC value of 0.89 [92].

Physiological Measures
Heart Rate Variability

A meta-analysis [93] found reduced HRV compared with healthy controls. In particular, LF-HRV was reduced but no differences in either HF or LF-to-HF ratio were found. However, these analyses did not consider different mood phases. Recent studies were interested in the exploration of HRV during a specific phase of BD. First, Wazen et al [94] showed that several HRV parameters (HF-HRV, time-domain, and non-linear domain measures) could increase when patients go from the mania phase to the euthymic phase. Conversely, Faurholt-Jepsen et al [93] reported an increased HRV in manic states compared with depressive and euthymic states, but no difference between the depressive and euthymic states. Hage et al [95] found that LF-HRV and heart period were significantly lower in patients with BD compared with healthy controls. However, after 8 weeks of treatment, there were no significant changes in HRV parameters.

The link between disease severity and HRV has also been explored. Benjamin et al [96] found a significant association between HF-HRV and disease severity. In the same way, Ortiz et al [97] found that longer illness duration, higher number of depressive episodes, longer duration of most severe manic/hypomanic episode, comorbid anxiety disorders, and a family history of suicide were associated with a reduced HRV. Moreover, the severity during a depressive episode was associated with lower HRV. Finally, Freyberg et al [98] compared 20 newly diagnosed BD cases, 20 unaffected first-degree relatives, and 20 healthy controls, and demonstrated that HRV did not differ in any measures between the 3 groups.

Lastly, 2 studies have tried to distinguish bipolar depression and MDD, in terms of HRV. First, Chang et al [99] studied a population of patients with depression and BD type II and found that, compared with patients with MDD, they had lower total HRV, with lower HF and LF-HRV but higher LF-to-HF ratio. The authors concluded that HRV may aid in the differential diagnosis of bipolar depression type II and unipolar depression. Second, Hage et al [100] supported these findings by comparing patients with MDD with those with BD depression including BD type I, type II, and “not otherwise specified.” Indeed, patients with MDD had significantly higher baseline sinus arrythmia (ie, HF-HRV) and LF-HRV in comparison to patients with BD.

Electrodermal Activity

Greco et al [101] showed that EDA levels could differentiate the phases of BD (depressive, euthymic, and mixed states). Indeed, electrodermal hypoactivity could be a rather stable trait of patients with depression and may reflect euthymia or remission when EDA increases [101]. Lemaire et al [102] measured the intensity and duration of physiological responses to affective stimuli. For this purpose, they used EDA and the heart interbeat intervals. The authors concluded that affective dysregulation, a key dimension of BD, could be measured as an increased affective reactivity to neutral pictures and reduced maintenance of subjective affective responses to all type of pictures, irrespective of the clinical state.

Concluding Remarks

For the digital phenotype of BD, we found that, in the depressed phase, speech changed in temporal (ie, decreased speech pause) and prosodic features (ie, decreased F0), but both increased when patients go into the hypomanic phase [84-88].

Independent of the phase, HRV appears reduced (ie, LF) [93], but the change in HRV between phases remains unclear [94,95,103]. For EDA, an electrodermal hypoactivity in a depressive state is reported, which increases when patients move toward euthymia [101].

PTSD and Psychological Trauma

Speech Analysis

The link between trauma and voice has been likewise explored. Monti et al [104] reported a significant relationship between voice fundamental frequency and the “total trauma” reported with the Childhood Trauma Questionnaire. Interestingly, after trauma recall, vocal jitter and voice irregularity (roughness) were strongly related to the existence of childhood trauma [104]. This suggests that variations in vibratory patterns of the vocal folds could have a relationship with childhood trauma recall. In another study, Monti et al [105] also found an association of anxious attachment and childhood neglect with intensity/loudness measures in singer’s voice. Anxious attachment was also positively correlated with jitter and irregularity. Moreover, in speech content, trauma narrative shows an increased use of first-person singular pronouns [106,107] and changes in narrative coherence [108,109].

As in depression, voice markers including F0, jitter, shimmer, and HNR are also found in PTSD [110,111]. Speech rate in both depression and PTSD was shown to be negatively correlated with severity of symptoms. Reduced tonality in vowel space is found in self-reported symptoms of both PTSD and depression [112]. However, in PTSD without MDD, voice markers showed slower and more monotonous speech and less change in tonality [113]. Finally, a recent study showed that audio intensity and reduced pitches per frame could predict PTSD, whereas reduced frequency of positive words seemed to rather predict depression [106].

Nonverbal Behavior Analysis

A core symptom of PTSD is an exaggerated startle response [2]. In their study on patients with PTSD, McTeague et al [114] focused on the eye-blink startle response to narrative texts corresponding to traumatic stories. The authors found a heightened startle reflex response during trauma imagery, which was also evident in reactivity to anger, panic, or physical danger compared with controls. Surprisingly, participants with multiple traumas showed a blunted reactivity compared with participants with a single trauma. Furthermore, participants with multiple traumas had more chronic and severe PTSD with more anxiety and mood comorbidity [114].

In the same paradigm, Blechert et al [115] studied the bodily startle response to electric stimuli in individuals with trauma, with and without PTSD. The results implied a primary response characterized by reflexlike facial and torso movement and a secondary response characterized by an emotional facial response. Besides, participants with PTSD had a stronger initial startle response and a more negatively valenced secondary facial expression compared with controls with trauma. Beyond the startle response to threat, several studies with eye tracking showed that higher levels of PTSD symptoms were associated with an increased attentional bias toward threat [116]. Some authors assumed that negative affect in PTSD could have an important role in maintaining this attention bias [117].

More generally, Katz et al [118] emphasized the importance of trauma disclosure. The authors analyzed 40 video interviews of children who have suffered an abuse, but only half of them disclosed it. Children who did not disclose the abuse showed specific nonverbal cues, such as more physical disengagement, than children who disclosed. Schultebraucks et al [106] performed deep learning audio- and video-based classification of depression versus PTSD for 81 participants, 1 month after their admission to a trauma center following a traumatic exposure. The authors successfully classified between PTSD and MDD status with a high discriminatory accuracy. Likewise, several voice and video markers have been identified as predictive of either PTSD or MDD. In a video analysis, higher fear and anger expressivity were predictive of PTSD, while higher contempt expressivity was important for the classification of MDD. In an audio analysis, an increased use of first-person singular pronouns, lower audio intensity, and reduced pitch were important features for the PTSD classification, whereas a reduced frequency of positive words was predictive of depression.

Physiological Measures
Heart Rate Variability

Several studies have investigated HRV in PTSD and suggested reduced parasympathetic activity in PTSD. Indeed, 2 meta-analyses found that HRV is lower both at rest and during stress in PTSD compared with controls. Particularly, RMSSD and HF-HRV were reduced, while SDNN and LF-HRV were also reduced, but with a smaller effect. Furthermore, a higher LF-to-HF ratio was found in PTSD [119,120]. Another meta-analysis [121] investigated the relationship between early life maltreatment (ELM) and resting-state HRV. Although no evidence for a relationship was found in comparative and correlational meta-analyses, the meta-regression analysis disclosed interesting results regarding both age and presence of psychopathology. In fact, studies including older-aged samples reported greater reductions in resting-state HRV in association with more severe ELM. Moreover, in clinical samples, compared with nonclinical samples, the authors found stronger reductions in resting-state HRV in ELM-exposed individuals compared with those nonexposed. The authors assumed that ELM could lead to alterations in the development of the autonomic nervous system, especially to an insufficient increase in vagal activity during childhood and adolescence, leading particularly to deficiencies in affective regulation that could increase the risk for psychopathology [121].

Finally, Stone et al [122] compared HF-HRV in women with depression with and without a history of childhood emotional abuse (CEA) and in controls without CEA, and reported that women with depression with CEA exhibited lower HF-HRV than both groups without CEA.

Electrodermal Activity

In patients with PTSD, higher subjective levels of intrusive reexperiencing have been linked to physiological reactivity during trauma recall, while acute dissociative symptoms could show reduced physiological reactivity to trauma memories. Actually, increased physiological reactivity was not associated with the self-report of memory avoidance or dissociation but rather with the degree of memory intrusiveness felt [123,124]. Thus, it can be hypothesized that persistently high physiological reactivity may facilitate chronic arousal dysregulation that disrupts neural systems and could lead to maintain the intrusiveness of remembering [125,126].

Patients with PTSD showed an increased physiological reactivity during trauma recall, while patients with depression had a low physiological response to the memory of an event that they associated with the onset of their depression [126]. Although some researchers suggest a possible attenuation of physiological reactivity in patients with PTSD by depressive symptoms [114], when comparing trauma recall between PTSD with and without depression, no blunting effect was found by depressive mood on physiological reactivity [126]. D’Andrea et al [127] studied individuals exposed to trauma and showed different autonomic reactivities in response to startling sounds according to the type and symptomatology of trauma. Indeed, blunted autonomic reactivity was found in participants with significant symptoms and early exposure to multiple traumas. Conversely, more attenuated trauma exposure had large heart rate acceleration and SCRs.

Concluding Remarks

In PTSD, modifications in speech prosodic and source features are found as in depression [104,110-113], but specific changes are reported in the speech content, especially during trauma narrative or trauma recall, that is, use of first-person singular pronouns [105,106], and changes in narrative coherence [108,109].

In the same way, behavior analysis shows a specific startle response during trauma recall (ie, eye blink or bodily startle) [114,115]. Besides, facial expressivity with higher fear or anger could be predictive of the presence of PTSD [106].

In physiological measures, for HRV, PTSD studies indicated a reduced parasympathetic activity (ie, HF, LF, and time domains) [119,120] as in depression. However, the presence of trauma in people with depression could further decrease HRV (ie, HF) [122]. For EDA, the presence of PTSD, especially with memory intrusiveness felt, leads to an increased EDA reactivity during trauma recall while acute dissociative symptoms could have the opposite effect [123,124,126].


Principal Expectations

In this review, we were interested in the state-of-the-art application of new digital measurements to assess symptoms in patients with several psychiatric conditions. In particular, we explored digital markers through speech and behavioral analysis, as well as physiological measures such as HRV and EDA. We focused our research on depression and its different clinical characteristics.

Limitations

Although several databases have been included, relevant papers may have been missed due to the choice of keywords. This paper is not intended to be a systematic review and a broader review could explore other interesting areas. We included a wide range of study designs due to the exploratory nature of our review but the design of the included studies also limited the review. Other reviewers have attempted to reduce the study selection bias but they have acknowledged the possibility of subjectivity in their analyses of the findings.

In addition, we noticed that overall, most studies only investigated the use of 1 digital measure, except for a few that tried to combine, for instance, the measure of EDA with HRV or speech with behavioral analysis.

Apart from a few ethological analyses, most studies analyzed the patient alone, without being in interaction. However, social interaction, especially during clinical interviews, seems to be the most important source of information for clinical assessment.

Moreover, we frequently found that each study used different measures that raise a problem of harmonization. For example, in speech analysis, some studies assessed the type of word used, prosodic or spectral features, or even analyzed a “toolkit” with hundreds of features. Further, certain studies investigated features extracted from free speech, whereas others focused on more constrained speech tasks.

Video analysis can be heterogeneous as well, in that some studies are interested in eyes, head, or torso movements at rest or during an evaluation. In HRV assessment, several studies used electrocardiogram but others used a finger clip sensor. In the same way, EDA can be measured with or without a concurrent task and with different types of technological tools, with variable degrees of precision.

Finally, we did not take into consideration articles dealing with depression with comorbid anxiety. Future research on this topic would be relevant to complete our findings.

Implications for Practice

Assuming that digital markers would help in the differential diagnosis for an MDE, clinicians will be able to make an early diagnosis. Thus, psychotherapy and drug treatment could be introduced early, and diagnostic and therapeutic mistakes could be limited.

Despite the lack of comparative studies, promising digital markers appear to emerge from this review, particularly within physiological measures such as HRV. In fact, patients with bipolar depression had more severe decrease than those with unipolar depression (ie, HF and LF-HRV) [99,100], whereas the presence of comorbid PTSD in MDD seems to induce a greater reactivity of physiological measures, especially when the clinician touched upon psychological trauma [126]. This reactivity with PTSD also seems to be reflected in speech and nonverbal behavioral analysis, which suggests other candidates for digital markers [108,109,114,115].

Conclusion and Future Directions

Despite many limitations, the studies detailed in our review showed promising results on the usefulness of digital phenotyping for the differential diagnosis of MDE. The authors believe that additional research is needed to better understand the potential value of digital phenotyping in clinical practice. Indeed, what should be relevant is a prospective and observational study in which different digital tools will be combined during recordings of social interaction. At least three distinct groups would be required: MDD, bipolar depression, and depression comorbid with PTSD. For each group, it would be interesting to merge speech, behavior, and physiological measures extracted from recordings of clinical interaction between patients and clinicians. By designing larger and comparative studies, a distinct digital phenotype could be defined for each of the aforesaid 3 groups. Thus, digital markers to help differential diagnosis could emerge.

Moreover, if large prospective studies are conducted, validated digital markers could be used for follow-up and for predicting treatment response or risk of relapse. Digital phenotyping could therefore contribute to personalized and precision medicine.

Acknowledgments

This research is funded by the MEPHESTO project (BMBF grant number 01IS20075; DRI-0120105/291). We thank the “Innovation Alzheimer” Association for its financial support to the publication.

Authors' Contributions

EE, AK, PM, JH, BG, DP, HL, MB, AL, and PR conceptualized the study. EE and AK wrote and prepared the original draft. AK, PM, JH, and PR provided visualization and supervision support. All authors have read and agreed to the published version of the manuscript.

Conflicts of Interest

None declared.

Multimedia Appendix 1

Search strings.

DOCX File , 72 KB

Multimedia Appendix 2

Summary of speech analysis, nonverbal behavioral analysis, HRV, and EDA studies in individuals with BD, PTSD or psychological trauma. BD: Bipolar Disorder; EDA: electrodermal activity; HRV: heart rate variability; MDD: major depressive disorder; PTSD: Post Traumatic Stress Disorder.

DOCX File , 34 KB

  1. GBD 2017 DiseaseInjury Incidence Prevalence Collaborators. Global, regional, and national incidence, prevalence, and years lived with disability for 354 diseases and injuries for 195 countries and territories, 1990-2017: a systematic analysis for the Global Burden of Disease Study 2017. Lancet 2018 Nov 10;392(10159):1789-1858 [FREE Full text] [CrossRef] [Medline]
  2. Diagnostic and statistical manual of mental disorders (DSM), 5th Edition. Washington, DC: American Psychiatric Association; 2013:XIV-947.
  3. Ostergaard SD, Jensen SOW, Bech P. The heterogeneity of the depressive syndrome: when numbers get serious. Acta Psychiatr Scand 2011 Dec;124(6):495-496. [CrossRef] [Medline]
  4. Gaynes BN, Warden D, Trivedi MH, Wisniewski SR, Fava M, Rush AJ. What did STAR*D teach us? Results from a large-scale, practical, clinical trial for patients with depression. Psychiatr Serv 2009 Nov;60(11):1439-1445. [CrossRef] [Medline]
  5. Hirschfeld R. Differential diagnosis of bipolar disorder and major depressive disorder. J Affect Disord 2014 Dec;169:S12-S16. [CrossRef]
  6. Judd LL, Akiskal HS, Schettler PJ, Endicott J, Maser J, Solomon DA, et al. The long-term natural history of the weekly symptomatic status of bipolar I disorder. Arch Gen Psychiatry 2002 Jun 01;59(6):530-537. [CrossRef] [Medline]
  7. Angst J, Merikangas KR, Cui L, Van Meter A, Ajdacic-Gross V, Rössler W. Bipolar spectrum in major depressive disorders. Eur Arch Psychiatry Clin Neurosci 2018 Dec 21;268(8):741-748. [CrossRef] [Medline]
  8. Hirschfeld RMA, Lewis L, Vornik LA. Perceptions and Impact of Bipolar Disorder. J Clin Psychiatry 2003 Feb 15;64(2):161-174. [CrossRef]
  9. Galvão F, Sportiche S, Lambert J, Amiez M, Musa C, Nieto I, et al. Clinical differences between unipolar and bipolar depression: interest of BDRS (Bipolar Depression Rating Scale). Compr Psychiatry 2013 Aug;54(6):605-610. [CrossRef] [Medline]
  10. Rytwinski NK, Scur MD, Feeny NC, Youngstrom EA. The co-occurrence of major depressive disorder among individuals with posttraumatic stress disorder: a meta-analysis. J Trauma Stress 2013 Jun 20;26(3):299-309. [CrossRef] [Medline]
  11. Gootzeit J, Markon K. Factors of PTSD: differential specificity and external correlates. Clin Psychol Rev 2011 Aug;31(6):993-1003. [CrossRef] [Medline]
  12. Gros DF, Price M, Magruder KM, Frueh BC. Symptom overlap in posttraumatic stress disorder and major depression. Psychiatry Res 2012 Apr 30;196(2-3):267-270 [FREE Full text] [CrossRef] [Medline]
  13. McFarlane AC, Papay P. Multiple diagnoses in posttraumatic stress disorder in the victims of a natural disaster. J Nerv Ment Dis 1992 Aug;180(8):498-504. [CrossRef] [Medline]
  14. Alarcon RD, Glover SG, Deering CG. The cascade model: an alternative to comorbidity in the pathogenesis of posttraumatic stress disorder. Psychiatry 1999 Sep 26;62(2):114-124. [CrossRef] [Medline]
  15. Insel TR. Digital Phenotyping: Technology for a New Science of Behavior. JAMA 2017 Oct 03;318(13):1215-1216. [CrossRef] [Medline]
  16. Waring OM, Majumder MS. Introduction to Digital Phenotyping for Global Health. In: Celi LA, Majumder MS, Ordóñez P, Osorio JS, Paik KE, Somai M, editors. Leveraging Data Science for Global Health. Cham, Switzerland: Springer International Publishing; 2020:251-261.
  17. Andrea A, Agulia A, Serafini G, Amore M. Digital biomarkers and digital phenotyping in mental health care and prevention. Eur J Public Health 2020;30(Supplement_5):161-166. [CrossRef]
  18. Brietzke E, Hawken ER, Idzikowski M, Pong J, Kennedy SH, Soares CN. Integrating digital phenotyping in clinical characterization of individuals with mood disorders. Neurosci Biobehav Rev 2019 Sep;104:223-230. [CrossRef] [Medline]
  19. Schilbach L. Using interaction-based phenotyping to assess the behavioral and neural mechanisms of transdiagnostic social impairments in psychiatry. Eur Arch Psychiatry Clin Neurosci 2019 Apr 13;269(3):273-274. [CrossRef] [Medline]
  20. Huckvale K, Venkatesh S, Christensen H. Toward clinical digital phenotyping: a timely opportunity to consider purpose, quality, and safety. NPJ Digit Med 2019 Sep 6;2(1):88 [FREE Full text] [CrossRef] [Medline]
  21. Alghowinem S. Detecting depression: A comparison between spontaneous and read speech. New York, NY: IEEE; 2013 Presented at: IEEE International Conference on Acoustics, Speech and Signal Processing; May 26-31, 2013; Vancouver, Canada p. 7547-7551. [CrossRef]
  22. Arevian AC, Bone D, Malandrakis N, Martinez VR, Wells KB, Miklowitz DJ, et al. Clinical state tracking in serious mental illness through computational analysis of speech. PLoS ONE 2020 Jan 15;15(1):e0225695 [FREE Full text] [CrossRef] [Medline]
  23. Hönig F. Automatic modelling of depressed speech: relevant features and relevance of gender. 2014 Presented at: Interspeech 2014; September 14-18, 2014; Singapore City, Singapore p. 1248-1252   URL: https://tinyurl.com/bdem6mj7 [CrossRef]
  24. Horwitz R. On the relative importance of vocal source, system,prosody in human depression. New York, NY: IEEE; 2013 Presented at: IEEE International Conference on Body Sensor Networks; May 6-9, 2013; Cambridge, MA p. 1-6. [CrossRef]
  25. Mundt JC, Vogel AP, Feltner DE, Lenderking WR. Vocal acoustic biomarkers of depression severity and treatment response. Biol Psychiatry 2012 Oct 01;72(7):580-587 [FREE Full text] [CrossRef] [Medline]
  26. Quatieri T. Vocal-Source Biomarkers for Depression: A Link to Psychomotor Activity. 2012 Presented at: Interspeech 2012; September 9-13, 2012; Portland, OR p. 1059-1062   URL: https:/​/www.​researchgate.net/​publication/​283629170_Vocal-source_biomarkers_for_depression_A_link_to_psychomotor_activity [CrossRef]
  27. Shinohara S, Nakamura M, Omiya Y, Higuchi M, Hagiwara N, Mitsuyoshi S, et al. Depressive Mood Assessment Method Based on Emotion Level Derived from Voice: Comparison of Voice Features of Individuals with Major Depressive Disorders and Healthy Controls. Int J Environ Res Public Health 2021 May 19;18(10):5435 [FREE Full text] [CrossRef] [Medline]
  28. Taguchi T, Tachikawa H, Nemoto K, Suzuki M, Nagano T, Tachibana R, et al. Major depressive disorder discrimination using vocal acoustic features. J Affect Disord 2018 Jan 01;225:214-220. [CrossRef] [Medline]
  29. Wang J, Zhang L, Liu T, Pan W, Hu B, Zhu T. Acoustic differences between healthy and depressed people: a cross-situation study. BMC Psychiatry 2019 Oct 15;19(1):300 [FREE Full text] [CrossRef] [Medline]
  30. Xu S, Yang Z, Chakraborty D, Chua YHV, Dauwels J, Thalmann D, et al. Automated Verbal and Non-verbal Speech Analysis of Interviews of Individuals with Schizophrenia and Depression. New York, NY: IEEE; 2019 Presented at: 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC); July 23-27, 2019; Berlin, Germany p. 225. [CrossRef]
  31. Yamamoto M, Takamiya A, Sawada K, Yoshimura M, Kitazawa M, Liang K, et al. Using speech recognition technology to investigate the association between timing-related speech features and depression severity. PLoS One 2020 Sep 11;15(9):e0238726 [FREE Full text] [CrossRef] [Medline]
  32. Alghowinem S, Goecke R, Cohn JF, Wagner G. Cross-cultural detection of depression from nonverbal behaviour. 2015 Presented at: 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition; May 4-8; Ljubljana, Slovenia p. 1-8. [CrossRef]
  33. Bhatia S, Goecke R, Hammal Z, Cohn JF. Automated Measurement of Head Movement Synchrony during Dyadic Depression Severity Interviews. 2019 Presented at: 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019); May 14-18; Lille, France p. 1-8. [CrossRef]
  34. Dibeklioglu H, Hammal Z, Cohn JF. Dynamic Multimodal Measurement of Depression Severity Using Deep Autoencoding. IEEE J Biomed Health Inform 2018 Mar;22(2):525-536. [CrossRef]
  35. Fiquer JT, Boggio PS, Gorenstein C. Talking bodies: nonverbal behavior in the assessment of depression severity. J Affect Disord 2013 Sep 25;150(3):1114-1119. [CrossRef] [Medline]
  36. Fiquer JT, Moreno RA, Canales JZ, Cavalcanti A, Gorenstein C. Is nonverbal behavior in patients and interviewers relevant to the assessment of depression and its recovery? A study with Dutch and Brazilian patients. Psychiatry Res 2017 Apr;250:59-64. [CrossRef] [Medline]
  37. Fiquer JT, Moreno RA, Brunoni AR, Barros VB, Fernandes F, Gorenstein C. What is the nonverbal communication of depression? Assessing expressive differences between depressive patients and healthy volunteers during clinical interviews. J Affect Disord 2018 Oct 01;238:636-644. [CrossRef] [Medline]
  38. Girard JM, Cohn JF, Mahoor MH, Mavadati S, Rosenwald DP. Social risk and depression: Evidence from manual and automatic facial expression analysis. 2013 Presented at: 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition; April 22-26; Shanghai, China p. 1-8. [CrossRef]
  39. Girard JM, Cohn JF, Mahoor MH, Mavadati SM, Hammal Z, Rosenwald DP. Nonverbal Social Withdrawal in Depression: Evidence from manual and automatic analysis. Image Vis Comput 2014 Oct;32(10):641-647 [FREE Full text] [CrossRef] [Medline]
  40. Guo W, Yang H, Liu Z, Xu Y, Hu B. Deep Neural Networks for Depression Recognition Based on 2D and 3D Facial Expressions Under Emotional Stimulus Tasks. Front Neurosci 2021 Apr 23;15:609760 [FREE Full text] [CrossRef] [Medline]
  41. Jiang Z, Harati S, Crowell A, Mayberg HS, Nemati S, Clifford GD. Classifying Major Depressive Disorder and Response to Deep Brain Stimulation Over Time by Analyzing Facial Expressions. IEEE Trans Biomed Eng 2021 Feb;68(2):664-672. [CrossRef]
  42. Kacem A, Hammal Z, Daoudi M, Cohn J. Detecting Depression Severity by Interpretable Representations of Motion Dynamics. 2018 Presented at: 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018); May 15-19; Xi'an, China p. 739-745. [CrossRef]
  43. Adolph D, Teismann T, Forkmann T, Wannemüller A, Margraf J. High frequency heart rate variability: Evidence for a transdiagnostic association with suicide ideation. Biol Psychol 2018 Oct;138:165-171. [CrossRef] [Medline]
  44. Fernandes RA, Fiquer JT, Gorenstein C, Razza LB, Fraguas R, Borrione L, et al. Nonverbal behaviors are associated with increased vagal activity in major depressive disorder: Implications for the polyvagal theory. J Affect Disord 2017 Feb;209:18-22. [CrossRef] [Medline]
  45. Giurgi-Oncu C, Tudoran C, Enatescu VR, Tudoran M, Pop GN, Bredicean C. Evolution of Heart Rate Variability and Heart Rate Turbulence in Patients with Depressive Illness Treated with Selective Serotonin Reuptake Inhibitors. Medicina (Kaunas) 2020 Nov 05;56(11):590 [FREE Full text] [CrossRef] [Medline]
  46. Hartmann R, Schmidt FM, Sander C, Hegerl U. Heart Rate Variability as Indicator of Clinical State in Depression. Front Psychiatry 2018 Jan 17;9:735 [FREE Full text] [CrossRef] [Medline]
  47. Kircanski K, Williams LM, Gotlib IH. Heart rate variability as a biomarker of anxious depression response to antidepressant medication. Depress Anxiety 2019 Jan 12;36(1):63-71 [FREE Full text] [CrossRef] [Medline]
  48. Lee M, Seo YE, Mok YE, Lee SH. Heart Rate Variability after Treatment for Depression in North Korean Defectors. Appl Psychophysiol Biofeedback 2021 Mar 19;46(1):11-18. [CrossRef] [Medline]
  49. Neyer S, Witthöft M, Cropley M, Pawelzik M, Lugo RG, Sütterlin S. Reduction of depressive symptoms during inpatient treatment is not associated with changes in heart rate variability. PLoS One 2021 Mar 23;16(3):e0248686 [FREE Full text] [CrossRef] [Medline]
  50. Sarlon J, Staniloiu A, Kordon A. Heart Rate Variability Changes in Patients With Major Depressive Disorder: Related to Confounding Factors, Not to Symptom Severity? Front Neurosci 2021 Jul 5;15:675624 [FREE Full text] [CrossRef] [Medline]
  51. Kim AY, Jang EH, Kim S, Choi KW, Jeon HJ, Yu HY, et al. Automatic detection of major depressive disorder using electrodermal activity. Sci Rep 2018;8(1):1. [CrossRef]
  52. Kim AY, Jang EH, Choi KW, Jeon HJ, Byun S, Sim JY, et al. Skin conductance responses in Major Depressive Disorder (MDD) under mental arithmetic stress. PLoS One 2019 Apr 3;14(4):e0213140 [FREE Full text] [CrossRef] [Medline]
  53. Litwińska-Bołtuć M, Święcicki ?, Spreco A, Timpka T. Clinical effectiveness of the electrodermal orienting reactivity test for evaluating relapse and recurrence risk in patients hospitalized for depression. BMC Psychiatry 2021 Feb 10;21(1):88 [FREE Full text] [CrossRef] [Medline]
  54. Pedrelli P, Fedor S, Ghandeharioun A, Howe E, Ionescu DF, Bhathena D, et al. Monitoring Changes in Depression Severity Using Wearable and Mobile Sensors. Front Psychiatry 2020 Dec 18;11:584711 [FREE Full text] [CrossRef] [Medline]
  55. Smith LT, Levita L, Amico F, Fagan J, Yek JH, Brophy J, et al. Using Resting State Heart Rate Variability and Skin Conductance Response to Detect Depression in Adults. Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC) 2020:5004-5007. [CrossRef]
  56. Cummins N, Scherer S, Krajewski J, Schnieder S, Epps J, Quatieri TF. A review of depression and suicide risk assessment using speech analysis. Speech Communication 2015 Jul;71:10-49. [CrossRef]
  57. Low DM, Bentley KH, Ghosh SS. Automated assessment of psychiatric disorders using speech: A systematic review. Laryngoscope Investig Otolaryngol 2020 Feb 31;5(1):96-116 [FREE Full text] [CrossRef] [Medline]
  58. Kiss G, Vicsi K. Mono- and multi-lingual depression prediction based on speech processing. Int J Speech Technol 2017 Sep 20;20(4):919-935. [CrossRef]
  59. Geerts E, Brune M. Ethological approaches to psychiatric disorders: focus on depression and schizophrenia. Aus & New Zealand J of Psychiatry 2009 Nov;43(11):1007-1015. [CrossRef]
  60. Alghowinem SM, Gedeon T, Goecke R, Cohn J, Parker G. Interpretation of Depression Detection Models via Feature Selection Methods. IEEE Trans Affective Comput 2020:1-1. [CrossRef]
  61. Servant D, Logier R, Mouster Y, Goudemand M. [Heart rate variability. Applications in psychiatry]. Encephale 2009 Oct;35(5):423-428. [CrossRef] [Medline]
  62. Malik M, Bigger JT, Camm AJ, Kleiger RE, Malliani A, Moss AJ, et al. Heart rate variability: Standards of measurement, physiological interpretation, and clinical use. European Heart Journal 1996 Mar 01;17(3):354-381. [CrossRef]
  63. Appelhans BM, Luecken LJ. Heart rate variability and pain: associations of two interrelated homeostatic processes. Biol Psychol 2008 Feb;77(2):174-182. [CrossRef] [Medline]
  64. Schwerdtfeger AR, Schwarz G, Pfurtscheller K, Thayer JF, Jarczok MN, Pfurtscheller G. Heart rate variability (HRV): From brain death to resonance breathing at 6 breaths per minute. Clin Neurophysiol 2020 Mar;131(3):676-693. [CrossRef] [Medline]
  65. Hillebrand S, Gast KB, de Mutsert R, Swenne CA, Jukema JW, Middeldorp S, et al. Heart rate variability and first cardiovascular event in populations without known cardiovascular disease: meta-analysis and dose-response meta-regression. Europace 2013 May;15(5):742-749. [CrossRef] [Medline]
  66. Koch C, Wilhelm M, Salzmann S, Rief W, Euteneuer F. A meta-analysis of heart rate variability in major depression. Psychol Med 2019 Jun 26;49(12):1948-1957. [CrossRef]
  67. Sarchiapone M, Gramaglia C, Iosue M, Carli V, Mandelli L, Serretti A, et al. The association between electrodermal activity (EDA), depression and suicidal behaviour: A systematic review and narrative synthesis. BMC Psychiatry 2018 Jan 25;18(1):22 [FREE Full text] [CrossRef] [Medline]
  68. Bloch V. New aspects of the psychogalvanic or electrodermographic method as a criterion of emotional tensions. L'annee Psychologique 1952:329-362.
  69. Sequeira H, Hot P, Silvert L, Delplanque S. Electrical autonomic correlates of emotion. Int J Psychophysiol 2009 Jan;71(1):50-56. [CrossRef] [Medline]
  70. Balconi M, Brambilla E, Falbo L. Appetitive vs. defensive responses to emotional cues. Autonomic measures and brain oscillation modulation. Brain Res 2009 Nov 03;1296:72-84. [CrossRef] [Medline]
  71. Crider A. Personality and electrodermal response lability: an interpretation. Appl Psychophysiol Biofeedback 2008 Sep 29;33(3):141-148. [CrossRef] [Medline]
  72. Hamm A. Affective blindsight: intact fear conditioning to a visual cue in a cortically blind patient. Brain 2003:267-275. [CrossRef]
  73. Bauer RM, Verfaellie M. Electrodermal discrimination of familiar but not unfamiliar faces in prosopagnosia. Brain and Cognition 1988 Oct;8(2):240-252. [CrossRef]
  74. Silvert L, Delplanque S, Bouwalerh H, Verpoort C, Sequeira H. Autonomic responding to aversive words without conscious valence discrimination. Int J Psychophysiol 2004 Jul;53(2):135-145. [CrossRef] [Medline]
  75. Bernstein AS, Schnur DB, Bernstein P, Yeager A, Wrable J, Smith S. Differing patterns of electrodermal and finger pulse responsivity in schizophrenia and depression. Psychol Med 1995;25(1):51-62. [CrossRef]
  76. Lader M, Wing L. Physiological measures in agitated and retarded depressed patients. J Psychiatr Res 1969 Dec;7(2):89-100. [CrossRef]
  77. Thorell L. Electrodermal activity in suicidal and nonsuicidal depressive patients and in matched healthy subjects. Acta Psychiatr Scand 1987 Oct;76(4):420-430. [CrossRef]
  78. Byrne DG. A Psychophysiological Distinction between Types of Depressive States. Aust N Z J Psychiatry 1975;9(3):181-185. [CrossRef]
  79. Mirkin AM, Coppen A. Electrodermal Activity in Depression: Clinical and Biochemical Correlates. Br J Psychiatry 1980;137(1):93-97. [CrossRef]
  80. Bouhuys A, Jansen C, van den Hoofdakker R. Analysis of observed behaviors displayed by depressed patients during a clinical interview: relationships between behavioral factors and clinical concepts of activation. J Affect Disord 1991 Feb;21(2):79-88. [CrossRef]
  81. Bouhuys A, Van den Hoofdakker R. A longitudinal study of interaction patterns of a psychiatrist and severely depressed patients based on observed behaviour: an ethological approach of interpersonal theories of depression. J Affect Disord 1993 Feb;27(2):87-99. [CrossRef]
  82. Geerts E, Bouhuys AL, Bloem GM. Nonverbal support giving induces nonverbal support seeking in depressed patients. J Clin Psychol 1997 Jan;53(1):35-39. [CrossRef]
  83. Bouhuys A, Albersnagel F. Do interactional capacities based on observed behaviour interfere with improvement in severely depressed patients? J Affect Disord 1992 Jun;25(2):107-116. [CrossRef]
  84. Maxhuni A, Muñoz-Meléndez A, Osmani V, Perez H, Mayora O, Morales EF. Classification of bipolar disorder episodes based on analysis of voice and motor activity of patients. Pervasive and Mobile Computing 2016 Sep;31:50-66. [CrossRef]
  85. Guidi A, Schoentgen J, Bertschy G, Gentili C, Scilingo E, Vanello N. Features of vocal frequency contour and speech rhythm in bipolar disorder. Biomedical Signal Processing and Control 2017 Aug;37:23-31. [CrossRef]
  86. Guidi A. Voice quality in patients suffering from bipolar disease. New York, NY: IEEE; 2015 Presented at: 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC); August 25-29, 2015; Milan, Italy p. 6106-6109. [CrossRef]
  87. Guidi A. Analysis of running speech for the characterization of mood state in bipolar patients. New York, NY: IEEE; 2015 Presented at: 2015 AEIT International Annual Conference (AEIT); October 14-16, 2015; Naples, Italy p. 1-6. [CrossRef]
  88. Vanello N. Speech analysis for mood state characterization in bipolar patients. New York, NY: IEEE; 2012 Presented at: 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society; August 28, 2012, to September 1, 2012; San Diego, CA, USA p. 2104-2107. [CrossRef]
  89. Zhang J, Pan Z, Gui C, Xue T, Lin Y, Zhu J, et al. Analysis on speech signal features of manic patients. J Psychiatr Res 2018 Mar;98:59-63. [CrossRef] [Medline]
  90. Gideon J, Provost EM, McInnis M. Mood state prediction from speech of varying acoustic quality for individuals with bipolar disorder. 2016 Mar Presented at: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP); 2016; Shanghai, China p. 2359-2363   URL: http://europepmc.org/abstract/MED/27570493 [CrossRef]
  91. Karam ZN. Ecologically valid long-term mood monitoring of individuals with bipolar disorder using speech. New York, NY: IEEE; 2014 Presented at: 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP); May 4-9, 2014; Florence, italy p. 4858-4862. [CrossRef]
  92. Faurholt-Jepsen M, Busk J, Frost M, Vinberg M, Christensen EM, Winther O, et al. Voice analysis as an objective state marker in bipolar disorder. Transl Psychiatry 2016 Jul 19;6(7):e856-e856 [FREE Full text] [CrossRef] [Medline]
  93. Faurholt-Jepsen M, Kessing LV, Munkholm K. Heart rate variability in bipolar disorder: A systematic review and meta-analysis. Neurosci Biobehav Rev 2017 Feb;73:68-80. [CrossRef] [Medline]
  94. Wazen GLL, Gregório ML, Kemp AH, Godoy MFD. Heart rate variability in patients with bipolar disorder: From mania to euthymia. J Psychiatr Res 2018 Apr;99:33-38. [CrossRef] [Medline]
  95. Hage B, Britton B, Daniels D, Heilman K, Porges SW, Halaris A. Diminution of Heart Rate Variability in Bipolar Depression. Front Public Health 2017 Dec 06;5:312 [FREE Full text] [CrossRef] [Medline]
  96. Benjamin BR, Valstad M, Elvsåshagen T, Jönsson EG, Moberget T, Winterton A, et al. Heart rate variability is associated with disease severity in psychosis spectrum disorders. Prog Neuropsychopharmacol Biol Psychiatry 2021 Dec 20;111:110108 [FREE Full text] [CrossRef] [Medline]
  97. Ortiz A, Bradler K, Moorti P, MacLean S, Husain MI, Sanches M, et al. Reduced heart rate variability is associated with higher illness burden in bipolar disorder. J Psychosom Res 2021 Jun;145:110478. [CrossRef] [Medline]
  98. Freyberg J, Brage S, Kessing LV, Faurholt-Jepsen M. Differences in psychomotor activity and heart rate variability in patients with newly diagnosed bipolar disorder, unaffected relatives, and healthy individuals. J Affect Disord 2020 Apr 01;266:30-36 [FREE Full text] [CrossRef] [Medline]
  99. Chang H, Chang C, Kuo TBJ, Huang S. Distinguishing bipolar II depression from unipolar major depressive disorder: Differences in heart rate variability. World J Biol Psychiatry 2015 Mar 24;16(5):351-360. [CrossRef] [Medline]
  100. Hage B, Britton B, Daniels D, Heilman K, Porges SW, Halaris A. Low cardiac vagal tone index by heart rate variability differentiates bipolar from major depression. World J Biol Psychiatry 2019 Jun 05;20(5):359-367. [CrossRef] [Medline]
  101. Greco A, Valenza G, Lanata A, Rota G, Scilingo EP. Electrodermal activity in bipolar patients during affective elicitation. IEEE J Biomed Health Inform 2014 Nov;18(6):1865-1873. [CrossRef] [Medline]
  102. Lemaire M, El-Hage W, Frangou S. Increased affective reactivity to neutral stimuli and decreased maintenance of affective responses in bipolar disorder. Eur Psychiatry 2015 Oct;30(7):852-860. [CrossRef] [Medline]
  103. Faurholt-Jepsen M, Brage S, Kessing LV, Munkholm K. State-related differences in heart rate variability in bipolar disorder. J Psychiatr Res 2017 Jan;84:169-173 [FREE Full text] [CrossRef] [Medline]
  104. Monti E, D’Andrea W, Freed S, Kidd DC, Feuer S, Carroll LM, et al. Does Self-Reported Childhood Trauma Relate to Vocal Acoustic Measures? Preliminary Findings at Trauma Recall. J Nonverbal Behav 2021 Jan 22;45(3):389-408. [CrossRef]
  105. Monti E, Kidd DC, Carroll LM, Castano E. What's in a singer's voice: The effect of attachment, emotions and trauma. Logoped Phoniatr Vocol 2017 Jul;42(2):62-72. [CrossRef] [Medline]
  106. Schultebraucks K, Yadav V, Shalev AY, Bonanno GA, Galatzer-Levy IR. Deep learning-based classification of posttraumatic stress disorder and depression following trauma utilizing visual and auditory markers of arousal and mood. Psychol Med 2020 Aug 03;52(5):957-967. [CrossRef]
  107. Kleim B, Horn AB, Kraehenmann R, Mehl MR, Ehlers A. Early Linguistic Markers of Trauma-Specific Processing Predict Post-trauma Adjustment. Front Psychiatry 2018;9:645 [FREE Full text] [CrossRef] [Medline]
  108. He Q, Veldkamp BP, Glas CAW, de Vries T. Automated Assessment of Patients' Self-Narratives for Posttraumatic Stress Disorder Screening Using Natural Language Processing and Text Mining. Assessment 2017 Mar;24(2):157-172. [CrossRef] [Medline]
  109. Huon P, Stutz N. Les marqueurs linguistiques du temps et de la subjectivité dans le récit de psychotraumatisme. L'Évolution Psychiatrique 2020 Oct;85(4):479-508. [CrossRef]
  110. Scherer S. Investigating voice quality as a speaker-independent indicator of depression and PTSD. Interspeech 2013:851. [CrossRef]
  111. Xu R, Mei G, Zhang G, Gao P, Judkins T, Cannizzaro M, et al. A voice-based automated system for PTSD screening and monitoring. Stud Health Technol Inform 2012;173:552-558. [Medline]
  112. Scherer S. Self-Reported Symptoms of Depression and PTSD Are Associated with Reduced Vowel Space in Screening Interviews. IEEE Transactions on Affective Computing 2016 Jan 1;7(1):59-73. [CrossRef]
  113. Marmar CR, Brown AD, Qian M, Laska E, Siegel C, Li M, et al. Speech-based markers for posttraumatic stress disorder in US veterans. Depress Anxiety 2019 Jul 22;36(7):607-616 [FREE Full text] [CrossRef] [Medline]
  114. McTeague LM, Lang PJ, Laplante M, Cuthbert BN, Shumen JR, Bradley MM. Aversive imagery in posttraumatic stress disorder: trauma recurrence, comorbidity, and physiological reactivity. Biol Psychiatry 2010 Feb 15;67(4):346-356 [FREE Full text] [CrossRef] [Medline]
  115. Blechert J, Michael T, Wilhelm FH. Video-Based Analysis of Bodily Startle and Subsequent Emotional Facial Expression in Posttraumatic Stress Disorder. J Exp Psychopathol 2019 May 21;4(4):435-447 [FREE Full text] [CrossRef] [Medline]
  116. Lazarov A, Suarez-Jimenez B, Tamman A, Falzon L, Zhu X, Edmondson DE, et al. Attention to threat in posttraumatic stress disorder as indexed by eye-tracking indices: a systematic review. Psychol Med 2018 Sep 04;49(5):705-726. [CrossRef]
  117. Mekawi Y, Murphy L, Munoz A, Briscione M, Tone EB, Norrholm SD, et al. The role of negative affect in the association between attention bias to threat and posttraumatic stress: An eye-tracking study. Psychiatry Res 2020 Feb;284:112674 [FREE Full text] [CrossRef] [Medline]
  118. Katz C, Hershkowitz I, Malloy LC, Lamb ME, Atabaki A, Spindler S. Non-verbal behavior of children who disclose or do not disclose child abuse in investigative interviews. Child Abuse Negl 2012 Jan;36(1):12-20. [CrossRef] [Medline]
  119. Schneider M, Schwerdtfeger A. Autonomic dysfunction in posttraumatic stress disorder indexed by heart rate variability: a meta-analysis. Psychol Med 2020 Aug 28;50(12):1937-1948. [CrossRef]
  120. Ge F, Yuan M, Li Y, Zhang W. Posttraumatic Stress Disorder and Alterations in Resting Heart Rate Variability: A Systematic Review and Meta-Analysis. Psychiatry Investig 2020 Jan;17(1):9-20 [FREE Full text] [CrossRef] [Medline]
  121. Sigrist C, Mürner-Lavanchy I, Peschel SK, Schmidt SJ, Kaess M, Koenig J. Early life maltreatment and resting-state heart rate variability: A systematic review and meta-analysis. Neurosci Biobehav Rev 2021 Jan;120:307-334. [CrossRef] [Medline]
  122. Stone LB, Amole MC, Cyranowski JM, Swartz HA. History of childhood emotional abuse predicts lower resting-state high-frequency heart rate variability in depressed women. Psychiatry Res 2018 Nov;269:681-687 [FREE Full text] [CrossRef] [Medline]
  123. Hopper JW, Frewen PA, Sack M, Lanius RA, van der Kolk BA. The Responses to Script-Driven Imagery Scale (RSDI): Assessment of State Posttraumatic Symptoms for Psychobiological and Treatment Research. J Psychopathol Behav Assess 2007 May 18;29(4):249-268. [CrossRef]
  124. Sack M, Cillien M, Hopper JW. Acute dissociation and cardiac reactivity to script-driven imagery in trauma-related disorders. Eur J Psychotraumatol 2012;3(1):17419 [FREE Full text] [CrossRef] [Medline]
  125. Nemeroff CB, Bremner JD, Foa EB, Mayberg HS, North CS, Stein MB. Posttraumatic stress disorder: a state-of-the-science review. J Psychiatr Res 2006 Feb;40(1):1-21. [CrossRef] [Medline]
  126. O'Kearney R, Parry L. Comparative physiological reactivity during script-driven recall in depression and posttraumatic stress disorder. J Abnorm Psychol 2014 Aug;123(3):523-532. [CrossRef] [Medline]
  127. D'Andrea W, Pole N, DePierro J, Freed S, Wallace DB. Heterogeneity of defensive responses after exposure to trauma: blunted autonomic reactivity in response to startling sounds. Int J Psychophysiol 2013 Oct;90(1):80-89. [CrossRef] [Medline]


AUC: area under the curve
BD: bipolar disorder
CEA: childhood emotional abuse
DSM-5: Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition
EDA: electrodermal activity
ELM: early life maltreatment
F0: fundamental frequency
GAF: Global Assessment of Functioning Scale
HAM-D: Hamilton Rating Scale for Depression
HC: healthy control
HDRS: Hamilton Depression Rating Scale
HF: high frequency
HNR: harmonic-to-noise ratio
HR: heart rate
HRV: heart rate variability
ICD-11: International Classification of Diseases 11th Revision
LF: low frequency
MDD: major depressive disorder
MDE: major depressive episode
MFCC: mel-frequency cepstral coefficient
PANSS: Positive and Negative Syndrome Scale
PNN50: proportion of NN50 divided by the total number of NN (R-R) intervals
PNS: parasympathetic nervous system
PTSD: posttraumatic stress disorder
RMSSD: root-mean-square surface distance
RR: beat-to-beat interval
RSA: respiratory sinus arrhythmia
SC: skin conductance
SCL: skin conductance level
SCR: skin conductance response
SD1: SD of points perpendicular to the major axis of the Poincaré plot
SD2: SD of points along the major axis of the Poincaré plot
SDNN: SD of the NN (R-R) intervals
SNS: sympathetic nervous system
TAT: Thematic Appreciation Test
VLF: very-low frequency


Edited by J Torous; submitted 11.02.22; peer-reviewed by S Mangelsdorf, G Mayer, K Reuter; comments to author 30.05.22; revised version received 02.09.22; accepted 30.09.22; published 23.01.23

Copyright

©Eric Ettore, Philipp Müller, Jonas Hinze, Michel Benoit, Bruno Giordana, Danilo Postin, Amandine Lecomte, Hali Lindsay, Philippe Robert, Alexandra König. Originally published in JMIR Mental Health (https://mental.jmir.org), 23.01.2023.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Mental Health, is properly cited. The complete bibliographic information, a link to the original publication on https://mental.jmir.org/, as well as this copyright and license information must be included.