Prosodic abnormalities in patients with depression are well known and a monotonous speech or reduced prosody can be easily and frequently heard. It is assumed that cognitive and physiological changes in depression could affect speech production, and changes in the automatic and somatic nervous system cause disturbances in muscle tension and respiratory rate. These changes will not only influence vocal folds and vocal tract dynamics, but also constrain articulatory movement [56].

Speech analysis, which consists of automatically extracting vocal and linguistic features from audio signals, offers the possibility to detect and measure these changes and serves as an additional objective assessment of depression. Traditionally, acoustic features can be divided into 5 types: temporal, prosodic, spectral, source, and filter features. Temporal features, defined as speech prosodic timing measures, appear to be a promising measure of depression and its severity [57].

Yamamoto et al [31] used 3 temporal features in a prospective follow-up study: speech rate, pause time, and response time. They found a correlation with scores on the Hamilton Rating Scale for Depression (HAM-D). Mundt et al [25] found 5 prosodic timing measures to be significantly correlated with depression severity: total speech time, total pause time, percentage pause time, speech pause ratio, and speaking rate. Xu et al [30] found that people with depression have a longer response time and interrupt their communication partner more often. To summarize, people with depression used a much larger percentage of words related to the past and sadness emotions. Similarly, Shinohara et al [27] used a “vitality” score based on emotional elements of speech (ie,  joy, anger) and showed a negative correlation between the vitality score and HAM-D scores. Arevian et al [22] analyzed the types of word used and found, among others, more negative emotion and less complex word use in patients with depression.

Finally, Alghowinem et al [21] and Hönig et al [23] found that the average syllable duration could be positively correlated with depression severity.

With prosodic features in people with depression, reduced fundamental frequency (F0) range and average are frequently found [23,24,26,58]. F0 corresponds to the lowest frequency of the speech signal perceived as pitch. It could explain the depressive monotonous speech and could be a consequence of disturbances in laryngeal muscle tension due to psychomotor retardation. The latter may also explain reduction in F0 variability [56,57]. However, some studies also report no significant correlation between F0 variables and depression [25,28].

Several studies on depression showed a shift in spectral energy, and mel-frequency cepstral coefficients (MFCCs) are often used as spectral features. Spectral features characterize the spectrum of speech, which at a given time correspond to the frequency distribution of the speech signal. Taguchi et al [28] showed that the second dimension of MFCC allowed a significant discrimination between patients with depression and controls. Further, Wang et al [29] found that MFCC 5, MFCC 7, and loudness were consistently lower in people with depression. According to the task designed during speech recording, MFCC 3 and F0 were significantly lower than in people with depression than in healthy people.

Finally, source features such as harmonic-to-noise ratio (HNR), jitter, and shimmer also tend to increase with depression severity, supporting the hypothesis of a more breathy phonation in depressed speech [26]. This could be explained by a more open and turbulent glottis, linked to a reduction in laryngeal muscle tension. The study by Hönig et al [23] also supports this hypothesis. Alghowinem et al [21] showed that not only shimmer and jitter but also loudness and MFCC features are part of the general characteristics of depressive speech.

Traditionally, evaluation of clinical depression is based on patients’ verbal information from psychological interviews and rating scales or self-report questionnaires. However, verbal analysis has several limitations. First, patients may underreport or overreport depressive symptoms due to different reasons (eg, social stigma). Second, it can be assumed that most human communication is taking place through nonverbal behavior. Indeed, a large part of nonverbal behavior is mostly outside of conscious control, and these cues and signals may differ from verbal reports. Facial expression, gestures, and body postures are mainly involuntary, and therefore represent a privileged way of expressing feelings and emotions [35,59]. Girard et al [38,39] showed that when the severity of depressive symptoms was high, participants made fewer affiliative facial expression such as smiling, and more nonaffiliative facial expression such as contempt. They also showed that patients exhibited diminished head motion (amplitude and velocity). These results support the social withdrawal hypothesis in depression. In fact, rather than affective valence with an increase or decrease in facial expression, it is the social-communicative value with affiliative expression that seems to be affected in depression. Thus, nonverbal behavior may serve to maintain interpersonal distance while facilitating social withdrawal.

Fiquer et al [35] supported the assumption that depression involves behaviors related to social withdrawal and negative feelings. Crying, asymmetric smiling, and motionless head/eyebrows were associated with higher levels of depression severity, while patients’ illustrative gestures, defined as “the hand and arm movements used to support the speech,” were associated with a lower level of depression severity.

Moreover, decrease of negative facial and head indicators and increase of eye contact and yes nodding accompanied clinical improvement, whereas illustrative gestures did not [35]. In another study, Fiquer et al [37] compared nonverbal behavior in patients with depression versus controls. They showed that patients with depression displayed higher levels of negative nonverbal behaviors (shrug, head and lips down, adaptative hand gesture, frown, and cry) and lower levels of positive nonverbal behaviors (eye contact and smile). The authors raise the important role of these behaviors in social withdrawal [37]. Nevertheless, the aforementioned nonverbal behaviors was not associated with depression severity before treatment, which suggests its independence.

The authors concluded that certain typically found nonverbal behaviors could represent a predisposition to depression possibly influenced by personality features. These social symptoms expressed through nonverbal behavior may be the last ones to vanish [37].

On the contrary, Fiquer et al [36] studied nonverbal variables such as speaking effort, encouragement, attunement, and change in attunement, and found no association between these behaviors and the severity of depression. Finally, several recent studies achieved classifying depression with high accuracy using video analysis with head and facial movements [34,42], only head movement [33], facial expressivity [41], or with head pose and eye activity [32]. Guo et al [40] found that the highest accuracy of depression recognition was when patients were watching film clips of positive and negative emotional stimuli compared with other tasks.

In a recent paper, Alghowinem et al [60] used feature selection methods and found that some features have a high capability for distinguishing between depression severities. For speech analysis, the strongest features were the temporal features, F0, HNR, formants, and MFCC; and for behavioral analysis, it was the left-right eye movement and gaze direction and the yaw head movement. The authors showed that these features outperformed all other features in depression detection.

The studies mentioned so far demonstrate the usefulness of behavioral analysis in understanding the mechanisms that underlie the onset and course of depression. These analyses are less influenced by conscious control compared with verbal communication [59], which could make them a reliable objective measure.

Based on these aforementioned studies, the digital phenotype of MDD could be identified as follows: a reduction in certain speech temporal features (ie, speech rate or speech time), an increase in others (ie, pause time or response time) [25,31], and a change in the type of words used (ie, less complex and more emotionally negative) [22,27]. In addition, prosodic features (ie, F0 range and average) [23,24,26,58], variability [56,57], spectral features [28,29], and source features (ie, HNR, jitter, shimmer [21,26]) are changed.

The behavior of patients with depression could be characterized by a more negative nonverbal behavior (ie, head motion, facial expressivity, and hand or body gestures) [35,38,39]. But more than the valence, it seems that it is the affiliative behaviors that are impacted, supporting a social withdrawal [38,39]. In fact, certain patterns of behavior when interacting with others may predict the severity of symptoms or treatment response [36,59,80-83]. Finally, physiological measures are mainly lowered for both HRV (ie, HF, LF, VLF, and time domains) [66] and EDA (ie, SCR and SCL) [67].

Several studies managed to classify the course of mood episodes or relapses with high confidence. For instance, speech pause duration [84] and number of longer pauses [85] are significantly higher in depressive states than in a hypomanic or euthymic state. Concerning prosodic and source features, Guidi et al [85-87] repeatedly found an increase of F0 in hypomanic states compared with euthymic or depressive states. Similar results were also obtained by Vanello et al [88]. Likewise, filter features seem to increase with manic mood [89], especially F1 and F2 formants, which correspond to the first and the second peak in the spectrum that results from a resonance of the vocal tract.

Finally, many studies extracted several speech features (ie, the openSMILE toolkit), achieving good classification accuracies [90,91]. Classification of depressive states could be obtained with an area under the curve (AUC) value of 0.78 and manic and mixed states with an AUC value of 0.89 [92].

For the digital phenotype of BD, we found that, in the depressed phase, speech changed in temporal (ie, decreased speech pause) and prosodic features (ie, decreased F0), but both increased when patients go into the hypomanic phase [84-88].

Independent of the phase, HRV appears reduced (ie, LF) [93], but the change in HRV between phases remains unclear [94,95,103]. For EDA, an electrodermal hypoactivity in a depressive state is reported, which increases when patients move toward euthymia [101].

The link between trauma and voice has been likewise explored. Monti et al [104] reported a significant relationship between voice fundamental frequency and the “total trauma” reported with the Childhood Trauma Questionnaire. Interestingly, after trauma recall, vocal jitter and voice irregularity (roughness) were strongly related to the existence of childhood trauma [104]. This suggests that variations in vibratory patterns of the vocal folds could have a relationship with childhood trauma recall. In another study, Monti et al [105] also found an association of anxious attachment and childhood neglect with intensity/loudness measures in singer’s voice. Anxious attachment was also positively correlated with jitter and irregularity. Moreover, in speech content, trauma narrative shows an increased use of first-person singular pronouns [106,107] and changes in narrative coherence [108,109].

As in depression, voice markers including F0, jitter, shimmer, and HNR are also found in PTSD [110,111]. Speech rate in both depression and PTSD was shown to be negatively correlated with severity of symptoms. Reduced tonality in vowel space is found in self-reported symptoms of both PTSD and depression [112]. However, in PTSD without MDD, voice markers showed slower and more monotonous speech and less change in tonality [113]. Finally, a recent study showed that audio intensity and reduced pitches per frame could predict PTSD, whereas reduced frequency of positive words seemed to rather predict depression [106].

A core symptom of PTSD is an exaggerated startle response [2]. In their study on patients with PTSD, McTeague et al [114] focused on the eye-blink startle response to narrative texts corresponding to traumatic stories. The authors found a heightened startle reflex response during trauma imagery, which was also evident in reactivity to anger, panic, or physical danger compared with controls. Surprisingly, participants with multiple traumas showed a blunted reactivity compared with participants with a single trauma. Furthermore, participants with multiple traumas had more chronic and severe PTSD with more anxiety and mood comorbidity [114].

In the same paradigm, Blechert et al [115] studied the bodily startle response to electric stimuli in individuals with trauma, with and without PTSD. The results implied a primary response characterized by reflexlike facial and torso movement and a secondary response characterized by an emotional facial response. Besides, participants with PTSD had a stronger initial startle response and a more negatively valenced secondary facial expression compared with controls with trauma. Beyond the startle response to threat, several studies with eye tracking showed that higher levels of PTSD symptoms were associated with an increased attentional bias toward threat [116]. Some authors assumed that negative affect in PTSD could have an important role in maintaining this attention bias [117].

More generally, Katz et al [118] emphasized the importance of trauma disclosure. The authors analyzed 40 video interviews of children who have suffered an abuse, but only half of them disclosed it. Children who did not disclose the abuse showed specific nonverbal cues, such as more physical disengagement, than children who disclosed. Schultebraucks et al [106] performed deep learning audio- and video-based classification of depression versus PTSD for 81 participants, 1 month after their admission to a trauma center following a traumatic exposure. The authors successfully classified between PTSD and MDD status with a high discriminatory accuracy. Likewise, several voice and video markers have been identified as predictive of either PTSD or MDD. In a video analysis, higher fear and anger expressivity were predictive of PTSD, while higher contempt expressivity was important for the classification of MDD. In an audio analysis, an increased use of first-person singular pronouns, lower audio intensity, and reduced pitch were important features for the PTSD classification, whereas a reduced frequency of positive words was predictive of depression.

In PTSD, modifications in speech prosodic and source features are found as in depression [104,110-113], but specific changes are reported in the speech content, especially during trauma narrative or trauma recall, that is, use of first-person singular pronouns [105,106], and changes in narrative coherence [108,109].

In the same way, behavior analysis shows a specific startle response during trauma recall (ie, eye blink or bodily startle) [114,115]. Besides, facial expressivity with higher fear or anger could be predictive of the presence of PTSD [106].

In physiological measures, for HRV, PTSD studies indicated a reduced parasympathetic activity (ie, HF, LF, and time domains) [119,120] as in depression. However, the presence of trauma in people with depression could further decrease HRV (ie, HF) [122]. For EDA, the presence of PTSD, especially with memory intrusiveness felt, leads to an increased EDA reactivity during trauma recall while acute dissociative symptoms could have the opposite effect [123,124,126].