This section presents a method that enables quick inference of a person’s perceived stress level using smartphone-integrated PPG and thermography. We call these measurements instant measurements to differentiate them from the short measurements (typically between 2 min and 5 min), which have been previously defined in the literature [47].

First, we describe software we implemented. This includes a recording setup and a set of techniques to produce reliable PPG-derived HRV profiles and sequential nose tip thermal variations (called hereafter the thermal variability sequence) from the thermal imaging sensor. We then introduce our study protocol to induce different levels of mental stress and collect short sequences (20 seconds) of cardiac pulse–related and thermal events together with self-reports of perceived mental stress scores. Third, we extract low-level (1-dimensional PP intervals and thermal variability sequences) and high-level hand-engineered features, comparing the performance of our system over the 2 sets of features and sensor modalities. We conclude by comparing our approach to data labeling with standard approaches to discuss the effect of intersubjective variability in reporting stress scores.

The main cardiovascular sensing channels of this work are the rear RGB camera of a mobile phone (LG Nexus 5) and a low-cost thermal camera (FLIR One 2G) attached to the phone. Figure 1 shows the smartphone with the attached thermal camera, the required finger placement and light emission for PPG, and the physiological measurement interface.

Although the smartphone imaging–based PPG measurement can be performed in either a contact [6,7] or a contactless manner [8], in our work, we only focus on a contact-based imaging PPG. The reason is based upon previously repeated investigations within clinical studies [6,10] reporting its high accuracy. In addition, given that a normal RGB camera is only sensitive to a narrow electromagnetic spectral range of visible light in the so-called visible spectrum [48], adequate lighting is required before it can be used as a PPG sensor. Hence, a light emission from the rear flash light-emitting diode (LED) is used and a user is required to hold the smartphone body and place his/her finger over both the back camera and flash light (Figure 1). Unfortunately, the use of the back flash limits the duration of the measurements in some devices since its heat can potentially burn a person’s skin. As shown in Figure 2, a large amount of heat is produced by the LED emission from the chosen smartphone (LG Nexus 5) in just 25-30 seconds of operation. A similar amount of heat was observed from another mobile phone (Samsung Galaxy 6 in Figure 1). Since temperatures above 50°C are potentially damaging to human skin tissues, for example, skin erythema could occur from 25 seconds heating at 51.07°C [49], we limit the cardiovascular measurement to a 20-second time period. This is also the required minimum duration for obtaining valid HRV metrics values, particularly LF/HF [47].

To capture a time series of apparent thermal sequences, we developed bespoke recording software using the FLIR One library (FLIR Systems). The interface is shown in Figure 1. Considering the thermal properties of human skin, the emissivity of the thermal imaging sensor was fixed at 0.98 [50]. As the thermal imaging system does not guarantee a consistent frame rate [48], the recording interface stores the time stamp with each image frame.

A data collection study was carried out to gather physiological data from participants during different tasks that induced different levels of mental load. The data collection protocol is described below.

First of all, we tested the reliability of the physiological measurements. From the 17 participants, we collected 102 sets of the estimated BVP signals, PP intervals, and thermal variability sequences from 20-second instant measurements taken after each Stroop and Math task and after each resting session. However, 2 sets of data were not recorded because of phone battery issues at the end of 1 experiment, and 1 set was not recorded as 1 participant clicked the turn-off button on the phone by mistake. A total of 6 further sets had to be discarded because some participant’s nose was not visible on thermal images (nose outside of the range of view because of sudden severe coughing during the 20 seconds, or because of head turned toward the experimenter, or the nose was covered by a person’s hand). Although these disturbances were often transient, they meant that data could not be collected within the 20 seconds immediately following the end of the stressor. An analysis of the thermal data from Rest 1 also showed some extreme patterns in the nose tip temperature (eg, sudden increase in temperature). This may be explained by the fact that the experiment was conducted during the winter and temperatures outside of the experimental room were often significantly lower. This included both outdoors and indoors, in the corridor where the participants waited for the experiment. Despite the temperature changes, the Rest 1 data were kept in the dataset. A total of 93 sets were used for the study.

As the measurement capability of smartphone PPG has previously been thoroughly investigated in earlier studies [6,9,10], we only tested the reliability of the cardiac pulse signals measured with our approach and compared it with the mean brightness intensity–based method, which has been dominantly used [6,7,9]. For this, we used the relative power Signal Quality Index (pSQI), which is to assess the strength of physiological signals in a frequency range of interest as a measure of quality [11,53,73,74]. The pSQI for the BVP signals can be expressed in the following manner (equation 3):

where 0≤P ≤1, is the power spectral density of BVP signals (in our case, in equation 2), and are the lower and upper boundary of expected HRs, respectively. Here, we set the expected HR range to 0.8 Hz (48 bpm) to 2.0 Hz (120 bpm) given that HRs of healthy adults mostly fall into this range [54]. To minimize effects of the baseline wander and high-frequency noise on this signal quality test [6,74], we used band-pass filtered BVP signals (0.7 Hz-4.0 Hz) as in a study by Chan P-H et al [6]. Figure 6 shows the better quality of the estimated BVP signals B̂ from the proposed method—Equation (2)—than that from the mean intensity method (Proposed: mean 0.755, SD 0.068; Traditional: mean 0.692, SD 0.075).

Figure 7 shows examples of thermal images taken from the participants during the data collection study. From our observations, we found that respiration influences the nasal tip temperature measurement in some cases. For instance, in Figure 7, thermal images of a person’s nose tip surface, which were sequentially captured, show that inhaled air changed the nose tip temperature. Hence, we tested how much participants’ respiratory cycled events affected the nose tip temperature measurements by using the pSQI in Equation (3) with the expected respiratory rate of interest (from 0.1 Hz-0.85 Hz) as used by Cho Y et al [11]. Figure 7 demonstrates how the measured nose tip temperatures involved respiratory cyclic patterns (respiratory pSQI: mean 0.714, SD 0.163), indicating that such affected temperature patterns may lead to wrong stress-level classification. On the other hand, the processing technique we propose to use (Figure 4) instead led to reducing respiratory artifacts on the measurement (respiratory pSQI: mean 0.157, SD 0.091).

An important step was the analysis and possible normalization of the self-reported stress scores. The boxplot in Figure 8 (top) shows the distribution of the self-reported scores over the resting periods and the different sessions and tasks. It is clear that the stress elicitation procedures did overall produce the wanted levels of stress with the hard sessions scoring higher than the easy sessions and the latter scoring higher than the resting periods (Rest from Session 1: mean 1.49, SD 1.94; Rest from Session 2: mean 1.30, SD 1.26; Stroop Easy: mean 2.17, SD 1.46; Math Easy: mean 2.66, SD 1.80; Stroop Hard: mean 3.92, SD 2.11; Math Hard: mean 5.17, SD 2.55) despite 2 outliers. However, the wide boxplots also show intersubject variability in self-reporting. In addition, the ranges (maximum-minimum) in scores for each participant differ quite highly (Maximum range: 8.75, Minimum range: 1.5; mean 4.7, SD 2.1), further suggesting the need for normalization of the scores.

Therefore, we normalized the data for each participant with respect to their range of scores over all the sessions. Figure 8 (middle) shows the original data and Figure 8 (bottom) shows the normalized data. The normalization helps to identify 2 main modes in the score distributions, suggesting the presence of 2 main clusters of stress levels. Given the subjectivity of stress ratings and the limited amount of data sets to carry a multilevel model, in this paper, we focused on binary classification of perceived mental stress: no/low stress versus medium/high (or very high) stress. The K-means separation between the 2 clusters is represented by each different color in Figure 8 (bottom).

We tested the correlations among the original self-reported scores, normalized self-reported scores, and the high-level hand-crafted PRV and thermal metrics as summarized in Table 1 (using Pearson correlation coefficients). The normalized self-scores maintained a high correlation with the original scores (r=.752, P<.001). Although some metrics of each physiological sensing channel were significantly correlated among themselves (eg, PRV F2-F4: r=.838, P<.001; Thermal F1-F3: r=.803, P<.001), the correlation values were lower across sensing channels. In addition, only SDSTV shows approaching significance but low correlation with the self-report scores (r=.196, P=.059), indicating that each individual engineered metric alone could not lead to high discrimination among perceived levels of stress.

Figure 9 shows values of each precrafted metric across the sessions (rest and 3 stressful events, ie, Stroop: easy/hard and Math: easy/hard) and across the labels produced by the labeling technique. As shown in Figure 9, there was no common pattern found between 2 easy or hard tasks, although they were designed to induce similar levels of mental stress (eg, easy: low stress level, hard: high stress level). For example, Thermal F1 appeared to strongly decrease during the Math hard task but not during the Stroop hard task, Thermal F2 increased with the Stroop hard task but less during the Math hard task. PRV F5 was generally high after both Math easy and hard task sessions than the Stroop hard session. This can further indicate that each metric alone from the instant measurement is less likely to contribute to the inference of each session. On the other hand, when we applied our labeling technique, Thermal F1 values grouped into Stress were generally lower than No-Stress data as shown in Figure 9, consistent with findings from the literature [38,40,41].

To evaluate the performance of instant stress recognition, we used a 17-fold leave-one-subject (participant)-out (LOSO) cross-validation. LOSO was chosen to test the ability to generalize to unseen participants (one size fits all) [36,70]. Figure 10 summarizes the accuracy results of the 3 classifiers (NN1, NN2, and kNN) using LOSO (N=17) for 3 different cases: (1) multimodal approach by simply combining features from both sensing channels (PRV, Thermal), (2) unimodal approach using thermal features, and (3) unimodal approach using PRV features. Both NN1 and NN2 used our proposed low-level features only (ie, PP intervals and thermal variability sequences). Overall, the NN2-based multimodal approach produced the highest mean accuracy of 78.33% (SD 15.43), mean F1 score of 77.92%, in discriminating between no-stress and perceived stress (see confusion matrix in Figure 10 for details). The NN1 (whose hidden layer is smaller than that for NN2) produced a lower accuracy (mean 66.76%, SD 21.75). From all cases of modality, the kNN with the high-level features (ie, using the hand-engineered 6 PRV and 3 thermal metrics) performed worst. A similar pattern can be seen for the PRV unimodal channel (NN1: mean 65.78%, SD 20.55; NN2: mean 68.53%, SD 18.89; kNN: mean 50.20%, SD 19.63). For the thermal channel, the NN1 appears to perform marginally better (mean 58.82%, SD 21.11) than the NN2 (mean 56.67%, SD 18.79), but both NNs again perform better than the kNN (mean 48.14%, SD 16.52).

However, it should be noted that, for all the models, the confusion matrices for the thermal case (Figure 10, Thermal) show a clear bias toward the no-stress class. Given this bias and the fact that thermal data from the Rest 1 sessions appeared to be affected by the large variation in temperature between the waiting space and the experiment room (in addition, some participants had just arrived from the outside while others had already been indoor for sometimes), we reran the models, discarding the data from the Rest 1 sessions. Although the overall performance over this modality did not largely change (NN1: mean 58.14%, SD 23.33; NN2: mean 58.14%, SD 21.59; kNN: mean 55.88%, SD 22.38) and NN1 and NN2 still perform better than the kNN with hand-engineered features, all the confusion matrices (Figure 10 bottom) show more balanced results and a better prediction of the stress class overall.

A repeated measures analysis of variance was carried out on results from the 17 folds (including the Rest 1 data) to compare the 2 NN modeling approaches (that use our proposed low-level features) with the kNN (that uses hand-engineered metrics) to determine whether there was a statistical mean difference in performance. The results show significant differences between the methods for the multi and the PRV modalities—PRV+Thermal: F_2,32=3.763, P=.034, η_p²=.190; PRV: F_2,32=6.001, P=.006, η_p²=.273. No differences were found for the thermal case—Thermal: F_2,32=2.304, P=.116, η_p²=.126. Posthoc paired t test with Bonferroni correction (see Figure 11) showed that NN2 performed significantly better than kNN for the unimodal PRV case (PRV: P=.023). For the multimodal case, NN2 approached significantly better performance than kNN (PRV+Thermal: P=.064) and NN1 (PRV+Thermal: P=.052). NN1 did not significantly perform better than kNN; however, it presented a positive trend in the unimodal PRV case (PRV: P=.091). Even if no significance differences were found over the unimodal thermal case, the graphs in Figure 11 show how the 2 NN models performed slightly better than the kNN for all cases including the thermal one. It could be expected that in the case of deployment, a larger sample of data for each class could indeed lead to statistical significance.

Lastly, we investigated the effect of the normalization and K-means clustering of self-reported scores in inferring the perceived stress levels. For this part of the study, we removed the Rest 1 data. There were 2 reasons for this. First, we wanted to avoid the noise from the set of data affecting the comparison among the labeling methods. Second, this was also to obtain a more balanced number of instances in each class for testing different labeling methods, less biasing the learning process. The comparison of models over the different labeling techniques did not aim to obtain better performance but to understand how normalization and different clustering approaches could affect the modeling by acting on class separation and interperson variability in subjective self-reports. We were also interested in understanding how sensitive the system was in separating stress scores by using the same dataset and merging the intermediate levels with 1 of the 2 classes (L1 and L2).

We tested the 3 models (NN1, NN2, and kNN) for the multimodal approach with the different labeling strategies (L2-L4, introduced in the previous section). Figure 12 summarizes the accuracy results for 4 different strategies—L1: the main method, L2: K-means with k=3, but combining no-stress and moderate level stress scores as 1 group, L3: K-means with k=2, dissecting the moderate level scores into no-stress and stress, and L4: original scores divided by a point between no-stress and moderate levels (ie, 3.334 of 10, see Figure 5c). The results showed that the L1 performed best in separating the bimodal distribution of normalized self-reported scores and helped address the interpersonal variability issue. Indeed, all 3 models obtained the best accuracy with L1 and the worst performance for L3 and L4 with L4 being marginally better than L3. Finally, it should be noted that in the case of L3 and L4, the best performance was obtained with NN2 rather than NN1. This may indicate that mapping feature values to perceived stress scores may benefit from a larger hidden layer to capture the complexity of the relation.

Our first contribution is to develop a new set of preprocessing techniques to enhance the quality of the signal extracted from either the PPG channel, which detects blood pulse variability, or the thermal camera, which detects vasoconstriction/dilation induced nasal temperature variability. This is particularly important in mobile, ubiquitous settings where physiological sensing setups are still of lower quality and have to be less controlled in comparison with the ones generally used in medical environments.

With the data collected from our stress-inducing tasks, we wanted to test the possibility of building algorithms that can reliably and continuously capture (1) a person’s BVP pattern from the smartphone camera and (2) nose tip temperature sequence from the add-on thermal camera. Reliable BVP recording is critical, particularly for short-term measurements [26,47]. The conducted signal quality test with the pSQI showed that our method produced higher quality BVP signals than the ones obtained with traditional camera-based PPG approaches [6,7,9] (see Figure 6). In addition, we found that a person’s respiratory cycles interfered with capturing thermal variations accurately from a person’s nasal area (Figure 7). Hence, we built a new technique to minimize such effects and gather a more reliable nose tip thermal signature. This was achieved through the use of an advanced thermal ROI tracking [11] and signal processing techniques to filter out breathing cyclic events (Figure 4) on measured temperatures from the nose area.

However, it should be noted that despite the use of the quantization approach that helps handle environmental temperature changes [11], thermal data during Rest 1 was affected by the difference in temperature between the waiting area and experiment area. This effect was further enhanced when the participants just arrived from outdoors with body temperature being strongly influenced by the cold weather outdoors (winter season). This is important because if the system has to be used, it is crucial for the person to use it in the same environment where stressful events occur. It should also be tested in future studies if a decrease in nose tip temperature may be saturated by very cold environments and therefore be less informative in such situations for automatically detecting mental stress.

We found that the capability of the HRV metrics, used as high-level features in the literature [18,20,25], in instantly quantifying stress was very limited (see Table 1). This is important as despite their general use (eg, literature in psychology or affective computing), there have still been arguments of such metrics with regard to the possibility of oversimplifying physiological responses [33-35]. It should be noted that although we used PPG-derived metrics rather than the more investigated ECG-derived metrics, strong correlations have been found between the 2 signal metrics in the case of healthy participants and limited physical movement [29,45]. Stressors in general affect cardiac pulse–related events even if the 2 types of events (heart rate and BVP) may be differently affected within nonhealthy or elderly population and extreme situations (hot temperature) [75-77]. It should also be noted that although mathematically, a shorter measurement period could lead to a lower resolution of data in the frequency domain resulting in a lower accuracy in computing metrics such as LF/HF [21], recent studies have validated the use of them with very short measurements, from 10 seconds to 30 seconds [47].

Similarly, the metrics applied to short-term nasal thermal data (eg, TD: Temperature Difference) did also weakly contribute to stress quantification. This may explain inconsistent findings in the literature where such metrics have been used to capture thermal responses to stressful events [41,78]. All in all, the results suggested the need to develop a novel way that describes dynamical information of BVP and vasoconstriction/ dilation-related nasal temperature to help improve the understanding and capturing of their complex phenomenon.

On the basis of the low correlation between perceived mental stress levels and typically engineered metrics for these 2 signals, we proposed to use thermal variability and PP interval sequences as a novel set of low-level features to capture stress responses of cardiovascular activities. With this, we investigated how to benefit from automatic feature learning capabilities of machine learning classifiers (ie, NNs) in instantly inferencing mental stress. The results showed clear improvements in performance. Indeed, our proposed method with the 2 cardiovascular signals achieved 78.33% correct recognition accuracy with the NN2, whereas only 60.59% from the kNN with the hand-engineered features. Similarly, using the HRV-related features only, there was an improvement by 18.33% with respect to the traditional approach (50.20%). The improvement on the thermal channel was smaller but still evident from the results.

In addition, 2 further contributions can be highlighted from our approach to the modeling of automatic stress inference: instant measurements and no need for baseline. First, previous work required relatively long-term measurements of between 2 minutes and 5 minutes [25,41,54]. Indeed, our results demonstrated the possibility to use just a 20-second measurement to automatically discriminate between stress and nonstress moments. This approach achieved state-of-the-art performance when compared with approaches using much longer measurements, up to around 70%-80% correct recognition from LOSO cross-validation [70]. This is very important given that stillness is critical during PPG measurements and for thermal imaging to a certain extent. In fact, even if automatic ROI-tracking methods may help with thermal measurements, people tend to easily move away from the camera or cover their nose with their hands (5 participants did so at least once even for 20 seconds).

Second, our approach (more reliable signal and richer features) led to state-of-the-art results without the use of a baseline. This is critical to everyday life settings as in everyday life, such baselines may be difficult to establish. Resting periods just before a stressful event cannot be planned, and continuously gathering such measures can be costly, whereas at the same time, nonstressful resting periods would also need to be automatically detected. In addition, our data from resting periods show that such a gold standard resting situation does not exist and environment temperature may change drastically, affecting skin temperature. This could have been because of a lab effect but general everyday life may also have specific effects on the data. Even when using differential features (eg, temperature differences between 2 areas of the face-forehead and nose tip), a baseline period was used [42]. The lack of a baseline is overcome here by proposing richer features capturing informative physiological variations over time.

Setting the ground truth is a difficult process when dealing with subjective reports. How to use self-reports to label the data is a critical issue in the field because of their subjectivity. Interpersonal variability has been repeatedly reported as a critical barrier for building stress inference or quantification systems that can generalize across people [24,70]. The intersubjectivity of self-reports and the need to reduce the number of classes along with types of applications or the size of the dataset require some decisions on how to refine the labels to be taken. In doing so, there is the danger to add noise to the dataset and hence to the modeling process. We explored how different labeling techniques may affect the modeling process.

We proposed to address this problem. The first step was to use a standard normalization technique to take into account personal score ranges over all tasks that aimed to induce a wide range of stress levels (from none to medium to quite high). This transformation led to a bimodal distribution highlighting at least 2 opposite levels of stress (low and high), whereas it still maintained its strong correlation with the original scores (r=.752, P<.001). The bimodal distribution is interesting as, given the low number of participants, it suggests the moderate level of stress is not well separated from the other 2 classes. A binary classification was hence a sensible approach to take in this paper; however, with larger datasets, a more refined analysis and modeling should be carried out. Second, we used a machine learning clustering technique, K-means, to improve separation of the scores into 2 classes of stress. The results obtained from the comparison of our approach (L1) with its variation (L2) and the more typically used approaches (L3 and L4) led to an interesting lesson on how to create a more reliable ground truth rather than increase noise in labeling.

Then, how should the data be clustered? According to the number of stress levels to be recognized or according to the number of stress levels the data collection experiment was set to induce? The latter approach appeared to be more successful. All labeling methods using K=3 (L1, L2, and to a certain extent L4) performed better than L3 using K=2. This suggests that directly clustering according to the number of classes to be recognized (2 in our case) may spread instances with similar stress-level responses (in this case, medium responses) across classes introducing noise rather than overcoming the problems of intersubjectivity. However, it should be noted that the normalization step was important. Indeed, the models built on either L1 and L2 using the normalized scores performed better than L4 where the original scores were used instead.

Another important issue to be addressed is how should the data be grouped when the number of classes to be detected is smaller than the number of levels induced? This decision could be needed either because there were no sufficient instances for a more refined inference or because the application at hand did not require such level of granularity (at the risk of introducing noise because of intersubjective variability). The results showed that L1, collapsing the moderate level into the high-level class, led to better performance than L2, where medium and no/low stress scores were instead combined. This may suggest that unless the stress level is very low, stress responses share more similarities than with no-stress responses. A more in-depth analysis of this aspect could be part of a future work and it may require an in-depth analysis of individual responses and validations over other datasets.

Although the results provide some interesting insights on how to cluster data from experiments, a question remains on how to deal with data from real-life situations. It is expected that in real-life situations, larger datasets may enable finer levels of discrimination personalized to a specific person. In such situations, as the dataset grows, parameters for labeling may need to be adapted to optimize the personalization. However, such rules we used could be helpful to bootstrap models on the basis of experimental datasets or well-structured initial real-life data collections. The bootstrapped models could then be personalized to specific users and recognition levels as data would be continuously collected by the person.

Despite the findings and contributions described above, there is still space for improvement. First, our proposed approach did not perform properly on multiple levels of stress (labeling the data using perceived self-scores). As discussed, this was most probably because of the limited size of the dataset, especially for the medium level of stress (out of 3 levels). Deploying built software in real life could be a way to build a larger dataset. With a function to collect self-reported person’s perceived stress scores (eg, digitalized VAS sliding bar in an app), this data collection in the wild could produce a sufficient size of cardiovascular signal sets to support more reliable performance in inferencing multiple levels. In addition, it would be interesting to investigate how the transformation of the self-reported scores could be used to support multiclass classification.

Second, this work focused on sedentary situations (but without constraining one’s mobility) and did not include physical activity (eg, walking). It is well-known that physical activity induces cardiovascular changes, in turn affecting stress inference performance [58]. Hence, it would be interesting to test the instant stress inference ability of our system in situations where there is a considerable amount of physical activity, for example, industrial factory work floor.

Finally, investigating the reliability of mobile sensing technologies themselves was outside the scope of this paper—see reviews on this topic [79]. We aimed to contribute a better stress inference method that can be used independently regardless of what sensing technology is used. This may be even more crucial when the sensing technology may not be as accurate and fine-grain as more expensive and medically approved technology.

With the long-term aim of building a stress monitoring system for mobile, everyday use, this paper focuses on the use of smartphone-based imaging capabilities: PPG and thermal imaging. To overcome the difficulties in using smartphone imaging for long period measurements, we propose a novel method that quickly infers a person's perceived level of stress from instant physiological measurements. This is achieved by (1) developing a more reliable PPG-sensing technique to extract a person’s BVP and its variability, (2) building a thermal imaging–based vasoconstriction monitoring system, (3) investigating the performance of widely used high-level features from PPG and nasal temperature in instant stress inference tasks, (4) proposing novel low-level features to represent HRV and thermal variability, (5) building an automatic feature learning–based multimodal perceived stress recognizer, and (6) investigating effects of clustering self-report scores to take into account the subjectivity of self-reports and ensure clear separation among the levels of stress to be modeled.

Through the data collection study with 17 participants and a series of stress-inducing tasks with different levels, we demonstrated how this system was able to achieve state-of-the-art performance using 20 seconds of data, rather than 2 to 5 minutes typically required by existing methods. This work makes smartphone imaging–based physiological computing capabilities more feasible for real-life applications, opening new possibilities for the development of mental-stress support apps and research.