TY  - JOUR
AU  - Song, Meishu
AU  - Yang, Zijiang
AU  - Triantafyllopoulos, Andreas
AU  - Zhang, Zixing
AU  - Nan, Zhe
AU  - Tang, Muxuan
AU  - Takeuchi, Hiroki
AU  - Nakamura, Toru
AU  - Kishi, Akifumi
AU  - Ishizawa, Tetsuro
AU  - Yoshiuchi, Kazuhiro
AU  - Schuller, Björn
AU  - Yamamoto, Yoshiharu
PY  - 2024
DA  - 2024/10/18
TI  - Empowering Mental Health Monitoring Using a Macro-Micro Personalization Framework for Multimodal-Multitask Learning: Descriptive Study
JO  - JMIR Ment Health
SP  - e59512
VL  - 11
KW  - multimodal
KW  - multitask
KW  - daily mental health
KW  - mental health
KW  - monitoring
KW  - macro
KW  - micro
KW  - framework
KW  - personalization
KW  - strategies
KW  - prediction
KW  - emotional state
KW  - wristbands
KW  - smartphone
KW  - mobile phones
KW  - physiological
KW  - signals
KW  - speech data
AB  - Background: The field of mental health technology presently has significant gaps that need addressing, particularly in the domain of daily monitoring and personalized assessments. Current noninvasive devices such as wristbands and smartphones are capable of collecting a wide range of data, which has not yet been fully used for mental health monitoring. Objective: This study aims to introduce a novel dataset for personalized daily mental health monitoring and a new macro-micro framework. This framework is designed to use multimodal and multitask learning strategies for improved personalization and prediction of emotional states in individuals. Methods: Data were collected from 298 individuals using wristbands and smartphones, capturing physiological signals, speech data, and self-annotated emotional states. The proposed framework combines macro-level emotion transformer embeddings with micro-level personalization layers specific to each user. It also introduces a Dynamic Restrained Uncertainty Weighting method to effectively integrate various data types for a balanced representation of emotional states. Several fusion techniques, personalization strategies, and multitask learning approaches were explored. Results: The proposed framework was evaluated using the concordance correlation coefficient, resulting in a score of 0.503. This result demonstrates the framework’s efficacy in predicting emotional states. Conclusions: The study concludes that the proposed multimodal and multitask learning framework, which leverages transformer-based techniques and dynamic task weighting strategies, is superior for the personalized monitoring of mental health. The study indicates the potential of transforming daily mental health monitoring into a more personalized app, opening up new avenues for technology-based mental health interventions. 
SN  - 2368-7959
UR  - https://mental.jmir.org/2024/1/e59512
UR  - https://doi.org/10.2196/59512
UR  - http://www.ncbi.nlm.nih.gov/pubmed/39422993
DO  - 10.2196/59512
ID  - info:doi/10.2196/59512
ER  -