Human Emotion Recognition Using Multi-modal Deep Learning: A Review of Methods, Datasets, and Challenges

Human emotion recognition plays a vital role in the field of affective computing and finds wide-ranging applications in healthcare, education, robotics, and human-computer interaction. Traditional unimodal approaches—based solely on facial expressions, speech, or physiological signals—often face limitations due to varying environmental conditions, individual differences, and signal noise. To address these challenges, the use of multi-modal deep learning has gained significant momentum, as it combines multiple data streams such as visual, auditory, textual, and physiological inputs to enhance the accuracy and robustness of emotion detection. This review paper presents a detailed examination of recent developments in the area of multi-modal deep learning for human emotion recognition. We explore various deep learning models such as Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), Long Short-Term Memory (LSTM), and Transformer-based architectures, and their effectiveness in processing different modalities. In addition, we study various fusion strategies—early, late, and hybrid fusion—and their respective contributions towards improving recognition performance. The aim of this paper is to provide researchers and practitioners with valuable insights into the current landscape, ongoing challenges, and future opportunities in this rapidly growing domain.