Cross Modal Emotion Detection: Leveraging Speech and Facial Expression Features

Cross Modal Emotion Detection: Leveraging Speech and Facial Expression Features

1. Mrs.R.Suchitra, Professor, 2.Ms. P.Manasa, 3.Ms. S.Manamma,

4.Ms. T.Sai Durga, 5.Ms. T. Roshini Rachel

Department of Computer Science Lendi Institute of Engineering and technology JNTU GV, Vizianagaram,Andhra Pradesh,India

reyya1243@gmail.com,manasapaluri26@gmail.com, simmalamani@gmail.com, roshinirachel2003@gmail.com

Abstract— Emotion recognition plays a vital role in enhancing human-computer interaction by enabling machines to interpret human affective states. This project proposes a cross-modal emotion detection system that leverages both speech and facial expressions to accurately classify emotions. The speech modality utilizes the RAVDESS dataset, where Mel-Spectrogram and MFCC features are extracted and processed through a combination of pre-trained DenseNet and custom CNN models. For the visual modality, facial expressions are analyzed using a Convolutional Neural Network trained on the FER2013 dataset, with real-time emotion detection enabled via live webcam feed.By integrating predictions from both modalities, the system improves the reliability and accuracy of emotion recognition compared to unimodal approaches. The final application is deployed as a user-friendly web interface using Streamlit, allowing users to upload audio files and capture live facial
expressions for emotion analysis. This cross-modal approach demonstrates the effectiveness of multimodal learning in understanding complex human emotions and sets the foundation for more empathetic AI systems.

Keywords— Emotion Detection, Speech Recognition, Facial Expression, Cross-Modal Fusion, BiLSTM, Machine Learning