Multi-modal Emotion Detection System
Akanksha Unde1, Shashank Kulkarni2, Harshada Sutar3, Pradnya Suryanwanshi4
Department Of Information Technology
Marathwada Mitra Mandal's College of Engineering, Karvenagar, Savitribai Phule University, Pune
---------------------------------------------------------------------***---------------------------------------------------------------------
Abstract –
Emotions are fundamental aspects of human communication and play a crucial role in our daily lives. Accurately detecting and understanding emotions can have profound implications in various domains, including human- computer interaction, mental health analysis, and personalized user experiences. However, existing emotion detection systems often focus on individual modalities, such as facial expressions or textual cues, which limit their effectiveness and real-world applicability. To address this, The Multi-Modal Emotion Detection System presented in this project integrates multi- modal data to effectively detect and classify emotions. By leveraging Convolutional Neural Networks (CNN) for real- time emotion recognition from webcam input and utilizing the Natural Language Toolkit (NLTK) for text processing and analysis, the system aims to capture a more comprehensive understanding of emotional states. The motivation behind this project stems from the need for a reliable and versatile emotion. This project encompasses several key components, including data collection, pre-processing, feature extraction, and fusion of multi-modal information. A diverse dataset comprising facial expression images and corresponding textual data is collected and annotated. The collected data is then pre-processed to ensure its quality and consistency. Feature extraction techniques tailored to each modality are employed to extract meaningful representations from the input data. To effectively combine the modalities, fusion strategies are employed, enabling the integration of facial expressions and textual features. Convolutional Neural Networks (CNNs) are trained to learn the underlying patterns and relationships within the fused data, facilitating accuracy.
Keywords: Deep learning, Image processing, Feature Extraction, CNN Model, Face Detection, Regression, Real Time