Automated Analysis of Emotional and Behavioural States Using AI

Automated Analysis of Emotional and Behavioral States Using AI (AAEBSAI)” addresses the critical need for comprehensive emotional state analysis in therapy sessions through multi-modal AI integration. Traditional manual analysis methods are time-intensive, subjective, and often miss subtle emotional cues that could inform clinical decisions. This research presents a framework that combines speech-to-text transcription, text emotion analysis, acoustic feature extraction, and facial expression recognition to create a unified, timestamped emotional timeline. By integrating OpenAI Whisper, BERT-based emotion classification, Librosa acoustic analysis, and MediaPipe/FER facial recognition, the system provides therapists with comprehensive insights into patient emotional states across three complementary modalities. Evaluation demonstrates that this multi-modal approach achieves 80-90% accuracy across different emotion types while processing typical therapy sessions in 3-8 minutes. The AAEBSAI framework offers a privacy-focused, locally-deployed solution with applications in mental health assessment, therapy supervision, and clinical research.