Machine Learning Based Prognostic Analysis of a Hybrid Speech Emotion Recognition System Integrated with Gemini Live API: A Comprehensive Performance Assessment

This paper presented VerbiQ, a hybrid Speech Emotion Recognition system for real-time professional soft-skill coaching, and evaluated it through a comprehensive machine learning prognostic framework on the RAVDESS corpus. All results reported below are derived directly from experimental measurements. The principal findings are:   •                  Baseline DNN is the Optimal Architecture: The two-hidden-layer Dense Neural Network (Dense 256 → Dense 128 → Softmax 8) achieves the best overall performance: test accuracy 0.8194, Macro F1-score 0.8229, Weighted F1-score 0.8194, Macro Precision 0.8248, Macro Recall 0.8222, and Macro AUC 0.9706 ,  the highest Macro AUC of any model evaluated. It is identified as the optimal backbone for the VerbiQ deployment pipeline. •                  Improved DNN Underperforms Due to Premature Early Stopping: The deeper Improved DNN achieves only test accuracy 0.3403 and Macro AUC 0.7522, completing training after ~14 epochs ,  substantially fewer than required for a deeper, more regularised network on a small dataset. These results highlight a practical constraint: deeper architectures require larger datasets or extended training budgets to outperform simpler baselines. •                  1D-CNN Fails on Pre-Extracted Feature Vectors: The 1D-CNN achieves near-chance classification (accuracy 0.1435, Macro AUC 0.6006, Macro F1 0.05) because its convolutional inductive bias ,  designed to detect local sequential patterns ,  is incompatible with the tabular structure of pre-extracted 125-dimensional feature vectors. This result motivates the application of 1D-CNN directly to raw waveforms in future work. •                  87 Principal Components Capture 95% of Feature Variance: PCA analysis confirms that the 125-dimensional vector is information-dense, requiring 87 principal components to reach the 95% explained variance threshold. Aggressive dimensionality reduction would discard discriminative variance from auxiliary features, particularly the Pause Ratio and Spectral Centroid. •                  t-SNE Reveals Substantial Class Intermixing: The 2D t-SNE projection shows extensive overlap between all eight emotion classes rather than clean cluster separation. This confirms that acoustic emotion classes are not linearly separable in low-dimensional projections and that the Baseline DNN’s strong performance relies on high-dimensional decision boundaries not visible in t-SNE. •                  Pause Ratio Provides Orthogonal Coaching Signal: Violin plot analysis confirms that all eight emotion classes have Pause Ratio medians in the narrow range 0.43–0.50 with substantial distributional overlap, confirming that Pause Ratio is not an independent emotion classifier. However, its near-zero correlation with MFCC coefficients (Figure 1(c)) confirms that it adds orthogonal information to the feature space, and its linguistic relevance to fluency and hesitation measurement provides direct added value for the coaching application. •                  Dataset Imbalance Requires Attention: The Neutral class contains only 192 samples compared to 384 for all other classes, a structural RAVDESS imbalance that was mitigated through balanced class weighting. Future work using the full four-corpus combined dataset (RAVDESS + CREMA-D + TESS + SAVEE) would substantially reduce this imbalance by introducing a larger and more demographically diverse training set.   High-accuracy real-time affective coaching is achievable with a two-hidden-layer dense network on pre-extracted acoustic features, Combining local acoustic analysis with LLM-generated feedback produces a technically sound and educationally viable approach to automated soft-skill coaching.

[1] World Economic Forum. (2023). Future of Jobs Report 2023. World Economic Forum, Geneva. [2] El Ayadi, M., Kamel, M.S., & Karray, F. (2011). Survey on speech emotion recognition: Features, classification schemes, and databases. Pattern Recognition, 44(3), 572–587. [3] Rochester, S.R. (1973). The significance of pauses in spontaneous speech. Journal of Psycholinguistic Research, 2(1), 51–81. [4] Livingstone, S.R., & Russo, F.A. (2018). The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS). PLoS ONE, 13(5), e0196391. https://doi.org/10.1371/journal.pone.0196391 [5] Davis, S., & Mermelstein, P. (1980). Comparison of parametric representations for monosyllabic word recognition. IEEE Transactions on Acoustics, Speech, and Signal Processing, 28(4), 357–366. [6] Ancilin, J., & Milton, A. (2021). Improved speech emotion recognition with Mel frequency magnitude coefficient. Applied Acoustics, 179, 108046. [7] Zhao, J., Mao, X., & Chen, L. (2019). Speech emotion recognition using deep 1D & 2D CNN LSTM networks. Biomedical Signal Processing and Control, 47, 312–323. [8] Chen, T., & Guestrin, C. (2016). XGBoost: A scalable tree boosting system. Proceedings of KDD 2016, 785–794. [9] Gupta, H.V., Kling, H., Yilmaz, K.K., & Martinez, G.F. (2009). Decomposition of the mean squared error and NSE performance criteria. Journal of Hydrology, 377(1–2), 80–91. [10] Kasneci, E., et al. (2023). ChatGPT for good? On opportunities and challenges of large language models for education. Learning and Individual Differences, 103, 102274. [11] Puri, T., et al. (2022). Detection of emotion of speech for RAVDESS audio using hybrid convolution neural network. Journal of Healthcare Engineering. [12] Issa, D., Demirci, M.F., & Yazici, A. (2020). Speech emotion recognition with deep convolutional neural networks. Biomedical Signal Processing and Control, 59, 101894. [13] Priyadarshini, M.A., et al. (2024). A multi-feature approach with data augmentation for speech emotion recognition using deep learning. Proceedings of ICCIET-2024, Atlantis Press, 835–856.