NeuroAI: A Multimodal Framework for Text and Speech Emotion Recognition, Suicide/Self-Harm Risk Screening, and PHQ-9-Based Mental Health Assessment

This paper presented NeuroAI, a multimodal framework integrating text emotion recognition, separate suicide/selfharm risk screening, speech emotion recognition, deterministic PHQ-9 scoring, weighted fusion, and clinician-oriented application workflows. The reported component results were 56.50% accuracy and 55.92% weighted F1-score for 28-class text emotion recognition, 97.38% accuracy and F1-score for binary safety screening, and 76.28% held-out validation accuracy for six-class speech emotion recognition. The framework uses configurable weights, missing-modality renormalization, and rule-based safety overrides to make the combined workflow more inspectable. The main contribution is the integration of heterogeneous model outputs and structured questionnaire scoring into one software workflow, rather than a claim of clinical validation. Because the fusion engine has not been evaluated on an independent labeled multimodal clinical dataset, no overall system accuracy is reported. NeuroAI should be considered a non-diagnostic research and decision-support prototype. Future work must establish external validity, measure fusion performance, assess robustness and fairness, and evaluate safety under qualified human oversight before clinical deployment.

[1]   D. Demszky, D. Movshovitz-Attias, J. Ko, A. Cowen, G. Nemade, and S. Ravi, “GoEmotions: A Dataset of Fine-Grained Emotions,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020, pp. 4040–4054. [2]   J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” in Proceedings of NAACL-HLT, 2019, pp. 4171–4186. [3]   Y. Liu et al., “RoBERTa: A Robustly Optimized BERT Pretraining Approach,” arXiv:1907.11692, 2019. [4]   S. B. Davis and P. Mermelstein, “Comparison of Parametric Representations for Monosyllabic Word Recognition in Continuously Spoken Sentences,” IEEE Transactions on Acoustics, Speech, and Signal Processing, vol. 28, no. 4, pp. 357–366, 1980. [5]   H. Cao, D. G. Cooper, M. K. Keutmann, R. C. Gur, A. Nenkova, and R. Verma, “CREMA-D: Crowd-Sourced Emotional Multimodal Actors Dataset,” IEEE Transactions on Affective Computing, vol. 5, no. 4, pp. 377–390, 2014. [6]   Y.-H. H. Tsai et al., “Multimodal Transformer for Unaligned Multimodal Language Sequences,” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019, pp. 6558–6569. [7]   K. Kroenke, R. L. Spitzer, and J. B. W. Williams, “The PHQ-9: Validity of a Brief Depression Severity Measure,” Journal of General Internal Medicine, vol. 16, no. 9, pp. 606–613, 2001.