Multimodal Sentiment Analysis on Social Media Big Data: A Survey of Deep Fusion Architectures

This paper reviewed deep learning approaches to multimodal sentiment analysis on social media, from lexicons to multimodal LLMs and state-space models. The main finding is that, on the benchmarks surveyed, the quality of the pre-trained encoders and of the cross-modal alignment affects results more than the design of the fusion operator, and that beyond a strong text encoder the visual and acoustic channels add only a small increment: MAG-BERT [28] improved on text-only BERT by less than one accuracy point. Evaluation is a second concern. Differences in feature extractors, alignment settings and the has-0/non-0 convention make published results hard to compare, and MERBench [6] indicates that differences between fusion methods diminish when these factors are controlled. Standardized evaluation protocols are therefore needed. Open problems remain in sarcasm and meme detection, multilingual and code-mixed content, missing modalities, explainability and fairness, and efficiency at platform scale. State-space models and multimodal LLMs are promising for long sequences and low-resource settings but remain at an early stage.

1. Soleymani, M., Garcia, D., Jou, B., Schuller, B., Chang, S.-F., Pantic, M.: A survey of multimodal sentiment analysis. Image Vis. Comput. 65 (2017) 3–14 2. Liang, P.P., Zadeh, A., Morency, L.-P.: Foundations and trends in multimodal machine learning: Principles, challenges, and open questions. arXiv preprint arXiv:2209.03430 (2022) 3. Zadeh, A., Zellers, R., Pincus, E., Morency, L.-P.: MOSI: Multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos. arXiv preprint arXiv:1606.06259 (2016) 4. Bagher Zadeh, A., Liang, P.P., Poria, S., Cambria, E., Morency, L.-P.: Multimodal language analysis in the wild: CMU-MOSEI dataset and interpretable dynamic fusion graph. In: Proc. 56th Annu. Meeting Assoc. Comput. Linguistics (ACL), pp. 2236–2246 (2018) 5. Busso, C., et al.: IEMOCAP: Interactive emotional dyadic motion capture database. Lang. Resour. Eval. 42(4) (2008) 335–359 6. Lian, Z., et al.: MERBench: A unified evaluation benchmark for multimodal emotion recognition. arXiv preprint arXiv:2401.03429 (2024) 7. Kim, Y.: Convolutional neural networks for sentence classification. In: Proc. Conf. Empirical Methods Natural Lang. Process. (EMNLP), pp. 1746–1751 (2014) 8. Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8) (1997) 1735–1780 9. Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: BERT: Pre-training of deep bidirectional transformers for language understanding. In: Proc. NAACL-HLT, pp. 4171–4186 (2019) 10. Liu, Y., et al.: RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692 (2019)