Musimo : Music Emotion and Instrument Detection System

This paper presented Musimo, an intelligent audio analysis platform comprising two deep learning modules for music emotion recognition and multi-label instrument detection. The emotion analysis module employs a VGGish-BiLSTM-Attention architecture trained on the Emotify dataset to predict continuous GEMS-9 emotion intensities, achieving a test MAE of 0.1245 and per-emotion Pearson correlations of 0.399–0.676. The instrument detection module uses a CNN-CRNN architecture trained on OpenMIC-2018 to detect the presence of 20 instrument classes from Mel-spectrogram inputs, with sigmoid multi-label outputs appropriate for polyphonic audio. Both modules adopt AdamW/Adam optimization, ReduceLROnPlateau scheduling, early stopping, gradient clipping, and stochastic data augmentation for stable training. They are deployed as FastAPI REST services and integrated with a shared frontend that renders emotion timelines, radar charts, and instrument confidence bar charts, providing actionable insights for sound engineers, video editors, and music composers. Future work will explore music-specific pretrained embeddings (MERT, Music2Vec) to replace VGGish, hierarchical temporal modeling across segments to capture long-range musical structure, multi-modal fusion combining audio with lyric sentiment and symbolic features, class-weighted BCE and focal loss to address instrument class imbalance, and real-time streaming inference through model quantization and ONNX export.

[1]     S. Hershey et al., "CNN Architectures for Large-Scale Audio Classification," in Proc. IEEE ICASSP, 2017, pp. 131–135. [2]       M. Gemmeke et al., "Audio Set: An Ontology and Human-Labeled Dataset for Audio Events," in Proc. IEEE ICASSP, 2017, pp. 776–780. [3]          M. Schedl, H. Zamani, C.-W. Chen, Y. Deldjoo, and T. Elahi, "Current Challenges and Visions in Music Recommender Systems Research," Int. J. Multimedia Info. Retrieval, vol. 7, no. 2, pp. 95–116, 2018. [4]                J. A. Russell, "A Circumplex Model of Affect," J. Personality Social Psychology, vol. 39, no. 6, pp. 1161–1178, 1980. [5]      M. D. Zentner, D. Grandjean, and K. R. Scherer, "Emotions Evoked by the Sound of Music: Characterization, Classification, and Measurement," Emotion, vol. 8, no. 4, pp. 494–521, 2008. [6]      A. Vaswani et al., "Attention Is All You Need," in Advances in Neural Information Processing Systems (NeurIPS), 2017, pp. 5998–6008. [7]              J. Hu, L. Shen, and G. Sun, "Squeeze-and-Excitation Networks," in Proc. IEEE CVPR, 2018, pp. 7132–7141. [8]           R. Panda, R. Malheiro, and R. P. Paiva, "Novel Audio Features for Music Emotion Recognition," IEEE Trans. Affective Computing, vol. 11, no. 4, pp. 614–626, 2020. [9]       B. McFee et al., "librosa: Audio and Music Signal Analysis in Python," in Proc. Python in Science Conference, 2015, pp. 18–25. [10]      E. Fonseca et al., "OpenMIC-2018: An Open Dataset for Multi-Label Audio Classification," in Proc. ISMIR, 2018.