LLM based Music Captioning

LLM based Music Captioning

Ms. Yogita Chavan, Yash Palkar, Ajay Gore, Apurv More

Department of Computer Engineering

New Horizon Institute of Technology and Management Thane, India

Abstract—Automatic music captioning, a process that generates natural language descriptions for music tracks, is pivotal in enhancing the comprehension and organization of vast musical datasets. However, researchers encounter challenges due to the scarcity and labor-intensive nature of collecting existing music-language datasets, often limited in size. To address this data scarcity issue, this project proposes leveraging large language models (LLMs) to artificially generate description sentences from extensive tag datasets. This initiative leads to the creation of a novel Large Language Model based Pseudo music caption dataset. The project systematically evaluates  this  large-scale  music  captioning word embeddings for text generation. introduced a temporal Attention mechanism for alignment between audio and text by pairing a pre-trained harmonic CNN encoder with an LSTM layer. generated playlist titles and descriptions using pre- trained GPT-2 Currently, the primary challenge of track- level music captioning is the scarcity of large-scale public datasets. used private production music datasets. also used a private dataset with 44M music-text pairs on YouTube, to address this data issue, a community-driven data collection initiative has been proposed. As of now, the only publicly available dataset for track-level music captioning is Music- Caps.

Keywords—Automatic music captioning, natural language descriptions, large language models (LLMs), pseudo music caption dataset, MusicCaps, dataset generation, quantitative evaluation metrics, human evaluation, transformer-based model, zero-shot learning, transfer learning, music understanding, data scarcity, music-language datasets.