LLM based Music Captioning

Automatic music captioning, a process that generates natural language descriptions for music tracks, is pivotal in enhancing the comprehension and organization of vast musical datasets. However, researchers encounter challenges due to the scarcity and labor-intensive nature of collecting existing music-language datasets, often limited in size. To address this data scarcity issue, this project proposes leveraging large language models (LLMs) to artificially generate description sentences from extensive tag datasets. This initiative leads to the creation of a novel Large Language Model based Pseudo music caption dataset. The project systematically evaluates this large-scale music captioning word embeddings for text generation. introduced a temporal Attention mechanism for alignment between audio and text by pairing a pre-trained harmonic CNN encoder with an LSTM layer. generated playlist titles and descriptions using pre- trained GPT-2 Currently, the primary challenge of track- level music captioning is the scarcity of large-scale public datasets. used private production music datasets. also used a private dataset with 44M music-text pairs on YouTube, to address this data issue, a community-driven data collection initiative has been proposed. As of now, the only publicly available dataset for track-level music captioning is Music- Caps.