LLM BG: Singing Accompaniment Generation with LLM & Encodec

¹Trong-Hieu Nguyen Mau, ^1,2Quoc-Huy Trinh, ¹Truong-Tien Nguyen, ^1,3Minh-Van Nguyen, ¹Khoa Tran, ¹Thanh Do

¹SongGen Team, Ho Chi Minh city, Vietnam
²Aalto University, Espoo, Finland
³Technical University of Denmark, Kongens Lyngby, Denmark

Abstract

Singing Accompaniment Generation (SAG) is a crucial task in song production, aiming to create accompaniment that harmonizes seamlessly with the vocal track. Recently, various studies have proposed unconditional generation models based on Transformers or Stable Diffusion, achieving promising results. However, these methods face challenges in real-world applications due to their lack of control. To address this issue, we propose LLM-BG, a Large Language Model based on QwenV2, capable of generating accompaniment conditioned on vocal audio input and prompt instructions. Through extensive experiments, we successfully generated 12-seconds accompaniment segments that harmonize with the vocal input and can be extended to full-song accompaniment for long vocal tracks, which is promising approach for the Singing Accompaniment Generation task.

Comparing with SingSong