Google explores text-to-music generation with MusicLM

In a paper submitted on 26 January, Google researchers describe MusicLM, a system that generates music from text. The paper reports 24kHz audio lasting several minutes and demonstrations guided by humming or whistling. The authors are not releasing the models.

Sulat illustration; Google logo.

Two earlier ideas meet

MusicLM combines AudioLM’s hierarchical audio generation with MuLan’s connection between sound and language.

The September AudioLM paper treats sound as sequences of discrete tokens. One representation captures longer-term structure; another, produced by a neural audio codec, retains the acoustic detail needed to reconstruct a waveform. The hierarchy addresses a practical tension: tokens that carry useful structure do not necessarily preserve enough detail for convincing sound.

AudioLM’s earlier demonstrations continued short speech and piano recordings. Its piano model learned from recorded audio without requiring symbolic musical notation. These results concerned continuing an audio prompt, rather than following a written description of the desired music.

MuLan, published in August, addresses that language connection. It trains separate audio and text networks to place related recordings and descriptions close together in a shared representation. Its research includes finding music from text and assigning descriptions without restricting every query to a fixed menu of labels.

Evidence and availability

The paper introduces MusicCaps: expert descriptions for 5,500 music clips. The paper notes weaknesses with negation and precise ordering, plus risks of memorizing creative content.