Sponsored by
AdsCreator.com
declare-lab / mustango-pretrained
Model's Last Updated: December 17 2023
text-to-audio
Introduction of mustango-pretrained
Model Details of mustango-pretrained
Mustango: Toward Controllable Text-to-Music Generation
Demo | Model | Website and Examples | Paper | Dataset
Meet Mustango, an exciting addition to the vibrant landscape of Multimodal Large Language Models designed for controlled music generation. Mustango leverages Latent Diffusion Model (LDM), Flan-T5, and musical features to do the magic!
🔥 Live demo available on Replicate and HuggingFace .
Quickstart Guide
Generate music from a text prompt:
import IPython
import soundfile as sf
from mustango import Mustango
model = Mustango("declare-lab/mustango")
prompt = "This is a new age piece. There is a flute playing the main melody with a lot of staccato notes. The rhythmic background consists of a medium tempo electronic drum beat with percussive elements all over the spectrum. There is a playful atmosphere to the piece. This piece can be used in the soundtrack of a children's TV show or an advertisement jingle."
music = model.generate(prompt)
sf.write(f"{prompt}.wav", audio, samplerate=16000)
IPython.display.Audio(data=audio, rate=16000)
Installation
git clone https://github.com/AMAAI-Lab/mustango
cd mustango
pip install -r requirements.txt
cd diffusers
pip install -e .
Datasets
The MusicBench dataset contains 52k music fragments with a rich music-specific text caption.
Subjective Evaluation by Expert Listeners
| Model | Dataset | Pre-trained | Overall Match ↑ | Chord Match ↑ | Tempo Match ↑ | Audio Quality ↑ | Musicality ↑ | Rhythmic Presence and Stability ↑ | Harmony and Consonance ↑ |
|---|---|---|---|---|---|---|---|---|---|
| Tango | MusicCaps | ✓ | 4.35 | 2.75 | 3.88 | 3.35 | 2.83 | 3.95 | 3.84 |
| Tango | MusicBench | ✓ | 4.91 | 3.61 | 3.86 | 3.88 | 3.54 | 4.01 | 4.34 |
| Mustango | MusicBench | ✓ | 5.49 | 5.76 | 4.98 | 4.30 | 4.28 | 4.65 | 5.18 |
| Mustango | MusicBench | ✗ | 5.75 | 6.06 | 5.11 | 4.80 | 4.80 | 4.75 | 5.59 |
Training
We use the
accelerate
package from Hugging Face for multi-gpu training. Run
accelerate config
from terminal and set up your run configuration by the answering the questions asked.
You can now train Mustango on the MusicBench dataset using:
accelerate launch train.py \
--text_encoder_name="google/flan-t5-large" \
--scheduler_name="stabilityai/stable-diffusion-2-1" \
--unet_model_config="configs/diffusion_model_config_munet.json" \
--model_type Mustango --freeze_text_encoder --uncondition_all --uncondition_single \
--drop_sentences --random_pick_text_column --snr_gamma 5 \
The
--model_type
flag allows to choose either Mustango, or Tango to be trained with the same code. However, do note that you also need to change
--unet_model_config
to the relevant config: diffusion_model_config_munet for Mustango; diffusion_model_config for Tango.
The arguments
--uncondition_all
,
--uncondition_single
,
--drop_sentences
control the dropout functions as per Section 5.2 in our paper. The argument of
--random_pick_text_column
allows to randomly pick between two input text prompts - in the case of MusicBench, we pick between ChatGPT rephrased captions and original enhanced MusicCaps prompts, as depicted in Figure 1 in our paper.
Recommended training time from scratch on MusicBench is at least 40 epochs.
Model Zoo
We have released the following models:
Mustango Pretrained: https://huggingface.co/declare-lab/mustango-pretrained
Mustango: https://huggingface.co/declare-lab/mustango
Citation
Please consider citing the following article if you found our work useful:
@misc{melechovsky2023mustango,
title={Mustango: Toward Controllable Text-to-Music Generation},
author={Jan Melechovsky and Zixun Guo and Deepanway Ghosal and Navonil Majumder and Dorien Herremans and Soujanya Poria},
year={2023},
eprint={2311.08355},
archivePrefix={arXiv},
}
Runs of declare-lab mustango-pretrained on huggingface.co
13
Total runs
0
24-hour runs
2
3-day runs
4
7-day runs
2
30-day runs
More Information About mustango-pretrained huggingface.co Model
Url of mustango-pretrained
mustango-pretrained huggingface.co Url
Provider of mustango-pretrained huggingface.co
Other API from declare-lab
Total runs:
473
Run Growth:
-1.6K
Growth Rate:
-322.22%
Total runs:
283
Run Growth:
263
Growth Rate:
92.93%
Total runs:
211
Run Growth:
192
Growth Rate:
91.00%
Total runs:
166
Run Growth:
73
Growth Rate:
43.45%
Total runs:
156
Run Growth:
-22
Growth Rate:
-13.84%
Total runs:
148
Run Growth:
22
Growth Rate:
15.38%
Total runs:
144
Run Growth:
41
Growth Rate:
27.33%
Total runs:
127
Run Growth:
81
Growth Rate:
63.78%
Total runs:
106
Run Growth:
-80
Growth Rate:
-69.57%
Total runs:
91
Run Growth:
47
Growth Rate:
51.65%
Total runs:
69
Run Growth:
-6
Growth Rate:
-8.70%
Total runs:
50
Run Growth:
-68
Growth Rate:
-133.33%
Total runs:
33
Run Growth:
21
Growth Rate:
63.64%
Total runs:
32
Run Growth:
-72
Growth Rate:
-225.00%
Total runs:
26
Run Growth:
-12
Growth Rate:
-46.15%
Total runs:
21
Run Growth:
13
Growth Rate:
61.90%
Total runs:
20
Run Growth:
10
Growth Rate:
50.00%
Total runs:
20
Run Growth:
4
Growth Rate:
20.00%
Total runs:
19
Run Growth:
-18
Growth Rate:
-94.74%
Total runs:
19
Run Growth:
3
Growth Rate:
15.79%
Total runs:
19
Run Growth:
2
Growth Rate:
10.53%
Total runs:
15
Run Growth:
3
Growth Rate:
20.00%
Total runs:
14
Run Growth:
1
Growth Rate:
7.14%
Total runs:
13
Run Growth:
6
Growth Rate:
46.15%
Total runs:
11
Run Growth:
4
Growth Rate:
36.36%
Total runs:
9
Run Growth:
-15
Growth Rate:
-166.67%
Total runs:
7
Run Growth:
-1
Growth Rate:
-14.29%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%