multimodalart / omnivoice-word-control

OmniVoice Word-Control

OmniVoice (0.6B masked-diffusion LM TTS on Qwen3-0.6B) fine-tuned for five-dimensional word-level acoustic control , following the task of WordVoice (arXiv:2607.06461) and its dataset WordVoice-5A .

Control is injected as inline special tokens placed before any word (any subset works; untagged words are planned freely by the model):

<|dur_9|><|bnd_0|><|eng_15|><|pit_17|><|ton_rise|>word

Usage

from omnivoice import OmniVoice  # code: github.com/k2-fsa/OmniVoice

model = OmniVoice.from_pretrained("multimodalart/omnivoice-word-control", device_map="cuda")
audio = model.generate(
    text="I will <|pit_3|><|ton_ffall|>never agree to <|bnd_4|>this!",
    language="en",
    ref_audio="ref.wav",
    ref_text="Transcript of the reference clip.",
)

The tokenizer in this repo already contains the 116 control tokens. For strict overall timing, also pass duration= (the diffusion canvas is fixed before generation, so word duration tokens redistribute time within it). Try it in the demo Space .

Training

• Init from k2-fsa/OmniVoice ; all parameters trained; masked-diffusion objective unchanged (control tokens are conditioning only — no loss on text).
• Data: WordVoice-5A English , all 4 train shards (~2,138 h, 1.18 M utterances); on-the-fly tag construction with dropout (15% utterances untagged / 20% words / attributes kept @ 0.7) to preserve plain TTS and enable partial control.
• 40k steps × 8,192 effective tokens (~1 epoch), lr 5e-5 cosine, bf16, single A100-40GB, ≈6 h.
• Final eval loss 4.109 (best of run). Directional probes (hi/lo tag ratio, seed-matched): pitch 3.61, energy 2.52, duration 2.70.

Fine-tuning recipe: the word-control patch set on the OmniVoice trainer ( omnivoice/data/word_control.py — tag vocabulary, binning, alignment, dropout).

License & lineage

Weights are CC-BY-NC-4.0 (derivative of the CC-BY-NC OmniVoice release). Data: WordVoice-5A (CC-BY-4.0). Task formulation: WordVoice, arXiv:2607.06461.