rednote-hilab / dots.tts-base

dots.tts-base

dots.tts is a 2B-parameter fully continuous, end-to-end autoregressive (AR) text-to-speech system . The backbone pairs a semantic encoder, an LLM, and an autoregressive flow-matching acoustic head over a 48 kHz AudioVAE — no discrete codec tokens anywhere in the pipeline.

This repository hosts dots.tts-base , the end-to-end pretrained checkpoint trained on ~1.5M hours of speech. It is the foundation for the two post-trained variants and the recommended starting point for fine-tuning .

	← you are here — Pretrain (~1.5M h). Fine-tuning, full CFG / NFE control.
	+ Self-corrective Alignment. Highest zero-shot fidelity and speaker similarity; also recommended for fine-tuning.
	+ MeanFlow distillation. Few-step inference (NFE = 4), low latency.

Quick Start

Installation

conda create -n dots_tts python=3.10 -y
conda activate dots_tts

python -m pip install --upgrade pip
python -m pip install "git+https://github.com/rednote-hilab/dots.tts.git" \
  -c "https://raw.githubusercontent.com/rednote-hilab/dots.tts/main/constraints/recommended.txt"

CLI

# Continuation voice cloning (reference audio + transcript) — recommended
dots.tts \
  --model-name-or-path rednote-hilab/dots.tts-base \
  --text "Hello, this is a zero-shot voice cloning demonstration." \
  --prompt-audio /path/to/reference.wav \
  --prompt-text "The exact transcript of the reference audio." \
  --output clone.wav

Python API

from dots_tts.runtime import DotsTtsRuntime
import soundfile as sf

runtime = DotsTtsRuntime.from_pretrained(
    "rednote-hilab/dots.tts-base",
    precision="bfloat16",
)

result = runtime.generate(
    text="Hello, this is a quick speech synthesis test.",
    prompt_audio_path="/path/to/reference.wav",
    prompt_text="The exact transcript of the reference audio.",
    num_steps=10,
    guidance_scale=1.2,
)

sf.write("output.wav", result["audio"].float().cpu().squeeze().numpy(), result["sample_rate"])

Recommended sampling settings

Fine-tuning

dots.tts-base is the recommended starting point for fine-tuning. See the training script and smoke config in the source repository:

accelerate launch scripts/train_dots_tts.py --config configs/dots_tts.yaml

Architecture

A frozen AudioVAE encodes 48 kHz mono waveform into a continuous latent and decodes it back via a BigVGAN-style causal decoder. An autoregressive backbone predicts that latent one patch at a time:

• Semantic encoder — re-encodes each newly generated VAE patch into a compact embedding for the LLM, stripping high-variance acoustic detail.
• LLM — initialized from Qwen2.5-1.5B-Base , consumes BPE text directly (no phonemes), emits one hidden state per audio step.
• AR flow-matching head — a DiT that conditions on the LLM hidden state and the AR prefix to denoise the next VAE patch, with a frozen CAM++ speaker x-vector as side input.

Performance — dots.tts-base

Seed-TTS-Eval (zero-shot, ~3 s reference)

MiniMax Multilingual (24 languages, average)

See the project README for the full per-language breakdown, CV3-Eval and EmergentTTS-Eval results.

Risks and Limitations

• Misuse risk. High-fidelity zero-shot voice cloning can produce highly realistic synthetic speech. This checkpoint is intended for research and authorized deployment. Do not use it for impersonation, fraud, or disinformation. Combine downstream use with consent-aware reference-audio policies, robust synthetic-speech detection, and content watermarking. Clearly mark AI-generated audio.
• Low-resource WER gap. A BPE backbone inherits the text LLM's language coverage at the cost of a higher data appetite. On script-divergent and under-represented languages (Arabic, Hindi, Turkish, Vietnamese) WER is higher than on high-resource languages; speaker similarity is preserved.
• Speech-heavy training. The backbone is trained on a speech-heavy mixture. Singing and unified speech + sound generation are not covered.

Citation

@article{dotstts2026,
  title   = {dots.tts Technical Report},
  author  = {dots.tts Team},
  journal = {arXiv preprint},
  year    = {2026},
}

License

Released under Apache-2.0 .