ModalityDance / AR-Omni-Chat-v0.1

AR-Omni-Chat

Overview

AR-Omni-Chat is the chat variant of AR-Omni: a single-decoder , single-token-stream autoregressive any-to-any model that supports text , images , and speech in a unified next-token prediction framework.

This checkpoint is instruction-tuned on AR-Omni-Instruct-v0.1 and is intended for interleaved any-to-any conversation .

Usage

The recommended way to run AR-Omni v0.1 is via the official repository (vendored editable transformers and accelerate ).
For full setup details, please follow: https://github.com/ModalityDance/AR-Omni

Run AR-Omni-Chat

Requires CosyVoice2 and WavTokenizer assets/config.
If CosyVoice is not installed as a package, set PYTHONPATH to include its repo.

PYTHONPATH=/path/to/CosyVoice/third_party/Matcha-TTS:/path/to/CosyVoice${PYTHONPATH:+:$PYTHONPATH} \
python3 inference/inference_chat.py \
  --input ./infer_test.json \
  --output_dir ./test_results \
  --model_root /path/to/converted_model_root \
  --hf_tokenizer /path/to/converted_model_root \
  --cosyvoice_model_dir /path/to/CosyVoice2-0.5B \
  --wavtokenizer_cfg_path /path/to/wavtokenizer.yaml \
  --wavtokenizer_ckpt_path /path/to/wavtokenizer.ckpt \
  --save_speech --save_images

Common optional flags:

• --greedy : greedy decode for text
• --txt_temp , --txt_top_p : sampling settings for text
• --img_temp : sampling temp for image tokens
• --bandwidth_id : wavtokenizer bandwidth id

Input Example

{
  "dialog_id": "demo_0001",
  "turns": [
    {
      "text": "Describe the image in detail.",
      "image_paths": ["inference/demo_test.jpg"],
      "user_append_text": "Please acknowledge the user's vocal input, create a textual response.",
      "reset": true
    }
  ]
}

License

This project is licensed under the MIT License .
It also complies with the licenses of referenced third-party projects and dependencies, including the Chameleon Research License .

Citation

@misc{aromni2026,
  title        = {AR-Omni: A Unified Autoregressive Model for Any-to-Any Generation},
  author       = {TBD},
  year         = {2026},
  note         = {AR-Omni-Chat-v0.1},
  url          = {https://huggingface.co/ModalityDance/AR-Omni-Chat-v0.1}
}