ddwang2000 / EchoChat

EchoChat

EchoChat is an audio-conditioned conversational model with text and speech-token outputs. This repository provides the language model, tokenizer, and audio encoder weights.

Files and loading

• Root: language model weights, tokenizer, configuration and custom model code.
• audio_encoder/ : complete audio encoder weights, configuration and its custom implementation.
• LICENSE , licenses/ , THIRD_PARTY_NOTICES.md : applicable licenses and notices.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "ddwang2000/EchoChat"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    repo, trust_remote_code=True,
    torch_dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",
).to("cuda").eval()

The custom audio encoder class is WhisperEncoder in audio_encoder/modeling_audio_encoder.py . Instantiate it with the supplied configuration and strictly load audio_encoder/model.safetensors . Its custom attention implementation requires a compatible CUDA/FlashAttention runtime.

This is a component checkpoint, not a standard Transformers text-generation pipeline. End-to-end audio use requires compatible feature extraction, prompting, and text/speech-token generation logic. Waveform synthesis additionally requires a separately licensed compatible decoder, which is not included here.

Dependencies

The component configuration targets Transformers 4.51.3. Runtime dependencies include PyTorch, Transformers, safetensors, tiktoken, einops, and a compatible FlashAttention build for CUDA execution. Inspect the custom code before enabling trust_remote_code=True .