TuKoResearch / AuriStream-base

AuriStream - Speech Language Model

AuriStream is a speech language model by Greta Tuckute and Klemen Kotar .

This repository contains the shared model code for AuriStream models.

Overview

AuriStream is a GPT-like transformer model for cochlear token prediction with optional multi-token prediction (MTP) heads.

This model predicts cochlear tokens from a tokenizer such as WavCochCausalV8192 .

Usage

This repository is not meant to be used directly. Instead, use one of the checkpoint repositories that reference this base code:

• AuriStream7B_40Pred_BigAudioDataset_500k

To load a checkpoint:

from transformers import AutoModel, AutoConfig

model = AutoModel.from_pretrained(
    "TuKoResearch/AuriStream7B_40Pred_BigAudioDataset_500k",
    trust_remote_code=True,
)

Model Architecture

The AuriStream model includes:

• RMSNorm for layer normalization
• Rotary Position Embeddings (RoPE)
• SiLU activation in MLP layers
• Multi-token prediction heads

Configuration Options

Files

• configuration_auristream.py - Configuration class
• modeling_auristream.py - Model implementation

Tokenizer

This model uses cochlear tokens from WavCochCausalV8192 .