roskosmos19 / Dolphin-35B-A3B-preview

Dolphin-35B-A3B Preview

A 35B-class sparse MoE that runs in phone-class memory.

3 GiB active memory · 15 tok/s · 4-bit

Dolphin-35B-A3B — a 35B MoE LLM that runs at viable speed in under 3 GiB of active memory .

Preview status: this is an early preview release. The checkpoint ships as int4 quantization plus LoRA and prerouter adapters.

Highlights

• Runs in phone-class memory : the full 4-bit checkpoint stays on storage and experts are streamed on demand, so only the active weights are in RAM — under 3 GiB , with no sharding and no upfront download of the weights into memory.
• Fast enough for interactive use : 15 tok/s decode; long prompts fill in at 140 tok/s.
• Quality kept after quantization : Recover-LoRA distillation keeps the int4 model within 3.9 points of its fp16 base.
• Works out of the box : base, LoRA and prerouter adapters ship together.

Three mechanisms make this work:

• SSD expert offload : expert weights are streamed from storage on demand — fetched only as routed, so RAM holds just the active weights. Peak memory is bounded by the active set, not the parameter count.
• Prerouter : a trained head predicts expert routing one step ahead, so expert loads overlap the forward pass instead of stalling it — up to +59% decode throughput.
• Recover-LoRA : the int4 base is frozen and LoRA adapters are trained by distillation from the FP teacher, recovering most of the quantization loss at 4-bit. Adapters stay unmerged: one read-only base serves multiple adapter sets.

Model summary

Quality

All benchmarks were run with OpenCompass under identical settings. The loss of the int4 + adapters relative to the fp16 base model is small: 3.9 points on average .

Performance

Measured on a Mac mini M4 Pro, 24 GB:

*Short contexts; long contexts add KV cache. Expert weights stream from SSD on demand and are not resident.

Use cases

• Edge / on-device inference where GPU VRAM is scarce and storage is fast (NVMe, internal flash).
• Batch serving on a single commodity machine — one read-only base serves many LoRA adapter sets without re-quantization.
• Multilingual chat and reasoning with thinking mode enabled by the bundled chat template.

Limitations

• Preview release: coverage and quality are still being extended.
• Agent capability: this preview release is not yet optimized for agentic tasks.
• Long contexts grow the KV cache; use shorter contexts to keep peak memory at 3 GiB.

Quick start

# Example with edge0-style runner (or any compatible MLX / llama.cpp backend)
export DOLPHIN_35B_MODEL=$PWD/Dolphin-35B-A3B-preview
# then load with your preferred inference engine

License

Apache 2.0.

Citation

If you find this model useful, please cite the original technical work it builds upon.