mlx-community / DeepSeek-V4-Flash-4bit

DeepSeek-V4-Flash-4bit (MLX)

4-bit quantized MLX port of deepseek-ai/DeepSeek-V4-Flash for Apple Silicon.

158B total params (~37B active), 149 GB on disk, fits comfortably on a single M3/M5 Ultra (256GB+).

Requires the V4 mlx-lm port

DeepSeek-V4 is a new architecture (mHC, hash-routed MoE, sqrtsoftplus, Compressor + Indexer for compressed KV) and is not yet in stock mlx-lm . To use this model you need the V4 port:

git clone https://github.com/machiabeli/mlx-lm-1.git mlx-lm
cd mlx-lm && git checkout feat/deepseek-v4
pip install -e .

Tracking PR: ml-explore/mlx-lm#1189 . Once merged, pip install mlx-lm will work.

Usage

from mlx_lm import load, generate

model, tokenizer = load("mlx-community/DeepSeek-V4-Flash-4bit")
out = generate(model, tokenizer, prompt="Q: What is 2+2?\nA:", max_tokens=64)
print(out)

Performance

Measured on M3 Ultra (512GB) single-node, batch=1:

Generation throughput includes the fused Metal kernel for mHC Sinkhorn added in PR #1189 (1.83x over the Python reference).

Source quality caveat

The bf16 source weights used for this conversion were upcasted from DeepSeek's native FP8 release rather than re-quantized directly from FP8. This stacks two quantization passes (FP8 -> BF16 -> Q4) and may produce slightly worse outputs than a direct FP8 -> Q4 conversion. A re-conversion from native FP8 is planned.

Conversion

mlx_lm.convert \
  --hf-path deepseek-ai/DeepSeek-V4-Flash \
  --mlx-path DeepSeek-V4-Flash-4bit \
  -q --q-bits 4 --q-group-size 64

Result: 4.506 bits per weight, 33 sharded safetensors.

Architecture

V4 is a substantial step from V3:

• mHC (Manifold-constrained Hyper-Connections) — replaces residual connections with hc_mult=4 parallel hidden-state copies recombined via a doubly-stochastic Sinkhorn-normalized mix matrix.
• Hash-routed MoE — first 3 layers use a deterministic tid2eid table (token id -> expert id) instead of learned routing.
• sqrtsoftplus scoring — sqrt(softplus(x)) instead of softmax for expert scores.
• MLA with single shared 512-dim KV head — broadcast across 64 query heads (no kv_lora_rank up-projection step like V3).
• Compressor + Indexer for compressed KV attention with topk sparse selection (Indexer at compress_ratio=4).
• Per-head learnable attn_sink in softmax denominator.

Full details: DeepSeek V4 technical report .

License

MIT (matches upstream).