r0b0tlab / Ling-3.0-flash-NVFP4

Ling-3.0-flash-NVFP4

NVFP4 (4-bit) quantization of inclusionAI/Ling-3.0-flash (BailingMoeV3, 42-layer hybrid MoE + MTP) produced with NVIDIA ModelOpt using the DeepSeek-V4 experts-only recipe, served natively by the inclusionAI vllm-ling-v3 fork (FlashInfer-CUTLASS NVFP4 MoE).

• Size: 81.3 GB (24 shards) — 3.14x compression over the 255 GB BF16 source
• Quantization: weight-only NVFP4 (E2M1 packed 2-per-byte, E4M3 block scales, FP32 per-tensor scale), activation_scheme: dynamic
• Scope: the 61,440 routed-expert projections (layers 2–41) are NVFP4; attention (KDA + MLA), shared experts, routers, norms, embeddings, lm_head and the full MTP layer remain BF16
• Hardware: designed for NVIDIA GB10 / SM120/SM121-class GPUs

Quickstart

# vllm-ling-v3 fork @ 92c104112 (see the reproducibility repo for the container)
git clone --branch ling_3_0 https://github.com/inclusionAI/vllm-ling-v3
cd vllm-ling-v3 && git checkout 92c104112
VLLM_USE_PRECOMPILED=1 uv venv --python 3.12
VLLM_USE_PRECOMPILED=1 uv pip install -e . --torch-backend=auto
uv pip install "flashinfer==0.6.15.post1" \
  https://github.com/flashinfer-ai/flashinfer/releases/download/v0.6.15.post1/flashinfer_cubin-0.6.15.post1-py3-none-any.whl

vllm serve r0b0tlab/Ling-3.0-flash-NVFP4 \
  --trust-remote-code --kv-cache-dtype fp8 --mamba-cache-mode align \
  --max-model-len 24576 --enable-auto-tool-choice --tool-call-parser ling3 \
  --reasoning-parser ling3 \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 1}'

Two project-local fork patches are required on SM121 (Triton MLA decode smem and NVFP4 input-scale defaults) — see github.com/r0b0tlab/ling-3.0-flash-nvfp4 ( patches/apply_patches.py ).

Note on chat_template.jinja : this checkpoint ships with enable_thinking defaulting to off (harness-compatible; matches the published benchmark rows). Pass chat_template_kwargs: {"enable_thinking": true} per request for reasoning traces, or restore the upstream template from inclusionAI/Ling-3.0-flash for the native thinking-on default.

Quality (r0b0bench core-subset, official harnesses)

Performance (1x GB10 / SM121, MTP scale 1, CUDA graphs)

• Decode: 33.5 tok/s C1 (21.4 base AR graphs, 13.8 base AR eager)
• MTP acceptance: 73.2 %, mean acceptance length 1.73
• FP8 KV, 24K context, mamba cache align

Full sweep + methodology: github.com/r0b0tlab/ling-3.0-flash-nvfp4

Reproducibility

• BF16 source: inclusionAI/Ling-3.0-flash @ eb04638493fe9eac066ede636090a9425272c481 (SHA-256 manifest in the reproducibility repo)
• Converter + container: r0b0tlab/ling-3.0-flash-nvfp4 ( docker run ... convert reproduces this checkpoint)
• Bench harness: r0b0tlab/r0b0bench (results ledger entries for this model)

License

MIT (mirrors the source model). Weights derived from inclusionAI/Ling-3.0-flash (MIT, © inclusionAI) — see the upstream model card for any additional terms. Quantization and validation by r0b0tlab.