r0b0tlab / Ling-3.0-flash-VL-NVFP4-MP

Ling-3.0-flash-VL-NVFP4-MP

Max-performance mixed NVFP4 + FP8 quantization of inclusionAI/Ling-3.0-flash-VL (124 B total / 5.1 B active hybrid KDA + MLA MoE vision-language model, MIT), produced with NVIDIA Model Optimizer 0.48 and served by the inclusionAI vllm-ling-v3 fork ( e2e5751 + two patches) in a purpose-built SM121 container.

• Size: 68 GB (64 shards) — 3.7× compression over the 249.7 GB BF16 source
• Runtime: ghcr.io/r0b0tlab/ling30vl-nvfp4-mp-sm121:e2e5751 (arm64, SM12x)
• Repro package (click-run): r0b0tlab/ling30vl-nvfp4-mp-sm121
• Hardware validated: single NVIDIA GB10 / SM 12.1 (121 GB unified)

Quantization

Audits: G0 census 16/16 · canary NVFP4 cos ≥ 0.9950 / FP8 cos ≥ 0.9997 · metadata contract · coverage 61,847/61,847 modules · G3 audit PASS (186,104 index keys).

Size, memory, and KV cache (single GB10, 121 GiB unified)

• Artifact: 72.62 GB / 67.64 GiB (64 shards; 72,597,622,664 bytes of weights + 26.6 MB metadata) — 3.44× smaller than the 249.7 GB BF16 source.
• GPU memory at the winner config ( --gpu-memory-utilization 0.85 ): weights 67.97 GiB + CUDA graph capture ~1.0 GiB + KV pool 30.49 GiB ≈ 99.5 of 121 GiB unified.
• KV cache (fp8): 4,032 B/token (7 MLA layers × 576 B); 2,309,051 tokens capacity → 140.9× concurrency at 16 K , 70.4× at 32 K, 17.6× at the 131,072-token advertised window. fp8 is the maximum — NVFP4 KV is unavailable for this dense-MLA model on SM121.

Evaluation (2026-09-09, single GB10)

Q200v2 text180 (frozen quality-text-180-v2 , thinking on, max_tokens 8192): gsm8k 78/80 · humaneval 40/40 · ifeval 38/40 · hard_reasoning 20/20 (independent manual review) → 176/179 transported = 98.3 % . One row exceeded the 8192-token ceiling ( finish_reason=length ) and is excluded by the frozen contract (its grader passed) — disclosed, not hidden. End-to-end throughput 42.64 tok/s mean / 43.13 aggregate (n=180); telemetry load 38.05 W mean, 68.3 °C mean, 5.07 GiB min MemAvailable.

BFCL v4 multi_turn_base structural-hard20 (official bfcl_eval 2025.12.17 checker): 17/20 = 85.0 % , 20/20 rows, 0 errors.

Q200v2 combined: 193/199 transported (97.0 %).

NIAH — advertised window (25/50/90 % depth + multi-key 33/66): 15/15 across 32 K / 65 K / 131 K ladders, 0 infra errors.

r0b0bench-vision v1.0 (4 suites, 4,703 rows, thinking-off): 3,801/4,703 = 80.82 % — CV-Bench 78.9 % (Count 67.1 · Relation 84.5 · Depth 92.8 · Distance 74.5), MMVP 82.0 % (paired 66.7 %), RealWorldQA 78.3 %, OCRBench 874/1,000.

Quality (single GB10, this checkpoint + container)

Performance (single GB10 / SM121, c = 1)

MoE backend FLASHINFER_CUTLASS , attention TRITON_MLA , NVFP4 linear FlashInferCuteDslNvfp4W4A16 , CUDA graphs + compile (4.2× vs eager), async scheduling.

Use

git clone https://github.com/r0b0tlab/ling30vl-nvfp4-mp-sm121.git
cd ling30vl-nvfp4-mp-sm121 && ./scripts/click-run.sh

from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="EMPTY")
r = client.chat.completions.create(
    model="ling-3.0-flash-vl-nvfp4-mp",
    messages=[{"role": "user", "content": "What is 17*23? Think briefly."}],
    temperature=0, max_tokens=512)
print(r.choices[0].message.content)

Thinking is on by default; pass chat_template_kwargs={"enable_thinking": False} for deterministic short answers. Tool calling: --enable-auto-tool-choice --tool-call-parser ling3 (already set in the container).

Boundaries

• NVFP4 KV cache is unavailable for this dense-MLA model on SM121 (runtime-verified; fp8 is the maximum).
• Video input is not supported by the fork's VL implementation (image-only).
• n-gram speculative decoding is a throughput regression on general traffic (opt-in only).
• No MTP head ships in this checkpoint.

Citation / attribution

Quantized by r0b0tlab with NVIDIA Model Optimizer; base model by inclusionAI. Independent community quantization — not affiliated with or endorsed by inclusionAI or NVIDIA. Evidence bundle: evidence/ in the repro repository (sanitized; no prompts or responses).