nota-ai / GLM-5.3-Flash-Nota-NVFP4

Nota GLM-5.3-Flash · NVFP4 (W4A4)

A 4-bit (NVFP4) quantized release of Z.ai 's GLM-5.3-Flash — a 320B-parameter natively multimodal Mixture-of-Experts model with ~18B active per token.

Highlights

• NVFP4 (4-bit float, W4A4) — group_size=16 , packed in the compressed-tensors nvfp4-pack-quantized format for direct serving in vLLM . Both weights and activations are quantized to 4-bit floating point.

  • Requires NVIDIA Blackwell. NVFP4 relies on the FP4 tensor cores introduced in the Blackwell architecture (e.g. B200 / B300 / GB200), so inference must run on a Blackwell-class GPU. Earlier architectures (Hopper, Ada, Ampere) do not support NVFP4 execution.

• Only the routed experts are quantized. They hold 94.8% of the parameters, so the memory saving is captured almost in full while every precision-critical path stays in BF16 — the same split the reference GLM-5.3 NVFP4 release uses.
• No architecture change. Tensor names, layer count and expert count are identical to the base checkpoint, so stock vLLM serves it as-is — no patched modeling file.
• MTP and multimodal preserved. The multi-token-prediction block and the vision tower stay BF16, so speculative decoding and image/video inputs work as in the base model.

Quantization scope

Kept in BF16:

Calibration

512 conversations of exactly 4,096 tokens, rendered through the GLM chat template and drawn from the workloads this model is built for rather than generic web text: agentic tool use (20.5%), SWE agent trajectories (14.3%), instruction following (8.2%), terminal agents (7.8%), code (7.0%), STEM (5.9%), reasoning (4.7%), knowledge MCQ (2.3%), and 29.3% Korean sources. 71.7% of the samples carry reasoning traces inside <think> blocks.

Requirements

vllm >= 0.29.0
flashinfer >= 0.6.17     # sparse MLA

Quick Start

B300 × 1

CUDA_VISIBLE_DEVICES=0 \
vllm serve nota-ai/GLM-5.3-Flash-Nota-NVFP4 \
  --served-model-name nota-ai/GLM-5.3-Flash-Nota-NVFP4 \
  --tensor-parallel-size 1 \
  --gpu-memory-utilization 0.96 \
  --compilation-config '{"cudagraph_mode":"PIECEWISE"}' \
  --kv-cache-dtype fp8 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":5}' \
  --tool-call-parser glm47 \
  --reasoning-parser glm45 \
  --enable-auto-tool-choice

B200 × 2

CUDA_VISIBLE_DEVICES=0,1 \
vllm serve nota-ai/GLM-5.3-Flash-Nota-NVFP4 \
  --served-model-name nota-ai/GLM-5.3-Flash-Nota-NVFP4 \
  --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.90 \
  --compilation-config '{"cudagraph_mode":"PIECEWISE"}' \
  --kv-cache-dtype fp8 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":5}' \
  --tool-call-parser glm47 \
  --reasoning-parser glm45 \
  --enable-auto-tool-choice