gpustack / GLM-5.3-W4A8

GLM-5.3-W4A8

Hopper-native W4A8 quantization of GLM-5.3 . Routed experts are INT4 (group size 128); activations and non-expert layers stay FP8. On 8×H20 the model runs on CUTLASS W4A8 grouped GEMM — no dequantization to BF16. The same checkpoint loads in both vLLM and SGLang ; you do not need two copies.

Two things this checkpoint is for:

• Half the weights, roughly twice the KV cache. The checkpoint shrinks from 704 GB to 372 GB. On 8×H20 (~1128 GB), KV capacity goes from 608k tokens to 1.26M tokens (+107%).
• No measurable accuracy drop. GPQA-Diamond and teacher-forced NLL both sit inside the sampling noise of the official FP8 release. MTP speculative decoding works out of the box.

This model still has the full ~753B parameters of GLM-5.3. If the sidebar shows ~370B, that is a counting artifact: each int32 slot stores eight 4-bit weights, and the metadata counts storage slots, not parameters.

Fully verified on 8×H20-3e in both engines : vLLM nightly ( 0.26.1rc1.dev229+g124154a88 ) and SGLang dev ( 0.0.0.dev1+gbb5e61986 ). For SGLang use lmsysorg/sglang:dev ; v0.5.16 was not verified. A native-load fix is in sgl-project/sglang#37207 .

Storage format is compressed-tensors: routed experts pack-quantized INT4, non-experts float-quantized FP8 128×128 block. This is not SGLang's native w4afp8 packing. vLLM uses CompressedTensorsW4A8Fp8MoEMethod ; SGLang uses CompressedTensorsW4AFP8MoE . Both sit on the SM90-only CUTLASS W4A8 kernel. vLLM loads as-is; SGLang needs an extra PYTHONPATH (see Deploy / SGLang).

Also published on ModelScope: skyai/GLM-5.3-W4A8 .

How this differs from other 4-bit releases

Requantizing from FP8 adds a second quantization step, but measured NLL is only +0.001 nats/token. In return you get a checkpoint vLLM can serve today, without waiting on an upstream w4afp8 PR, and SGLang can use the same files.

Accuracy

Measured with vLLM on 8×H20 (TP8, --kv-cache-dtype fp8_ds_mla , temperature 1.0, top_p 0.95, thinking on by default). Official FP8 and this checkpoint used identical prompts, option shuffles, and sampling. The PhalaCloud column is from that model's card (SGLang, different protocol — not a direct subtraction).

Zero GPQA parse failures on both local runs. W4A8 pass@1 is slightly above FP8; the gap is about five samples inside 792 draws — sampling noise. majority@4 matches exactly. Window-to-window NLL SE is ~0.074; +0.001 is far below that.

PhalaCloud's GPQA is ~1.6pp higher, mostly from AWQ calibration, quantizing from BF16 (no second quant step), and a different engine / truncation-retry protocol.

Full tables: EVAL_DETAILS.md . Chinese card: README.zh.md .

Performance (8×H20-3e, single node)

MoE backend: vLLM auto selects CUTLASS W4A8. Attention: --kv-cache-dtype fp8_ds_mla → FLASHMLA_SPARSE. W4A8 requires --enable-expert-parallel ; FP8 is faster without EP.

Offline batching (no rate limit, max_model_len=16384 ). Prefill rows are input throughput; decode rows are output tokens / wall time including the first prefill.

A single TP8 replica is not faster than FP8 (prefill 0.94×, decode 0.85–0.96×): H20 has spare bandwidth, and CUTLASS dequant eats the bandwidth win. The real gains are:

1. 2.07× KV cache — longer context and higher concurrency on the same 8 GPUs.
2. TP4 dual replica becomes possible. 372 GiB / 4 = 93 GiB per GPU, which fits in 141 GiB; FP8's 704 GiB / 4 = 176 GiB does not. Dual-replica aggregate vs FP8 TP8: prefill 1.63× , decode 128 conc. 1.34× , 256 conc. 1.55× .

MTP / speculative decoding

vLLM loads the layer-78 draft head via glm_moe_dsa → deepseek_mtp → DeepSeekMTPModel . Test: 512-token input / 256-token output, num_speculative_tokens=1 .

Quantization does not break MTP. PhalaCloud reports EAGLE steps=3 acceptance length ~2.93, which is not comparable to num_speculative_tokens=1 .

Deploy

Hardware: Hopper GPUs only (H20 / H100 / H200, compute capability exactly SM90). Ada (SM89) and Blackwell (SM100 / SM120) are not supported — see Limitations. Pick one engine:

• vLLM nightly (0.26.1rc1 or newer recommended; the v0.26.0 release may not have full GLM-5.3 DSA + SM90 W4A8). The vLLM examples below work as written.
• SGLang dev ( lmsysorg/sglang:dev , verified at 0.0.0.dev1+gbb5e61986 ). Needs PYTHONPATH ; see the end of this section. Release v0.5.16 was not verified on this repo.

All performance numbers below are from vLLM.

Throughput: two replicas × TP4 (recommended)

M=camel-ai/GLM-5.3-W4A8
for i in 0 1; do
  [ $i -eq 0 ] && D='"device=0,1,2,3"' PORT=8000 || D='"device=4,5,6,7"' PORT=8001
  eval docker run -d --name glm53-r$i --gpus "$D" --ipc=host --shm-size=32g \
    -p $PORT:8000 \
    vllm/vllm-openai:nightly \
    --model $M --served-model-name GLM-5.3 \
    --tensor-parallel-size 4 --enable-expert-parallel \
    --kv-cache-dtype fp8_ds_mla \
    --gpu-memory-utilization 0.92 --max-model-len 65536 --trust-remote-code
done

Long context: single instance TP8

docker run -d --name glm53 --gpus all --ipc=host --shm-size=32g \
  -p 8000:8000 \
  vllm/vllm-openai:nightly \
  --model camel-ai/GLM-5.3-W4A8 \
  --tensor-parallel-size 8 --enable-expert-parallel \
  --kv-cache-dtype fp8_ds_mla \
  --gpu-memory-utilization 0.90 --max-model-len 262144 --trust-remote-code

To enable MTP speculative decoding, add:

  --speculative-config '{"method":"mtp","num_speculative_tokens":1}'

The server exposes a standard OpenAI-compatible API. Sampling defaults live in generation_config.json : temperature 1.0, top_p 0.95. Thinking mode is on by default via the chat template ( <think> ).

Do not set --moe-backend marlin or triton : on SM90, W4A8 has only the CUTLASS path, and auto will pick it. Do pass --enable-expert-parallel ; otherwise each GPU holds 256 experts × intermediate 256, and CUTLASS grouped GEMM slows down a lot.

SGLang

The same weights run on SGLang if you put the repo directory on PYTHONPATH . Use lmsysorg/sglang:dev ; release v0.5.16 was not verified here:

M=/path/to/GLM-5.3-W4A8
docker run -d --name glm53-sgl --gpus all --ipc=host --shm-size=32g \
  -p 30000:30000 -v $M:$M \
  -e PYTHONPATH=$M \
  lmsysorg/sglang:dev \
  python3 -m sglang.launch_server --model-path $M \
  --tp 8 --trust-remote-code --mem-fraction-static 0.85 \
  --reasoning-parser glm45 --tool-call-parser glm45 \
  --host 0.0.0.0 --port 30000

PYTHONPATH makes every worker load this repo's sitecustomize.py . On checkpoint load it renames non-expert FP8 scales from weight_scale to weight_scale_inv : SGLang's Fp8LinearMethod and DSA indexer fusion hard-code the latter, vLLM wants the former. Both names cannot live on disk at once — either engine KeyError s on a tensor with no matching parameter — so the rename has to happen at read time. The weight bytes themselves are unchanged.

vLLM does not need and will not load that file. On sglang:dev , TP8 used 48.2 GiB/GPU with max_total_num_tokens=803264 .

To drop the shim, SGLang upstream needs three hard-coded lookups fixed (tracked in sgl-project/sglang#37207 ):

The first two contradict each other, and the compressed-tensors schema forbids block_structure together with group strategy, so no legal config satisfies both.

Sampling defaults

Same as the base model: temperature 1.0, top_p 0.95. Without top-p truncation, thinking mode can occasionally loop.

Quantization

• Routed experts (~734B parameters, 96% of the model): dequantize FP8 blocks to float32, then symmetric INT4, group size 128. Each group searches 11 shrink ratios (1.00 → 0.70) for the MSE-best scale, then packs as compressed-tensors pack-quantized uint4b8 / int32. Measured relative L2 error ~0.103, better than naive RTN (0.124) and MXFP4 group-32 (0.113).
• Attention / shared experts / dense MLP / DSA indexer : official FP8 128×128 block, byte-identical . Only the scale tensor name changes from official weight_scale_inv to compressed-tensors weight_scale . SGLang still expects the former, so sitecustomize.py renames it back at read time.
• Embedding, lm_head, router, norms : stay BF16.
• Activations are FP8 per-token dynamic at runtime, same as the official FP8 path.

Hopper has no FP4 tensor cores, so MXFP4 would be Marlin W4A16 (half the peak FLOPS). This checkpoint uses INT4-W4A8 instead: 4-bit storage, dequant to FP8 inside the kernel, then wgmma at 296 TFLOPS peak.

Limitations

• Hopper (SM90) only — H100 / H200 / H20. This is not a conservative estimate: vLLM picks the W4A8 CUTLASS path with _check_scheme_supported(90, match_exact=True) , i.e. capability exactly 9.0 , not “9.0 or above”. Ada (SM89, L40S / L4 / 4090), Blackwell (SM100, B200 / GB200) , and RTX 5090 / Pro 6000 (SM120) are unsupported — even the newer ones. On Blackwell, use NVFP4 (native FP4 tensor cores).
• On non-SM90 cards you will not get a clean error . vLLM's _is_dynamic_token_w4a8_int only compares num_bits and ignores activation type, so this config can mis-hit CompressedTensorsW4A8Int8MoEMethod , which expects unpacked torch.int8 weights; this repo is pack-quantized int32 and fails at load with a shape/dtype error. SGLang is looser: its MoE path does not check capability and will call an SM90-only CUTLASS kernel at runtime.
• Format is compressed-tensors, not SGLang --quantization w4afp8 . vLLM loads directly; SGLang loads with PYTHONPATH pointing at this repo's sitecustomize.py , and you must use lmsysorg/sglang:dev (release v0.5.16 not verified). See Deploy / SGLang.
• Accuracy and speed numbers are from vLLM. SGLang was only smoke-tested for load + generation; the kernels differ, so do not copy the numbers across engines.
• Requantized from official FP8, so in theory slightly worse than BF16 + AWQ. Local GPQA / NLL put the gap inside noise.
• TP split: moe_intermediate_size after TP must be divisible by 256. TP8: 2048/8=256, OK. TP16: 128, not OK.
• Inherits the capabilities and limits of base GLM-5.3.

License

Same as GLM-5.3 (see LICENSE ). Please cite the original GLM-5.3 work.

---

Base model card (official)

Excerpted from the official base-model repo. Full text: zai-org/GLM-5.3 .

GLM-5.3 uses the same base as GLM-5.2 — every gain comes from post-training. Versus GLM-5.2 it is much stronger at complex coding and long-horizon tasks:

• Stronger coding : strongest open-weights coding model; +50% vs GLM-5.2 on the in-house Z.ai Code Bench; open-source SOTA on Terminal Bench 3.0 and Agents' Last Exam.
• Emergent security-research capability : post-training scaled those skills faster than expected.

Official reference numbers (model card): HLE 62.5 (with tools), Deep-SWE 66.9, Terminal-Bench 2.1 88.2, Terminal-Bench 3.0 28.3.