0xSero / GLM-5.1-555B-NVFP4

Support this work → · X · GitHub · REAP paper · Cerebras REAP

GLM-5.1-555B-NVFP4

NVFP4 quantization of 0xSero/GLM-5.1-555B .

At a glance

Which variant should I pick?

NVFP4 quantization of 0xSero/GLM-5.1-555B — a REAP-pruned variant of GLM-5.1 (192 experts per MoE layer, down from 256).

Target hardware: 8× RTX PRO 6000 Blackwell 96GB (sm120) via sglang . See deploy recipe below.

Model details

Layers kept in BF16 (per AutoRound ignore pattern)

• lm_head
• model.layers.[0-2].mlp.{gate,up,down}_proj (first 3 layers' experts — most sensitive)
• model.layers.[0-77].self_attn.indexer.weights_proj (DSA indexer, quant-sensitive)

Deploy on sm120 (RTX PRO 6000 Blackwell)

Uses pre-built voipmonitor/sglang:cu130 Docker image with all sm120 patches applied.

docker run --gpus all --ipc=host --shm-size=8g --network=host \
  --ulimit memlock=-1 --ulimit stack=67108864 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v jit-cache:/cache/jit \
  -e SGLANG_ENABLE_JIT_DEEPGEMM=0 \
  -e SGLANG_ENABLE_DEEP_GEMM=0 \
  -e FLASHINFER_DISABLE_VERSION_CHECK=1 \
  -e NCCL_IB_DISABLE=1 \
  -e NCCL_P2P_LEVEL=SYS \
  -e NCCL_MIN_NCHANNELS=8 \
  voipmonitor/sglang:cu130 \
  python3 -m sglang.launch_server \
    --model-path 0xSero/GLM-5.1-555B-NVFP4 \
    --served-model-name glm-5.1-reap \
    --reasoning-parser glm45 \
    --tool-call-parser glm47 \
    --tensor-parallel-size 8 \
    --quantization compressed-tensors \
    --kv-cache-dtype bf16 \
    --trust-remote-code \
    --mem-fraction-static 0.85 \
    --chunked-prefill-size 16384 \
    --attention-backend flashinfer \
    --fp4-gemm-backend b12x \
    --moe-runner-backend b12x \
    --host 0.0.0.0 --port 5000

Critical flags:

• --kv-cache-dtype bf16 — mandatory; fp8_e4m3 produces garbled output on sm120
• --attention-backend flashinfer — sm120-compatible (trtllm_mha, flashmla are not)
• SGLANG_ENABLE_DEEP_GEMM=0 — DeepGEMM needs WGMMA/TCGEN05 absent on sm120

Memory fit: 320 GB weights + KV cache fits on 8× 96GB (≈768 GB total VRAM). Minimum viable: 6× RTX PRO 6000 with --tp 2 --pp 3 .

Will not run on sm_90 (H100) : NVFP4 is Blackwell-native. Both vLLM (Marlin FP4 PTX mismatch) and sglang ( NotImplementedError: Current platform does not support w4a4 nvfp4 quantization ) explicitly block sm_90.

Quantization method

Produced via AutoRound 0.12.2 layerwise mode on 8× H100 80GB.

Settings

Calibration dataset

Custom mix targeting realistic use cases (1,190 samples total → 505 valid after packing):

Multi-dataset loading used AutoRound's :concat=true option (patched during build; upstreamable) to pack short instruction samples into full-seqlen sequences.

Wall time

• Model load + offload: ~55 min
• Calibration + quant: 6h 34m
• Save: 7 min
• Total: ~7.5 hours on 8× H100 80GB (brev compute)

Quality characteristics

Layer-level loss (iter 0 → iter 49) trajectory:

Expected quality impact: benchmarks on sm120 recommended to measure MMLU/GSM8K/IFEval gap vs BF16 source. Loss magnitudes alone suggest non-trivial degradation at deep layers; whether this matters in practice depends on task.

Provenance

• Source model: 0xSero/GLM-5.1-555B (BF16, 1.1 TB, 26 safetensors)
• Quantization compute: Nebius H100×8 via brev
• Quant tool: Intel AutoRound 0.12.2
• Deploy tool: voipmonitor/sglang:cu130

License

MIT (inherits from base model).

Acknowledgements

• Cerebras REAP team for the pruning recipe
• voipmonitor for the sm120 sglang deployment guide
• Intel AutoRound team for the quantization toolkit
• Nebius for the H100 compute

License & citation

License inherited from the base model.

@misc{lasby2025reap,
  title  = {REAP the Experts: Why Pruning Prevails for One-Shot MoE Compression},
  author = {Mike Lasby and Ivan Lazarevich and Nish Sinnadurai and Sean Lie and Yani Ioannou and Vithursan Thangarasa},
  year   = {2025}, eprint = {2510.13999}, archivePrefix = {arXiv}
}

Sponsors

Made possible by NVIDIA · TNG Technology · Lambda · Prime Intellect · Hot Aisle .