bg-digitalservices / Gemma-4-E2B-it-NVFP4

Gemma-4-E2B-it-NVFP4

NVFP4 quantization of google/gemma-4-E2B-it — the instruct (chat-optimized) variant of Google's Gemma 4 E2B Mixture-of-Experts model.

W4A4 — both weights and activations in FP4. Maximum speed on Blackwell GPUs. See also Gemma-4-E2B-it-NVFP4A16 for the weight-only W4A16 variant.

Key Specs

Serving with vLLM

Note: Requires a vLLM build with transformers >= 5.4 and the included gemma4_patched.py for NVFP4 MoE scale key loading.

docker run -d \
  --gpus all --ipc=host --network host \
  -e VLLM_NVFP4_GEMM_BACKEND=marlin \
  -v /path/to/model:/model \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v /path/to/gemma4_patched.py:/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/gemma4.py \
  <your-vllm-tf5-image> \
  vllm serve /model \
    --served-model-name gemma-4-e2b \
    --host 0.0.0.0 --port 8888 \
    --quantization modelopt \
    --dtype auto --kv-cache-dtype fp8 \
    --gpu-memory-utilization 0.30 \
    --max-model-len 131072 \
    --moe-backend marlin \
    --trust-remote-code

Key Flags

Testing

This is an instruct model — use the chat completions endpoint.

curl http://localhost:8888/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"gemma-4-e2b","messages":[{"role":"user","content":"Hello!"}],"max_tokens":200}'

How this was made

Gemma 4 MoE stores expert weights as fused 3D tensors that no existing quantization tool handles. We wrote a custom _QuantGemma4TextExperts plugin for NVIDIA Model Optimizer — see our Gemma-4-26B-A4B-it-NVFP4 model card for full technical details.

Calibration

• Tool: NVIDIA Model Optimizer v0.43, NVFP4_DEFAULT_CFG (W4A4)
• Data: 4096 samples from CNN/DailyMail, batch size 32, seq_len 1024
• Expert routing: Natural (no forced uniform routing)
• Vision encoder: Excluded from quantization (stays BF16)
• Hardware: NVIDIA H200

Known Issues

• Requires vLLM with transformers >= 5.4 and the included gemma4_patched.py
• --moe-backend marlin required for correct MoE computation
• Vision encoder stays in BF16 (~5.8GB of the 7.1GB total) — excluded from quantization to preserve image quality. Text inference still benefits from quantized MoE experts/attention layers
• Vision/video capabilities untested after quantization
• Community quantization, not an official NVIDIA or Google release
• vLLM issue filed: vllm-project/vllm#38912

Reproduce

python quantize_gemma4_moe.py --model google/gemma-4-E2B-it --qformat nvfp4

Full quantization script included as quantize_gemma4_moe.py .

License

Apache 2.0 — inherited from the base model.

Credits

Quantized by Mario Iseli on an NVIDIA H200. Built and validated with AI-engineering assistance from Anthropic.

Shout-out to eugr/spark-vllm-docker for the DGX Spark-optimized vLLM build.

📬 [email protected] ☕ Buy me a coffee if this makes your Spark go brrrrrr! 🚀