A production-grade FP8 dynamic quantization of
google/gemma-4-26B-A4B-it
using
llm-compressor
's
oneshot()
method with the
load_context()
MoE linearization pattern.
Model Details
Base Model
: google/gemma-4-26B-A4B-it
Quantization
: FP8 Dynamic (W8A8 per-token)
Method
: llm-compressor oneshot() with load_context()
Quantized Layers
: All Linear layers except router projections, norms, and lm_head
Router Weights
: Kept in full BF16 precision for numerical stability in expert dispatch
Target Runtime
: vLLM 0.20.2rc1+ with cu129 (CUDA 12.9)
Quantization Details
This checkpoint was quantized using
llm-compressor
's FP8 dynamic quantization. The quantization process carefully preserves model quality by keeping router weights and normalization layers in full precision to maintain numerical stability.
Throughput Gain vs BF16
: ~80% more concurrent calls at equivalent latency
Accuracy (vs BF16)
MMLU-Pro
: -0.26 percentage points (negligible)
Generation Quality
: No observable drift on multi-turn conversations
Router Stability
: Keeping router weights in BF16 eliminates topic drift in long-context calls
Key Design Decisions
Why
load_context()
+
oneshot()
?
The raw
google/gemma-4-26B-A4B-it
checkpoint stores MoE experts as fused 3D tensors
[128, dim, dim]
rather than separate
nn.Linear
modules. Generic quantization tools (including
model_free_ptq
) cannot handle this format: they linearize for calibration but fail to fuse weights back to the native format on save, leaving the checkpoint in a broken intermediate state.
Using
load_context()
with
oneshot()
triggers llm-compressor's internal
linearize_moe()
and automatic fuse-back-on-save path, which correctly handles Gemma 4's fused expert structure.
Why exclude router weights?
Router weights make discrete expert-selection decisions. Quantizing them to FP8 adds noise to this decision boundary, which compounds over long sequences (especially in voice call contexts with many tokens). Keeping them in full BF16 ensures numerically stable expert dispatch regardless of sequence length.
Hardware Requirements
Minimum
: NVIDIA A100 (40GB) or H100 (80GB)
Tested
: NVIDIA H100 NVL with vLLM 0.20.2rc1+
CUDA Driver
: 12.7+ (cu129 image requires 12.9 compatible driver)
Citation
@article{gemma4,
title={Gemma 4: Open Models for Everyone},
author={Google DeepMind},
year={2025}
}
@software{llm-compressor,
title={LLM Compressor: Compression of Large Language Models},
author={Neural Magic},
url={https://github.com/vllm-project/llm-compressor}
}
License
This quantized checkpoint is provided under the same license as the base
google/gemma-4-26B-A4B-it
model (Gemma Community License). See
https://huggingface.co/google/gemma-4-26B-A4B-it
for full license terms.
gemma-4-26B-A4B-it-FP8-DYNAMIC huggingface.co is an AI model on huggingface.co that provides gemma-4-26B-A4B-it-FP8-DYNAMIC's model effect (), which can be used instantly with this urs-rs gemma-4-26B-A4B-it-FP8-DYNAMIC model. huggingface.co supports a free trial of the gemma-4-26B-A4B-it-FP8-DYNAMIC model, and also provides paid use of the gemma-4-26B-A4B-it-FP8-DYNAMIC. Support call gemma-4-26B-A4B-it-FP8-DYNAMIC model through api, including Node.js, Python, http.
gemma-4-26B-A4B-it-FP8-DYNAMIC huggingface.co is an online trial and call api platform, which integrates gemma-4-26B-A4B-it-FP8-DYNAMIC's modeling effects, including api services, and provides a free online trial of gemma-4-26B-A4B-it-FP8-DYNAMIC, you can try gemma-4-26B-A4B-it-FP8-DYNAMIC online for free by clicking the link below.
urs-rs gemma-4-26B-A4B-it-FP8-DYNAMIC online free url in huggingface.co:
gemma-4-26B-A4B-it-FP8-DYNAMIC is an open source model from GitHub that offers a free installation service, and any user can find gemma-4-26B-A4B-it-FP8-DYNAMIC on GitHub to install. At the same time, huggingface.co provides the effect of gemma-4-26B-A4B-it-FP8-DYNAMIC install, users can directly use gemma-4-26B-A4B-it-FP8-DYNAMIC installed effect in huggingface.co for debugging and trial. It also supports api for free installation.
gemma-4-26B-A4B-it-FP8-DYNAMIC install url in huggingface.co:
