Total runs:
52.1K
Run Growth:
49.4K
Growth Rate:
94.91%
Sponsored by
ThumbnailCreator.com
coolthor / gemma-4-12B-it-FP8-dynamic
Model's Last Updated: July 21 2026
any-to-any
Introduction of gemma-4-12B-it-FP8-dynamic
Model Details of gemma-4-12B-it-FP8-dynamic
Gemma 4 12B-it — FP8 dynamic
Self-quantized
FP8 (dynamic)
of
google/gemma-4-12B-it
— Google's encoder-free
omni
model (text + image + audio + video). Quantized and benchmarked on an NVIDIA DGX Spark (GB10, sm_121a).
TL;DR: 13 GB on disk (from 23 GB BF16), 15.9 tok/s on a GB10 via vLLM, all four modalities intact. Data-free — no calibration needed.
Benchmark (GB10 / DGX Spark, vLLM 0.22.1 native, single-stream decode, warm)
| Format | Disk | tok/s (EN/ZH) | Omni |
|---|---|---|---|
| BF16 | 23 GB | 7.7 | yes |
| FP8 dynamic (this) | 13 GB | 15.9 | yes |
| NVFP4 W4A16 | 7.7 GB | 24.9 | yes |
If you want the smallest + fastest build, see the sibling NVFP4 weight-only repo . FP8 is the conservative choice (dynamic activations, no calibration, widest kernel support).
Quantization recipe
llmcompressor
, scheme
FP8_DYNAMIC
, data-free (no calibration data). Ignore list keeps the head and the multimodal projectors in BF16:
QuantizationModifier(targets="Linear", scheme="FP8_DYNAMIC",
ignore=["lm_head", "re:.*embed_vision.*", "re:.*embed_audio.*"])
Serving (vLLM)
Needs vLLM with native
Gemma4UnifiedForConditionalGeneration
(~0.22.x / main) and the
TRITON_ATTN
backend (Gemma 4 has heterogeneous head dims: head_dim 256 x 16 = 4096 != hidden 3840):
VLLM_ATTENTION_BACKEND=TRITON_ATTN \
vllm serve coolthor/gemma-4-12B-it-FP8-dynamic --max-model-len 4096
Environment (exact versions — this model is version-sensitive)
This is a brand-new arch on a brand-new GPU, so the toolchain matters more than usual. The versions I actually ran:
| Component | Version | Why it matters |
|---|---|---|
| vLLM |
0.22.1rc1.dev124
(main, post-PR)
|
Needs the
native
Gemma4UnifiedForConditionalGeneration
class, which only landed around
0.22.x
/main. On an older vLLM it falls back to the generic transformers backend, which mishandles Gemma 4's non-square attention and crashes on
o_proj
.
|
| transformers |
5.10.1
|
First release that knows
model_type: gemma4_unified
. Older transformers can't even load the config.
|
| torch |
2.11.0+cu130
|
The one that bit me.
vLLM main pins
torch==2.10
, but its
_C.abi3.so
was compiled against 2.11+cu130 — installing the pinned 2.10 (and pip silently pulling the
CPU
wheel on arm64) gives an
undefined symbol
import error and a CPU-only build. Force-align:
pip install --force-reinstall --no-deps torch==2.11.0 --index-url https://download.pytorch.org/whl/cu130
.
|
| compressed-tensors | bundled with llmcompressor | Reads the FP8 weight format. |
| GPU / arch |
DGX Spark
GB10
,
sm_121a
, CUDA 13.x
|
The torch-ABI dance above is specific to building vLLM from source for sm_121. |
| Attention backend |
VLLM_ATTENTION_BACKEND=TRITON_ATTN
|
Required, not optional — see the head-dim note above. |
On a normal CUDA GPU
(Hopper/Ada/Blackwell desktop) you don't need the torch-ABI overlay — that pain is specific to building vLLM from source for sm_121. A recent
pip install vllm
(with the native Gemma4Unified class) plus
VLLM_ATTENTION_BACKEND=TRITON_ATTN
is enough. FP8 is data-free, so the quantization recipe above is the whole reproduce step — no calibration set needed.
Validation (GB10, transformers)
- Text: coherent EN + ZH.
- Image: accurately described a studio-podcast photo (cat + Shiba Inu, headphones, studio mics, latte-art mug).
- Audio: understood a LibriSpeech clip (Mr. Quilter / apostle / middle classes).
- Video: correctly described a night-street clip.
Credits
-
Base model:
google/gemma-4-12B-it(Apache 2.0, Google DeepMind) -
Quantization:
llmcompressor+compressed-tensors - Quantized & benchmarked by coolthor on a DGX Spark (GB10)
Support
One-person effort on a single DGX Spark, no sponsor. If it saved you time, a coffee ☕ is appreciated.
Runs of coolthor gemma-4-12B-it-FP8-dynamic on huggingface.co
32.3K
Total runs
-38
24-hour runs
-12
3-day runs
-38.7K
7-day runs
-7.7K
30-day runs
More Information About gemma-4-12B-it-FP8-dynamic huggingface.co Model
Url of gemma-4-12B-it-FP8-dynamic
gemma-4-12B-it-FP8-dynamic huggingface.co Url
Provider of gemma-4-12B-it-FP8-dynamic huggingface.co
Other API from coolthor
Updated:September 16 2026
Total runs:
47.9K
Run Growth:
43.8K
Growth Rate:
91.45%
Total runs:
5.7K
Run Growth:
3.2K
Growth Rate:
56.32%
Total runs:
1.2K
Run Growth:
-2.1K
Growth Rate:
-178.76%
Total runs:
682
Run Growth:
-2.0K
Growth Rate:
-294.87%
Total runs:
491
Run Growth:
-14.0K
Growth Rate:
-2846.44%
Total runs:
364
Run Growth:
-337
Growth Rate:
-92.58%
Total runs:
346
Run Growth:
299
Growth Rate:
86.42%
Total runs:
253
Run Growth:
-283
Growth Rate:
-111.86%
Total runs:
171
Run Growth:
164
Growth Rate:
95.91%
Total runs:
139
Run Growth:
-100
Growth Rate:
-71.94%
Total runs:
113
Run Growth:
64
Growth Rate:
56.64%
Total runs:
83
Run Growth:
9
Growth Rate:
10.84%
Total runs:
78
Run Growth:
28
Growth Rate:
35.90%
Total runs:
43
Run Growth:
18
Growth Rate:
41.86%