ig1 / medgemma-27b-it-FP8-Dynamic

MedGemma-27B-IT-FP8-Dynamic

Overview

MedGemma-27B-IT-FP8-Dynamic is an FP8 Dynamic–quantized derivative of Google’s MedGemma-27B-IT model, optimized for high-throughput inference while preserving strong performance on medical and biomedical instruction-tuned tasks.

This version is intended for vLLM deployment on modern NVIDIA GPUs and follows a safe FP8 Dynamic quantization strategy that avoids known instability issues related to vision components.

Base Model

• Base model : google/medgemma-27b-it
• Architecture : Decoder-only Transformer (instruction-tuned)
• Domain : Medical / Biomedical NLP
• Modality : Multimodal (text + vision), text-focused FP8 quantization

Quantization Details

• Method : FP8 Dynamic
• Tooling : llmcompressor
• Quantized layers : Linear layers
• Excluded components :
  • lm_head
  • Vision tower and multimodal projection layers
    ( vision_tower , visual , vision_model , multi_modal_projector , etc.)

Rationale

• FP8 Dynamic reduces VRAM usage and improves throughput.
• Vision-related modules are intentionally excluded to avoid instability and unnecessary quantization for text-centric inference.
• The resulting model is stable and compatible with vLLM .

Weights are already quantized — do not apply runtime quantization.

Intended Use

• Medical and biomedical instruction-following
• Clinical text summarization and analysis
• Medical RAG pipelines
• Decision-support and research assistance

Deployment (vLLM)

Recommended

vllm serve ig1/medgemma-27b-it-FP8-Dynamic \
  --served-model-name medgemma-27b-it-fp8 \
  --dtype auto