Sponsored by APIMart
Toolify Toolify logo
AI Models
gemma-3-27b-it-quantized.w4a16

RedHatAI / gemma-3-27b-it-quantized.w4a16

Run on huggingface.co
huggingface.co
Total runs: 68.1K
24-hour runs: 0
7-day runs: 886
30-day runs: 53.3K
Model's Last Updated: June 09 2025
image-text-to-text

Introduction of gemma-3-27b-it-quantized.w4a16

Model Details of gemma-3-27b-it-quantized.w4a16

gemma-3-27b-it-quantized.w4a16

Model Overview
  • Model Architecture: google/gemma-3-27b-it
    • Input: Vision-Text
    • Output: Text
  • Model Optimizations:
    • Weight quantization: INT4
    • Activation quantization: FP16
  • Release Date: 6/4/2025
  • Version: 1.0
  • Model Developers: RedHatAI

Quantized version of google/gemma-3-27b-it .

Model Optimizations

This model was obtained by quantizing the weights of google/gemma-3-27b-it to INT4 data type, ready for inference with vLLM >= 0.8.0.

Deployment
Use with vLLM

This model can be deployed efficiently using the vLLM backend, as shown in the example below. 

from vllm import LLM, SamplingParams
from vllm.assets.image import ImageAsset
from transformers import AutoProcessor

# Define model name once
model_name = "RedHatAI/gemma-3-27b-it-quantized.w4a16"

# Load image and processor
image = ImageAsset("cherry_blossom").pil_image.convert("RGB")
processor = AutoProcessor.from_pretrained(model_name, trust_remote_code=True)

# Build multimodal prompt
chat = [
    {"role": "user", "content": [{"type": "image"}, {"type": "text", "text": "What is the content of this image?"}]},
    {"role": "assistant", "content": []}
]
prompt = processor.apply_chat_template(chat, add_generation_prompt=True)

# Initialize model
llm = LLM(model=model_name, trust_remote_code=True)

# Run inference
inputs = {"prompt": prompt, "multi_modal_data": {"image": [image]}}
outputs = llm.generate(inputs, SamplingParams(temperature=0.2, max_tokens=64))

# Display result
print("RESPONSE:", outputs[0].outputs[0].text)

vLLM also supports OpenAI-compatible serving. See the documentation for more details.

Creation

This model was created with llm-compressor by running the code snippet below:

Model Creation Code 
import base64
from io import BytesIO
import torch
from datasets import load_dataset
from transformers import AutoProcessor, Gemma3ForConditionalGeneration
from llmcompressor.modifiers.quantization import GPTQModifier
from llmcompressor.transformers import oneshot


# Load model.
model_id = "google/gemma-3-27b-it"
model = Gemma3ForConditionalGeneration.from_pretrained(
    model_id,
    device_map="auto",
    torch_dtype="auto",
)
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)

# Oneshot arguments
DATASET_ID = "neuralmagic/calibration"
DATASET_SPLIT = {"LLM": "train[:1024]"}
NUM_CALIBRATION_SAMPLES = 1024
MAX_SEQUENCE_LENGTH = 2048

# Load dataset and preprocess.
ds = load_dataset(DATASET_ID, split=DATASET_SPLIT)
ds = ds.shuffle(seed=42)

dampening_frac=0.07

def data_collator(batch):
    assert len(batch) == 1, "Only batch size of 1 is supported for calibration"
    item = batch[0]
    collated = {}
    import torch


    for key, value in item.items():
        if isinstance(value, torch.Tensor):
            collated[key] = value.unsqueeze(0)
        elif isinstance(value, list) and isinstance(value[0][0], int):
            # Handle tokenized inputs like input_ids, attention_mask
            collated[key] = torch.tensor(value)
        elif isinstance(value, list) and isinstance(value[0][0], float):
            # Handle possible float sequences
            collated[key] = torch.tensor(value)
        elif isinstance(value, list) and isinstance(value[0][0], torch.Tensor):
            # Handle batched image data (e.g., pixel_values as [C, H, W])
            collated[key] = torch.stack(value)  # -> [1, C, H, W]
        elif isinstance(value, torch.Tensor):
            collated[key] = value
        else:
            print(f"[WARN] Unrecognized type in collator for key={key}, type={type(value)}")
    
    return collated
   


# Recipe
recipe = [
    GPTQModifier(
        targets="Linear",
        ignore=["re:.*lm_head.*", "re:.*embed_tokens.*", "re:vision_tower.*", "re:multi_modal_projector.*"],
        sequential_update=True,
        sequential_targets=["Gemma3DecoderLayer"],
        dampening_frac=dampening_frac,
    )
]

SAVE_DIR=f"{model_id.split('/')[1]}-quantized.w4a16"

# Perform oneshot
oneshot(
    model=model,
    tokenizer=model_id,
    dataset=ds,
    recipe=recipe,
    max_seq_length=MAX_SEQUENCE_LENGTH,
    num_calibration_samples=NUM_CALIBRATION_SAMPLES,
    trust_remote_code_model=True,
    data_collator=data_collator,
    output_dir=SAVE_DIR
)
Evaluation

The model was evaluated using lm_evaluation_harness for OpenLLM v1 text benchmark. The evaluations were conducted using the following commands:

Evaluation Commands
OpenLLM v1 
lm_eval \
  --model vllm \
  --model_args pretrained="<model_name>",dtype=auto,add_bos_token=True,max_model_len=4096,tensor_parallel_size=<n>,gpu_memory_utilization=0.8,enable_chunked_prefill=True,trust_remote_code=True,enforce_eager=True \
  --tasks openllm \
  --batch_size auto
Accuracy
Category Metric google/gemma-3-27b-it RedHatAI/gemma-3-27b-it-quantized.w8a8 Recovery (%)
OpenLLM V1 ARC Challenge 72.53% 72.35% 99.76%
GSM8K 92.12% 91.66% 99.51%
Hellaswag 85.78% 84.97% 99.06%
MMLU 77.53% 76.77% 99.02%
Truthfulqa (mc2) 62.20% 62.57% 100.59%
Winogrande 79.40% 79.79%% 100.50%
Average Score 78.26% 78.02% 99.70%
Vision Evals MMMU (val) 50.89% 51.78% 101.75%
ChartQA 72.16% 72.20% 100.06%
Average Score 61.53% 61.99% 100.90%

Runs of RedHatAI gemma-3-27b-it-quantized.w4a16 on huggingface.co

68.1K
Total runs
0
24-hour runs
0
3-day runs
886
7-day runs
53.3K
30-day runs

More Information About gemma-3-27b-it-quantized.w4a16 huggingface.co Model

More gemma-3-27b-it-quantized.w4a16 license Visit here:

https://choosealicense.com/licenses/gemma

gemma-3-27b-it-quantized.w4a16 huggingface.co

gemma-3-27b-it-quantized.w4a16 huggingface.co is an AI model on huggingface.co that provides gemma-3-27b-it-quantized.w4a16's model effect (), which can be used instantly with this RedHatAI gemma-3-27b-it-quantized.w4a16 model. huggingface.co supports a free trial of the gemma-3-27b-it-quantized.w4a16 model, and also provides paid use of the gemma-3-27b-it-quantized.w4a16. Support call gemma-3-27b-it-quantized.w4a16 model through api, including Node.js, Python, http.

gemma-3-27b-it-quantized.w4a16 huggingface.co Url

https://huggingface.co/RedHatAI/gemma-3-27b-it-quantized.w4a16

RedHatAI gemma-3-27b-it-quantized.w4a16 online free

gemma-3-27b-it-quantized.w4a16 huggingface.co is an online trial and call api platform, which integrates gemma-3-27b-it-quantized.w4a16's modeling effects, including api services, and provides a free online trial of gemma-3-27b-it-quantized.w4a16, you can try gemma-3-27b-it-quantized.w4a16 online for free by clicking the link below.

RedHatAI gemma-3-27b-it-quantized.w4a16 online free url in huggingface.co:

https://huggingface.co/RedHatAI/gemma-3-27b-it-quantized.w4a16

gemma-3-27b-it-quantized.w4a16 install

gemma-3-27b-it-quantized.w4a16 is an open source model from GitHub that offers a free installation service, and any user can find gemma-3-27b-it-quantized.w4a16 on GitHub to install. At the same time, huggingface.co provides the effect of gemma-3-27b-it-quantized.w4a16 install, users can directly use gemma-3-27b-it-quantized.w4a16 installed effect in huggingface.co for debugging and trial. It also supports api for free installation.

gemma-3-27b-it-quantized.w4a16 install url in huggingface.co:

https://huggingface.co/RedHatAI/gemma-3-27b-it-quantized.w4a16

Url of gemma-3-27b-it-quantized.w4a16

gemma-3-27b-it-quantized.w4a16 huggingface.co Url

Provider of gemma-3-27b-it-quantized.w4a16 huggingface.co

RedHatAI
ORGANIZATIONS
huggingface.co

Other API from RedHatAI

 huggingface.co
RedHatAI/gpt-oss-120b

Total runs: 26.2K
Run Growth: -15.7K
Growth Rate: -59.90%
Updated:May 18 2026
huggingface.co
RedHatAI/gpt-oss-20b

Total runs: 20.9K
Run Growth: -11.8K
Growth Rate: -56.48%
Updated:April 29 2026