Bedovyy / Qwen3-32B.w8a8

W8A8 INT8 Quantization of Qwen3-32B

I made this for running on vLLM with Ampere GPU.

On 2xRTX3090, you may set context length to upto 16384 (or 24576 if you use --kv-cache-dtype fp8 ).

Quantization method

Quantized using

• Tool: llmcompressor 0.5.2.dev22 (db959a3) .
• System: 4x3090, DDR4 128GB + swap 32GB
• Time taken: 6 hours (wall time)

## modified based on the code from https://huggingface.co/nytopop/Qwen3-14B.w8a8

from transformers import AutoTokenizer, AutoModelForCausalLM
from datasets import load_dataset
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import GPTQModifier
from llmcompressor.modifiers.smoothquant import SmoothQuantModifier
from llmcompressor.transformers.compression.helpers import calculate_offload_device_map

model_id  = "Qwen/Qwen3-32B"
model_out = "Qwen3-32B.w8a8"

num_samples = 256
max_seq_len = 4096

tokenizer = AutoTokenizer.from_pretrained(model_id)

def preprocess_fn(example):
  return {"text": tokenizer.apply_chat_template(example["messages"], add_generation_prompt=False, tokenize=False)}

ds = load_dataset("neuralmagic/LLM_compression_calibration", split="train")
ds = ds.shuffle().select(range(num_samples))
ds = ds.map(preprocess_fn)

model = AutoModelForCausalLM.from_pretrained(
  model_id,
  device_map="auto",
  torch_dtype="bfloat16",
  max_memory={0: "10GiB", 1:"10GiB", 2:"10GiB", 3:"10GiB", "cpu":"96GiB"},
)

recipe = [
  SmoothQuantModifier(smoothing_strength=0.7),
  GPTQModifier(sequential=True,targets="Linear",scheme="W8A8",ignore=["lm_head"],dampening_frac=0.01),
]

oneshot(
  model=model,
  dataset=ds,
  recipe=recipe,
  max_seq_length=max_seq_len,
  num_calibration_samples=num_samples,
  output_dir=model_out,
)