AxisQuant / Qwen3.6-27b-gptq-int4

Qwen3.6-27B — GPTQ INT4

GPTQ INT4 quantization of Qwen/Qwen3.6-27B . 3× smaller. Negligible accuracy loss. Drop-in.

Author: Prashant Takale ( prashantcp8 ) Base model: Qwen/Qwen3.6-27B Method: GPTQ (weight-only, 4-bit, group_size=128, desc_act=True) Tooling: GPTQModel v7

---

At a glance

Quantization preserves accuracy to within ~1–2% of the BF16 baseline on standard benchmarks — typical for GPTQ INT4 on dense models of this scale.

---

Benchmarks

Both models evaluated under identical conditions with lm-evaluation-harness : greedy decoding, no thinking mode, apply_chat_template=True , max_gen_toks=512 , batch_size=8 .

Both deltas are within the per-task standard error, so accuracy is statistically indistinguishable from BF16.

---

Quantization recipe

The vision encoder ( model.visual.* ) is intentionally left in BF16 — only the language-model weights are quantized.

---

Usage

With GPTQModel (recommended)

from gptqmodel import GPTQModel
from transformers import AutoTokenizer

model_id  = "AxisQuant/Qwen3.6-27b-gptq-int4"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model     = GPTQModel.load(model_id, device_map="auto", trust_remote_code=True)

messages = [{"role": "user", "content": "Explain GPTQ in one sentence."}]
text   = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True, enable_thinking=False,
)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
out    = model.generate(**inputs, max_new_tokens=256, do_sample=False)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

With transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id  = "AxisQuant/Qwen3.6-27b-gptq-int4"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model     = AutoModelForCausalLM.from_pretrained(
    model_id, device_map="auto", trust_remote_code=True,
)

---

Hardware

• Weights : 18 GB on disk · ~14 GB VRAM at load
• Single-GPU friendly : comfortably fits on a 24 GB consumer card (RTX 3090 / 4090) for short-to-mid context
• Long context (64K+ tokens) : H100 80 GB or A100 80 GB recommended

---

Limitations

• Only the language-model weights are quantized; the vision encoder remains in BF16
• Calibration set was English C4 — heavy non-English or domain-specific workloads may benefit from re-quantizing on a matching corpus
• Thinking mode ( enable_thinking=True ) works but is significantly slower — enable only when reasoning quality matters more than latency

---

License

Inherits the license of the base model. See the Qwen/Qwen3.6-27B model page for terms.

---

Citation

Base model

@misc{qwen3.6-27b,
    title  = {{Qwen3.6-27B}: Flagship-Level Coding in a {27B} Dense Model},
    author = {{Qwen Team}},
    month  = {April},
    year   = {2026},
    url    = {https://qwen.ai/blog?id=qwen3.6-27b}
}

Quantization method

@article{frantar2022gptq,
    title   = {{GPTQ}: Accurate Post-training Compression for Generative Pretrained Transformers},
    author  = {Frantar, Elias and Ashkboos, Saleh and Hoefler, Torsten and Alistarh, Dan},
    journal = {arXiv preprint arXiv:2210.17323},
    year    = {2022}
}

This quantization

@misc{prashanttakale,
    title  = {Qwen3.6-27B GPTQ INT4 Quantization},
    author = {Takale, Prashant},
    year   = {2026},
    url    = {https://huggingface.co/prashantcp8/Qwen3.6-27b-gptq-int4}
}