AxisQuant / Qwen3.5-4B-INT8

Qwen3.5-4B-INT8

Author: Prashant Takale

Model Description

This is an INT8 quantized version of Qwen/Qwen3.5-4B using bitsandbytes LLM.int8() quantization.

What I Did

• Quantized the original Qwen3.5-4B model from BF16 to INT8 precision using bitsandbytes
• Reduced memory footprint by 42.4% while maintaining model quality
• Benchmarked on WikiText-2 (perplexity) and GSM8K (math reasoning)

Why I Did This

1. Memory Efficiency: Reduce GPU memory requirements to run on smaller GPUs
2. Faster Inference: INT8 operations can be faster on compatible hardware
3. Accessibility: Enable deployment on consumer-grade hardware with limited VRAM

Benchmark Results

Key Findings

• 1.74x memory reduction with minimal quality loss
• Only +1.30% perplexity increase on WikiText-2
• Math reasoning capabilities preserved on GSM8K benchmark

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "prashantcp8/Qwen3.5-4B-INT8",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("prashantcp8/Qwen3.5-4B-INT8")

messages = [{"role": "user", "content": "What is machine learning?"}]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
response = tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
print(response)

Technical Details

• Quantization Method: bitsandbytes LLM.int8()
• Base Model: Qwen/Qwen3.5-4B
• Original Precision: BF16
• Quantized Precision: INT8
• Framework: Transformers + bitsandbytes

Hardware Used

• GPU: NVIDIA L4
• PyTorch: 2.10.0+cu128

Limitations

• Requires bitsandbytes library for optimal performance
• Some operations may cast to FP16 during inference
• Best suited for NVIDIA GPUs with INT8 tensor core support

Citation

If you use this model, please cite the original Qwen3.5 model:

@misc{qwen3.5,
  title={Qwen3.5 Technical Report},
  author={Qwen Team},
  year={2025},
  publisher={Alibaba}
}

Author

Prashant Takale

Note: This model is for testing and experimental purposes only. There are still some improvements required in terms of inference speed and accuracy. I plan to explore better quantization methods like AWQ or GPTQ in future iterations for optimized performance.