bbkdevops / cass-vibe-8k

Qwen3.5-9B-DS-v4-Flash-v3.0-FP8

Qwen3.5-9B-DS-v4-Flash-v3.0-FP8 is a FP8 dynamic quantized version of prithivMLmods/Qwen3.5-9B-DS-v4-Flash-v3.0 , a reasoning-capable 9B-parameter language model built on Qwen/Qwen3.5-9B . This quantization reduces model size and memory footprint while preserving the model's long-form reasoning, mathematical problem solving, scientific analysis, coding, and instruction-following capabilities, making deployment more accessible on smaller GPUs. Qwen3.5-9B-DS-v4-Flash-v3.0 was trained through a multi-stage supervised fine-tuning pipeline on top of prithivMLmods/Q3.5-9B-DS-v4-Flash-v2.0 , using approximately 3.5K filtered samples drawn from DeepSeek V4 Flash reasoning traces along with additional high-quality reasoning datasets.

This model is an experimental release and may generate unexpected behaviors or reasoning artifacts in certain scenarios. Quantization to FP8 may introduce minor numerical differences relative to the bf16 source model.

Quantization Details

Quantization was performed using llmcompressor with the following recipe:

default_stage:
  default_modifiers:
    QuantizationModifier:
      targets: [Linear]
      ignore: ['re:.*lm_head', 're:.*embed_tokens$', 're:.*visual.*', 're:.*model.visual.*',
        're:.*linear_attn.*']
      scheme: FP8_DYNAMIC
      bypass_divisibility_checks: false
      requires_calibration_data: false

Linear layers are quantized to FP8 with dynamic per-tensor activation scaling, so no calibration dataset is required ( requires_calibration_data: false ). The lm_head , embedding table, any vision-tower ( visual ) components, and linear_attn layers are excluded from quantization and remain at full precision to preserve output-head fidelity and numerical stability.

Use with vLLM

Qwen3.5-9B-DS-v4-Flash-v3.0-FP8 is served through vLLM with native support for compressed-tensors FP8 checkpoints.

Requirements

• torch >= 2.11.0
• vllm >= 0.19.1
• A GPU with FP8 support recommended (Hopper or Blackwell class) for best throughput; also runs on Ampere with FP8 dequantized on the fly.

Serve

vllm serve prithivMLmods/Qwen3.5-9B-DS-v4-Flash-v3.0-FP8 \
  --max-model-len 32768

Client request

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

messages = [
    {
        "role": "user",
        "content": "Explain how a transformer model processes text."
    }
]

response = client.chat.completions.create(
    model="prithivMLmods/Qwen3.5-9B-DS-v4-Flash-v3.0-FP8",
    messages=messages,
    temperature=0.0,
    max_tokens=512,
)

print(response.choices[0].message.content)

Quick Start with Transformers

pip install transformers accelerate

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model = AutoModelForCausalLM.from_pretrained(
    "prithivMLmods/Qwen3.5-9B-DS-v4-Flash-v3.0-FP8",
    torch_dtype="auto",
    device_map="auto"
)

tokenizer = AutoTokenizer.from_pretrained(
    "prithivMLmods/Qwen3.5-9B-DS-v4-Flash-v3.0-FP8"
)

messages = [
    {
        "role": "user",
        "content": "Explain how a transformer model processes text."
    }
]

inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt"
).to(model.device)

outputs = model.generate(
    inputs,
    max_new_tokens=512
)

print(
    tokenizer.decode(
        outputs[0][inputs.shape[-1]:],
        skip_special_tokens=True
    )
)

Training Details (Base Model)

Intended Use and Limitations

Intended use, known limitations, training data composition, and responsible use guidance are unchanged from the base model. See the Qwen3.5-9B-DS-v4-Flash-v3.0 model card for full details.

• Reasoning Research : Studying long-context reasoning and multi-stage training techniques.
• Mathematical Reasoning : Solving complex mathematical problems with multi-step reasoning.
• Scientific Reasoning : Performing structured scientific analysis and problem solving.
• Coding Assistance : Improving code understanding and generation through long-context reasoning.
• Instruction Following : Evaluating and improving instruction-following capabilities.
• Efficient Local Deployment : Reduced memory footprint enables 9B reasoning inference on smaller GPUs.

Limitations

• Experimental Model : Behavior may differ from the base model in certain scenarios.
• Reasoning Artifacts : Complex reasoning chains may occasionally produce incorrect intermediate steps or conclusions.
• Training Biases : Performance reflects the characteristics and coverage of the reasoning datasets used during training.
• Quantization Drift : FP8 dynamic quantization may introduce minor numerical differences relative to the bf16 base model; downstream accuracy should be validated for production use cases.

License

Released under the Apache 2.0 License, consistent with the base model.

Acknowledgements

• Qwen/Qwen3.5-9B : Base model used for this project.
• TRL - Transformers Reinforcement Learning : TRL is a full stack library providing tools to train transformer language models with methods including Supervised Fine-Tuning (SFT), Group Relative Policy Optimization (GRPO), Direct Preference Optimization (DPO), Reward Modeling, and more.
• Transformers : Transformers provides state-of-the-art machine learning models for text, computer vision, audio, video, and multimodal tasks, supporting both inference and training.
• llmcompressor : Used to produce the FP8 dynamic quantization for this release.