bumbuk / Qwen3.5-4B-AWQ-text-flat

Qwen3.5-4B-AWQ-text-flat

Text-only AWQ quantum Qwen3.5-4B (2.91 GiB / 3.13 GB, one safetensors file) with tensors renamed to the flat naming scheme expected by vLLM.

Why is this needed?

The original checkpoint kumar2235/Qwen3.5-4B-AWQ is preserved in the VLM model markup: all tensors are prefixed with model.language_model.* (a legacy of Qwen3_5ForConditionalGeneration ). The vLLM implementation of the text architecture Qwen3_5ForCausalLM is flat: expects model.layers.* , model.embed_tokens , and model.norm . Because of this, vLLM doesn't load the original and crashes on startup:

ValueError: There is no module or parameter named 'language_model' in Qwen3_5Model.

What's changed

• All 1170 tensors have had their prefix renamed from model.language_model. to model. (e.g., model.language_model.layers.0.mlp.gate_proj.weight to model.layers.0.mlp.gate_proj.weight ).
• Only the safetensors JSON header has been rewritten (154,752 to 137,196 bytes). Data bytes are intact: the file size matches the original byte for byte — 3,127,716,724 bytes .
• config.json , tokenizer, and chat_template.jinja are copies of the original, unchanged.
• The model remains text-only : no vision tower, Qwen3_5ForCausalLM architecture, quantization — AWQ W4A16_ASYM group 128 (compressed-tensors, pack-quantized ), linear_attn (DeltaNet) and lm_head remain in BF16.

Verified

• vLLM 0.27.1 ( vllm/vllm-openai:latest ), NVIDIA GPU (Blackwell / Ada): loads and responds to requests. -Quantization is determined automatically by config.json ( --quantization compressed-tensors can be omitted).
• Note: when bind-mounting models on Windows (9P file system), vLLM recommends --safetensors-load-strategy=prefetch . On a Linux server (standard deployment), the flag is not needed.

Running in vLLM

vllm serve /models/Qwen3.5-4B-AWQ-text-flat \
--served-model-name Qwen3.5-9B \
--max-model-len 16384 \
--reasoning-parser qwen3 \
--language-model-only \
--default-chat-template-kwargs '{"enable_thinking": false}'

How it's done (reproducible)

Renaming is rewriting the safetensors header (stdlib Python, no dependencies: data_offsets in safetensors are calculated from the start of the data section, so the header can be changed freely, the data is not shifted):

import json, struct, shutil

SRC = "model.safetensors" # original (kumar2235/Qwen3.5-4B-AWQ)
DST = "model-flat.safetensors" # result

with open(SRC, "rb") as f: 
n = struct.unpack("<Q", f.read(8))[0] 
header = json.loads(f.read(n))

renamed = {}
for k, v in header.items(): 
if k == "__metadata__": 
renamed[k] = v 
elif k.startswith("model.language_model."): 
renamed["model." + k[len("model.language_model."):]] = v
else:
renamed[k] = v

new_header = json.dumps(renamed, separators=(",", ":")).encode()
with open(SRC, "rb") as fin, open(DST, "wb") as fout:
fout.write(struct.pack("<Q", len(new_header)))
fout.write(new_header)
fin.seek(8 + n) # data section immediately after the original header
shutil.copyfileobj(fin, fout)

Post-conversion check: there should be no names with language_model left in the header, and os.path.getsize(DST) should equal 8 + len(new_header) + (os.path.getsize(SRC) - 8 - n) .

Attribution

• Quantization: kumar2235/Qwen3.5-4B-AWQ (AWQ W4A16_ASYM g128, llm-compressor, calibration — 512 OpenPlatypus samples).
• Base: Qwen/Qwen3.5-4B (Apache 2.0).
• Derivative license — Apache 2.0 (inherited from base).