malaiwah / GLM-5.2-QSRT-Fruit-Instruct-exact

GLM-5.2 QSRT Fruit Instruct

This is the assistant-masked SFT form of a 5.04B-parameter GLM-5.2 serving proxy , not the 754B GLM-5.2 model. It is encoded in KQuant's canonical QSRT atom format and served without reconstructing dense expert weights.

The artifact qualifies the codec/storage/runtime integration on the instruction-tuned Fruit checkpoint. Packaging, provenance, exact state decoding, and the canonical single-GPU runtime path are implemented. No downstream task quality is claimed; see Known limitations .

What is included

• 13 transformer layers: 3 dense and 10 MoE layers, plus the packaged MTP expert layer.
• Hidden size 1,024; MoE intermediate size 512; 256 routed experts per MoE/MTP layer.
• 2,816 QSRT experts in 11 canonical atom files.
• SQG-XOR-Cheb-T12 E4M3 codebook, three-bit trellis payload, fixed P24/P33 pair records, and physical atom rotation.
• Canonical qsrt_atoms_v1 storage with complete per-file SHA-256 manifests and a fail-closed QSRT_COMPLETE.json marker.
• W4A16 prefill/reference execution and W4A8 decode execution through B12X.

The expert allocation selected by the frozen calibration evidence is recorded in each qsrt-layer-*.json sidecar. Aggregate allocation counts are:

Size and memory

The apples-to-apples baseline is the complete BF16 tensor set. All three rows below cover the same 5,040,368,896 logical parameters and count only Safetensors files; effective bpw is stored bytes times eight divided by that parameter count, so it includes container and quantization metadata.

The whole-model rates include 611,183,872 non-routed parameters retained in BF16. Isolating the 4,429,185,024 routed-expert weights gives:

QSRT is therefore 10.25% smaller than SIQ on the routed-expert component and 6.21% smaller on the compared tensor files. Package-level totals are not used for the comparison because tokenizer, card, and optional evaluation files are not model weights. The previous card's 7,593,020,594-byte BF16 row was not the complete BF16 tensor set and has been removed.

W4A8 and W4A16 use the same stored weights, so their loader weight storage is identical; W4A8 changes the decode execution path, not the checkpoint size.

Hugging Face repository-size audit

The Hugging Face model API with blobs=true reported the following immutable snapshot on 2026-08-08. Repository bytes sums every sibling's reported size; Safetensors bytes sums only *.safetensors . These are observed repository payloads, not parameter-count estimates.

The three full-model rows ground real download/storage scale only. They are not used for Fruit percentage claims because Fruit has 5.04B parameters while the production model has roughly 754B. The apples-to-apples Fruit tensor comparison above remains the codec-size result.

Adjacent-rate evidence

evaluation/fruit-uniform-rate-sweep.json re-encodes the same authenticated expert sample at uniform K2, K3, and K4, with fresh per-endpoint FP16 scales. It measured 16 of 19 predeclared assignments; 3 lacked the minimum routed calibration support and were skipped rather than imputed. These are pre-layer-deduplication expert-local endpoints, not package sizes.

Relative to K3, K2 is 3.948x / 4.039x / 4.254x on weight, captured-H, and validation-routed NMSE. K4 is 0.257x / 0.255x / 0.251x on the same metrics.

Evidence boundary

The completion seal covers every top-level package file and every regular file under evaluation/ . The sealed adjacent-rate report measures local routed expert reconstruction on authenticated, document-disjoint calibration rows. It does not establish chat quality, broad downstream task quality, or general serving throughput.

Reproducible runtime

The runtime is pinned to the reviewed commits below:

• KQuant encoder: local-inference-lab/kquant#4 , encoded with KQuant revision d3355b1b1088c5c9fa2afbcc608cc627a8a42f69 .
• B12X kernels: local-inference-lab/b12x#129 , tested revision 84df431be79d6897fb3872516a7da39139e4905b .
• vLLM loader: local-inference-lab/vllm#269 , tested revision 056c3e1f46c1870378d266f4ee5fa9049309bc04 .

The derived image starts from the content-addressed public base docker.io/voipmonitor/vllm@sha256:3230c25ff95f8678a8eeb52a463f0d3b9f96f6ad550418cc51ea12177a55b41c hard-coded by Dockerfile.fruit-qsrt . It installs the exact B12X checkout, copies the base's compiled vLLM extensions into the reviewed source tree, and seals the exact runtime package bytes. MODEL_REVISION resolves the Hub branch once; hf download then uses the resulting immutable commit SHA.

git clone https://github.com/malaiwah/vllm-voipmonitor.git vllm-fruit
git -C vllm-fruit checkout --detach 056c3e1f46c1870378d266f4ee5fa9049309bc04

docker build \
  --file vllm-fruit/Dockerfile.fruit-qsrt \
  --build-arg VLLM_REVISION=056c3e1f46c1870378d266f4ee5fa9049309bc04 \
  --build-arg B12X_REVISION=84df431be79d6897fb3872516a7da39139e4905b \
  --tag fruit-qsrt:056c3e1f46c1870378d266f4ee5fa9049309bc04 \
  vllm-fruit

MODEL_REVISION="$(
  curl -fsSL https://huggingface.co/api/models/malaiwah/GLM-5.2-QSRT-Fruit-Instruct-exact \
    | python3 -c 'import json,sys; print(json.load(sys.stdin)["sha"])'
)"
MODEL_DIR="Fruit-QSRT-${MODEL_REVISION}"
test ! -e "${MODEL_DIR}"
hf download malaiwah/GLM-5.2-QSRT-Fruit-Instruct-exact \
  --revision "${MODEL_REVISION}" \
  --local-dir "${MODEL_DIR}"

docker run --rm --gpus '"device=0"' --shm-size=16g \
  --read-only \
  --tmpfs /tmp:rw,exec,nosuid,size=8g \
  --tmpfs /cache:rw,exec,nosuid,size=16g \
  --tmpfs /root/.cache:rw,nosuid,size=1g \
  --publish 8000:8000 \
  --volume "$PWD/${MODEL_DIR}:/model:ro" \
  --env MODEL=/model \
  fruit-qsrt:056c3e1f46c1870378d266f4ee5fa9049309bc04

The qualified path is SM120 with CUDA 13.2.1 and PyTorch 2.12.0+cu132 in the content-addressed base, plus nvidia-cutlass-dsl == 4.6.0 in the derived image. The launcher rejects extra vLLM arguments and any value other than TP1, max_num_seqs=1 , max_model_len=4096 , and max_num_batched_tokens=4096 before importing the GPU runtime. The current B12X sparse-prefill backend requires single-request prefill chunks.

The runtime manifest is an integrity check rooted in the trusted immutable image, not an independent signature. The build host and container operator remain trusted. Run the container read-only, keep writable tmpfs mounts outside /opt/vllm-fruit , /opt/b12x-fruit , and /opt/fruit-runtime , and mount the authenticated model read-only as shown above.

W4A16 is used for prefill and any row count above the W4A8 decode ceiling. W4A8 is selected for decode-sized batches of at most 16 rows. Unsupported shapes, activation modes, metadata, or incomplete manifests fail closed.

Provenance and integrity

• Authenticated BF16 source: malaiwah/GLM-5.2-SIQ-Fruit-Instruct-bf16 at immutable revision 678954f65e056a0f508e21eeb9251c655bb9463f .
• Authenticated source manifest ( safetensors_manifest ) SHA-256: 8f23aed5e9b12000ed103a76da772a20730ca53ab7e352d6cb94da2709165245 .
• Calibration capture ID: 24b290abaddb9eff8d7328a2a22f3c33bb7f6f45b77692393b3549ba0fded0a2 .
• Calibration manifest SHA-256: be944c8dfc5b550319d26bc2899f0d2ea3f4ca81275ba53034f0cc8ef7b4e9c5 .
• The encoder authenticated 256 documents / 46,223 tokens from disjoint fit, confirmation, and validation splits.
• Full encoding: 2,816 experts, 1,347.09 GPU-seconds, 2.550 GiB peak CUDA allocation.
• MANIFEST.sha256 , qsrt-manifest.json , .qsrt-source-evidence.json , qsrt-calibration-evidence.json , and QSRT_COMPLETE.json bind the published package to the source and encoder fingerprints.

Known limitations

• Not assistant-quality. A matched TP1/eager four-prompt protocol on 2026-08-08 found that both BF16 and QSRT missed an exact-string instruction, returned 84 for 84 * 3 / 2 instead of 126 , and produced incoherent Chinese text; both honored a short named-system instruction. This is protocol evidence, not a representative benchmark, and no downstream task-quality claim is made.
• Narrow performance result. On one RTX 5090, three warmed eager TP1 requests generating 256 tokens measured median decode rates of 63.54 tokens/s for QSRT and 77.45 tokens/s for matched BF16. Model loading used 2.7 GiB versus 8.77 GiB. This max_num_seqs=1 protocol is not a general throughput benchmark.
• TP2 atom ownership is unit-tested, but only TP1 physical serving has been qualified for this Fruit package.
• The current sparse-attention prefill backend requires max_num_seqs=1 .
• This release qualifies the QSRT codec, storage, loader, and kernels. It does not establish broad downstream task quality.

License

The packaged model files are MIT, matching the authenticated Fruit BF16 source license. B12X and vLLM are Apache-2.0. KQuant is not redistributed in this model repository and remains subject to its upstream repository licensing.