cloudyu / GLM-5.3-GGUF

GLM-5.3-SLIM-E192 — IQ2_XXS GGUF: GLM 5.3 on a single machine

GLM 5.3, the 744B-parameter frontier MoE, running on one GPU or one Mac. Expert pruning (192 of 256 routed experts, "SLIM") took 25 % off the model; DwarfStar's 2-bit routed-expert recipe took the rest. The result is a 149.7 GiB file that stays fully resident on a single 180 GB GPU (B200, 21 tokens/s measured) or a single Mac Studio (256 GB; 192 GB with a small context), and streams from SSD on a 128 GB Mac — hardware where the unpruned model needs a multi-GPU node or does not fit at all.

The pruning is the enabler: at 2 bits the unpruned experts alone are 187 GB, so no single-device quantization of the original could fit a 180 GB card with room for a context. Pruning removes 47 GB of expert bytes at this precision and, per its author's A/B on the FP8 checkpoints, costs nothing measurable on coding, cybersecurity, tool calling and math (GPQA −3.6 pt, C-Eval −7.3 pt).

What the file is: GLM-5.3-SLIM-E192 (attention with MLA + DSA sparse indexer, shared experts, router, tokenizer and chat template identical to GLM 5.3; no MTP head) with routed experts in IQ2_XXS (2.06 bits/weight) and everything else in Q8_0 — the same recipe DwarfStar publishes for the full GLM 5.3, so per-token compute and memory traffic are unchanged; only the footprint drops.

It runs with the ds4-glm-slim fork of DwarfStar (a few changes on top of upstream antirez/ds4 , see Requirements ). It is not a llama.cpp GGUF: the tensor layout, quant mix and metadata follow DwarfStar's GLM-DSA format.

At a glance

What is in the file

Per-token decode reads the same amount of data as the full GLM 5.3 Q2 (8 routed experts per layer either way), so speed per token is the same as the 197 GiB file; the gain is memory .

Requirements

Upstream antirez/ds4 cannot load this file: its GLM-DSA loader pins the shape to the official checkpoint (256 experts, 79 blocks, 1 MTP block) and its CUDA backend has no kernels for IQ2_XXS down projections. The fork adds:

• runtime acceptance of expert-pruned GLM 5.2/5.3 shapes (expert count, block count and MTP count read from the GGUF);
• CUDA kernels for all-IQ2_XXS routed layers (prefill on the mmq tier, decode on mmvq vector kernels) and resident weights on discrete GPUs;
• a CUDA quantizer for IQ2_XXS (byte-identical to the C one) and the tooling that produced this file.

Everything else — CLI, agent, HTTP server, KV snapshots, tool calling, thinking control — is unchanged DwarfStar. Build it:

git clone https://github.com/yuhai-china/ds4-glm-slim.git ds4
cd ds4
make                # Apple Silicon / Metal
make cuda-generic   # NVIDIA, local GPU architecture (needs nvcc + cuBLAS)

Startup prints ds4: GLM DSA variant: 192 routed experts, 78 blocks, 0 MTP block(s) when the fork recognises the file.

Quick start

Put the file in gguf/ inside the repository (or pass a full path with -m ).

# interactive chat (thinking on by default; --nothink for direct answers)
./ds4 -m gguf/GLM-5.3-SLIM-E192-IQ2_XXS.gguf --ctx 32768

# OpenAI-compatible server on http://127.0.0.1:8000
./ds4-server -m gguf/GLM-5.3-SLIM-E192-IQ2_XXS.gguf --ctx 65536

# native coding agent
./ds4-agent -m gguf/GLM-5.3-SLIM-E192-IQ2_XXS.gguf --ctx 65536

Add --cuda on NVIDIA hosts. On a 128 GB Mac add --ssd-streaming . See RUNNING.md for memory planning, streaming, server usage and troubleshooting.

Quality

The pruned FP8 base ( model card ) was measured against GLM-5.3 by its author: coding, cybersecurity, tool calling and math within run-to-run noise; GPQA-Diamond −3.6 pt; C-Eval −7.3 pt (the deliberate trade-off). The 2-bit routed experts of this file add their own loss on top. Held-out perplexity of the FP8 base vs. the original: code +1.0 %, English chat +4.7 %, Chinese +7.3 %.

This GGUF, ds4-eval (DwarfStar's built-in harness: GPQA Diamond, SuperGPQA, AIME 2025; thinking on, default budgets), CUDA B200:

ds4-eval scores are integration checks, not leaderboard numbers; compare against the published full GLM 5.3 Q2 run on the same machine and suite.

An imatrix-guided variant (importance collected from this model's own routed activations with ds4 --imatrix-dataset ) is the natural next step and would replace this file's weight-energy importance.

Field notes: security-domain probe on Metal (2026-09-17, this file)

A 6-question offensive/defensive security probe (SQL-injection methodology, C code audit, ROP vs. Canary/NX/ASLR, AD lateral movement with blue-team detection points, Python tooling, RSA broadcast attack) plus a Chinese/English A/B on the same questions, run on the M2 Ultra resident setup below. Single run per question, thinking on — enough to map the failure mode, not a benchmark:

• Reasoning and structure survive the quant. The C audit found all five planted bugs including the two subtle logic flaws; the RSA answer's math (Håstad broadcast via CRT + integer cube root) and its reference code were correct; defense playbooks were coherent in both languages.
• Long-tail precise facts degrade — measurably more in Chinese than in English. The Chinese answers confabulated Windows event IDs, DRSUAPI UUIDs and NTSTATUS codes; the English answers got the same facts right (events 4624/4769/4776, etype 0x17, MITRE T1550.002/T1558.003) yet still mangled tool names ( krb5tgc for krb5tgs ) and invented one detection mechanism. Verify any concrete ID, command or parameter before use, in either language. This matches the pruning A/B above (Chinese +7.3 % perplexity vs. English +4.7 %, C-Eval as the largest score drop).
• Capacity beats precision on hard coding. On the same 13-task suite, this 2-bit/563B build scored 10/13 vs 9/13 for a 4-bit MLX build of a much smaller sibling model (~4× faster per task). The extra wins were the hardest items (persistent segment tree, optimizer comparison) — evidence that expert pruning + 2-bit costs long-tail facts (see above) far more than code reasoning.

Field notes: cybersecurity probe, English (2026-09-17, this file)

Eight expert-level offensive/defensive questions (Kerberoasting detection with event IDs and etype values, x86-64 gadget opcodes, DCSync IOCs, Flask code audit, CBC bit-flipping/padding-oracle math, DNS-tunneling detection design, Windows event-log chain reconstruction, CET/CFI/PAC bypass theory), greedy-free sampling ( temperature 1.0, top_p 0.95, min_p 0 ) via the OpenAI-compatible server. Grades against ground truth:

• Two failure modes, unpredictable. On precision identifiers (UUIDs, GUIDs, RFC numbers, MSR names, opcodes) the model sometimes confabulates confidently (DCSync answer: invented interface UUID, invented RPC method name) and sometimes explicitly refuses to guess and flags for verification (the CET answer). Which mode appears is not predictable.
• Sampling matters more here than anywhere else. Greedy (temp 0) drove the thinking trace into an endlessly repeating loop that consumed the entire budget six times in eight runs, delivering nothing. Recommended sampling for this build: temperature 1.0, top_p 0.95, min_p 0 — i.e. the upstream GLM defaults with min-p disabled. With thinking on, budget 8–13k tokens; with think:false , tasks complete 3–5× faster and code quality holds (10/13 on the coding benchmark).
• What is stable vs what is not. Reasoning chains, exploit-construction logic, code, and defense architecture are reproducible across runs. Long-tail identifiers (UUIDs, GUIDs, event names, opcodes, RFC numbers) fail stochastically — different wrong values each run, ~1 in 10 facts per identifier-dense answer. Treat every concrete identifier in its output as unverified, in any language; prose, code, and arithmetic survive the quant.
• English thinking traces run 3–5× longer than Chinese ones (5–7k tokens observed). With thinking on, budget ≥8–13k generation tokens for English deep-dive prompts ( --ctx 16384 -n 13000 worked) — a 3k-token cap was consumed entirely by the reasoning trace twice, delivering no answer.

Field notes: code-execution benchmark on Metal (2026-09-17, this file)

A 13-task suite (number theory, combinatorics, segmented sieve; algorithm / systems / data-structure / ML / graph coding; logic puzzles; physics and statistics simulations) with automatic code execution, grading against known answers, and one retry on runtime errors. ds4-server resident, thinking off, temperature 1.0, top-p 1.0, ctx 16384. Single run per task; sample is small — treat as a capability map, not a leaderboard:

• 10 / 13 PASS , keyword coverage 98 %. All three number-theory tasks passed, including a segmented sieve to 10^12 and an O(n) totient sieve. Harder coding passed: thread-safe sharded LRU with TTL under an 8-thread stress test, a persistent segment tree with versioned queries, numpy-only multi-head attention with RoPE.
• Failures are systematic, not flaky. One long task (Dijkstra + A* + benchmarking harness) exhausted an 8k-token cap three times and needs ~16k; one physics simulation mis-derived the potential-energy term (wrong total energy from the start — a knowledge gap, not instability); one MCMC task failed once and passed on a re-roll.
• Stability over a 4-hour server session (1,253 speed samples): decode 11.6–14.4 t/s, median 13.15, σ/median = 3.0 %. Zero crashes, zero restarts.
• Capacity beats precision on hard coding. On the same 13-task suite, this 2-bit/563B build scored 10/13 vs 9/13 for a 4-bit MLX build of a much smaller sibling model (~4× faster per task). The extra wins were the hardest items (persistent segment tree, optimizer comparison) — evidence that expert pruning + 2-bit costs long-tail facts (see above) far more than code reasoning.

Per-task results (thinking off, temp 1.0; gen = generation wall time):

Speed

Mac runbook (192 GB, measured 2026-09-17, ds4 Metal build): quit heavy apps, raise the Metal wired limit to 180 GiB ( sudo sysctl iogpu.wired_limit_mb=184320 ; the default 160 GiB fits the plan but leaves no headroom), then

DS4_GLM_MEMORY_GUARD_RESERVE_GB=16 ./ds4 -m GLM-5.3-SLIM-E192-IQ2_XXS.gguf --ctx 8192 -p "…"

Startup reports KV 0.73 GiB + buffers 4.07 GiB + resident model 149.71 GiB = 154.50 GiB planned ; generation ran with zero swap used. Decode holds at ~13 t/s regardless of prompt. Short-prompt prefill is dominated by fixed per-layer cost, so the per-token prefill rate rises with prompt length (25 t/s at 22 tokens → 55 t/s at 157 tokens). With the file in the page cache, model mapping takes ~2 s.

How it was built

python3 gguf-tools/glm53_full_quantize.py \
  --hf GLM-5.3-SLIM-E192 \
  --tokenizer-template GLM-5.3-UD-IQ2_XXS_RoutedIQ2XXS_blk78Q2K.gguf \
  --model-name GLM-5.3-SLIM-E192 \
  --repo-url https://huggingface.co/cloudyu/GLM-5.3-SLIM-E192 \
  --cuda --cuda-batch 32 \
  --out GLM-5.3-SLIM-E192-IQ2_XXS.gguf

• Source: the FP8 (block 128) safetensors of cloudyu/GLM-5.3-SLIM-E192 , dequantized exactly; tokenizer/chat template taken from the checkpoint, validated against the published full GLM 5.3 GGUF.
• Routed experts quantized on a B200 with the fork's PyTorch/CUDA port of DwarfStar's IQ2_XXS quantizer ( gguf-tools/iq2xxs_cuda.py ), which is byte-identical to the C implementation for the same importance vector (verified on 296,848 real blocks). 43,200 expert matrices in 29 minutes.
• Importance: DwarfStar's fallback importance[column] = Σ row[column]² (no activation imatrix).
• Attention, shared experts, dense FFN, embeddings and output head: Q8_0 via the C quantizer, as in the published GLM 5.3 Q2.

Limitations

• 2-bit routed experts. Expect a measurable drop versus the FP8 checkpoint on knowledge-heavy and Chinese-exam tasks; coding and agent use are the intended workloads. Measured on this file (see the Quality field notes): reasoning and code survive; long-tail precise facts (event IDs, UUIDs, tool names, opcodes) are the first casualty , more so in Chinese than in English — treat concrete identifiers in its output as unverified.
• No MTP head. Do not pass --mtp .
• No imatrix in this build (see Quality).
• 192 GB Macs are borderline: 149.7 GiB weights + ~5 GiB graph leaves little for macOS; DwarfStar's memory guard will ask for DS4_GLM_MEMORY_GUARD_RESERVE_GB to be lowered and the context kept small. Now measured on an M2 Ultra 192 GB (Metal, resident, --ctx 8192 , wired limit raised to 180 GiB): 12.3–13.3 t/s decode with zero swap — see Speed . Larger contexts on 192 GB remain untested.
• SSD streaming starts cold on pruned variants (the built-in GLM 5.2 hot seed uses the original expert numbering and is skipped).
• CUDA needs a single GPU with ≥ 158 GiB free (weights + 8 GiB) for the resident fast path; smaller cards fall back to a host mapping that streams experts over PCIe at ~1 t/s. Multi-GPU placement is untested with this file.
• general.source.revision in the GGUF metadata carries the quantizer's default (the official GLM-5.3 revision); the SLIM checkpoint revision used is e45b62eb3f5a22232f1e4980da255266ab933f31 .

Verify the download

size    160760301792 bytes
sha256  68abcb6effe4e7a4379d92f68607dfbd1aca0a1fad5dabda78a96f1a4a0e4725

License and credits

• Weights: GLM-5.3 License (Z.AI), including the Model-as-a-Service clause; this file is a derivative of zai-org/GLM-5.3 via cloudyu/GLM-5.3-SLIM-E192 .
• Expert pruning: cloudyu (GLM-5.3-SLIM-E192).
• Inference engine and quant recipe: Salvatore Sanfilippo and the DwarfStar contributors ( antirez/ds4 ), building on llama.cpp / GGML.
• Fork, CUDA quantizer and this build: yuhai-china — https://github.com/yuhai-china/ds4-glm-slim .