infosave / KAT-Coder-V2.5-CMF

KAT-Coder-V2.5 — CMF coding specialist (expert-defragmented MoE)

A 34.7B-A3B MoE coder in a single 12.7 GB file — 35% smaller than the full quantized model, ×1.8 faster on a 24 GB MacBook, +2.8% code perplexity. This is Kwaipilot/KAT-Coder-V2.5-Dev (Qwen3.6-35B-A3B architecture: 40 layers, 30 of them GatedDeltaNet linear attention, 256 routed experts top-8 + a shared expert), quantized to 4-bit tiles and then physically stripped of the experts that code generation never routes to .

The speedup is not a kernel trick: the full model does not fit a 24 GB machine and pages from disk on every token, while the specialist resides in memory entirely. On machines with plenty of RAM the two decode at similar speed and you simply save the 7 GB.

How to use

CMF is a single-file LLM format with a small pure-Rust runtime — no torch, no CUDA install, no Python. Install the CLI (one command, GPU backends included) and run:

cargo install cortiq-cli   # or a release binary: github.com/infosave2007/cmf/releases

cortiq run KAT-Coder-V2.5-CMF.cmf \
    --prompt "Write a Python function that checks if a number is prime." --max-tokens 300

cortiq serve KAT-Coder-V2.5-CMF.cmf --port 8080   # OpenAI-compatible API + dashboard
cortiq bench KAT-Coder-V2.5-CMF.cmf               # measure on your hardware

The tokenizer and chat template are embedded in the file — run is a real chat turn out of the box.

GPU: CMF_GPU=1 enables the GPU path. On discrete Vulkan/DX12 cards the entire decode — GatedDeltaNet recurrence, attention, the MoE router, the on-device top-k expert selection and every selected expert — executes as one GPU submit per token (the full-model variant of this pipeline decodes at 32.8 tok/s on an RTX 5090 vs 14.4 on its 32-core host CPU). On Apple silicon a runtime probe arbitrates Metal against CPU per operation and keeps whichever wins. Long contexts: --o1 all converts the 10 softmax-attention layers into a constant-memory streaming operator (KV+state at 4K context: 238 → 83 MB).

The technology

MoE expert usage turns out to be strongly task-conditional . Measured on the full KAT-Coder: the top-64 expert sets selected for code vs for natural-language prose overlap with a Jaccard index of just 0.25 — near-disjoint working sets — and a code-derived expert mask captures only ~39% of prose routing mass. A model serving one task therefore carries hundreds of experts it never routes to.

The pipeline that produced this file (two commands, reproducible with cortiq ≥ 0.5.27 ):

1. Record the routing field. A teacher-forced pass over a representative code corpus with CMF_MOE_STATS=stats.json records per-layer expert-selection frequencies — an empirical routing field over the expert lattice (the "B-field" of the CMF patent family's claim 12).
2. Drop what the task never uses. cortiq moe-defrag model.cmf --stats stats.json --cover 0.95 keeps, per layer, the smallest top set of experts reaching 95% of the recorded routing mass (mean 158 of 256 per layer here; 3,906 experts dropped in total), renumbers the kept experts into a contiguous prefix, slices the router's rows to match, and writes a standard .cmf . At inference the router's softmax renormalizes over the kept set — no retraining, weights byte-identical to the quantized originals.

The same restriction can be previewed at runtime without rewriting the file ( CMF_MOE_MASK=stats.json CMF_MOE_MASK_COVER=0.95 ), which is how the perplexity gate above was measured before the physical cut — the two are mathematically identical.

Scope, honestly. This is a code specialist: calibration was Rust-heavy source code, and quality outside the calibrated task degrades by design (prose routes to experts that are no longer there). For general-purpose use, take the full model through the step-by-step guide — it covers GGUF → CMF conversion, Vulkan and Metal, and carving your own specialist for any task with your own corpus.

Provenance

• Base model: Kwaipilot/KAT-Coder-V2.5-Dev (Apache-2.0)
• Quantized source: bartowski/Kwaipilot_KAT-Coder-V2.5-Dev-GGUF (Q4_K_M), imported to CMF q4_tiled by cortiq import-gguf (pure Rust; every llama.cpp storage convention undone on import)
• Expert defrag: cortiq moe-defrag , cover 0.95, code-calibrated (3,072 tokens of Rust source)
• Runtime: github.com/infosave2007/cmf (Apache-2.0), cortiq ≥ 0.5.27