Myric / KAT-Coder-V2.5-Dev-MTP-APEX-GGUF

KAT-Coder-V2.5-Dev — APEX GGUF with MTP head

MoE-aware, mixed-precision APEX quantization of Kwaipilot/KAT-Coder-V2.5-Dev , an agentic-coding fine-tune on the Qwen3.6-35B-A3B backbone ( qwen3_5_moe : 40 layers, 256 routed

• 1 shared expert, hybrid GatedDeltaNet linear-attention with periodic full attention). Kwaipilot reports SWE-bench Verified 69.40 for this size class, against Qwen3.6-35B-A3B's 64.40.

What makes this build different: it has a working MTP head. KAT-Coder ships mtp_num_hidden_layers: 0 — no multi-token-prediction head at all, so no speculative decoding is possible out of the box, and that is true of the vendor release and of every other quant of this model I am aware of. This repo transplants Qwen3.6-35B-A3B's own trained MTP head onto KAT-Coder and quantizes the result. Measured below: 2.03x on a hard agentic-coding suite, with correctness unchanged.

Files

Pick MTP-APEX-i-quality unless you are VRAM-constrained, in which case APEX-dynamic is the best quality available at its size. To run either you need only the .gguf . The safetensors shard and index are there for anyone who wants to redo the splice against the original checkpoint: drop those three files into a local copy of Kwaipilot/KAT-Coder-V2.5-Dev alongside its own 13 model-*-of-00013.safetensors shards. KAT-Coder's original shards are untouched and byte-identical; this is a drop-in addition, not a re-upload of the 65 GB checkpoint.

The MTP head — honest framing

This is the naive, un-fine-tuned transplant . The head's weights are copied straight from stock Qwen3.6 with zero training on KAT-Coder's own hidden-state distribution. It is architecturally clean rather than a hack — hidden_size, expert count, and attention/MoE shapes match exactly, because KAT-Coder is itself built on the Qwen3.6-35B-A3B backbone — but a fine-tuning pass (freezing KAT-Coder's 40 layers, training only the ~850M-parameter head) is a planned follow-up. These numbers are the baseline before that.

Draft-acceptance rate

llama-server --spec-type draft-mtp --spec-draft-n-max 2 , 5 real from-scratch coding prompts, n_predict=400 each:

The same quant tier without any MTP head runs ~70.8 tok/s solo. Acceptance tracks predictability: BST traversal boilerplate hits 89%, the LRU cache with genuine locking and eviction logic is hardest at 61%.

Agentic coding benchmarks

Two task suites run through opencode 's real tool-calling loop, MTP build vs the same quant without MTP. Correctness was identical — 100% on both suites either way (59/59 and 28/28 test cases), which is what the theory predicts: the MTP head only drafts, the main model always verifies against its own computation, so a bad draft is rejected and never reaches the output. Only speed moves.

The single slower result ( lru_ttl_cache , 0.81x) is most likely run-to-run variance in how much iteration that attempt needed rather than a systematic MTP cost — every other task improved.

The imatrix

llama-imatrix is impractical on this architecture: GatedDeltaNet linear-attention is a serial state-space recurrence, and the imatrix collection callback breaks the GPU path, falling back to a single CPU thread. kat-coder.imatrix was instead generated with a band-serialized PyTorch forward pass over a general text/code corpus. It is a standard GGUF-format importance matrix ( in_sum2 + counts per tensor), bit-compatible with llama-quantize --imatrix , so it is directly reusable if you want to build your own tiers.

One checkpoint quirk worth knowing if you do: KAT-Coder stores MoE experts as fully separate per-expert tensors ( mlp.experts.{e}.{gate,up,down}_proj.weight × 256) rather than the fused gate_up_proj / down_proj layout the HF module expects. Both imatrix generation and GGUF conversion have to fuse them.

Allocation

APEX structural mode: routed experts are bucketed into edge / near / middle depth bands (edge = first and last layers, generally most load-bearing) with decreasing precision toward the middle. For i-quality that is routed edge → Q6_K, near → Q4_K, middle → Q3_K. Attention, shared-expert, and router tensors stay at higher fixed precision (Q6_K–Q8_0).

Measured quality

Wikitext perplexity across the tiers that were built. The two marked hosted are in this repo; the other two were measured for comparison and are not uploaded:

Two things worth drawing out. i-quality lands within 1.3% of full bf16 at under a third of the size. And APEX-dynamic beats the stock IQ3_XXS baseline on both axes at once — smaller (12.13 vs 12.68 GiB) and lower perplexity (7.543 vs 7.881) — which is why it is the one hosted alongside the flagship rather than the stock tier.

That result is cleaner than the same method got on Laguna-XS, where it traded size for a small PPL cost. Whether role/depth-aware allocation beats a well-tuned stock quant appears to depend on the architecture and calibration, and is not something to assume without measuring.

The hosted MTP build uses the i-quality recipe; its file is slightly larger than the 19.23 GiB row because it carries the MTP head as well.

Attribution

• KAT-Coder base model: Kwaipilot — Kwaipilot/KAT-Coder-V2.5-Dev (Apache-2.0).
• MTP head weights: Qwen — Qwen/Qwen3.6-35B-A3B (Apache-2.0), transplanted unmodified.
• APEX recipe & toolkit: LocalAI — localai-org/apex-quant .
• Quantization engine: llama.cpp ( ggml-org ).

Unofficial community quantization and experiment; not affiliated with or endorsed by Kwaipilot or Qwen.