vinpix / Bonsai-8B-llama.cpp

Ternary-Bonsai-8B for llama.cpp

GGUF builds of PrismML's Ternary-Bonsai-8B for llama.cpp , covering both deployment targets: a native TQ2_0 build for CPU, and a ternary-aware zero-exact Q2_K build for GPU and every k-quant runtime — at matching perplexity.

Ternary-Bonsai is an end-to-end 1.58-bit model (weights in {-1, 0, +1} , trained with ternary QAT, no higher-precision escape hatches). It punches far above its size class, but on llama.cpp it runs into a practical wall: the tight ternary format has no GPU kernel. This repo provides both halves of the solution.

Files

Both decode to the same learned weights. They differ only in storage container, and therefore in which backends can run them and how fast.

TL;DR — which one? CPU box -> TQ2_0 (fewer bytes streamed, ~12-14 tok/s, smallest RAM). GPU offload, or any runtime that only speaks k-quants -> Q2_KT .

---

Why two files

TQ2_0 — the native CPU format

TQ2_0 is the tightest faithful ternary layout in ggml (~2.06 BPW). It is the right choice on CPU and it is what PrismML ships. Its one hard limit:

TQ2_0 has no GPU kernel. In current llama.cpp it implements a vec_dot for CPU only — no CUDA, Metal, or Vulkan path. -ngl silently keeps the ternary tensors on the host.

Q2_KT — the GPU-portable format

Q2_K is one of the most broadly supported quant types in the ecosystem — hand-written kernels for CUDA, Metal, Vulkan, ROCm, SYCL. Re-hosting the ternary weights inside a Q2_K container makes Bonsai portable everywhere Q2_K runs.

The catch: you cannot just run llama-quantize ... Q2_K on a ternary model. Doing so is actively harmful — for a reason worth explaining.

---

The trap: naively quantizing a ternary model to Q2_K

Q2_K dequantizes each weight as

w = d * (scale & 0xF) * q  -  dmin * (scale >> 4),   q in {0, 1, 2, 3}

The stock Q2_K quantizer ( make_qkx2_quants ) is an L2-optimal grid fitter for continuous FP16 distributions. Handed a ternary tensor whose values are exactly {-s, 0, +s} , it fits a 4-level grid over 3-level data and places the levels so the ternary zero lands between grid points — at q ~ 1.5 , rounding to an offset of +/-s/3 .

Measured directly on Bonsai-8B:

A --pure Q2_K of a ternary model gets two thirds of the signs wrong and shifts every zero — in practice a degenerate model that loops single tokens. The stock pipeline masks this by promoting sensitive tensors ( attn_output , ffn_down , attn_v ) to Q3_K / Q4_K , inflating the file to ~3.18 BPW and only partially repairing the damage. That promotion is a workaround, not a fix.

Q2_KT fixes the encoding at the source.

---

How Q2_KT is built (ternary-aware, zero-exact)

The ternary structure admits an exact Q2_K encoding with no residual. For a group of magnitude s :

scale nibble  sc = 15
min   nibble  m  = 15          (min-nibble == scale-nibble)
d = dmin = s / 15
q = t + 1                      t in {-1, 0, +1}  ->  q in {0, 1, 2}

Substituting into the dequant:

w = d*15*q - dmin*15 = d*15*(q - 1) = s*(q - 1) = s*t     <- exact ternary

The -1 bias falls out of the min term for free; the ternary zero ( t = 0 -> q = 1 ) reconstructs to exactly 0 ; every non-zero to exactly +/-s . The only error is the FP16 storage of the single per-group scale — which TQ2_0 also carries. Hence sign accuracy 100 % and PPL parity.

The qs nibbles are packed with the exact ggml Q2_K interleave (32-byte groups, stride-32 crumbs across 128-element spans), so the file is bit-compatible with the reference dequantize_row_q2_K and the SIMD vec_dot_q2_K_q8_K in the forward pass. Output projection and token embedding are kept at Q4_0 (not ternary in the source).

Layout note. TQ2_0 and Q2_K qs streams share the same stride-32 / 128-span interleave. Matching ggml 's canonical order is the whole game: an encoding that is self-consistent with a hand-rolled unpacker but disagrees with the canonical order will pass every fidelity check against itself and still emit garbage in the forward pass, because dequantize_row and vec_dot both assume the canonical order. This build matches it.

---

Usage

CPU (use TQ2_0 )

llama-cli -m Bonsai-8B-TQ2_0.gguf -p "The capital of France is" -n 64 -t 6

Fastest on CPU and smallest RAM footprint. ~12-14 tok/s on a low-end Zen2 desktop (Ryzen 3 PRO 4350GE, dual-channel DDR4).

GPU offload (use Q2_KT )

llama-cli -m Bonsai-8B-Q2_KT.gguf -ngl 99 -p "The capital of France is" -n 64

Verified running fully offloaded on an AMD Radeon Vega iGPU via the Vulkan RADV backend. Because Q2_K has real GPU kernels, -ngl 99 actually places the ternary body on the device — unlike TQ2_0 , which would stay on the host.

Any GGUF loader / llama.cpp server

Q2_KT loads as an ordinary Q2_K model — no custom code, no patched runtime. Every existing loader already understands the container.

---

Quality

Perplexity on wikitext-2-raw (test split, 20 chunks of 512 tokens, -t 4 , identical harness):

Q2_KT is perplexity-neutral vs. the native ternary build (delta within the +/-0.5 statistical error). You pay ~0.85 BPW over TQ2_0 for universal backend support; you do not pay for it in quality — expected, since the encoding is lossless on the ternary levels.

For task benchmarks (MMLU-Redux, GSM8K, HumanEval+, IFEval, ...), see PrismML's numbers for the base model — these files change the container, not the learned weights.

---

Technical notes & reproducibility

• Q2_KT container: 252 ternary body tensors -> pure Q2_K (zero-exact); output + token_embd -> Q4_0 ; 145 norms -> F32 .
• No tensor promotion. Every ternary tensor is pure Q2_K ; the zero-exact encoding makes the Q3_K / Q4_K promotion unnecessary — that is why Q2_KT is 2.91 BPW, not 3.18.
• Bit-exactness: validated against ggml 's reference dequantize_row_q2_K (100 % sign accuracy, max abs error 1e-5 from the FP16 scale) and end-to-end against the forward vec_dot (coherent generation, PPL parity).
• Architecture: Qwen3, 36 layers, hidden 4096 , FFN 12288 , GQA 32/8, full 65536-token context preserved in metadata.
• Reproduce PPL: llama-perplexity -m <file> -f wiki.test.raw --chunks 20 .

A note for the llama.cpp community

The +/-s/3 zero-error affects any end-to-end ternary model (Bonsai, BitNet, TriLM, ...) run through the stock Q2_K quantizer. A ternary-aware branch in quantize_row_q2_K — detect a block with <=3 symmetric levels, emit d = dmin = s/15 , sc = m , q = t+1 — would let the official pipeline produce zero-exact Q2_K for the whole class, without the promotion workaround. Q2_KT is a concrete demonstration that the encoding is exact and backend-portable. Contributions upstream welcome.

---

Attribution & license

• Base model: Ternary-Bonsai-8B by PrismML (Babak Hassibi et al., Caltech). Ternary QAT, Qwen3 architecture, end-to-end 1.58-bit.
• These files: GGUF re-encodes for llama.cpp . TQ2_0 is the native ternary layout; Q2_KT is a lossless-on-ternary Q2_K re-encode for backend portability. Learned weights unchanged.
• License: Apache 2.0, inherited from the base model.

@techreport{ternarybonsai,
  title  = {Ternary Bonsai: 1.58-bit Language Models at 8B, 4B, and 1.7B Scale},
  author = {Prism ML},
  year   = {2026},
  month  = {April},
  url    = {https://prismml.com}
}

---

Community re-encodes, not affiliated with or endorsed by PrismML. Quality figures are measured on the specific files in this repository.