eaddario / IQ2_K-IQ6_K-Test-Models

Experimental IQ2_K, IQ3_K, IQ4_K, IQ5_K & IQ6_K Tests

Please note that these models cannot be used with standard llama.cpp . See [TBD]

Quantization test results for K2-Horizon-7B , comparing standard llama.cpp quantization against a fork that adds five new non-linear super-block types: IQ2_K to IQ6_K .

Summary

The five new types are non-linear (NL) super-block (K) types. Each one splits a super-block into 16 sub-blocks of 16 weights, gives every sub-block a signed scale under one fp16 super-block scale, maps every index through a non-linear codebook, and stores one phase bit per sub-block that shifts that sub-block's codebook by a fixed step. Each new type spends exactly the same index payload as its K-quant counterpart:

IQ3_K , IQ4_K and IQ5_K files are therefore exactly as large as their K-quant counterparts:

K2-Horizon-7B-Q3_K.log:  llama_model_quantize_impl: quant size  =  4927.77 MiB (4.59 BPW)
K2-Horizon-7B-IQ3_K.log: llama_model_quantize_impl: quant size  =  4927.77 MiB (4.59 BPW)

IQ2_K is 8 bytes per super-block smaller than Q2_K , and IQ6_K is 2 bytes larger than Q6_K . Measured against the F16 reference:

Four of the five new types perform better than their counterpart on every quality metric. IQ2_K is the exception.

1. Benefits

a) Same bytes, lower error: IQ3_K , IQ4_K and IQ5_K

These three are size-for-size replacements: same super-block, same bytes per block, same file size to the byte. On K2-Horizon-7B , with the recipe, imatrix and the quantization parameters all fixed, the whole quality difference comes solely from the format, proving the new types to be superior on every metric at all three levels:

IQ3_K is the standout. At the same 4.81 GiB it halves the mean KLD of Q3_K and cuts the PPL penalty from +17.17% to +6.97%. That gain is almost as large as the whole step from Q2_K to Q3_K (−52.1% mean KLD), which costs 673 MiB. IQ3_K is also faster on Metal: tg128 65.49 vs 39.40 t/s (+66.2%) and pp1024+tg1024 105.44 vs 84.28 t/s (+25.1%).

IQ4_K and IQ5_K gain less, but consistently: −20.3% and −17.1% mean KLD, −18.4% and −19.7% at the 99th percentile, and 10% and 9% fewer top-1 changes.

b) IQ6_K : less than half the KLD of Q6_K

IQ6_K is the only type that is larger than its counterpart: 212 against 210 bytes per super-block. That is +0.95% on the block tensors but only +0.69% (+51.75 MiB) on the file. In exchange:

Key Takeaways

• IQ3_K halves the error of Q3_K at the same size. Both are 4.81 GiB; mean KLD is 0.0875 vs 0.1798 ( −51.3% ), PPL +6.97% vs +17.17% over F16, top-1 agreement 85.93% vs 80.05%, and tg128 is 66% faster on Metal.
• IQ4_K and IQ5_K also improve. Same file sizes as Q4_K and Q5_K , −20.3% and −17.1% mean KLD, and better on every other quality metric.
• IQ6_K outperforms Q6_K by more than Q6_K beats Q5_K . +52 MiB (+0.7%) buys −55.7% mean KLD; the step from Q5_K to Q6_K costs 880 MiB for −41.0%.
• IQ2_K is the exception. It is 4.9% smaller than Q2_K , but its mean KLD is 18.5% higher and it runs about 20% slower.

2. Disadvantages

1. IQ2_K loses to Q2_K . It is the only type that is worse than its counterpart: mean KLD 0.4447 vs 0.3752 (+18.5%), PPL +46.44% vs +37.85% over F16, 99% KLD 3.16 vs 2.84. It is also smaller (76 vs 84 bytes per super-block), so the comparison is not like-for-like.
2. Throughput is mixed ( llama-bench , Metal, 12 threads). Only IQ3_K is clearly faster than its counterpart. IQ2_K is about 20% slower on all three tests ( pp512 −22.3%, tg128 −20.2%, pp1024+tg1024 −18.4%). IQ6_K is about 10% slower ( pp512 −9.7%, pp1024+tg1024 −11.4%), and IQ4_K is 10.4% slower on pp1024+tg1024 . IQ5_K is roughly even ( pp512 −4.0%, pp1024+tg1024 +4.3%).
3. Quantization is 2–10× slower. Wall-clock time with 16 threads: IQ2_K 86.8 s vs 39.9 s for Q2_K (2.2×), IQ3_K 155.3 s vs 15.6 s (9.9×), IQ4_K 167.0 s vs 33.6 s (5.0×), IQ5_K 201.8 s vs 37.6 s (5.4×) and IQ6_K 117.1 s vs 14.0 s (8.4×). However, it is a one-off cost per model.
4. Single architecture, single model. Every conclusion here is drawn from one 9B dense checkpoint, one calibration corpus (imatrix), one KLD corpus (275 chunks of 1024 tokens) and one benchmark machine.

3. Comparative Performance Tables

Lower is better for every metric except Top-1 .

Columns prefixed S· describe the standard (naive) build; columns prefixed F· describe the fork build.

Bold marks the better value of each standard/fork pair, here and in the extended metrics below. Rows 2–4 compare files of identical size.

Extended Metrics

Prompt Processing + Text Generation Benchmarks

4. Logs and Test Results

5. Quantization Command

llama-quantize --pure --token-embedding-type q8_0 --output-tensor-type q8_0 --imatrix imatrix/imatrix-K2-Horizon-7B-medium.gguf K2-Horizon-7B-F16.gguf K2-Horizon-7B-<TYPE>.gguf <TYPE>