This repo contains specialized MoE-quants for XiaomiMiMo/MiMo-V2.6-Flash-RL. The idea being that given the huge size of the FFN tensors compared to the rest of the tensors in the model, it should be possible to achieve a better quality while keeping the overall size of the entire model smaller compared to a similar naive quantization. To that end, the quantization type default is kept in high quality and the FFN UP + FFN GATE tensors are quanted down along with the FFN DOWN tensors.
The MXFP4 quant is the "full quality" version, as the model has MXFP4 experts. I'm working on some new quants using
ed's bpw-size PR
and will upload them if they look good.
|
Quant
|
Size
|
Mixture
|
PPL
|
1-(Mean PPL(Q)/PPL(base))
|
KLD
|
|
MXFP4
|
162.89 GiB (4.52 BPW)
|
BF16 / MXFP4
|
5.149210 ± 0.030596
|
+0.0715%
|
-0.000000 ± 0.000000
|
|
Q3_K
|
137.75 GiB (3.82 BPW)
|
Q8_0 / Q3_K / Q3_K / mxfp4
|
5.177623 ± 0.030900
|
+0.6237%
|
0.123607 ± 0.000648
|
|
IQ2_S
|
106.31 GiB (2.95 BPW)
|
Q6_K / IQ2_S / IQ2_S / Q3_K
|
5.404774 ± 0.032158
|
+5.0382%
|
0.178092 ± 0.000878
|