0xSero / Inkling-Small-EXL3-4.5bpw

Inkling-Small-EXL3-4.5bpw

Calibrated EXL3 trellis quantization of the routed MoE experts in thinkingmachines/Inkling-Small , targeting 4.5 bits per routed-expert weight .

Status: The assembled weight archive has been uploaded. Consult EXL3_MANIFEST.json for the current structural and runtime validation state.

Runtime validation: Text generation, multimodal generation, and MTP validation are pending. The uploaded archive and its packed tensors have structural validation only.

What is quantized

This fractional target uses whole routed layers encoded at integer EXL3 K=4 or K=5. Twenty of the forty routed layers use each K value. The higher-error layers measured by the K=4 calibration proxy receive K=5. No tensor is labeled with a fractional bit width.

• Calibration: 1,048,576 naturally routed tokens selected with seeded, no-repeat axis water-filling across general, legal, code/agentic, and reasoning/termination data
• Maximum calibration sequence/sample span: 4,096 tokens
• Routing: Inkling's natural top-6 routed-expert assignments
• Source revision: b2d4f225a02032c5d154bff748ab5a00c5ca26e4
• Achieved routed-trellis rate: 4.500000 bpw
• Assembled repository payload: 150.75 GiB
• Per-layer allocation, tensor inventory, sizes, and validation state: EXL3_MANIFEST.json

Compatibility and how to use it

Download the repository with:

hf download 0xSero/Inkling-Small-EXL3-4.5bpw \
  --local-dir Inkling-Small-EXL3-4.5bpw

This repository is not a drop-in Transformers checkpoint . The routed experts use EXL3 trellis tensors while the rest of Inkling remains in source precision. It requires an Inkling-aware EXL3 loader/runtime that understands the tensor layout described by quantization_config.json and EXL3_MANIFEST.json .

Stock ExLlamaV3 v1.2.1 does not yet include an InklingForConditionalGeneration architecture adapter. The upstream BF16 Inkling model has vLLM and SGLang recipes, but those recipes do not by themselves add support for this experts-only EXL3 layout. Do not infer text, image, audio, or MTP runtime support from a successful download or structural assembly alone.

All EXL3 variants

Method

The source model is loaded once for calibration. Hidden states and natural expert assignments are captured for all forty routed layers. Each expert's gate, up, and down projections are calibrated, Hadamard-transformed, and encoded as EXL3/MCG trellis weights. The full sweep checks finite Hessians and scales, exact trellis byte counts, safetensor key counts, and per-file checksums. Before the sweep, a bounded H200 proof on a real Inkling expert also passed trellis pack/unpack/repack equality and finite reconstruction. That bounded kernel proof is not a full-model generation test. Integer-K caches are reused to assemble the seven public variants without repeating the full model calibration.

The target bpw applies to routed-expert trellis weights. The complete repository is larger than a whole-model quantization at the same nominal bpw because attention, shared experts, multimodal components, the LM head, and MTP remain in source precision.

Credits

• Thinking Machines Lab / Inkling team for releasing Inkling-Small .
• TurboDerp for ExLlamaV3 and the EXL3 format .
• JarvisLabs for providing the 8× NVIDIA H200 compute used for this quantization sweep.

This is an independent community quantization and is not an official release from Thinking Machines Lab, TurboDerp, or JarvisLabs.

License and use

This derivative follows the upstream Apache 2.0 license and the upstream acceptable-use policy . Review the base model card for intended uses, limitations, and safety information.