eatang / Kimi-K2.5-2layer

Kimi-K2.5-2layer

A 2-layer INT4 slice of moonshotai/Kimi-K2.5 , built as a GPU CI fixture for SkyRL . This is not a usable language model — two layers of a 61-layer network produce incoherent text. It exists so tests can exercise the real architecture, with real (truncated) weights, at a size that fits a single node.

Contents

• language_model.model.layers.0 — dense MLP layer ( first_k_dense_replace=1 )
• language_model.model.layers.1 — MoE layer with all 384 routed experts + 1 shared expert
• language_model.model.embed_tokens , language_model.lm_head , language_model.model.norm
• vision_tower.* and mm_projector.* , unchanged

Weights are byte-identical to the source release: the routed-expert weight_packed / weight_scale / weight_shape triples and its quantization_config (compressed-tensors pack-quantized INT4, group_size=32 ) are kept as they ship. Only text_config.num_hidden_layers is changed, to 2.

Companion

eatang/Kimi-K2.5-2layer-BF16 is the same slice with its routed experts dequantized. SkyRL serves this INT4 checkpoint from the inference engine and loads the BF16 one as fake-INT4 QAT masters in the trainer ( trainer.policy.model.fake_int4_qat.bf16_base_path ), since Megatron-Bridge cannot load compressed-tensors.

Total: 16.4 GB.