lightseekorg / kimi-k2.6-eagle3-mla

kimi-k2.6-eagle3-mla

Model Overview

kimi-k2.6-eagle3-mla is an Eagle3 MTP draft model with MLA (Multi-Latent Attention) for accelerating inference of Kimi-K2.6 , trained with TorchSpec — an online speculative decoding training framework that runs FSDP training and inference concurrently. If you find this draft model useful, please give our project TorchSpec a star on GitHub .

Why an MLA (Multi-Latent Attention) Draft Model

Compared with an MHA draft model, the MLA variant is a better fit for Kimi-K2.6 deployment:

• Uses less KV cache, which reduces serving memory pressure.
• Matches Kimi-K2.6's MLA architecture, so it fits more naturally into the inference engine's KV-cache handling under different serving scenarios such as PD-Disaggregation.

Training Setup

• Cluster : 3 nodes × 8× B200 (24 GPUs total)
• Training : 1 node (8 GPUs), FSDP
• Inference : 2 nodes (16 GPUs), vLLM (TP=8 per node)
• Continual training : Initialized from kimi-k2.5-eagle3-mla checkpoint
• Iterations : 9,279 steps
• Learning rate : 2e-5, cosine schedule

Performance

The primary metric is accept_length — the average number of tokens accepted per speculation step with num_speculative_tokens=3 . Higher is better.

Benchmarks were run on vLLM 0.20.0 with 8× B200 GPUs.

Quick Start

Requirements

• NVIDIA GPU with CUDA 12.0+
• vLLM >= 0.20.0

Launch Server (vLLM)

vllm serve moonshotai/Kimi-K2.6 \
    --tensor-parallel-size 8 \
    --speculative-config '{"model": "lightseekorg/kimi-k2.6-eagle3-mla", "method": "eagle3", "num_speculative_tokens": 3}' \
    --trust-remote-code

Launch Server (SGLang)

MLA Eagle3 draft model is not yet supported in SGLang. Will update once support is available.

Citation

@misc{torchspec2026,
  title={TorchSpec: An Online Speculative Decoding Training Framework},
  url={https://github.com/torchspec-project/TorchSpec},
  year={2026}
}