lightseekorg / kimi-k2.5-eagle3

Model Overview

kimi-k2.5-eagle3 is an Eagle3 MTP draft model for accelerating inference of Kimi-K2.5 , trained with TorchSpec — an online speculative decoding training framework that runs FSDP training and inference concurrently.

Training data is available at lightseekorg/kimi-mtp-dataset .

Training Setup

• Cluster : 4 nodes × 8× H200 (32 GPUs total)
• Training : 2 nodes (16 GPUs), FSDP
• Inference : 2 nodes (16 GPUs), Engine (TP=8 per node)
• Duration : ~14 hours per phase

Training ran in two phases, each 20k steps (~300k samples):

• Phase 1 : Regenerated open-perfectblend dataset
• Phase 2 : Mixed dataset (English, VL, Chinese, function-call, agent, creative writing)

All training responses were regenerated by Kimi-K2.5 via Engine to match the base model's exact token distribution.

Training Curves

The plots show loss, token acceptance accuracy, and simulated accept_length during training. Both eval sets contain 256 samples drawn from each phase's own training corpus.

Phase 1 (steps 0 → 20k):

Phase 2 (steps 20k → 40k):

Performance

The primary metric is accept_length — the average number of tokens accepted per speculation step with topk=1, num_steps=3, num_draft_tokens=4 . Higher is better.

Benchmarks were run using SpecForge 's bench_eagle3.py . BFCL v3 benchmarks (†) use a custom extension to the original script.

Quick Start

Requirements

• NVIDIA GPU with CUDA 12.0+
• SGLang ≥ 0.5.8

Launch Server

python -m sglang.launch_server \
    --model-path /path/to/Kimi-K2.5 \
    --tp 8 \
    --trust-remote-code \
    --speculative-algorithm EAGLE3 \
    --speculative-draft-model-path lightseekorg/kimi-k2.5-eagle3 \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --mem-fraction-static 0.75 \
    --dtype bfloat16

Run Benchmarks

python bench_eagle3.py \
    --model-path /path/to/Kimi-K2.5 \
    --port 30000 \
    --config-list 1,3,1,4 \
    --benchmark-list <benchmark_name> \
    --skip-launch-server

--config-list format: topk,num_steps,topk,num_draft_tokens .