lightseekorg / kimi-k3-dflash2

kimi-k3-dflash2

Model Overview

kimi-k3-dflash2 is a DFlash2 MTP draft model for accelerating inference of Kimi-K3 , trained with TorchSpec - an online speculative decoding training framework that runs FSDP training and inference concurrently. If you find this draft model useful, please give our project TorchSpec a star on GitHub .

Performance

The primary metric is accept_length - the average number of tokens accepted per speculation step with topk=1, num_steps=7, num_draft_tokens=8 . Higher is better; the ceiling for this configuration is 8.

Measured against an NVFP4 Kimi-K3 target on GB300 (TP=8), at temperature=1.0, top_p=0.95 , reasoning effort max, one request at a time. Acceptance is reported per request by the scheduler, so N is the number of requests measured.

Quick Start

Requirements

• NVIDIA GPU with CUDA 12.0+
• TokenSpeed with DFlash2 speculative decoding support, added in tokenspeed#1211

Launch Server (TokenSpeed)

python3 -m tokenspeed.cli serve /models/Kimi-K3 \
  --served-model-name kimi-k3-dflash2 \
  --speculative-algorithm DFLASH \
  --speculative-draft-model-path /models/dflash2 \
  --speculative-num-steps 7 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 8 \
  --speculative-draft-model-quantization unquant \
  --attention-backend mla \
  --drafter-attention-backend mla \
  --sampling-backend flashinfer \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --mm-encoder-tp-mode data \
  --moe-backend flashinfer_trtllm \
  --gpu-memory-utilization 0.90 \
  --max-num-seqs 4 \
  --max-cudagraph-capture-size 4 \
  --cudagraph-capture-sizes 1 2 3 4 \
  --disable-prefill-graph \
  --disable-kvstore \
  --host 0.0.0.0 \
  --port 8000

/models/Kimi-K3 is the target checkpoint and /models/dflash2 is this draft; point both at your local paths.

--sampling-backend flashinfer keeps rejection sampling faithful to the requested temperature / top_p , which is required to reproduce the acceptance numbers above.