modal-labs / Inkling-NVFP4-DFlash

Inkling-NVFP4-DFlash

Paper | Github | Blog

This repository contains a DFlash draft model for thinkingmachines/Inkling-NVFP4 . It is not a standalone language model. It is intended to be paired with the target model in a speculative decoding server.

This is an early preview release, and the drafter is still training. It uses all causal sliding-window attention (SWA) layers.

DFlash uses a lightweight block diffusion draft model to propose multiple tokens in parallel. The target model verifies those proposals, improving serving throughput while preserving the target model's output distribution.

Quick Start

Here is an example deployment. Some options, including trtllm_mha for draft attention, are pending upstream SGLang support.

sglang serve \
  --model-path thinkingmachines/Inkling-NVFP4 \
  --tp 8 \
  --trust-remote-code \
  --quantization modelopt_fp4 \
  --fp4-gemm-backend flashinfer_trtllm \
  --moe-runner-backend flashinfer_trtllm_routed \
  --enable-torch-symm-mem \
  --attention-backend fa4 \
  --mamba-radix-cache-strategy extra_buffer \
  --disable-custom-all-reduce \
  --page-size 128 \
  --reasoning-parser inkling \
  --tool-call-parser inkling \
  --enable-multimodal \
  --cuda-graph-backend-prefill breakable \
  --enable-scattered-sconv \
  --mem-fraction-static 0.78 \
  --max-running-requests 32 \
  --swa-full-tokens-ratio 0.10 \
  --mamba-full-memory-ratio 0.10 \
  --chunked-prefill-size 16384 \
  --watchdog-timeout 900 \
  --weight-loader-prefetch-checkpoints \
  --weight-loader-prefetch-num-threads 8 \
  --cuda-graph-max-bs-decode 32 \
  --cuda-graph-max-bs-prefill 4096 \
  --cuda-graph-bs-prefill 128 256 384 512 768 1024 1536 2048 2560 3072 3584 4096 \
  --kv-cache-dtype mxfp8 \
  --speculative-algorithm DFLASH \
  --speculative-draft-model-path modal-labs/Inkling-DFlash \
  --speculative-dflash-block-size 16 \
  --speculative-draft-model-quantization fp8 \
  --speculative-draft-attention-backend trtllm_mha \
  --speculative-draft-kv-cache-dtype fp8_e4m3 \
  --speculative-draft-window-size 4096 \
  --host 0.0.0.0 \
  --port 30000

Benchmark Results

This preview includes a preliminary accept-length evaluation at concurrency 1 and block size 16. A full benchmark suite, including throughput and higher-concurrency measurements, will follow.

Setup

• Runtime: SGLang on 8x NVIDIA B300 GPUs, tensor parallel size 8
• Target model: thinkingmachines/Inkling-NVFP4
• Target weights and KV cache: ModelOpt NVFP4 weights with an mxfp8 KV cache
• Backends: fa4 target attention and trtllm_mha DFlash draft attention
• Draft model: FP8 weights with an fp8_e4m3 KV cache and a 4096-token draft window
• Workloads: GSM8K, MATH500, HumanEval, MBPP, and MT-Bench with Inkling's native chat renderer
• Decoding: greedy, reasoning effort 0.9, max output length 2048 tokens
• Measurement: DFlash block size 16 at concurrency 1, with up to 64 measured generation requests per workload; warmup and warmdown requests are excluded
• Accept length: completion_tokens / spec_verify_ct per generation turn, averaged across generation turns

Accept Length

Mean DFlash accept length at concurrency 1.

Citation

If you find DFlash useful, please cite the original paper:

@article{chen2026dflash,
  title   = {{DFlash: Block Diffusion for Flash Speculative Decoding}},
  author  = {Chen, Jian and Liang, Yesheng and Liu, Zhijian},
  journal = {arXiv preprint arXiv:2602.06036},
  year    = {2026}
}