AngelSlim / Hy3-DFly-Block8

A more accessible, comprehensive, and efficient toolkit for large model compression.

✒️ TechnicalReport |    📖 Documentation |   🤗 Hugging Face |   🤖 ModelScope

💬 WeChat |   🫨 Discord

Hy3-A21B No-Thinking Drafters: MTP and DFly

Model Summary

This repository contains no-thinking speculative-decoding drafter checkpoints for Hy3-A21B , including an MTP drafter and a DFly drafter . These draft models are trained with AngelSpec and are intended to accelerate Hy3-A21B inference by proposing candidate tokens that are verified by the target model.

They should be deployed together with the corresponding Hy3-A21B target model under the no-thinking inference setting.

• Target model: Hy3-A21B
• Inference mode: No-thinking
• Drafter types: MTP and DFly
• Training / inference framework: AngelSpec
• Recommended serving backend: vLLM speculative decoding
• Recommended MTP speculative tokens: 3
• Recommended DFly speculative tokens: 8

Training Data

The no-thinking Hy3 drafter training starts from Open-PerfectBlend-style instruction prompts. Target responses are regenerated with Hy3-A21B under the no-thinking setting so that the drafter learns the target-side token distribution used during deployment.

Domain-Specific Data Expansion for DFly

To further strengthen code generation and mathematical reasoning, the DFly training corpus augments Open-PerfectBlend with 700K domain-specific prompts :

• 500K code prompts from OpenCodeInstruct and OpenCodeReasoning.
  • OpenCodeInstruct provides broad code-instruction coverage.
  • OpenCodeReasoning emphasizes distilled reasoning traces for competitive programming.
  • Together, they improve both general code completion and multi-step program synthesis.
• 200K math prompts from Big-Math, adding coverage of diverse and challenging reasoning problems.

Before response generation, prompts that share a contiguous sequence of at least 16 tokens with an evaluation example are removed. Responses for the filtered prompts are then regenerated with Hy3-A21B , mixed with Open-PerfectBlend, and used to train the combined corpus for six epochs with the same objective.

This expansion mainly improves math/code acceptance length. The gain is consistent across structured benchmarks, while chat-oriented performance may not improve uniformly because the final checkpoint is specialized toward code and math.

Evaluation: Accepted Length

The following table reports the mean accepted length on Hy3-A21B at temperature 1 under the no-thinking setting. The average is the arithmetic mean over the six benchmarks.

Evaluation: Throughput Benefit

The following results are measured on HY3 295B-A21B with TP=8 at temperature 1 across concurrency levels. Tok/s denotes output-token throughput. Spd. denotes throughput relative to autoregressive decoding ( AR ). Each cell uses 3 x 120s measurement windows, and Avg. is the arithmetic mean across the six datasets.

Deployment

Set TARGET to the Hy3-A21B target model path or repository and DRAFT to the corresponding drafter checkpoint. The relevant inference code can be found in PR https://github.com/vllm-project/vllm/pull/50246 .

MTP-3

export TARGET="<path-or-hf-repo-to-hy3-a21b-target>"
export DRAFT="<path-or-hf-repo-to-hy3-mtp-drafter>"

vllm serve "$TARGET" \
  --tensor-parallel-size 8 \
  --speculative-config "{"method":"mtp", "model":"$DRAFT", "num_speculative_tokens":3}"

DFly-8

export TARGET="<path-or-hf-repo-to-hy3-a21b-target>"
export DRAFT="<path-or-hf-repo-to-hy3-dfly-no-thinking-drafter>"

vllm serve "$TARGET" \
  --tensor-parallel-size 8 \
  --speculative-config "{"method":"dspark", "model":"$DRAFT","num_speculative_tokens":7}"

Technical Report

For more details, see the technical report:

AngelSpec: Towards Real-World High Performance Inference with Speculative Decoding
arXiv: https://arxiv.org/abs/2607.25852