AngelSlim / Hy3-DFlash-B8

English | 简体中文

A more accessible, comprehensive, and efficient toolkit for large model compression.

✒️ TechnicalReport |    📖 Documentation |   🤗 Hugging Face |   🤖 ModelScope

💬 WeChat |   🫨 Discord

📣Latest News

• [26/07/06] We now support FP8-Static quantization , SmoothQuant for Hy3 (MoE A21B). [Docs]
• [26/06/04] We have released Stem , a sparse attention algorithm that accelerates the Prefill stage of long-context LLMs by dynamically selecting top-k key blocks for block-sparse attention, significantly reducing latency while preserving generation quality. [Docs]
• [26/06/01] We have released DFlare , a block-diffusion speculative decoding framework with layer-wise fusion that achieves up to 5.52× end-to-end speedup . [Docs]
• [26/05/27] We have released D-Cut , an adaptive verification depth pruning technique for speculative decoding. [Docs]
• [26/05/20] We support Distillation for full-precision HuggingFace models and quantized QAT-style models, as detailed in the distillation documentation .
• [26/05/08] We have released STQ1_0 kernel for 1.25-bit model and given a PR to llama.cpp PR #22836 ! If you have any questions or suggestions for STQ_0, welcome to comment under the PR !🔥🔥🔥
• [26/04/29] We have released 2-bit and 1.25-bit versions of Tencent Hy-MT1.5-1.8B Translation Model: Hy-MT1.5-1.8B-2bit and Hy-MT1.5-1.8B-1.25bit . Additionally, we have make an offline translation demo for you to try out. We invite you to give it a spin! 🔥🔥🔥
• [26/04/23] We now support FP8-Static quantization for Hy3-preview (MoE A20B).
• [26/03/25] We have released DAQ , the quantization algorithm that preserves the knowledge acquired while the update of parameters is relatively small during post-training training. [Paper] | [Docs]
• [26/02/09] We have released HY-1.8B-2Bit, 2bit on-device large language model, [Huggingface] .
• [26/01/13] We have released v0.3. We support the training and deployment of Eagle3 for all-scale LLMs/VLMs/Audio models, as detailed in the guidance documentation . And We released Sherry , the hardware-efficient 1.25 bit quantization algorithm [Paper] | [Code] 🔥🔥🔥

🌟Key Features

• Highly Integrated : This toolkit integrates mainstream compression algorithms into a unified framework, offering developers one-click access with exceptional ease of use.
• Continuous Innovation : Beyond integrating widely-used industry algorithms, we are continuously researching better compression algorithms, which will be gradually open-sourced in the future.
• Performance-Driven : We continuously optimize end-to-end performance in model compression workflows and algorithm deployment, such as enabling quantization of models like Qwen3-235B and DeepSeek-R1 on a single GPU.

💼Technical Overview

🛎️How to Use

1. Install AngelSlim

We recommend using pip to install the latest stable version of AngelSlim :

pip install angelslim

Alternatively, you can clone the repository and install from source in editable mode:

cd AngelSlim && python setup.py install

For more detailed installation instructions and platform-specific guidance, please refer to the Installation Documentation .

2. Quick Start

2.1 Speculative Decoding

After installing AngelSlim, you can quickly start Eagle3 training with the following scripts:

# Start the vLLM server
bash scripts/speculative/run_vllm_server.sh
# Generate training data
bash scripts/speculative/generate_data_for_target_model.sh
# Perform online training for the Eagle3 model
bash scripts/speculative/train_eagle3_online.sh

Training and Deployment Guide for Eagle3: LLM | VLM | Audio(ASR) | Audio(TTS) .

2.2 LLM/VLM/Audio Model Quantization

After installing AngelSlim , you can launch static FP8 quantization for the Qwen3-1.7B model with the following one-command script:

python3 tools/run.py -c configs/qwen3/fp8_static/qwen3-1_7b_fp8_static.yaml

This example produces quantized model weights by performing PTQ calibration on a model loaded from HuggingFace.

For Hy3-preview (MoE A20B) FP8-Static quantization:

python tools/run.py -c configs/hunyuan/fp8_static/hunyuanv3_a20b_fp8_static_c8.yaml

from angelslim.engine import Engine

slim_engine = Engine()
# Prepare model
slim_engine.prepare_model(model_name="Qwen", model_path="Qwen/Qwen3-1.7B",)
# Initialize compressor
slim_engine.prepare_compressor("PTQ", default_method="fp8_dynamic")
# Compress model
slim_engine.run()
# Save compressed model
slim_engine.save("./output")

For more details, please refer to the Quick Start Documentation .

2.3 Diffusion Model Quantization

Use the scripts/diffusion/run_diffusion.py for quantization and inference:

# Online quantization and inference
python scripts/diffusion/run_diffusion.py \
  --model-name-or-path black-forest-labs/FLUX.1-schnell \
  --quant-type fp8-per-tensor \
  --prompt "A cat holding a sign that says hello world" \
  --height 1024 --width 1024 --steps 4 --guidance 0.0 --seed 0

For more quantization inference methods, please refer to the Diffusion Model Quantization Documentation .

2.4 Token Compression (VLM)

AngelSlim provides a universal metadata-driven framework for vision token pruning and merging. You can quickly verify a compression strategy (e.g., VisionZip ) with a smoke test:

python tools/test_universal_pruning.py \
    --model_path "Qwen/Qwen2.5-VL-3B-Instruct" \
    --config "configs/qwen2_5_vl/pruning/visionzip_r0.9.yaml"

For more details on implementing new strategies, please refer to the Token Compressor Documentation .

3. Deployment and Testing

3.1 Offline Inference

To test offline inference with a quantized model loaded via transformers .

python scripts/deploy/offline.py $MODEL_PATH "Hello, my name is"

3.2 API Service Deployment

After specifying the quantized model path MODEL_PATH , you can deploy an OpenAI-compatible API service using vLLM and SGLang inference frameworks.

3.3 Service Invocation

Invoke requests via OpenAI's API format .

bash scripts/deploy/openai.sh -m $MODEL_PATH -p "Hello, my name is" --port 8080 --max-tokens 4096 --temperature 0.7 --top-p 0.8 --top-k 20 --repetition-penalty 1.05 --system-prompt "You are a helpful assistant."

3.4 Performance Evaluation

Evaluate the performance of quantized model using lm-evaluation-harness , recommended version lm-eval>=0.4.8 .

bash scripts/deploy/lm_eval.sh -d 0,1 -t 2 -g 0.8 -r $RESULT_PATH -b "auto" --tasks ceval-valid,mmlu,gsm8k,humaneval -n 0 $MODEL_PATH

For more detaileds, please refer to the Deployment Documentation .

📈 Benchmark

1. Speculative Decoding

We evaluated the Eagle3 model trained by AngelSlim on tasks including code generation, mathematical reasoning, instruction following, text generation, and multimodal understanding using vLLM. The inference acceleration and context length performance of our trained model under the settings of num_speculative_tokens = 2 or 4 are presented as follows, with an accept length of 1.8–3.5 and a maximum speedup of 1.4–1.9×.

1.1 Qwen3 Series Models

Benchmark results for Qwen3 series models using Eagle3 speculative decoding on vLLM (v0.11.2) across MT-bench , HumanEval , GSM8K and Alpaca , using a single GPU ( tp=1, ep=1, num_speculative_tokens=2, batch_size=1, output_len=1024 ).

1.2 VLM Models

1.2.1 Qwen3-VL Series Models

Benchmark results for Qwen3-VL series models using Eagle3 speculative decoding on vLLM (v0.12.0) across language and multimodal tasks, using a single GPU ( tp=1, ep=1, num_speculative_tokens=4, batch_size=1, output_len=1024 ).

1.2.2 HunyuanOCR Model

Benchmark results for HunyuanOCR using Eagle3 speculative decoding on vLLM (v0.13.0) across OmniDocBench dataset, using a single GPU ( tp=1, ep=1, num_speculative_tokens=4, batch_size=1, output_len=1024 ).

1.3 Audio Models

1.3.1 Qwen2-Audio Model

Benchmark results for Qwen2-Audio using Eagle3 speculative decoding on vLLM (v0.12.0) across LibriSpeech dataset, using a single GPU ( tp=1, ep=1, num_speculative_tokens=4, batch_size=1, output_len=1024 ).

1.3.2 Fun-CosyVoice3 Model

Benchmark results for Fun-CosyVoice3 using Eagle3 speculative decoding across LibriTTS dataset, using a single GPU ( tp=1, ep=1, num_speculative_tokens=4, batch_size=1, output_len=1024 ).

Adapted for Transformers backend inference, only displays accept length. vLLM speedup ~1.6×, estimated from baseline LLM speedup.

2. Quantization

The performance test results for selected models are shown below. For the complete benchmark, refer to the Benchmark documentation

2.1 Hunyuan Series Models

Benchmark results for the Hunyuan-Instruct model with FP8 , INT4-AWQ and INT4-GPTQ quantization algorithms on datasets including OlympiadBench , AIME 2024 and DROP :

2.2 Qwen3 Series Models

Benchmark results for Qwen3 series models with FP8-Static , FP8-Dynamic , INT4-GPTQ , and INT4-AWQ quantization algorithms on datasets including CEVAL , MMLU , GSM8K , and HUMANEVAL :

2.3 DeepSeek Series Models

Benchmark results for DeepSeek-R1-0528 series models with FP8-Block-Wise and W4A8-FP8 quantization algorithms on datasets including GPQA Diamond 、 AIME 2024 、 SimpleQA and LiveCodeBench ：

2.4 Qwen-VL Series Models

Qwen3-VL Benchmark

Benchmark results for Qwen3VL series models with BF16 、 FP8-Static and FP8-Dynamic quantization algorithms on datasets including MMMU_VAL 、 DocVQA_VAL and ChartQA_TEST ：

2.5 Qwen-Omni Series Models

Qwen3-Omni Text to Text Benchmark

Benchmark results for Qwen3-Omni series models in BF16, FP8-Static, and FP8-Dynamic on aime25, gpqa_diamond, and mmlu_redux are as follows:

2.6 Other Models

Other models such as GLM-4.6, Qwen2.5, and Seed-OSS have been evaluated on benchmarks like CEVAL , MMLU , and GSM8K using quantization strategies including FP8-Static , FP8-Dynamic , INT4-GPTQ , and INT4-AWQ .

3. Token Compression (VLM)

We evaluated various vision token compression strategies on the Qwen2.5-VL-3B-Instruct model across multiple multimodal benchmarks. You can replicate these results using the following command:

python tools/run_pruning_eval.py \
    --model_path "Qwen/Qwen2.5-VL-3B-Instruct" \
    --configs "configs/qwen2_5_vl/pruning/visionzip_r0.9.yaml" \
    --tasks "textvqa" \
    --output_dir "./results/visionzip_test"

📝 License

The code for this project is open-sourced under the License for AngelSlim .

🔗 Citation

@article{angelslim2026,
  title={AngelSlim: A more accessible, comprehensive, and efficient toolkit for large model compression},
  author={Hunyuan AI Infra Team},
  journal={arXiv preprint arXiv:2602.21233},
  year={2026}
}

💬 Technical Discussion

• AngelSlim is developed by the Tencent Hunyuan AI Infra team, with new features being iteratively updated. If you have any questions or suggestions, please submit them on GitHub Issues or join our WeChat discussion group .

• ⭐ Star this repo to follow our latest progress. And if you are interested in joining us for an internship or full-time position, send your resume to: [email protected] .