AngelSlim / Qwen3-VL-4B-Instruct_eagle3

Dedicated to building a more intuitive, comprehensive, and efficient LLMs compression toolkit.

📖 Documentation |   🤗 Hugging Face |   🤖 ModelScope |   💬 WeChat

📣Latest News

• [26/01/13] We have released v0.3. We support the training and deployment of Eagle3 for all-scale LLMs/VLMs/Audio models, as detailed in the guidance documentation . And We released Sherry , the hardware-efficient 1.25 bit quantization algorithm [Paper Comming soon] | [Code] 🔥🔥🔥
• [25/11/05] We have released v0.2. Quantization support for new models, such as GLM-4.6 , Qwen3-VL and Qwen3-Omni , open-sources the Eagle3 speculative decoding training framework, and updates the Diffusion model quantization tools.
• [25/09/30] We have released SpecExit , the reasoning early-exit algorithm: [Paper] | [Docs] | [vLLM Code]
• [25/09/26] We have released TEQUILA , the ternary quantization algorithm [Paper] | [Code]
• [25/09/24] We now support the PTQ quantization of NVFP4 for the Qwen3 series models. We also opensource Qwen3-32B-NVFP4 and Qwen3-235B-A22B-NVFP4 weights.

🌟Key Features

• Highly Integrated : This toolkit integrates mainstream compression algorithms into a unified framework, offering developers one-click access with exceptional ease of use.
• Continuous Innovation : Beyond integrating widely-used industry algorithms, we are continuously researching better compression algorithms, which will be gradually open-sourced in the future.
• Performance-Driven : We continuously optimize end-to-end performance in model compression workflows and algorithm deployment, such as enabling quantization of models like Qwen3-235B and DeepSeek-R1 on a single GPU.

💼Technical Overview

🛎️How to Use

1. Install AngelSlim

We recommend using pip to install the latest stable version of AngelSlim :

pip install angelslim

Alternatively, you can clone the repository and install from source in editable mode:

cd AngelSlim && python setup.py install

For more detailed installation instructions, please refer to the Installation Documentation .

2. Quick Start

2.1 Speculative Decoding

After installing AngelSlim, you can quickly start Eagle3 training with the following scripts:

# Start the vLLM server
bash scripts/speculative/run_vllm_server.sh
# Generate training data
bash scripts/speculative/generate_data_for_target_model.sh
# Perform online training for the Eagle3 model
bash scripts/speculative/train_eagle3_online.sh

Training and Deployment Guide for Multimodal Model Eagle3—Supporting LLM, VLM, and Audio (ASR & TTS) Models: LLM | VLM | Audio(ASR) | Audio(TTS) .

2.2 LLM/VLM Model Quantization

After installing AngelSlim , you can launch static FP8 quantization for the Qwen3-1.7B model with the following one-command script:

python3 tools/run.py -c configs/qwen3/fp8_static/qwen3-1_7b_fp8_static.yaml

This example produces quantized model weights by performing PTQ calibration on a model loaded from HuggingFace.

from angelslim.engine import Engine

slim_engine = Engine()
# Prepare model
slim_engine.prepare_model(model_name="Qwen", model_path="Qwen/Qwen3-1.7B",)
# Initialize compressor
slim_engine.prepare_compressor("PTQ", default_method="fp8_dynamic")
# Compress model
slim_engine.run()
# Save compressed model
slim_engine.save("./output")

For more details, please refer to the Quick Start Documentation .

2.3 Diffusion Model Quantization

Use the scripts/diffusion/run_diffusion.py for quantization and inference:

# Online quantization and inference
python scripts/diffusion/run_diffusion.py \
  --model-name-or-path black-forest-labs/FLUX.1-schnell \
  --quant-type fp8-per-tensor \
  --prompt "A cat holding a sign that says hello world" \
  --height 1024 --width 1024 --steps 4 --guidance 0.0 --seed 0

For more quantization inference methods, please refer to the Diffusion Model Quantization Documentation .

3. Deployment and Testing

3.1 Offline Inference

To test offline inference with a quantized model loaded via transformers , run the following command:

python scripts/deploy/offline.py $MODEL_PATH "Hello, my name is"

Where MODEL_PATH is the path to the quantized model output.

3.2 API Service Deployment

After specifying the quantized model path MODEL_PATH , you can deploy an OpenAI-compatible API service using the following LLMs inference frameworks:

• vLLM

  Use the following script to launch a vLLM server, recommended version vllm>=0.8.5.post1 . For MOE INT8 quantized models, vllm>=0.9.0 is required.

  bash scripts/deploy/run_vllm.sh --model-path $MODEL_PATH --port 8080 -d 0,1,2,3 -t 4 -p 1 -g 0.8 --max-model-len 4096
  

  Where -d is the visible devices, -t is tensor parallel size, -p is pipeline parallel size, and -g is the GPU memory utilization.

• SGLang

  Use the following script to launch a SGLang server, recommended version sglang>=0.4.6.post1 .

  bash scripts/deploy/run_sglang.sh --model-path $MODEL_PATH --port 8080 -d 0,1,2,3 -t 4 -g 0.8
  

3.3 Service Invocation

Invoke requests via OpenAI's API format :

bash scripts/deploy/openai.sh -m $MODEL_PATH -p "Hello, my name is" --port 8080 --max-tokens 4096 --temperature 0.7 --top-p 0.8 --top-k 20 --repetition-penalty 1.05 --system-prompt "You are a helpful assistant."

where -p is the input prompt.

3.4 Performance Evaluation

Evaluate the performance of quantized model using lm-evaluation-harness , recommended version lm-eval>=0.4.8

bash scripts/deploy/lm_eval.sh -d 0,1 -t 2 -g 0.8 -r $RESULT_PATH -b "auto" --tasks ceval-valid,mmlu,gsm8k,humaneval -n 0 $MODEL_PATH

📈 Benchmark

1. Speculative Decoding

We evaluated the Eagle3 model trained by AngelSlim on tasks including code generation, mathematical reasoning, instruction following, text generation, and multimodal understanding using vLLM. The inference acceleration and context length performance of our trained model under the settings of num_speculative_tokens = 2 or 4 are presented as follows, with an accept length of 1.8–3.5 and a maximum speedup of 1.4–1.9×.

1.1 Qwen3 Series Models

Benchmark results for Qwen3 series models using Eagle3 speculative decoding on vLLM (v0.11.2) across MT-bench , HumanEval , GSM8K and Alpaca , using a single NVIDIA H20 GPU ( tp=1, ep=1, num_speculative_tokens=2, batch_size=1, output_len=1024 ).

1.2 VLM Models

1.2.1 Qwen3-VL Series Models

Benchmark results for Qwen3-VL series models using Eagle3 speculative decoding on vLLM (v0.12.0) across language and multimodal tasks, using a single NVIDIA H20 GPU ( tp=1, ep=1, num_speculative_tokens=4, batch_size=1, output_len=1024 ).

1.2.2 HunyuanOCR Model

Benchmark results for HunyuanOCR using Eagle3 speculative decoding on vLLM (v0.13.0) across OmniDocBench dataset, using a single NVIDIA H20 GPU ( tp=1, ep=1, num_speculative_tokens=4, batch_size=1, output_len=1024 ).

1.3 Audio Models

1.3.1 Qwen2-Audio Model

Benchmark results for Qwen2-Audio using Eagle3 speculative decoding on vLLM (v0.12.0) across LibriSpeech dataset, using a single NVIDIA H20 GPU ( tp=1, ep=1, num_speculative_tokens=4, batch_size=1, output_len=1024 ).

1.3.2 Fun-CosyVoice3 Model

Benchmark results for Fun-CosyVoice3 using Eagle3 speculative decoding across LibriTTS dataset, using a single NVIDIA H20 GPU ( tp=1, ep=1, num_speculative_tokens=4, batch_size=1, output_len=1024 ).

Adapted for Transformers backend inference, only displays accept length. vLLM speedup ~1.6×, estimated from baseline LLM speedup.

2. Quantization

The performance test results for selected models are shown below. For the complete benchmark, refer to the Benchmark documentation

2.1 Hunyuan Series Models

Benchmark results for the Hunyuan-Instruct model with FP8 , INT4-AWQ and INT4-GPTQ quantization algorithms on datasets including OlympiadBench , AIME 2024 and DROP :

2.2 Qwen3 Series Models

Benchmark results for Qwen3 series models with FP8-Static , FP8-Dynamic , INT4-GPTQ , and INT4-AWQ quantization algorithms on datasets including CEVAL , MMLU , GSM8K , and HUMANEVAL :

2.3 DeepSeek Series Models

Benchmark results for DeepSeek-R1-0528 series models with FP8-Block-Wise and W4A8-FP8 quantization algorithms on datasets including GPQA Diamond 、 AIME 2024 、 SimpleQA and LiveCodeBench ：

2.4 Qwen-VL Series Models

Qwen3-VL Benchmark

Benchmark results for Qwen3VL series models with BF16 、 FP8-Static and FP8-Dynamic quantization algorithms on datasets including MMMU_VAL 、 DocVQA_VAL and ChartQA_TEST ：

2.5 Qwen-Omni Series Models

Qwen3-Omni Text to Text Benchmark

Benchmark results for Qwen3-Omni series models in BF16, FP8-Static, and FP8-Dynamic on aime25, gpqa_diamond, and mmlu_redux are as follows:

2.6 Other Models

Other models such as GLM-4.6, Qwen2.5, and Seed-OSS have been evaluated on benchmarks like CEVAL , MMLU , and GSM8K using quantization strategies including FP8-Static , FP8-Dynamic , INT4-GPTQ , and INT4-AWQ .

📝 License

The code for this project is open-sourced under the License for AngelSlim .

🔗 Citation

@software{AngelSlim2025,
    title={{AngelSlim}},
    author={Tencent AngelSlim Project Contributors},
    year={2025},
    month={6},
    url={https://github.com/Tencent/AngelSlim},
}

💬 Technical Discussion

• AngelSlim is continuously iterating and new features will be released soon. If you have any questions or suggestions, please open an issue on GitHub Issues or join our WeChat discussion group .