Total runs:
22.9K
Run Growth:
-3.1K
Growth Rate:
-13.69%
Sponsored by
APIMart
inference-optimization / Qwen3.5-397B-A17B-speculator.dflash-extra
Model's Last Updated: July 31 2026
Introduction of Qwen3.5-397B-A17B-speculator.dflash-extra
Model Details of Qwen3.5-397B-A17B-speculator.dflash-extra
RedHatAI/Qwen3.5-397B-A17B-speculator.dflash
This is a DFlash speculator model for Qwen/Qwen3.5-397B-A17B .
Training Details
This model was trained using the
Speculators
library on a subset of
Magpie-Align/Magpie-Llama-3.1-Pro-300K-Filtered
and the
train_sft
split of
HuggingFaceH4/ultrachat_200k
. Responses were regenerated by
RedHatAI/Qwen3.5-397B-A17B
and stored at
Qwen3.5-397B-A17B-responses
Commands
Using the Speculators library and the helper scripts provided in the repo.
Prepare data
# In virtual environment with speculators installed
python scripts/prepare_data.py \
--model Qwen/Qwen3.5-397B-A17B
--data ./regenerated_data.jsonl \
--assistant-pattern "<\|im_start\|>assistant\s*([\s\S]*?)<\|im_end\|>" \
--output ./output \
--seq-length 16384
Launch vLLM
# In (separate) virtual environment with [vLLM](https://github.com/imargulis/vllm/tree/fix/extract-hidden-states-hybrid-block-size) installed
CUDA_VISIBLE_DEVICES=0,1,2,3 vllm_venv/bin/python scripts/launch_vllm.py \
Qwen/Qwen3.5-397B-A17B \
--target-layer-ids 3 15 23 35 47 \
--max-model-len 32000 \
--reasoning-parser qwen3 \
--language-model-only \
--kv-cache-dtype fp8 \
--max-num-batched-tokens 32768\
--tensor-parallel-size 4 \
--async-scheduling
Launch training
Must be run once vLLM has finished launching and is running in the background.
# In virtual environment with speculators installed
CUDA_VISIBLE_DEVICES=4,5,6,7 torchrun \
--standalone \
--nproc_per_node 4 \
scripts/train.py \
--verifier-name-or-path Qwen/Qwen3.5-397B-A17B \
--data-path ./output \
--on-missing generate \
--on-generate delete \
--scheduler-type cosine \
--draft-vocab-size 24576 \
--max-anchors 3072 \
--target-layer-ids 3 15 23 35 47 \
--speculator-type dflash \
--logger trackio \
--lr 0.0006 \
--epochs 5 \
--draft-config 397_config.json
Model Specifications
| Base Model | Qwen/Qwen3.5-397B-A17B |
| Chat Template |
Qwen/Qwen3.5-397B-A17B (use
/chat/completions
endpoint)
|
| Format | Safetensors |
| License | Apache 2.0 |
| Validation Hardware | Nvidia H200 |
Deployment
# Install vLLM from the required PR
pip install git+https://github.com/vllm-project/vllm.git
# Deploy with speculative decoding
vllm serve Qwen/Qwen3.5-397B-A17B \
--tensor-parallel-size 8 \
--max-num-batched-tokens 32768 \
--attention-backend FLASH_ATTN \
--speculative-config '{
"model": "RedHatAI/Qwen3.5-397B-A17B-speculator.dflash",
"num_speculative_tokens": 7,
"method": "dflash"
}'
Acceptance Rates
Per-position token acceptance rates across datasets DFLASH:
(with reasoning enabled)
| Dataset | Pos 0 | Pos 1 | Pos 2 | Pos 3 | Pos 4 | Pos 5 | Pos 6 | Avg. Length |
|---|---|---|---|---|---|---|---|---|
| HumanEval | 83.9% | 68.6% | 56.6% | 46.9% | 39.1% | 32.4% | 26.2% | 4.54 |
| math_reasoning | 85.5% | 71.9% | 61.1% | 52.3% | 44.6% | 37.6% | 31.2% | 4.84 |
| qa | 69.8% | 47.3% | 33.1% | 24.5% | 18.8% | 14.0% | 10.1% | 3.18 |
| question | 73.6% | 52.0% | 37.6% | 28.6% | 22.7% | 18.0% | 13.9% | 3.46 |
| rag | 78.7% | 59.3% | 45.0% | 34.7% | 26.9% | 20.7% | 15.6% | 3.81 |
| summarization | 72.5% | 49.6% | 34.8% | 25.5% | 19.1% | 14.4% | 10.6% | 3.27 |
| tool_call | 73.9% | 52.2% | 36.7% | 26.2% | 18.8% | 13.3% | 9.7% | 3.31 |
| translation | 67.1% | 41.3% | 23.6% | 13.0% | 7.0% | 3.8% | 2.0% | 2.58 |
| writing | 73.6% | 52.0% | 37.6% | 28.6% | 22.6% | 18.0% | 13.9% | 3.46 |
Per-position token acceptance rates across datasets MTP:
(with reasoning enabled)
| Dataset | Pos 0 | Pos 1 | Pos 2 | Pos 3 | Pos 4 | Pos 5 | Pos 6 | Avg. Length |
|---|---|---|---|---|---|---|---|---|
| HumanEval | 93.4% | 84.8% | 75.7% | 67.0% | 59.3% | 52.1% | 45.9% | 5.78 |
| math_reasoning | 91.9% | 81.7% | 71.8% | 63.0% | 54.9% | 47.3% | 40.6% | 5.51 |
| qa | 83.5% | 64.5% | 49.4% | 38.7% | 30.6% | 24.0% | 18.7% | 4.09 |
| question | 84.7% | 66.6% | 52.0% | 41.3% | 33.4% | 27.2% | 22.1% | 4.27 |
| rag | 87.2% | 72.0% | 59.2% | 49.2% | 40.9% | 34.2% | 28.7% | 4.71 |
| summarization | 85.6% | 68.6% | 54.4% | 43.3% | 34.7% | 27.9% | 22.5% | 4.37 |
| tool_call | 88.1% | 73.0% | 59.6% | 48.5% | 39.9% | 32.9% | 27.0% | 4.69 |
| translation | 85.0% | 65.4% | 47.7% | 34.2% | 24.5% | 17.5% | 12.5% | 3.87 |
| writing | 84.7% | 66.6% | 52.0% | 41.2% | 33.3% | 27.2% | 22.1% | 4.27 |
Latency Speedup
Speedup comparisons of DFlash speculative decoding vs. baseline (no speculation) at varying request rates on Nvidia H200:
References
Paper : DFlash: Block Diffusion for Flash Speculative Decoding
Runs of inference-optimization Qwen3.5-397B-A17B-speculator.dflash-extra on huggingface.co
33
Total runs
0
24-hour runs
-1
3-day runs
0
7-day runs
18
30-day runs
More Information About Qwen3.5-397B-A17B-speculator.dflash-extra huggingface.co Model
Url of Qwen3.5-397B-A17B-speculator.dflash-extra
Qwen3.5-397B-A17B-speculator.dflash-extra huggingface.co Url
Provider of Qwen3.5-397B-A17B-speculator.dflash-extra huggingface.co
Other API from inference-optimization
Updated:June 16 2026
Total runs:
16.7K
Run Growth:
-1.9K
Growth Rate:
-11.51%
Total runs:
7.4K
Run Growth:
-380
Growth Rate:
-5.13%
Total runs:
5.1K
Run Growth:
-2.6K
Growth Rate:
-50.13%
Total runs:
4.5K
Run Growth:
-962
Growth Rate:
-21.30%
Total runs:
4.5K
Run Growth:
-867
Growth Rate:
-19.43%
Total runs:
3.8K
Run Growth:
-1.4K
Growth Rate:
-37.57%
Total runs:
3.4K
Run Growth:
-1.9K
Growth Rate:
-55.85%
Total runs:
2.6K
Run Growth:
1.3K
Growth Rate:
48.53%
Total runs:
1.7K
Run Growth:
1.6K
Growth Rate:
98.32%
Total runs:
1.3K
Run Growth:
201
Growth Rate:
15.38%
Total runs:
1.1K
Run Growth:
125
Growth Rate:
11.86%
Total runs:
995
Run Growth:
936
Growth Rate:
94.07%
Total runs:
959
Run Growth:
-56.7K
Growth Rate:
-5914.60%
Total runs:
702
Run Growth:
76
Growth Rate:
10.83%
Total runs:
603
Run Growth:
159
Growth Rate:
26.37%
Total runs:
546
Run Growth:
546
Growth Rate:
100.00%
Total runs:
528
Run Growth:
282
Growth Rate:
53.41%
Total runs:
501
Run Growth:
175
Growth Rate:
34.93%
Total runs:
443
Run Growth:
443
Growth Rate:
100.00%
Total runs:
330
Run Growth:
26
Growth Rate:
7.88%
Total runs:
312
Run Growth:
312
Growth Rate:
100.00%
Total runs:
287
Run Growth:
287
Growth Rate:
100.00%
Total runs:
285
Run Growth:
50
Growth Rate:
17.54%
Total runs:
245
Run Growth:
245
Growth Rate:
100.00%
Total runs:
244
Run Growth:
244
Growth Rate:
100.00%
Total runs:
240
Run Growth:
114
Growth Rate:
47.50%
Total runs:
199
Run Growth:
199
Growth Rate:
100.00%
Total runs:
186
Run Growth:
19
Growth Rate:
10.22%
Total runs:
182
Run Growth:
180
Growth Rate:
98.90%
Total runs:
168
Run Growth:
168
Growth Rate:
100.00%
Total runs:
152
Run Growth:
152
Growth Rate:
100.00%
Total runs:
151
Run Growth:
19
Growth Rate:
12.58%
Total runs:
150
Run Growth:
76
Growth Rate:
50.67%
Total runs:
144
Run Growth:
105
Growth Rate:
72.92%
Total runs:
144
Run Growth:
144
Growth Rate:
100.00%
Total runs:
139
Run Growth:
139
Growth Rate:
100.00%
Total runs:
126
Run Growth:
126
Growth Rate:
100.00%
Total runs:
123
Run Growth:
123
Growth Rate:
100.00%
Total runs:
120
Run Growth:
120
Growth Rate:
100.00%
Total runs:
115
Run Growth:
4
Growth Rate:
3.48%
Total runs:
110
Run Growth:
110
Growth Rate:
100.00%
Total runs:
109
Run Growth:
91
Growth Rate:
83.49%
Total runs:
109
Run Growth:
91
Growth Rate:
83.49%
Total runs:
108
Run Growth:
25
Growth Rate:
23.15%
Total runs:
106
Run Growth:
106
Growth Rate:
100.00%
Total runs:
99
Run Growth:
99
Growth Rate:
100.00%
Total runs:
98
Run Growth:
49
Growth Rate:
50.00%
Total runs:
94
Run Growth:
18
Growth Rate:
19.15%
Total runs:
94
Run Growth:
75
Growth Rate:
79.79%
Total runs:
93
Run Growth:
93
Growth Rate:
100.00%
Total runs:
90
Run Growth:
52
Growth Rate:
57.78%
Total runs:
89
Run Growth:
89
Growth Rate:
100.00%
Total runs:
88
Run Growth:
88
Growth Rate:
100.00%
Total runs:
84
Run Growth:
60
Growth Rate:
71.43%
Total runs:
84
Run Growth:
84
Growth Rate:
100.00%
Total runs:
80
Run Growth:
80
Growth Rate:
100.00%
Total runs:
80
Run Growth:
25
Growth Rate:
31.25%
Total runs:
79
Run Growth:
39
Growth Rate:
49.37%
Total runs:
72
Run Growth:
72
Growth Rate:
100.00%
Total runs:
72
Run Growth:
46
Growth Rate:
63.89%
Total runs:
71
Run Growth:
52
Growth Rate:
73.24%
Total runs:
69
Run Growth:
37
Growth Rate:
53.62%
Total runs:
69
Run Growth:
69
Growth Rate:
100.00%
Total runs:
68
Run Growth:
68
Growth Rate:
100.00%
Total runs:
68
Run Growth:
6
Growth Rate:
8.82%
Total runs:
67
Run Growth:
67
Growth Rate:
100.00%
Total runs:
66
Run Growth:
66
Growth Rate:
100.00%
Total runs:
64
Run Growth:
30
Growth Rate:
46.88%
Total runs:
64
Run Growth:
46
Growth Rate:
71.88%
Total runs:
64
Run Growth:
39
Growth Rate:
60.94%
Total runs:
63
Run Growth:
32
Growth Rate:
50.79%
Total runs:
61
Run Growth:
-13
Growth Rate:
-21.31%
Total runs:
61
Run Growth:
25
Growth Rate:
40.98%
Total runs:
60
Run Growth:
60
Growth Rate:
100.00%
Total runs:
58
Run Growth:
58
Growth Rate:
100.00%
Total runs:
56
Run Growth:
56
Growth Rate:
100.00%
Total runs:
56
Run Growth:
27
Growth Rate:
48.21%
Total runs:
56
Run Growth:
31
Growth Rate:
55.36%
Total runs:
55
Run Growth:
31
Growth Rate:
56.36%
Total runs:
54
Run Growth:
0
Growth Rate:
0.00%
Total runs:
53
Run Growth:
27
Growth Rate:
50.94%
Total runs:
53
Run Growth:
53
Growth Rate:
100.00%
Total runs:
52
Run Growth:
2
Growth Rate:
3.85%
Total runs:
51
Run Growth:
43
Growth Rate:
84.31%
Total runs:
46
Run Growth:
46
Growth Rate:
100.00%
Total runs:
45
Run Growth:
17
Growth Rate:
37.78%
Total runs:
45
Run Growth:
39
Growth Rate:
86.67%
Total runs:
44
Run Growth:
16
Growth Rate:
36.36%
Total runs:
43
Run Growth:
43
Growth Rate:
100.00%
Total runs:
42
Run Growth:
27
Growth Rate:
64.29%
Total runs:
40
Run Growth:
-15
Growth Rate:
-37.50%
Total runs:
38
Run Growth:
38
Growth Rate:
100.00%