rasyosef / Qwen3.5-2B-DSpark-v2

Qwen3.5-2B-DSpark-v2

A DSpark draft model for speculative decoding with Qwen/Qwen3.5-2B as the verifier, trained with speculators . The drafter proposes 8 tokens at a time and the verifier checks them in one forward pass, so greedy output is identical to running the verifier alone — a lossless speedup. At temperature 0, mean acceptance length is 3.77 tokens committed per verification step, up to 5.49 on math_reasoning. Compared with the verifier alone, throughput is 2.25× higher on average, up to 2.98× on math_reasoning and 2.87× on HumanEval, and it is the fastest configuration on all nine benchmark subsets, ahead of both the verifier's native MTP drafter and v1 .

This is a smaller and faster successor to rasyosef/Qwen3.5-2B-DSpark : 4 layers instead of 5, about 0.3B parameters instead of 0.5B, trained on more data. It accepts slightly fewer tokens per step than v1 and is faster anyway.

Training code: rasyosef/train-dspark-draft-models .

Trained on 180,000 samples.

Usage

vLLM

vLLM loads the verifier automatically from the config — don't pass it separately.

vllm serve rasyosef/Qwen3.5-2B-DSpark-v2 --port 8000 \
  --gpu-memory-utilization 0.8 \
  --override-generation-config '{"temperature": 0}'

Then query the OpenAI-compatible endpoint at http://localhost:8000/v1 .

To serve with the sampling settings used in the evaluation below, swap the last flag:

  --override-generation-config '{"temperature":1.0,"top_p":0.95,"top_k":20}'

Details

4 Qwen3 layers (hidden size 2048, intermediate size 6144, 16 attention heads over 4 KV heads, head dim 128, all layers sliding-window attention with a 2048-token window), ~0.3B params, bfloat16. Block size 8, draft vocabulary reduced to 32,000 from the verifier's 248,320 (selected by token frequency over the training data), aux hidden-state layers 2/8/16/22, confidence head with Markov (vanilla, rank 256).

Trained for 3 epochs at lr 3e-4 (AdamW, cosine schedule with 4% warmup) on 180,000 filtered Open PerfectBlend prompts with responses regenerated by the verifier itself, split 96/4 into train and validation, with a {"ce": 0.1, "tv": 0.9} loss. Samples prepared at 2048 tokens; training sequence length 8192, up to 1024 anchors per sample. Verifier hidden states were pulled on demand from a running vLLM server during training and deleted after use rather than staged to disk up front.

Changes from v1

Hidden size, intermediate size, attention heads, block size, loss and the rest of the training setup are unchanged.

Evaluation

Results are from evaluate.py throughput across the nine RedHatAI/speculator_benchmarks subsets, with acceptance measured at two settings: temperature 0 (greedy decoding), and sampling with {"temperature":1.0,"top_p":0.95,"top_k":20} . Throughput is reported at temperature 0 only. acceptance_length is mean tokens committed per verification step, including the bonus token — floor 1.0, ceiling 9.0 at block size 8. Rows are in the same order in every table, sorted by acceptance length at temperature 0.

Acceptance at temperature 0

Weighted across all subsets: 3.772 over 118,426 verification steps. math_reasoning and HumanEval are highest, and with qa they are the three subsets that still accept more than 15% of drafts at pos_7. translation and summarization are lowest and fall below 1% by pos_7.

Acceptance with sampling

Same benchmark with {"temperature":1.0,"top_p":0.95,"top_k":20} .

Weighted across all subsets: 2.868 over 173,922 verification steps, against 3.772 at temperature 0. Acceptance length is lower on every subset. The drop is largest on qa (3.675 to 2.239) and HumanEval (4.693 to 3.352), and smallest on translation (2.516 to 2.406). math_reasoning stays highest at 4.265. Acceptance at pos_7 is under 5% on every subset except math_reasoning (18.5%) and HumanEval (8.1%).

Throughput at temperature 0 (tokens/s)

Mean output throughput with a single concurrent request. Baseline is Qwen3.5-2B with no speculative decoding, and MTP is the verifier's native MTP drafter at 4 and 8 draft tokens; the baseline, MTP and v1 columns are the figures reported on the v1 card , measured on one A100. The fastest configuration in each row is bolded. The last column gives this model's throughput divided by baseline throughput for each subset.

This model has the highest average speedup at 2.25× (unweighted mean of per-subset speedups), ahead of v1 (2.00×), MTP at 4 draft tokens (1.57×) and MTP at 8 (1.31×), and it has the highest mean throughput on all nine subsets. The gains are largest on math_reasoning (2.98×), HumanEval (2.87×) and tool_call (2.55×), and smallest on summarization (1.60×) and translation (1.28×).

Against v1, mean throughput is higher on every subset, by 2% on tool_call and 4% on math_reasoning up to 29% on writing and 31% on question. Translation, where v1 trailed MTP at 4 draft tokens, is now ahead of it on mean throughput (215.6 against 196.6).

Median throughput is close to the mean on math_reasoning (683.4) and HumanEval (666.9). It is well below the mean on qa (423.6 against 563.7) and on translation (90.6 against 215.6); translation is the smallest subset, with fewer than 1,000 verification steps.

Acceptance comparison with MTP and v1

Acceptance length at temperature 0 for this model, v1 and the verifier's native MTP drafter on the same subsets. MTP and v1 figures are from the v1 card. The highest value in each row is bolded and named in the last column. At 4 draft tokens the ceiling on acceptance_length is 5.0 rather than 9.0.

MTP at 8 draft tokens has the highest acceptance length on seven of the nine subsets and on the weighted average. This model is highest on math_reasoning (5.488) and writing (3.332). MTP's lead over it is widest on rag (4.844 against 3.857) and narrowest on HumanEval (4.862 against 4.693). Against MTP at 4 draft tokens, this model is higher on six subsets and lower on question, summarization and translation.

Against v1, acceptance length is lower on seven subsets, most on qa (4.074 to 3.675), and higher on math_reasoning and writing. The weighted average goes from 3.892 to 3.772.

This model is faster than both v1 and MTP at 8 draft tokens on every subset while accepting fewer tokens per step than each of them on seven, which points to a cheaper draft-and-verify step rather than better drafts as the source of the gain.

Limitations

Works only with Qwen3.5-2B and is not usable as a standalone model. The verifier is vision-language, but the drafter was trained on text-only prompts and was evaluated on text-only benchmarks, so acceptance on image inputs is untested. All results are for a single concurrent request; throughput under load is not reported. With sampling at temperature 1.0, weighted acceptance length falls from 3.772 to 2.868; throughput with sampling is not reported. The verifier's native MTP drafter at 8 draft tokens has a higher acceptance length on seven of the nine subsets, though lower throughput on all nine, and v1 has a higher acceptance length on seven. Acceptance at the last draft position at temperature 0 is under 1% on summarization and translation and under 10% on rag, tool_call, writing and question, so a block size of 8 is partly wasted there; with sampling it is under 5% on seven subsets. The throughput mean on translation rests on a small sample and is far above its median. Real-world speedup depends on your traffic mix, and because verification is lossless, the verifier's own behavior and biases carry through unchanged.

License

Apache 2.0, inherited from the verifier. The speculators training code is Apache-2.0.