rasyosef / Qwen3.5-2B-DSpark

Qwen3.5-2B-DSpark

A DSpark draft model for speculative decoding with Qwen/Qwen3.5-2B as the verifier, trained with speculators . The drafter proposes 8 tokens at a time and the verifier checks them in one forward pass, so output is identical to running the verifier alone — a lossless speedup. Mean acceptance length is 2.31 tokens committed per verification step, up to 3.16 on math_reasoning.

Training code: rasyosef/train-dspark-draft-models .

Trained on 50,000 samples.

Usage

vLLM loads the verifier automatically from the config — don't pass it separately.

vllm serve rasyosef/Qwen3.5-2B-DSpark --port 8000 --gpu-memory-utilization 0.8

Then query the OpenAI-compatible endpoint at http://localhost:8000/v1 .

Details

3 Qwen3 layers (hidden size 2048, intermediate size 6144, 8 attention heads over 2 KV heads, head dim 256, sliding-window attention with a 2048-token window), ~0.3B params, bfloat16. Block size 8, draft vocabulary reduced to 32,000 from the verifier's 248,320, aux hidden-state layers 2/8/16/22, confidence head with Markov (vanilla, rank 256).

Trained for 3 epochs at lr 3e-4 (AdamW, cosine schedule with 4% warmup) on 50,000 Open PerfectBlend prompts regenerated by the verifier itself at up to 1280 tokens, split 96/4 into train and validation, with a {"ce": 0.1, "tv": 0.9} loss. Prompts prepared at 1024 tokens; training sequence length 8192, up to 1024 anchors per sample. Verifier hidden states were pulled on demand from a running vLLM server during training and deleted after use rather than staged to disk up front.

Evaluation

evaluate.py throughput across the nine RedHatAI/speculator_benchmarks subsets. acceptance_length is mean tokens committed per verification step, including the bonus token — floor 1.0, ceiling 9.0 at block size 8.

Weighted across all subsets: 2.306 over 245,451 verification steps.

Acceptance is highest where the verifier's next token is most predictable, but the ordering is less clean than the usual code-and-math split. math_reasoning is the only subset above 3.0 and the only one still accepting more than one draft in twenty at pos_7 (6.1%); HumanEval sits well behind it at 2.56 and collapses after pos_3. Everything else lands in a narrow 1.7–2.2 band, with tool_call (2.11) offering no real advantage over open-ended writing (2.17) or retrieval-grounded answering (2.09) — the structure of a tool call is apparently not buying much predictability here. The bottom three (translation, qa, summarization) are close to the floor: they accept under 5% from pos_3 onward, so on that traffic the drafter is contributing roughly one extra token per step.

First-position acceptance spans only 40–67% across all nine subsets, and every subset except math_reasoning is below 10% by pos_4, so most of an 8-token block goes unused outside of math.

Limitations

Works only with Qwen3.5-2B and is not usable as a standalone model. The verifier is vision-language, but the drafter was trained on text-only prompts and was evaluated on text-only benchmarks, so acceptance on image inputs is untested. Acceptance falls off steeply past the first few positions on nearly all traffic, so a block size of 8 is mostly wasted outside math_reasoning — a shorter block would capture most of the same gain at lower drafting cost. Real-world speedup depends on your traffic mix, and because verification is lossless, the verifier's own behavior and biases carry through unchanged.

License

Apache 2.0, inherited from the verifier. The speculators training code is Apache-2.0.