rasyosef / Qwen3.5-2B-DFlash2

Qwen3.5-2B-DFlash2

A DFlash2 draft model for speculative decoding with Qwen/Qwen3.5-2B as the verifier, trained with speculators . With a block size of 8, the drafter proposes 7 tokens at a time and the verifier checks them in one forward pass, so output is identical to running the verifier alone — a lossless speedup. Mean acceptance length is 2.38 tokens committed per verification step, up to 3.44 on math_reasoning.

Training code: rasyosef/train-dspark-draft-models .

Trained on 100,000 samples. This is the epoch 2 checkpoint of a 4-epoch run.

Usage

vLLM loads the verifier automatically from the config — don't pass it separately.

vllm serve rasyosef/Qwen3.5-2B-DFlash2 --port 8000 --gpu-memory-utilization 0.8

Then query the OpenAI-compatible endpoint at http://localhost:8000/v1 .

Details

4 Qwen3 layers (hidden size 2048, intermediate size 6144, 16 attention heads over 4 KV heads, head dim 128), with the first three layers using sliding-window attention (2048-token window) and the last using full attention. Block size 8 (7 draft tokens per step), aux hidden-state layers 1/5/9/13/17/21, bfloat16.

Training was configured for 4 epochs; this checkpoint was saved after epoch 2. Trained at lr 4e-4 (AdamW, cosine schedule with 4% warmup) on 100,000 filtered Open PerfectBlend prompts with responses regenerated by the verifier itself, split 96/4 into train and validation, with a {"ce": 0.1, "tv": 0.9} loss. Samples prepared at 1024 tokens; training sequence length 4096, up to 512 anchors per sample. Verifier hidden states were pulled on demand from a running vLLM server during training and deleted after use rather than staged to disk up front.

Evaluation

evaluate.py throughput across the nine RedHatAI/speculator_benchmarks subsets. acceptance_length is mean tokens committed per verification step, including the bonus token — floor 1.0, ceiling 8.0 at block size 8 (7 draft tokens).

Weighted across all subsets: 2.379 over 229,381 verification steps.

Acceptance is highest where the verifier's next token is most predictable. math_reasoning is the only subset above 3.0 and still accepts more than one draft in seven at pos_6 (14.4%); HumanEval follows at 2.64 and stays above 10% through pos_4. The next four — tool_call, rag, question and writing — sit in a tight 2.16–2.21 band, with tool_call (2.21) only marginally ahead of open-ended writing (2.16), so the structure of a tool call is apparently not buying much predictability here. qa (1.88) and translation (1.86) come next; translation actually starts above the middle group at pos_0 (49.8%) but collapses after pos_1, falling to 8.0% at pos_2. It is also the smallest subset (1,169 steps), so treat its number as noisy. summarization (1.80) sits closest to the floor, accepting under 10% from pos_2 onward, so on that traffic the drafter contributes about four-fifths of an extra token per step.

First-position acceptance spans 41–70% across all nine subsets, and only math_reasoning and HumanEval remain above 10% at pos_4, so most of a 7-token draft goes unused outside math and code.

Limitations

Works only with Qwen3.5-2B and is not usable as a standalone model. The verifier is vision-language, but the drafter was trained on text-only prompts and was evaluated on text-only benchmarks, so acceptance on image inputs is untested. Acceptance falls off steeply past the first few positions on most traffic, so much of each 7-token draft is wasted outside math_reasoning and code — a shorter block would capture most of the same gain at lower drafting cost. On qa, translation and summarization, acceptance length is below 1.9, so gains there are likely smaller. Throughput has not been benchmarked for this checkpoint; real-world speedup depends on your hardware and traffic mix, and because verification is lossless, the verifier's own behavior and biases carry through unchanged.

License

Apache 2.0, inherited from the verifier. The speculators training code is Apache-2.0.