rasyosef / Llama-3.2-3B-Instruct-DSpark

Llama-3.2-3B-Instruct-DSpark

A DSpark draft model for speculative decoding with unsloth/Llama-3.2-3B-Instruct as the verifier, trained with speculators . The drafter proposes 8 tokens at a time and the verifier checks them in one forward pass, so output is identical to running the verifier alone — a lossless speedup. Mean acceptance length is 3.27 tokens committed per verification step, up to 4.92 on math_reasoning.

Training code: rasyosef/train-dspark-draft-models .

Trained on 100,000 samples.

Usage

vLLM loads the verifier automatically from the config — don't pass it separately.

vllm serve rasyosef/Llama-3.2-3B-Instruct-DSpark --port 8000 --gpu-memory-utilization 0.8

Then query the OpenAI-compatible endpoint at http://localhost:8000/v1 .

Details

5 Qwen3 layers (hidden size 3072, intermediate size 8192, 24 attention heads over 8 KV heads, sliding-window attention with a 2048-token window), ~0.9B params, bfloat16. Block size 8, draft vocabulary reduced to 50,000, aux hidden-state layers 2/8/14/20/26, confidence head with Markov (rank 256).

Trained for 3 epochs at lr 3e-4 (AdamW, cosine schedule with 4% warmup) on 100,000 Open PerfectBlend prompts regenerated by the verifier itself, split 96/4 into train and validation, with a {"ce": 0.1, "tv": 0.9} loss. Prompts prepared at 1024 tokens; training sequence length 8192, up to 1024 anchors per sample. Verifier hidden states were pulled on demand from a running vLLM server during training and deleted after use rather than staged to disk up front. speculators 0.8.0.dev207 , vLLM 0.28.0 , transformers 5.15.1 , torch 2.13.0 .

Evaluation

evaluate.py throughput across the nine RedHatAI/speculator_benchmarks subsets. acceptance_length is mean tokens committed per verification step, including the bonus token — floor 1.0, ceiling 9.0 at block size 8.

Weighted across all subsets: 3.269 over 73,086 verification steps.

Acceptance is still highest where the verifier's next token is most predictable — math, code, structured tool calls. math_reasoning and HumanEval are nearly indistinguishable and hold their lead all the way through the block: math_reasoning's pos_4 (40.8%) is above summarization's pos_1 (39.3%), and both still accept about one token in five at pos_7. tool_call sits on its own at 3.34, roughly halfway between the two groups.

The prose-like subsets all start in a narrow band at pos_0 (57–66%) and separate only deeper into the block. question and writing decay gently and are still above 4% at pos_7, while summarization, translation, and qa are under 1.5% there — so the tail of the block is worth something on open-ended generation but essentially nothing on summarization and qa.

Limitations

Works only with Llama-3.2-3B-Instruct and is not usable as a standalone model. Acceptance falls off steeply past the first few positions on prose-like traffic (summarization, translation, qa), so a block size of 8 is mostly wasted there — the gains concentrate in math, code, and tool calls. Real-world speedup depends on your traffic mix, and because verification is lossless, the verifier's own behavior and biases carry through unchanged.

License

Llama 3.2 Community License, inherited from the verifier. The speculators training code is Apache-2.0.