rasyosef / Llama-3.2-1B-Instruct-speculator.dspark

Llama-3.2-1B-Instruct-speculator.dspark

A DSpark draft model for speculative decoding with unsloth/Llama-3.2-1B-Instruct as the verifier, trained with speculators . The drafter proposes 4 tokens at a time and the verifier checks them in one forward pass, so output is identical to running the verifier alone — a lossless speedup.

Training code: rasyosef/train-dspark-draft-models .

Trained on 5,000 samples as a pipeline demonstration, not a deployment-ready drafter.

Usage

vLLM loads the verifier automatically from the config — don't pass it separately.

vllm serve rasyosef/Llama-3.2-1B-Instruct-speculator.dspark --port 8000 --gpu-memory-utilization 0.75

Then query the OpenAI-compatible endpoint at http://localhost:8000/v1 .

Details

3 Qwen3 layers (hidden size 2048, intermediate size 8192, 32 attention heads over 8 KV heads, sliding-window attention with a 2048-token window), ~0.3B params, bfloat16. Block size 4, draft vocabulary reduced to 32,000, aux hidden-state layers 2/5/8/11/14, confidence head with Markov (rank 256).

Trained for 3 epochs at lr 4e-4 on 5,000 Magpie prompts regenerated by the verifier itself, with a {"ce": 0.1, "tv": 0.9} loss. Sequence length 2048, up to 384 anchors per sample. Offline mode: verifier hidden states were generated to disk by a vLLM server before training. speculators 0.8.0.dev207 , vLLM 0.28.0 , transformers 5.15.1 , torch 2.13.0 .

Evaluation

evaluate.py throughput across the nine RedHatAI/speculator_benchmarks subsets. acceptance_length is mean tokens committed per verification step, including the bonus token — floor 1.0, ceiling 5.0 at block size 4.

Weighted across all subsets: 1.776 over 120,771 verification steps. Acceptance is highest where the verifier's next token is most predictable — code and math — and lowest on translation and summarization. Note that math_reasoning has the best first-position acceptance of any subset but falls behind HumanEval overall, because code stays predictable deeper into the block while math drops off sharply after the first token.

Validation at the end of training: accept rate 0.441, acceptance length 2.069, full-block accuracy 0.477, per-position accuracy 0.572 / 0.495 / 0.444 / 0.396. The gap between these and the benchmark numbers is the usual in-distribution effect — validation is held-out Magpie, the benchmarks are not.

Limitations

Trained on only 5,000 samples, so acceptance would improve with more data. Works only with Llama-3.2-1B-Instruct and is not usable as a standalone model. Real-world speedup depends on your traffic mix, and because verification is lossless, the verifier's own behavior and biases carry through unchanged.

Related

yosefw/Qwen3-0.6B-DSpark — same pipeline with Qwen/Qwen3-0.6B as the verifier.

License

Llama 3.2 Community License, inherited from the verifier. The speculators training code is Apache-2.0.