rasyosef / Llama-3.2-1B-Instruct-DSpark

Llama-3.2-1B-Instruct-DSpark

A DSpark draft model for speculative decoding with unsloth/Llama-3.2-1B-Instruct as the verifier, trained with speculators . The drafter proposes 8 tokens at a time and the verifier checks them in one forward pass, so output is identical to running the verifier alone — a lossless speedup. Mean acceptance length is 3.15 tokens committed per verification step, up to 4.57 on HumanEval.

Training code: rasyosef/train-dspark-draft-models .

Trained on 100,000 samples.

Usage

vLLM loads the verifier automatically from the config — don't pass it separately.

vllm serve rasyosef/Llama-3.2-1B-Instruct-DSpark --port 8000 --gpu-memory-utilization 0.8

Then query the OpenAI-compatible endpoint at http://localhost:8000/v1 .

Details

3 Qwen3 layers (hidden size 2048, intermediate size 8192, 32 attention heads over 8 KV heads, sliding-window attention with a 2048-token window), ~0.3B params, bfloat16. Block size 8, draft vocabulary reduced to 32,000, aux hidden-state layers 2/8/14, confidence head with Markov (rank 256).

Trained for 3 epochs at lr 3e-4 (AdamW, cosine schedule with 4% warmup) on 100,000 Open PerfectBlend prompts regenerated by the verifier itself, split 96/4 into train and validation, with a {"ce": 0.1, "tv": 0.9} loss. Prompts prepared at 1024 tokens; training sequence length 8192, up to 1024 anchors per sample. Verifier hidden states were pulled on demand from a running vLLM server during training and deleted after use rather than staged to disk up front. speculators 0.8.0.dev207 , vLLM 0.28.0 , transformers 5.15.1 , torch 2.13.0 .

Evaluation

evaluate.py throughput across the nine RedHatAI/speculator_benchmarks subsets. acceptance_length is mean tokens committed per verification step, including the bonus token — floor 1.0, ceiling 9.0 at block size 8.

Weighted across all subsets: 3.148 over 89,102 verification steps.

Acceptance is still highest where the verifier's next token is most predictable — code, math, structured tool calls. HumanEval and math_reasoning are far ahead of everything else and hold their lead deep into the block: HumanEval's pos_4 (36.8%) is above summarization's pos_1 (31.0%), and both still accept better than one token in six at pos_7. The prose-like subsets cluster tightly at 2.0–2.6 and fall off sharply after pos_3, where the longer block buys little — translation is under 2% accepted from pos_4 onward.

Limitations

Works only with Llama-3.2-1B-Instruct and is not usable as a standalone model. Acceptance falls off steeply past the first few positions on prose-like traffic (summarization, qa, translation), so a block size of 8 is mostly wasted there — the gains concentrate in code, math, and tool calls. Real-world speedup depends on your traffic mix, and because verification is lossless, the verifier's own behavior and biases carry through unchanged.

License

Llama 3.2 Community License, inherited from the verifier. The speculators training code is Apache-2.0.