RedHatAI / GLM-5.2-speculator.dspark

GLM-5.2 DSpark speculator

Overview

A DSpark speculator model for the zai-org/GLM-5.2-FP8 base model, enabling faster inference through speculative decoding. DSpark extends the DFlash parallel draft backbone with two lightweight heads: a Markov logit-bias head (low-rank intra-block token dependency) and a per-position confidence head (accept-rate prediction). Trained with the speculators library.

Model Specifications

• Base Model : zai-org/GLM-5.2-FP8
• Chat Template : GLM-5.2 (compatible with /chat/completions )
• Format : Safetensors
• License : MIT
• Draft : 5 layers, block_size=8 , full vocabulary (154,880), aux layers [8, 23, 39, 55, 70]
• Validation Hardware : NVIDIA B300

Checkpoint series

This repo publishes per-epoch checkpoints of a single 3-epoch run. main tracks the latest available epoch; each epoch is also a permanent revision.

from transformers import AutoModel
model = AutoModel.from_pretrained(
    "mgoin/GLM-5.2-speculator.dspark", revision="epoch-2", trust_remote_code=True
)

Training Details

The model was trained using the Speculators library on prompts from Magpie-Align/Magpie-Llama-3.1-Pro-300K-Filtered and HuggingFaceH4/ultrachat_200k , with responses regenerated by GLM-5.2-FP8 itself (published as mgoin/GLM-5.2-FP8-magpie-ultrachat ).

Training is online : the draft consumes hidden states streamed on-the-fly from a live GLM-5.2-FP8 vLLM server, with the trainer running FSDP data-parallel on separate GPUs. The three commands below (data prep → server → trainer) reproduce the run. Install speculators and vLLM from main. GPU indices/parallelism are examples — adjust to your hardware.

Data Preparation

python scripts/prepare_data.py \
  --model zai-org/GLM-5.2-FP8 \
  --trust-remote-code \
  --data ./regenerated_data.jsonl \
  --output ./output \
  --seq-length 8192 \
  --assistant-pattern '<\|assistant\|>((?:(?!<\|user\|>|<\|assistant\|>).)*)'

--assistant-pattern is currently needed for GLM-5.2's inline-reasoning chat format (the <think>...</think> trace is kept inside the assistant turn); it may be auto-detected by future speculators versions.

vLLM Server Launch (hidden-states server)

CUDA_VISIBLE_DEVICES=0,1,2,3 python scripts/launch_vllm.py \
  zai-org/GLM-5.2-FP8 \
  --target-layer-ids 8 23 39 55 70 \
  -- --port 8000 \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.9 \
  --max-model-len 8192 \
  --trust-remote-code

Training Command

CUDA_VISIBLE_DEVICES=4,5,6,7 torchrun \
  --standalone \
  --nproc_per_node 4 \
  scripts/train.py \
  --verifier-name-or-path zai-org/GLM-5.2-FP8 \
  --speculator-type dspark \
  --num-layers 5 \
  --block-size 8 \
  --data-path ./output \
  --vllm-endpoint http://localhost:8000/v1 \
  --save-path ./output/checkpoints \
  --epochs 3 \
  --lr 0.0006 \
  --scheduler-type cosine \
  --total-seq-len 4096 \
  --draft-arch qwen3 \
  --draft-hidden-act silu \
  --target-layer-ids 8 23 39 55 70 \
  --max-anchors 1024 \
  --markov-rank 256 \
  --enable-confidence-head \
  --confidence-head-with-markov \
  --loss-fn '{"ce": 0.1, "tv": 0.9}' \
  --confidence-head-alpha 1.0 \
  --checkpoint-freq 0.2 \
  --on-missing generate \
  --on-generate delete \
  --seed 42 \
  --log-freq 100 \
  --prefetch-factor 2 \
  --num-workers 8 \
  --trust-remote-code

Notes:

• Omitting --draft-vocab-size trains on the full vocabulary ; pass --draft-vocab-size 32000 for a reduced draft vocab.
• DSpark-specific flags: --markov-rank , --enable-confidence-head , --confidence-head-with-markov , --confidence-head-alpha . Dropping them (and using --speculator-type dflash ) recovers a plain DFlash draft.
• --checkpoint-freq 0.2 writes sub-epoch checkpoints (~5 per epoch) for resumability.

Deployment

DSpark inference support is available on the vLLM nightly ( uv pip install vllm --extra-index-url https://wheels.vllm.ai/nightly ):

vllm serve zai-org/GLM-5.2-FP8 \
    --tensor-parallel-size 4 \
    --max-model-len 16384 \
    --trust-remote-code \
    --speculative-config '{
        "model": "RedHatAI/GLM-5.2-speculator.dspark",
        "num_speculative_tokens": 7,
        "method": "dspark",
        "draft_sample_method": "probabilistic"
    }'

Evaluation Results

Train-set metrics at the end of epoch 2 (per-epoch validation passes did not complete due to server restarts; per-dataset acceptance evaluation will accompany the final checkpoint):

Per-position accuracy (positions 1-7), epoch 2: 0.809 / 0.693 / 0.619 / 0.549 / 0.515 / 0.478 / 0.445

Still improving epoch-over-epoch (epoch 3 in training).

Acceptance length in vLLM (revision epoch-2 )

Measured end-to-end in vLLM speculative decoding (nightly 0.23.1rc1.dev709+g2b753ad20 ), serving zai-org/GLM-5.2-FP8 on 4xB300 with num_speculative_tokens=7 , draft_sample_method="probabilistic" , greedy sampling, batch size 1, 64 single-turn chat prompts (32 HumanEval + 32 math_reasoning from RedHatAI/speculator_benchmarks ), 1024 output tokens each. Acceptance is computed from vLLM's spec_decode_num_accepted_tokens_per_pos / num_drafts counter deltas. The earlier GLM-5.2-speculator.dspark-preview is included for reference (measured on a 16-prompt subset of the same set).

This table is updated as later epochs are published; main currently points to epoch-2 .

The epoch-2 gain comes from deeper draft positions (position-1 acceptance is unchanged), consistent with continued training. For reference, the same server without speculative decoding decodes at 102 tok/s (2.2x speedup for epoch-2 ).

References

• DFlash : Block Diffusion for Flash Speculative Decoding (arXiv:2602.06036) — the parallel draft backbone DSpark builds on.
• DSpark (DeepSeek) — the Markov + confidence-head additions replicated here.
• speculators — training library.

AI assistance was used to build the training pipeline and run these experiments.