EntityDeletr / gpt-oss-20b-DFlash-GGUF

Quantized version of z-lab/gpt-oss-20b-DFlash .

Works with mainline llama.cpp, will NOT work with forks.

Their model card is pasted as is below.

Files:

• model.safetensors - original unquantized safetensors
• model.gguf - unquantized bf16 GGUF
• gpt-oss-20b-DFlash.gguf - GGUF quantized to Q5_K_M

gpt-oss-20b-DFlash

Paper | GitHub | Blog

DFlash is a novel speculative decoding method that utilizes a lightweight block diffusion model for drafting. It enables efficient, high-quality parallel drafting that pushes the limits of inference speed.

This model serves as the drafter component and contains 0.8B parameters . It must be used in conjunction with the target model openai/gpt-oss-20b .

📊 Training Data

gpt-oss-20b-DFlash is trained on 800K samples , drawn from:

• nvidia/Nemotron-Post-Training-Dataset-v2
• theblackcat102/evol-codealpaca-v1

For all samples, the response portion was regenerated using the target model openai/gpt-oss-20b .

🚀 Quick Start

SGLang

Installation

uv pip install "git+https://github.com/sgl-project/sglang.git@refs/pull/20547/head#subdirectory=python"

Launch Server

# Optional: enable schedule overlapping (experimental, may not be stable)
# export SGLANG_ENABLE_SPEC_V2=1
# export SGLANG_ENABLE_DFLASH_SPEC_V2=1
# export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1

python -m sglang.launch_server \
    --model-path openai/gpt-oss-20b \
    --speculative-algorithm DFLASH \
    --speculative-draft-model-path z-lab/gpt-oss-20b-DFlash \
    --tp-size 1 \
    --dtype bfloat16 \
    --attention-backend fa3 \
    --mem-fraction-static 0.75 \
    --trust-remote-code

Usage

from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")

response = client.chat.completions.create(
    model="openai/gpt-oss-20b",
    messages=[{"role": "user", "content": "Write a quicksort in Python."}],
    max_tokens=2048,
    temperature=0.0,
)
print(response.choices[0].message.content)

vLLM

Installation

uv pip install vllm
uv pip install -U vllm --torch-backend=auto --extra-index-url https://wheels.vllm.ai/nightly

Launch Server

vllm serve openai/gpt-oss-20b \
  --speculative-config '{"method": "dflash", "model": "z-lab/gpt-oss-20b-DFlash", "num_speculative_tokens": 7}' \
  --attention-backend flash_attn \
  --max-num-batched-tokens 32768

Usage

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

response = client.chat.completions.create(
    model="openai/gpt-oss-20b",
    messages=[{"role": "user", "content": "Write a quicksort in Python."}],
    max_tokens=2048,
    temperature=0.0,
)
print(response.choices[0].message.content)

Evaluation

We use a block size of 8 (7 draft tokens) during speculation. DFlash consistently achieves high acceptance lengths and speedups across different concurrency levels. All experiments are conducted using SGLang on a single H200 GPU .

The numbers reported are end-to-end speedup (including prefill time). You can specify different block size during inference by passing --speculative-num-draft-tokens arguments when launch the server.

The reasoning effort is set to medium for all tasks. Low reasoning effort will give even higher acceptance length.

Acknowledgement

We are grateful to Yotta Labs for their compute support in training this draft model.

Citation

If you find DFlash useful for your research or applications, please cite our project.

@misc{chen2026dflash,
  title         = {DFlash: Block Diffusion for Flash Speculative Decoding},
  author        = {Chen, Jian and Liang, Yesheng and Liu, Zhijian},
  year          = {2026},
  eprint        = {2602.06036},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2602.06036}
}