0xSero / deepseek-v4-flash-0731-spark

DeepSeek-V4-Flash-0731 Spark — REAP K216 EXL3 3.0 bpw

UNTESTED SINGLE-DGX-SPARK RECIPE. The checkpoint is structurally verified, but this exact K216 EXL3 artifact has not completed generation on one DGX Spark. The recipe below requires a patched ARM64 SparkInfer runtime with a TP1 loader for the published TP4 rank-sliced weights. Do not describe it as working until weight load, CUDA graph capture, KV allocation, natural generation, structured output, tool calling, and coding probes all pass.

This is a REAP-pruned 3.0 bpw EXL3 derivative of 0xSero/DeepSeek-V4-Flash-0731-EXL3-3.0bpw , itself derived from deepseek-ai/DeepSeek-V4-Flash-0731 . It is sized for memory-constrained research systems such as DGX Spark.

Size and pruning

• Original EXL3 weight files: 124,867,114,600 bytes
• This checkpoint: 106,816,685,560 bytes (99.48 GiB)
• Removed: 18,050,429,040 bytes (18.05 GB)
• Routed experts: 216 of 256 retained per MoE scope
• Router top-k: 6, unchanged
• Quantization: EXL3 3.0 bpw, MCG codebook, TP4 rank-sliced layout

REAP ranking selects the least-salient experts for removal. The K216 plan additionally makes the top 32 specialists from both tool_calling and agentic_tool_trajectory mandatory in every layer before filling remaining slots by global REAP saliency. Backbone and all three DSpark/MTP expert scopes are compacted consistently. Attention, shared experts, embeddings, output head, mHC, compressor, and indexer tensors are retained.

Observation provenance

• Dataset: 0xSero/deepseek-v4-flash-reap-observations-v2
• Snapshot: partial-v2-21289
• Rows used: 21,289
• All experts observed: true
• Structured categories: 2,253 agentic-tool rows and 4,096 tool-calling rows
• Keep-plan SHA-256: cc2bdaf189ebdaa98178e35abac4b1b336808c4f86b11e9b8eccde6cb3e44001
• Source EXL3 revision: dde26d81896abe613403ab3d6168e1195da4400e

The observation ranking was transferred from the closely aligned prior DeepSeek-V4-Flash revision. All 43 router matrices passed identity alignment; the weakest individual same-index cosine was 0.9460793734 and the three hash routing tables were bit-identical.

Validation boundary

The files, compact expert IDs, router rows, hash remapping, tensor index, byte counts, and SHA-256 manifest were structurally validated before publication. End-to-end generation for this K216 rank-sliced EXL3 artifact has not yet passed. Do not infer coding, structured-output, long-context, or quality parity from structural validation alone. A compatible DeepSeek-V4 EXL3 runtime must support 216 experts and the TP4 rank-sliced schema.

The original K160 source-format REAP checkpoint passed a bounded DGX Spark generation smoke, but that does not validate this separately transformed K216 EXL3 checkpoint.

What REAP, EXL3, Trellis, and SparkInfer each do

• REAP is the pruning decision. Router-weighted activation observations rank routed experts; this checkpoint retains 216/256 experts per MoE scope and remaps the router and hash tables. Structured-output and agentic specialists were made mandatory before global saliency filled the remaining slots.
• EXL3 is the weight-compression format. It stores retained routed experts near 3.0 bits per weight while carried DeepSeek FP8 tensors remain FP8. It does not choose experts and does not prove quality preservation.
• Trellis is the EXL3 low-bit reconstruction and execution scheme used by the fused W4A16 MoE path. Here TR3 means the K3 Trellis tier. Trellis is not pruning; it decodes packed expert weights during matrix multiplication.
• SparkInfer is the optimized CUDA-kernel layer used by the patched vLLM runtime. It must provide Trellis MoE and DeepSeek-V4 attention kernels compiled for the DGX Spark's GB10/SM121 GPU.

The pipeline is: REAP selects and compacts experts, EXL3/Trellis stores and executes those retained experts, and SparkInfer supplies the GPU kernels.

Candidate recipe: one DGX Spark with SparkInfer

This is a bring-up candidate, not a supported or measured recipe . The target is exactly one DGX Spark: one GB10/SM121 GPU, 128 GB unified memory, ARM64, and CUDA 13.

The published tensors use a TP4 rank-sliced storage schema. One Spark has one GPU, so the runtime must expose TP1 while its loader coalesces rank slices 0–3 into each local tensor. This can be a streaming TP1 repack or an in-loader operation, but this repository does not yet ship or validate that component. Setting --tensor-parallel-size 1 alone is not sufficient.

Before launch, the pinned ARM64 image must prove all of the following:

• DeepSeek-V4 hybrid EXL3 plus carried-FP8 loading is present;
• the TP1 loader reconstructs all TP4 projections without changing EXL3 codes, scales, expert IDs, router rows, or hash tables;
• exllamav3_ext and SparkInfer K3 Trellis kernels import and are compiled for sm_121a /SM121 rather than SM120-only or SM90-only code;
• n_routed_experts=216 is accepted in all 43 backbone and three DSpark/MTP expert scopes;
• expert parallel and MTP remain disabled for first acceptance; and
• regular full-and-piecewise CUDA graphs remain enabled.

Download the pinned weight revision on the Spark:

MODEL_REPO=0xSero/deepseek-v4-flash-0731-spark
MODEL_REVISION=22f28d32b9b29b4352eaa380ff8c2c170b2847ab
MODEL_DIR=/models/deepseek-v4-flash-0731-spark

hf download "$MODEL_REPO" \
  --revision "$MODEL_REVISION" \
  --local-dir "$MODEL_DIR" \
  --max-workers 8
hf cache verify "$MODEL_REPO" \
  --revision "$MODEL_REVISION" \
  --local-dir "$MODEL_DIR" \
  --fail-on-missing-files \
  --fail-on-extra-files

Use an immutable ARM64 SparkInfer image digest. The public Anemll 0.1.1 recipe in MiaAI-Lab PR #14 is useful GB10 evidence for sm_121a , checkpoint-owned DeepSeek-V4 encoding, chunked prefill, async scheduling, and regular CUDA graphs. It is a two-Spark FP8 recipe, however, and is not evidence that this one-Spark EXL3 command works.

After building and pinning an image containing the TP1 loader and SM121 SparkInfer dependencies above, start with this deliberately small profile:

export MODEL_DIR=/models/deepseek-v4-flash-0731-spark
export CACHE_DIR=/var/cache/deepseek-v4-flash-0731-spark
export IMAGE='<arm64-sparkinfer-image@sha256:...>'
mkdir -p "$CACHE_DIR"

docker run --rm --name deepseek-v4-flash-0731-spark \
  --gpus all \
  --network host \
  --ipc host \
  --shm-size 16g \
  --ulimit memlock=-1 \
  -v "$MODEL_DIR:/model:ro" \
  -v "$CACHE_DIR:/cache:rw" \
  -e CUDA_DEVICE_ORDER=PCI_BUS_ID \
  -e CUDA_VISIBLE_DEVICES=0 \
  -e CUTE_DSL_ARCH=sm_121a \
  -e VLLM_USE_BREAKABLE_CUDAGRAPH=0 \
  -e VLLM_EXL3_TRELLIS_MIN_M=1 \
  -e VLLM_EXL3_TRELLIS_MAX_M=32 \
  -e VLLM_EXL3_PREFILL_TRELLIS=1 \
  -e VLLM_CACHE_ROOT=/cache \
  "$IMAGE" \
  python -m vllm.entrypoints.cli.main serve /model \
    --served-model-name deepseek-v4-flash-0731-spark \
    --host 0.0.0.0 \
    --port 8000 \
    --tensor-parallel-size 1 \
    --pipeline-parallel-size 1 \
    --decode-context-parallel-size 1 \
    --no-enable-expert-parallel \
    --moe-backend auto \
    --quantization exl3 \
    --load-format safetensors \
    --attention-backend FLASHMLA_SPARSE_DSV4 \
    --kv-cache-dtype fp8 \
    --block-size 256 \
    --gpu-memory-utilization 0.88 \
    --max-model-len 8192 \
    --max-num-seqs 1 \
    --max-num-batched-tokens 1024 \
    --max-cudagraph-capture-size 1 \
    --cudagraph-capture-sizes 1 \
    --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE","custom_ops":["all"]}' \
    --enable-chunked-prefill \
    --async-scheduling \
    --enable-prefix-caching \
    --tokenizer-mode deepseek_v4 \
    --tool-call-parser deepseek_v4 \
    --reasoning-parser deepseek_v4 \
    --enable-auto-tool-choice \
    --generation-config vllm

This follows the live GLM-5.2 EXL3 recipe's acceptance shape: immutable image, read-only weights, writable compile cache, full-and-piecewise graphs, conservative first context, and post-graph KV evidence. That GLM recipe runs different weights on four RTX PRO 6000 GPUs, so it supplies method rather than compatibility or performance evidence for this checkpoint.

Do not raise context or concurrency until logs prove that the TP1 loader consumed all rank slices, weights loaded without an OOM, SparkInfer selected K3 Trellis on SM121, graph capture finished, and post-graph KV allocation exceeds 8,192 tokens with margin. Then require /health , /v1/models , a naturally completed deterministic generation, strict JSON Schema output, a forced tool call, multi-turn role separation, and a coding probe. If memory fails, first reduce context to 4,096 and batched tokens to 512; never substitute eager mode for the CUDA-graph acceptance gate.

Weight size alone is not runtime proof: 99.48 GiB of weights must share unified memory with TP1 assembly, CUDA kernels, graph workspaces, activations, and KV cache. For a separately validated single-Spark source-format fallback, see 0xSero/DeepSeek-V4-Flash-0731-REAP-K160 .

Credits

Thanks to DeepSeek for the base model, Cerebras Research for REAP, TurboDerp and contributors for ExLlamaV3/EXL3, and the DGX Spark community. The upstream model license and terms continue to apply.