cloudyu / DeepSeek-V4-Flash-4Expert-GGUF

DeepSeek V4 Flash 4Expert — Q4_K GGUF

4-bit quantized GGUF of the 4Expert variant of DeepSeek V4 Flash, for use with ds4 .

Model Summary

Independent Evaluation Results

We evaluated the model against the original top_k=6 configuration on HumanEval (code generation)

HumanEval (Pass@1)

## eval details

GGUF Evaluation Report — 4Expert Q4_K GGUF BY ds4-eval

Model: cloudyu/DeepSeek-V4-Flash-4Expert-GGUF Source safetensors: cloudyu/DeepSeek-V4-Flash-4Expert Date: 2026-06-29

Summary

Quantization Strategy

Compiled with deepseek4-quantize using a layer-specific policy:

How to Use

Requires ds4 built from the 4Expert PR . The upstream ds4 defaults to 6 active experts and cannot load this GGUF. The PR is submitted upstream; until merged, use the branch:

git clone https://github.com/yuhai-china/ds4
cd ds4
git checkout 4expert
make cpu -j$(nproc)            # Linux
make -C gguf-tools -j$(nproc)

Then run:

ln -sfn DeepSeek-V4-Flash-4Expert-Q4K.gguf ds4flash.gguf
./ds4 -p "The weather is great today" -n 100

Reproduce: Convert Safetensors to This GGUF

This GGUF was produced by the following pipeline. Anyone with the source safetensors can reproduce it.

One-Click Script

git clone https://github.com/yuhai-china/ds4
cd ds4
git checkout 4expert
bash test-4expert.sh /path/to/DeepSeek-V4-Flash-4Expert $(nproc)

This runs all 5 steps (clone, build, download, convert, test) in one go.

Manual Steps

For transparency, here is exactly how this GGUF was produced.

Step 1 — Download source safetensors

pip install huggingface_hub
python3 -c "
from huggingface_hub import snapshot_download
snapshot_download('cloudyu/DeepSeek-V4-Flash-4Expert', local_dir='./DeepSeek-V4-Flash-4Expert')
"

Step 2 — Build ds4 and gguf-tools

git clone https://github.com/yuhai-china/ds4
cd ds4
git checkout 4expert
make -C gguf-tools -j$(nproc)
make cpu -j$(nproc)

Step 3 — Generate GGUF template from safetensors metadata

python3 gguf-tools/gen_gguf_template.py \
  --hf ./DeepSeek-V4-Flash-4Expert \
  --out template.gguf

The template (~5.6 MB) contains metadata, tokenizer, and tensor descriptors (names, shapes, types) but no weight data . It describes where each tensor goes in the final GGUF.

Step 4 — Quantize weights into the final GGUF

./gguf-tools/deepseek4-quantize \
  --hf ./DeepSeek-V4-Flash-4Expert \
  --template template.gguf \
  --out DeepSeek-V4-Flash-4Expert-Q4K.gguf \
  --experts q4_k \
  --attention-proj q8_0 \
  --attention f16 \
  --shared q8_0 \
  --output q8_0 \
  --embedding f16 \
  --dense f16 \
  --threads $(nproc)

The quantizer reads each safetensors tensor, dequantizes from the storage format (F8_E4M3 or packed FP4 with E8M0 scales for experts, BF16/F32 for others), applies the target quantization, and writes to the output GGUF. Output is ~153 GiB.

Step 5 — Test the GGUF

ln -sfn DeepSeek-V4-Flash-4Expert-Q4K.gguf ds4flash.gguf
./ds4 -p "The weather is great today" -n 100

Expected output: coherent English text continuation at ~26 t/s (CPU, 20 threads).

Technical Notes

Why Q4_K for experts and F16 for norms?

deepseek4-quantize applies the quantization policy selectively by tensor shape:

• 1D tensors (norms, scales, biases): the policy never overrides the template type. They stay F32 regardless of what --dense or --attention say.
• 2D+ tensors : the policy applies the most specific matching flag:
  • Expert tensors ( blk.*.ffn_*_exps.weight ) → --experts
  • Attention projections ( attn_q_a/b , attn_kv , attn_output_a/b ) → --attention-proj
  • Shared expert weights → --shared
  • Output head → --output
  • Token embedding → --embedding
  • Other attention/indexer/compressor → --attention
  • Everything else 2D+ → --dense

How the template maps HF names to GGUF names

gen_gguf_template.py uses the same layer_map table as deepseek4-quantize.c . For example:

The script also automatically converts the ffn.gate.tid2eid routing table from I64 to I32, which is the only non-F32/F16 tensor type override in the template.

4Expert vs 6Expert: What Changed in ds4

The upstream ds4 hardcodes 6 active routed experts per token ( n_expert_used = 6 ). For this 4Expert model to work:

1. Default changed to 4 — DS4_SHAPE_FLASH.n_expert_used and g_ds4_shape.n_expert_used now default to 4.
2. Backward compatible — When loading a GGUF with n_expert_used = 6 in its metadata, ds4 preserves 6 at runtime. Old 6-expert GGUF files continue to work.
3. Template generator — gen_gguf_template.py handles the full tensor mapping, replacing manual template construction.

Full details: PR #474

GGUF Evaluation Report — DeepSeek V4 Flash 4Expert Q4_K GGUF BY ds4-eval

Model: cloudyu/DeepSeek-V4-Flash-4Expert-GGUF Source safetensors: cloudyu/DeepSeek-V4-Flash-4Expert Date: 2026-06-29

Summary

80 of 92 tests passed. The 12 failures are detailed below.

Evaluation Methodology

All tests were run using ds4-eval , the built-in evaluation tool shipped with the ds4 inference engine . Each test case consists of a prompt and a set of valid ground-truth answers (e.g., A , B , C , D for multiple choice; integer answers for AIME; ranges or enumerations for COMPSEC).

The evaluator feeds the prompt to the model, reads the generated completion, and extracts the final answer using framework-specific parsers. A test passes if the extracted answer matches any of the valid ground-truth values.

Evaluation Frameworks

• AIME 2025 (25 tests): American Invitational Mathematics Examination. Integer answers (0–999). Tests mathematical reasoning.
• GPQA Diamond (25 tests): Graduate-level multiple-choice science questions. Options A–D. Tests deep domain knowledge.
• SuperGPQA (25 tests): Expanded graduate-level multiple choice. Options A–J. Broader and harder than GPQA.
• COMPSEC (17 tests): Computer security questions. Answers are integer codes or ranges (e.g., 5 , 10-15 , 3,13-15 ). Tests specialized security knowledge.

Scoring Rules

• AIME: exact integer match.
• GPQA / SuperGPQA: exact option letter match (A–J).
• COMPSEC: answer must fall within one of the accepted integer values or ranges.

Hardware & Build

Build Configuration

git clone https://github.com/yuhai-china/ds4
cd ds4
git checkout 4expert
make -j$(sysctl -n hw.ncpu)

No flags passed — standard release build ( -O3 -ffast-math -mcpu=native ).

Runtime Configuration

GGUF loaded via memory-mapped I/O. Key runtime parameters from ds4-eval output:

ds4: Metal device Apple M2 Ultra, 192.00 GiB RAM
ds4: Metal 4 tensor API disabled for pre-M5/pre-A19 devices
ds4: drift-patch flags hc_stable=on norm_unify=on kv_raw_f32=off rope_exp2_log2=off
ds4-eval: context auto-sized to 16777 tokens
ds4-eval: context buffers 630.30 MiB
ds4-eval: model shape DeepSeek V4 Flash

No environment variables or config overrides were set beyond the default.

Detailed Results

AIME 2025 (13/15, 86.7%)

All 5 AIME failures are generation truncation — the model hit the 16,000 token budget before finishing the chain-of-thought and producing a final answer. The budget was auto-sized by ds4-eval as largest_prompt + 16,000 .

GPQA Diamond (8/10, 80.0%)

SuperGPQA (12/15, 80.0%)

COMPSEC (14/15, 93.3%)

Failure Analysis (12 failed)

Of the 12 failures:

• 5 are AIME chain-of-thought truncation (context budget = 16,777 tokens, generation budget = 16,000 tokens). The model needed more tokens to finish reasoning. These would likely pass with a larger context window.
• 7 are genuine incorrect answers (3 GPQA, 3 SuperGPQA, 1 COMPSEC).

Excluding truncation failures, the pass rate is 80/87 = 92.0% .

Reproduction

git clone https://github.com/yuhai-china/ds4 && cd ds4 && git checkout 4expert
make -j$(sysctl -n hw.ncpu)

pip install huggingface_hub
python3 -c "
from huggingface_hub import hf_hub_download
hf_hub_download('cloudyu/DeepSeek-V4-Flash-4Expert-GGUF', 'DeepSeek-V4-Flash-4Expert-Q4K.gguf', local_dir='.')
"

./ds4-eval -m DeepSeek-V4-Flash-4Expert-Q4K.gguf

On Linux, replace the build step with make cpu -j$(nproc) . On CUDA systems, use make cuda-generic -j$(nproc) .