This model is a mixed-precision quantized version of DeepSeek-V3.1-Terminus, with
dense layer keep the FP8 quantization of the original model, while MoE layers
uses INT4 weights and FP8 activation, also called W4AFP8.
Benchmark
The accuracy below was obtained with SGLang V0.5.3 in non-thinking mode.
Model
math_500
gpqa
aime2024
mmlu-pro
DeepSeek-V3.1-Terminus-W4AFP8
89.83
78.28
0.8
83.66
Inference with SGLang
We have already supported deploying this model using tensor parallel in sglang for
better performance. The releated PR
https://github.com/sgl-project/sglang/pull/8118
has been merged in SGLang V0.5.2, so you can deploy this model using SGLang
version 0.5.2 or later with tensor parallel.
DeepSeek-V3.1-Terminus-W4AFP8 huggingface.co is an AI model on huggingface.co that provides DeepSeek-V3.1-Terminus-W4AFP8's model effect (), which can be used instantly with this tencent DeepSeek-V3.1-Terminus-W4AFP8 model. huggingface.co supports a free trial of the DeepSeek-V3.1-Terminus-W4AFP8 model, and also provides paid use of the DeepSeek-V3.1-Terminus-W4AFP8. Support call DeepSeek-V3.1-Terminus-W4AFP8 model through api, including Node.js, Python, http.
DeepSeek-V3.1-Terminus-W4AFP8 huggingface.co is an online trial and call api platform, which integrates DeepSeek-V3.1-Terminus-W4AFP8's modeling effects, including api services, and provides a free online trial of DeepSeek-V3.1-Terminus-W4AFP8, you can try DeepSeek-V3.1-Terminus-W4AFP8 online for free by clicking the link below.
tencent DeepSeek-V3.1-Terminus-W4AFP8 online free url in huggingface.co:
DeepSeek-V3.1-Terminus-W4AFP8 is an open source model from GitHub that offers a free installation service, and any user can find DeepSeek-V3.1-Terminus-W4AFP8 on GitHub to install. At the same time, huggingface.co provides the effect of DeepSeek-V3.1-Terminus-W4AFP8 install, users can directly use DeepSeek-V3.1-Terminus-W4AFP8 installed effect in huggingface.co for debugging and trial. It also supports api for free installation.
DeepSeek-V3.1-Terminus-W4AFP8 install url in huggingface.co: