Sponsored by PoYo.AI
Toolify Toolify logo
AI Models
DeepSeek-Coder-V2-Instruct-IMat-GGUF

legraphista / DeepSeek-Coder-V2-Instruct-IMat-GGUF

Run on huggingface.co
huggingface.co
Total runs: 1.3K
24-hour runs: 0
7-day runs: 492
30-day runs: 760
Model's Last Updated: June 19 2024
text-generation

Introduction of DeepSeek-Coder-V2-Instruct-IMat-GGUF

Model Details of DeepSeek-Coder-V2-Instruct-IMat-GGUF

DeepSeek-Coder-V2-Instruct-IMat-GGUF

Llama.cpp imatrix quantization of deepseek-ai/DeepSeek-Coder-V2-Instruct

Original Model: deepseek-ai/DeepSeek-Coder-V2-Instruct
Original dtype: BF16 ( bfloat16 )
Quantized by: llama.cpp b3166
IMatrix dataset: here

Files
IMatrix

Status: ✅ Available
Link: here

Common Quants
Filename Quant type File Size Status Uses IMatrix Is Split
DeepSeek-Coder-V2-Instruct.Q8_0/* Q8_0 250.62GB ✅ Available ⚪ Static ✂ Yes
DeepSeek-Coder-V2-Instruct.Q6_K/* Q6_K 193.54GB ✅ Available ⚪ Static ✂ Yes
DeepSeek-Coder-V2-Instruct.Q4_K/* Q4_K 142.45GB ✅ Available 🟢 IMatrix ✂ Yes
DeepSeek-Coder-V2-Instruct.Q3_K/* Q3_K 112.67GB ✅ Available 🟢 IMatrix ✂ Yes
DeepSeek-Coder-V2-Instruct.Q2_K/* Q2_K 85.95GB ✅ Available 🟢 IMatrix ✂ Yes
All Quants
Filename Quant type File Size Status Uses IMatrix Is Split
DeepSeek-Coder-V2-Instruct.Q8_0/* Q8_0 250.62GB ✅ Available ⚪ Static ✂ Yes
DeepSeek-Coder-V2-Instruct.Q6_K/* Q6_K 193.54GB ✅ Available ⚪ Static ✂ Yes
DeepSeek-Coder-V2-Instruct.Q5_K/* Q5_K 167.22GB ✅ Available ⚪ Static ✂ Yes
DeepSeek-Coder-V2-Instruct.Q5_K_S/* Q5_K_S 162.31GB ✅ Available ⚪ Static ✂ Yes
DeepSeek-Coder-V2-Instruct.Q4_K/* Q4_K 142.45GB ✅ Available 🟢 IMatrix ✂ Yes
DeepSeek-Coder-V2-Instruct.Q4_K_S/* Q4_K_S 133.88GB ✅ Available 🟢 IMatrix ✂ Yes
DeepSeek-Coder-V2-Instruct.IQ4_NL/* IQ4_NL 132.91GB ✅ Available 🟢 IMatrix ✂ Yes
DeepSeek-Coder-V2-Instruct.IQ4_XS/* IQ4_XS 125.56GB ✅ Available 🟢 IMatrix ✂ Yes
DeepSeek-Coder-V2-Instruct.Q3_K/* Q3_K 112.67GB ✅ Available 🟢 IMatrix ✂ Yes
DeepSeek-Coder-V2-Instruct.Q3_K_L/* Q3_K_L 122.37GB ✅ Available 🟢 IMatrix ✂ Yes
DeepSeek-Coder-V2-Instruct.Q3_K_S/* Q3_K_S 101.68GB ✅ Available 🟢 IMatrix ✂ Yes
DeepSeek-Coder-V2-Instruct.IQ3_M/* IQ3_M 103.37GB ✅ Available 🟢 IMatrix ✂ Yes
DeepSeek-Coder-V2-Instruct.IQ3_S/* IQ3_S 101.68GB ✅ Available 🟢 IMatrix ✂ Yes
DeepSeek-Coder-V2-Instruct.IQ3_XS/* IQ3_XS 96.30GB ✅ Available 🟢 IMatrix ✂ Yes
DeepSeek-Coder-V2-Instruct.IQ3_XXS/* IQ3_XXS 90.85GB ✅ Available 🟢 IMatrix ✂ Yes
DeepSeek-Coder-V2-Instruct.Q2_K/* Q2_K 85.95GB ✅ Available 🟢 IMatrix ✂ Yes
DeepSeek-Coder-V2-Instruct.Q2_K_S/* Q2_K_S 79.60GB ✅ Available 🟢 IMatrix ✂ Yes
DeepSeek-Coder-V2-Instruct.IQ2_M/* IQ2_M 76.92GB ✅ Available 🟢 IMatrix ✂ Yes
DeepSeek-Coder-V2-Instruct.IQ2_S/* IQ2_S 69.87GB ✅ Available 🟢 IMatrix ✂ Yes
DeepSeek-Coder-V2-Instruct.IQ2_XS/* IQ2_XS 68.71GB ✅ Available 🟢 IMatrix ✂ Yes
DeepSeek-Coder-V2-Instruct.IQ2_XXS/* IQ2_XXS 61.50GB ✅ Available 🟢 IMatrix ✂ Yes
DeepSeek-Coder-V2-Instruct.IQ1_M/* IQ1_M 52.68GB ✅ Available 🟢 IMatrix ✂ Yes
DeepSeek-Coder-V2-Instruct.IQ1_S/* IQ1_S 47.39GB ✅ Available 🟢 IMatrix ✂ Yes
Downloading using huggingface-cli

If you do not have hugginface-cli installed: 

pip install -U "huggingface_hub[cli]"

Download the specific file you want: 

huggingface-cli download legraphista/DeepSeek-Coder-V2-Instruct-IMat-GGUF --include "DeepSeek-Coder-V2-Instruct.Q8_0.gguf" --local-dir ./

If the model file is big, it has been split into multiple files. In order to download them all to a local folder, run: 

huggingface-cli download legraphista/DeepSeek-Coder-V2-Instruct-IMat-GGUF --include "DeepSeek-Coder-V2-Instruct.Q8_0/*" --local-dir ./
# see FAQ for merging GGUF's
Inference
Simple chat template 
<｜begin▁of▁sentence｜>User: {user_prompt}

Assistant: {assistant_response}<｜end▁of▁sentence｜>User: {next_user_prompt}
Chat template with system prompt 
<｜begin▁of▁sentence｜>{system_prompt}

User: {user_prompt}

Assistant: {assistant_response}<｜end▁of▁sentence｜>User: {next_user_prompt}
Llama.cpp 
llama.cpp/main -m DeepSeek-Coder-V2-Instruct.Q8_0.gguf --color -i -p "prompt here (according to the chat template)"
FAQ
Why is the IMatrix not applied everywhere?

According to this investigation , it appears that lower quantizations are the only ones that benefit from the imatrix input (as per hellaswag results).

How do I merge a split GGUF?
  1. Make sure you have gguf-split available
  2. Locate your GGUF chunks folder (ex: DeepSeek-Coder-V2-Instruct.Q8_0 )
  3. Run gguf-split --merge DeepSeek-Coder-V2-Instruct.Q8_0/DeepSeek-Coder-V2-Instruct.Q8_0-00001-of-XXXXX.gguf DeepSeek-Coder-V2-Instruct.Q8_0.gguf
    • Make sure to point gguf-split to the first chunk of the split.

Got a suggestion? Ping me @legraphista !

Runs of legraphista DeepSeek-Coder-V2-Instruct-IMat-GGUF on huggingface.co

1.3K
Total runs
0
24-hour runs
204
3-day runs
492
7-day runs
760
30-day runs

More Information About DeepSeek-Coder-V2-Instruct-IMat-GGUF huggingface.co Model

More DeepSeek-Coder-V2-Instruct-IMat-GGUF license Visit here:

https://choosealicense.com/licenses/deepseek-license

DeepSeek-Coder-V2-Instruct-IMat-GGUF huggingface.co

DeepSeek-Coder-V2-Instruct-IMat-GGUF huggingface.co is an AI model on huggingface.co that provides DeepSeek-Coder-V2-Instruct-IMat-GGUF's model effect (), which can be used instantly with this legraphista DeepSeek-Coder-V2-Instruct-IMat-GGUF model. huggingface.co supports a free trial of the DeepSeek-Coder-V2-Instruct-IMat-GGUF model, and also provides paid use of the DeepSeek-Coder-V2-Instruct-IMat-GGUF. Support call DeepSeek-Coder-V2-Instruct-IMat-GGUF model through api, including Node.js, Python, http.

DeepSeek-Coder-V2-Instruct-IMat-GGUF huggingface.co Url

https://huggingface.co/legraphista/DeepSeek-Coder-V2-Instruct-IMat-GGUF

legraphista DeepSeek-Coder-V2-Instruct-IMat-GGUF online free

DeepSeek-Coder-V2-Instruct-IMat-GGUF huggingface.co is an online trial and call api platform, which integrates DeepSeek-Coder-V2-Instruct-IMat-GGUF's modeling effects, including api services, and provides a free online trial of DeepSeek-Coder-V2-Instruct-IMat-GGUF, you can try DeepSeek-Coder-V2-Instruct-IMat-GGUF online for free by clicking the link below.

legraphista DeepSeek-Coder-V2-Instruct-IMat-GGUF online free url in huggingface.co:

https://huggingface.co/legraphista/DeepSeek-Coder-V2-Instruct-IMat-GGUF

DeepSeek-Coder-V2-Instruct-IMat-GGUF install

DeepSeek-Coder-V2-Instruct-IMat-GGUF is an open source model from GitHub that offers a free installation service, and any user can find DeepSeek-Coder-V2-Instruct-IMat-GGUF on GitHub to install. At the same time, huggingface.co provides the effect of DeepSeek-Coder-V2-Instruct-IMat-GGUF install, users can directly use DeepSeek-Coder-V2-Instruct-IMat-GGUF installed effect in huggingface.co for debugging and trial. It also supports api for free installation.

DeepSeek-Coder-V2-Instruct-IMat-GGUF install url in huggingface.co:

https://huggingface.co/legraphista/DeepSeek-Coder-V2-Instruct-IMat-GGUF

Url of DeepSeek-Coder-V2-Instruct-IMat-GGUF

DeepSeek-Coder-V2-Instruct-IMat-GGUF huggingface.co Url

Provider of DeepSeek-Coder-V2-Instruct-IMat-GGUF huggingface.co

legraphista
ORGANIZATIONS
huggingface.co

Other API from legraphista