legraphista / granite-20b-code-instruct-IMat-GGUF

granite-20b-code-instruct-IMat-GGUF

Llama.cpp imatrix quantization of ibm-granite/granite-20b-code-instruct

Original Model: ibm-granite/granite-20b-code-instruct
Original dtype: BF16 ( bfloat16 )
Quantized by: llama.cpp b3649
IMatrix dataset: here

• Files
  • IMatrix
  • Common Quants
  • All Quants
• Downloading using huggingface-cli
• Inference
  • Simple chat template
  • Chat template with system prompt
  • Llama.cpp
• FAQ
  • Why is the IMatrix not applied everywhere?
  • How do I merge a split GGUF?

Files

IMatrix

Status: ✅ Available
Link: here

Common Quants

All Quants

Downloading using huggingface-cli

If you do not have hugginface-cli installed:

pip install -U "huggingface_hub[cli]"

Download the specific file you want:

huggingface-cli download legraphista/granite-20b-code-instruct-IMat-GGUF --include "granite-20b-code-instruct.Q8_0.gguf" --local-dir ./

If the model file is big, it has been split into multiple files. In order to download them all to a local folder, run:

huggingface-cli download legraphista/granite-20b-code-instruct-IMat-GGUF --include "granite-20b-code-instruct.Q8_0/*" --local-dir ./
# see FAQ for merging GGUF's

Inference

Simple chat template

Question:
{user_prompt}

Answer:
{assistant_response}

Question:
{next_user_prompt}

Chat template with system prompt

System:
{system_prompt}

Question:
{user_prompt}

Answer:
{assistant_response}

Question:
{next_user_prompt}

Llama.cpp

llama.cpp/main -m granite-20b-code-instruct.Q8_0.gguf --color -i -p "prompt here (according to the chat template)"

FAQ

Why is the IMatrix not applied everywhere?

According to this investigation , it appears that lower quantizations are the only ones that benefit from the imatrix input (as per hellaswag results).

How do I merge a split GGUF?

1. Make sure you have gguf-split available
   • To get hold of gguf-split , navigate to https://github.com/ggerganov/llama.cpp/releases
   • Download the appropriate zip for your system from the latest release
   • Unzip the archive and you should be able to find gguf-split
2. Locate your GGUF chunks folder (ex: granite-20b-code-instruct.Q8_0 )
3. Run gguf-split --merge granite-20b-code-instruct.Q8_0/granite-20b-code-instruct.Q8_0-00001-of-XXXXX.gguf granite-20b-code-instruct.Q8_0.gguf
   • Make sure to point gguf-split to the first chunk of the split.

Got a suggestion? Ping me @legraphista !