Converted version of
Qwen2.5-32B-Instruct
to 4-bit using bitsandbytes. For more information about the model,
refer to the model's page.
Impact on performance
Impact of quantization on a set of models.
Evaluation of the model was conducted using the PoLL (Pool of LLM) technique, assessing performance on
100 French questions
with scores aggregated from six evaluations
(two per evaluator). The evaluators included GPT-4o, Gemini-1.5-pro, and Claude3.5-sonnet.
Performance Scores (on a scale of 5):
Model
Score
# params (Billion)
size (GB)
gpt-4o
4.13
N/A
N/A
gpt-4o-mini
4.02
N/A
N/A
Qwen/Qwen2.5-32B-Instruct
3.99
32.8
65.6
cmarkea/Qwen2.5-32B-Instruct-4bit
3.98
32.8
16.4
mistralai/Mixtral-8x7B-Instruct-v0.1
3.71
46.7
93.4
cmarkea/Mixtral-8x7B-Instruct-v0.1-4bit
3.68
46.7
23.35
meta-llama/Meta-Llama-3.1-70B-Instruct
3.68
70.06
140.12
gpt-3.5-turbo
3.66
175
350
cmarkea/Meta-Llama-3.1-70B-Instruct-4bit
3.64
70.06
35.3
TheBloke/Mixtral-8x7B-Instruct-v0.1-GPTQ
3.56
46.7
46.7
meta-llama/Meta-Llama-3.1-8B-Instruct
3.25
8.03
16.06
mistralai/Mistral-7B-Instruct-v0.2
1.98
7.25
14.5
cmarkea/bloomz-7b1-mt-sft-chat
1.69
7.07
14.14
cmarkea/bloomz-3b-dpo-chat
1.68
3
6
cmarkea/bloomz-3b-sft-chat
1.51
3
6
croissantllm/CroissantLLMChat-v0.1
1.19
1.3
2.7
cmarkea/bloomz-560m-sft-chat
1.04
0.56
1.12
OpenLLM-France/Claire-Mistral-7B-0.1
0.38
7.25
14.5
The impact of quantization is negligible.
Prompt Pattern
Here is a reminder of the command pattern to interact with the model:
Qwen2.5-32B-Instruct-4bit huggingface.co is an AI model on huggingface.co that provides Qwen2.5-32B-Instruct-4bit's model effect (), which can be used instantly with this cmarkea Qwen2.5-32B-Instruct-4bit model. huggingface.co supports a free trial of the Qwen2.5-32B-Instruct-4bit model, and also provides paid use of the Qwen2.5-32B-Instruct-4bit. Support call Qwen2.5-32B-Instruct-4bit model through api, including Node.js, Python, http.
Qwen2.5-32B-Instruct-4bit huggingface.co is an online trial and call api platform, which integrates Qwen2.5-32B-Instruct-4bit's modeling effects, including api services, and provides a free online trial of Qwen2.5-32B-Instruct-4bit, you can try Qwen2.5-32B-Instruct-4bit online for free by clicking the link below.
cmarkea Qwen2.5-32B-Instruct-4bit online free url in huggingface.co:
Qwen2.5-32B-Instruct-4bit is an open source model from GitHub that offers a free installation service, and any user can find Qwen2.5-32B-Instruct-4bit on GitHub to install. At the same time, huggingface.co provides the effect of Qwen2.5-32B-Instruct-4bit install, users can directly use Qwen2.5-32B-Instruct-4bit installed effect in huggingface.co for debugging and trial. It also supports api for free installation.
Qwen2.5-32B-Instruct-4bit install url in huggingface.co: