Total runs:
333.7K
Run Growth:
0
Growth Rate:
0.00%
Sponsored by
YouArt
neuralmagic / Mistral-7B-Instruct-v0.3-GPTQ-4bit
Model's Last Updated: June 11 2024
text-generation
Introduction of Mistral-7B-Instruct-v0.3-GPTQ-4bit
Model Details of Mistral-7B-Instruct-v0.3-GPTQ-4bit
Model Card for Mistral-7B-Instruct-v0.3 quantized to 4bit weights
- Weight-only quantization of Mistral-7B-Instruct-v0.3 via GPTQ to 4bits with group_size=128
- GPTQ optimized for 99.75% accuracy recovery relative to the unquantized model
Open LLM Leaderboard evaluation scores
| Mistral-7B-Instruct-v0.3 |
Mistral-7B-Instruct-v0.3-GPTQ-4bit
(this model) |
|
|---|---|---|
|
arc-c
25-shot |
63.48 | 63.40 |
|
mmlu
5-shot |
61.13 | 60.89 |
|
hellaswag
10-shot |
84.49 | 84.04 |
|
winogrande
5-shot |
79.16 | 79.08 |
|
gsm8k
5-shot |
43.37 | 45.41 |
|
truthfulqa
0-shot |
59.65 | 57.48 |
|
Average
Accuracy |
65.21 | 65.05 |
| Recovery | 100% | 99.75% |
vLLM Inference Performance
This model is ready for optimized inference using the Marlin mixed-precision kernels in vLLM: https://github.com/vllm-project/vllm
Simply start this model as an inference server with:
python -m vllm.entrypoints.openai.api_server --model neuralmagic/Mistral-7B-Instruct-v0.3-GPTQ-4bit
Runs of neuralmagic Mistral-7B-Instruct-v0.3-GPTQ-4bit on huggingface.co
2.7K
Total runs
0
24-hour runs
0
3-day runs
0
7-day runs
0
30-day runs
More Information About Mistral-7B-Instruct-v0.3-GPTQ-4bit huggingface.co Model
Url of Mistral-7B-Instruct-v0.3-GPTQ-4bit
Mistral-7B-Instruct-v0.3-GPTQ-4bit huggingface.co Url
Provider of Mistral-7B-Instruct-v0.3-GPTQ-4bit huggingface.co
Other API from neuralmagic
Updated:March 26 2025
Total runs:
210.5K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
163.8K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
111.8K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
105.6K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
83.2K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
62.1K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
58.7K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
48.2K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
29.2K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
21.7K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
16.5K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
15.6K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
12.6K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
12.1K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
11.9K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
10.9K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
8.6K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
8.4K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
8.1K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
7.5K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
7.2K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
6.6K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
5.6K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
5.2K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
4.7K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
4.5K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
4.4K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
4.2K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
4.1K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
4.0K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
3.6K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
3.4K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
2.6K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
2.4K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
2.0K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
1.8K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
1.8K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
1.6K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
1.5K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
1.5K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
1.3K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
1.2K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
1.2K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
1.2K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
1.2K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
1.1K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
1.1K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
1.1K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
1.0K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
959
Run Growth:
0
Growth Rate:
0.00%
Total runs:
949
Run Growth:
0
Growth Rate:
0.00%
Total runs:
915
Run Growth:
0
Growth Rate:
0.00%
Total runs:
906
Run Growth:
0
Growth Rate:
0.00%
Total runs:
901
Run Growth:
0
Growth Rate:
0.00%
Total runs:
896
Run Growth:
0
Growth Rate:
0.00%
Total runs:
875
Run Growth:
0
Growth Rate:
0.00%
Total runs:
774
Run Growth:
0
Growth Rate:
0.00%
Total runs:
759
Run Growth:
0
Growth Rate:
0.00%
Total runs:
709
Run Growth:
0
Growth Rate:
0.00%
Total runs:
667
Run Growth:
0
Growth Rate:
0.00%
Total runs:
647
Run Growth:
0
Growth Rate:
0.00%
Total runs:
647
Run Growth:
0
Growth Rate:
0.00%
Total runs:
628
Run Growth:
0
Growth Rate:
0.00%
Total runs:
617
Run Growth:
0
Growth Rate:
0.00%
Total runs:
597
Run Growth:
0
Growth Rate:
0.00%
Total runs:
592
Run Growth:
0
Growth Rate:
0.00%
Total runs:
567
Run Growth:
0
Growth Rate:
0.00%
Total runs:
546
Run Growth:
0
Growth Rate:
0.00%
Total runs:
545
Run Growth:
0
Growth Rate:
0.00%
Total runs:
543
Run Growth:
0
Growth Rate:
0.00%
Total runs:
525
Run Growth:
0
Growth Rate:
0.00%
Total runs:
524
Run Growth:
0
Growth Rate:
0.00%
Total runs:
482
Run Growth:
0
Growth Rate:
0.00%
Total runs:
466
Run Growth:
0
Growth Rate:
0.00%
Total runs:
465
Run Growth:
0
Growth Rate:
0.00%
Total runs:
456
Run Growth:
0
Growth Rate:
0.00%
Total runs:
443
Run Growth:
0
Growth Rate:
0.00%
Total runs:
438
Run Growth:
0
Growth Rate:
0.00%
Total runs:
362
Run Growth:
0
Growth Rate:
0.00%
Total runs:
351
Run Growth:
0
Growth Rate:
0.00%
Total runs:
348
Run Growth:
0
Growth Rate:
0.00%
Total runs:
346
Run Growth:
0
Growth Rate:
0.00%
Total runs:
344
Run Growth:
0
Growth Rate:
0.00%
Total runs:
334
Run Growth:
0
Growth Rate:
0.00%
Total runs:
332
Run Growth:
0
Growth Rate:
0.00%
Total runs:
282
Run Growth:
0
Growth Rate:
0.00%
Total runs:
269
Run Growth:
0
Growth Rate:
0.00%
Total runs:
257
Run Growth:
0
Growth Rate:
0.00%
Total runs:
245
Run Growth:
0
Growth Rate:
0.00%
Total runs:
234
Run Growth:
0
Growth Rate:
0.00%
Total runs:
203
Run Growth:
0
Growth Rate:
0.00%
Total runs:
199
Run Growth:
0
Growth Rate:
0.00%
Total runs:
194
Run Growth:
0
Growth Rate:
0.00%
Total runs:
190
Run Growth:
0
Growth Rate:
0.00%
Total runs:
182
Run Growth:
0
Growth Rate:
0.00%
Total runs:
181
Run Growth:
0
Growth Rate:
0.00%
Total runs:
178
Run Growth:
0
Growth Rate:
0.00%