Sponsored by
PoYo.AI
internlm / internlm2_5-7b-chat-4bit
Model's Last Updated: July 02 2024
text-generation
Introduction of internlm2_5-7b-chat-4bit
Model Details of internlm2_5-7b-chat-4bit
INT4 Weight-only Quantization and Deployment (W4A16)
LMDeploy adopts AWQ algorithm for 4bit weight-only quantization. By developed the high-performance cuda kernel, the 4bit quantized model inference achieves up to 2.4x faster than FP16.
LMDeploy supports the following NVIDIA GPU for W4A16 inference:
-
Turing(sm75): 20 series, T4
-
Ampere(sm80,sm86): 30 series, A10, A16, A30, A100
-
Ada Lovelace(sm90): 40 series
Before proceeding with the quantization and inference, please ensure that lmdeploy is installed.
pip install lmdeploy[all]
This article comprises the following sections:
Inference
Trying the following codes, you can perform the batched offline inference with the quantized model:
from lmdeploy import pipeline, TurbomindEngineConfig
engine_config = TurbomindEngineConfig(model_format='awq')
pipe = pipeline("internlm/internlm2_5-7b-chat-4bit", backend_config=engine_config)
response = pipe(["Hi, pls intro yourself", "Shanghai is"])
print(response)
For more information about the pipeline parameters, please refer to here .
Evaluation
Please overview this guide about model evaluation with LMDeploy.
Service
LMDeploy's
api_server
enables models to be easily packed into services with a single command. The provided RESTful APIs are compatible with OpenAI's interfaces. Below are an example of service startup:
lmdeploy serve api_server internlm/internlm2_5-7b-chat-4bit --backend turbomind --model-format awq
The default port of
api_server
is
23333
. After the server is launched, you can communicate with server on terminal through
api_client
:
lmdeploy serve api_client http://0.0.0.0:23333
You can overview and try out
api_server
APIs online by swagger UI at
http://0.0.0.0:23333
, or you can also read the API specification from
here
.
Runs of internlm internlm2_5-7b-chat-4bit on huggingface.co
248
Total runs
0
24-hour runs
0
3-day runs
0
7-day runs
0
30-day runs
More Information About internlm2_5-7b-chat-4bit huggingface.co Model
Url of internlm2_5-7b-chat-4bit
internlm2_5-7b-chat-4bit huggingface.co Url
Provider of internlm2_5-7b-chat-4bit huggingface.co
Other API from internlm
Total runs:
87.6K
Run Growth:
17.6K
Growth Rate:
19.96%
Total runs:
74.4K
Run Growth:
-3.0K
Growth Rate:
-4.07%
Total runs:
58.7K
Run Growth:
25.6K
Growth Rate:
45.07%
Total runs:
48.5K
Run Growth:
3.2K
Growth Rate:
6.70%
Total runs:
32.7K
Run Growth:
3.4K
Growth Rate:
10.30%
Total runs:
21.2K
Run Growth:
2.3K
Growth Rate:
10.72%
Total runs:
21.1K
Run Growth:
2.1K
Growth Rate:
9.97%
Total runs:
17.3K
Run Growth:
738
Growth Rate:
4.27%
Total runs:
8.0K
Run Growth:
1.8K
Growth Rate:
23.38%
Total runs:
7.7K
Run Growth:
2.6K
Growth Rate:
34.31%
Total runs:
5.6K
Run Growth:
1.8K
Growth Rate:
31.32%
Total runs:
5.6K
Run Growth:
-6
Growth Rate:
-0.11%
Total runs:
4.4K
Run Growth:
1.8K
Growth Rate:
42.02%
Total runs:
4.2K
Run Growth:
-191
Growth Rate:
-4.51%
Total runs:
3.2K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
3.0K
Run Growth:
-534
Growth Rate:
-18.05%
Total runs:
1.8K
Run Growth:
1.6K
Growth Rate:
100.00%
Total runs:
1.7K
Run Growth:
0
Growth Rate:
0.00%
Total runs:
1.6K
Run Growth:
-134.7K
Growth Rate:
-8320.57%
Total runs:
1.6K
Run Growth:
421
Growth Rate:
26.80%
Total runs:
1.5K
Run Growth:
1.4K
Growth Rate:
100.00%
Total runs:
1.3K
Run Growth:
53
Growth Rate:
3.99%
Total runs:
1.3K
Run Growth:
1.2K
Growth Rate:
100.00%
Total runs:
1.1K
Run Growth:
-449
Growth Rate:
-40.89%
Total runs:
1.0K
Run Growth:
778
Growth Rate:
75.90%
Total runs:
894
Run Growth:
-2
Growth Rate:
-0.22%
Total runs:
855
Run Growth:
-3.0K
Growth Rate:
-356.02%
Total runs:
757
Run Growth:
516
Growth Rate:
68.16%
Total runs:
731
Run Growth:
731
Growth Rate:
100.00%
Total runs:
683
Run Growth:
668
Growth Rate:
100.00%
Total runs:
657
Run Growth:
575
Growth Rate:
100.00%
Total runs:
570
Run Growth:
-767
Growth Rate:
-136.23%
Total runs:
471
Run Growth:
-305
Growth Rate:
-64.76%
Total runs:
470
Run Growth:
-890
Growth Rate:
-193.48%
Total runs:
454
Run Growth:
0
Growth Rate:
0.00%
Total runs:
423
Run Growth:
133
Growth Rate:
31.44%
Total runs:
382
Run Growth:
34
Growth Rate:
8.90%
Total runs:
363
Run Growth:
69
Growth Rate:
19.01%
Total runs:
358
Run Growth:
128
Growth Rate:
35.75%
Total runs:
326
Run Growth:
0
Growth Rate:
0.00%
Total runs:
320
Run Growth:
286
Growth Rate:
89.38%
Total runs:
306
Run Growth:
121
Growth Rate:
39.54%
Total runs:
304
Run Growth:
5
Growth Rate:
1.64%
Total runs:
294
Run Growth:
59
Growth Rate:
20.07%
Total runs:
294
Run Growth:
-30
Growth Rate:
-10.20%
Total runs:
293
Run Growth:
2
Growth Rate:
0.68%
Total runs:
281
Run Growth:
119
Growth Rate:
42.35%
Total runs:
280
Run Growth:
70
Growth Rate:
25.27%
Total runs:
250
Run Growth:
161
Growth Rate:
67.65%
Total runs:
244
Run Growth:
62
Growth Rate:
25.51%
Total runs:
242
Run Growth:
-26
Growth Rate:
-11.21%
Total runs:
230
Run Growth:
73
Growth Rate:
32.74%
Total runs:
215
Run Growth:
102
Growth Rate:
47.44%
Total runs:
211
Run Growth:
-50
Growth Rate:
-23.70%
Total runs:
211
Run Growth:
0
Growth Rate:
0.00%
Total runs:
205
Run Growth:
76
Growth Rate:
38.19%
Total runs:
204
Run Growth:
53
Growth Rate:
26.90%
Total runs:
190
Run Growth:
51
Growth Rate:
26.84%
Total runs:
186
Run Growth:
31
Growth Rate:
16.67%
Total runs:
179
Run Growth:
-169
Growth Rate:
-94.41%
Total runs:
178
Run Growth:
110
Growth Rate:
61.80%
Total runs:
149
Run Growth:
58
Growth Rate:
40.00%
Total runs:
149
Run Growth:
120
Growth Rate:
83.33%
Total runs:
145
Run Growth:
0
Growth Rate:
0.00%
Total runs:
132
Run Growth:
-35
Growth Rate:
-26.52%
Total runs:
132
Run Growth:
132
Growth Rate:
100.00%
Total runs:
124
Run Growth:
0
Growth Rate:
0.00%
Total runs:
122
Run Growth:
50
Growth Rate:
40.98%
Total runs:
115
Run Growth:
20
Growth Rate:
17.86%
Total runs:
111
Run Growth:
44
Growth Rate:
41.51%
Total runs:
102
Run Growth:
2
Growth Rate:
1.96%
Total runs:
101
Run Growth:
0
Growth Rate:
0.00%
Total runs:
96
Run Growth:
50
Growth Rate:
51.02%
Total runs:
93
Run Growth:
13
Growth Rate:
14.13%
Total runs:
90
Run Growth:
11
Growth Rate:
12.36%
Total runs:
86
Run Growth:
1
Growth Rate:
1.16%
Total runs:
81
Run Growth:
39
Growth Rate:
45.35%
Total runs:
80
Run Growth:
-63
Growth Rate:
-78.75%