Total runs:
942
Run Growth:
264
Growth Rate:
28.03%
Sponsored by
AdsCreator.com
mgoin / Nemotron-4-340B-Base-hf
Model's Last Updated: August 08 2024
text-generation
Introduction of Nemotron-4-340B-Base-hf
Model Details of Nemotron-4-340B-Base-hf
Nemotron-4-340B-Base-hf
Converted checkpoint of nvidia/Nemotron-4-340B-Base . Specifically it was produced from the v1.2 .nemo checkpoint on NGC .
You can deploy this model with
vllm>=0.5.4
(
PR#6611
):
vllm serve mgoin/Nemotron-4-340B-Base-hf --tensor-parallel-size 16
Evaluations
All the below evaluations were run with the
FP8 checkpoint
using
lm-eval==0.4.3
on 8xA100 GPUs.
lm_eval --model vllm --model_args pretrained=/home/mgoin/code/Nemotron-4-340B-Base-hf-FP8,tensor_parallel_size=8,distributed_executor_backend="ray",max_model_len=4096,gpu_memory_utilization=0.6 --tasks truthfulqa_mc2 --num_fewshot 0 --batch_size 16
vllm (pretrained=/home/mgoin/code/Nemotron-4-340B-Base-hf-FP8,tensor_parallel_size=8,distributed_executor_backend=ray,max_model_len=4096,gpu_memory_utilization=0.6), gen_kwargs: (None), limit: None, num_fewshot: 0, batch_size: 16
| Tasks |Version|Filter|n-shot|Metric| |Value | |Stderr|
|--------------|------:|------|-----:|------|---|-----:|---|-----:|
|truthfulqa_mc2| 2|none | 0|acc |↑ |0.4869|± |0.0142|
lm_eval --model vllm --model_args pretrained=/home/mgoin/code/Nemotron-4-340B-Base-hf-FP8,tensor_parallel_size=8,distributed_executor_backend="ray",max_model_len=4096,gpu_memory_utilization=0.6 --tasks winogrande --num_fewshot 5 --batch_size 16
vllm (pretrained=/home/mgoin/code/Nemotron-4-340B-Base-hf-FP8,tensor_parallel_size=8,distributed_executor_backend=ray,max_model_len=4096,gpu_memory_utilization=0.6), gen_kwargs: (None), limit: None, num_fewshot: 5, batch_size: 16
| Tasks |Version|Filter|n-shot|Metric| |Value | |Stderr|
|----------|------:|------|-----:|------|---|-----:|---|-----:|
|winogrande| 1|none | 5|acc |↑ |0.8887|± |0.0088|
The original paper evals for reference:
The Minitron paper has more evals as well:
Runs of mgoin Nemotron-4-340B-Base-hf on huggingface.co
5
Total runs
0
24-hour runs
-2
3-day runs
-1
7-day runs
-2
30-day runs
More Information About Nemotron-4-340B-Base-hf huggingface.co Model
Url of Nemotron-4-340B-Base-hf
Nemotron-4-340B-Base-hf huggingface.co Url
Provider of Nemotron-4-340B-Base-hf huggingface.co
Other API from mgoin
Updated:April 19 2024
Total runs:
457
Run Growth:
207
Growth Rate:
45.30%
Total runs:
201
Run Growth:
-240
Growth Rate:
-119.40%
Total runs:
199
Run Growth:
76
Growth Rate:
38.97%
Total runs:
100
Run Growth:
34
Growth Rate:
34.00%
Total runs:
87
Run Growth:
33
Growth Rate:
37.93%
Total runs:
66
Run Growth:
24
Growth Rate:
36.92%
Total runs:
64
Run Growth:
26
Growth Rate:
40.63%
Total runs:
64
Run Growth:
0
Growth Rate:
0.00%
Total runs:
59
Run Growth:
0
Growth Rate:
0.00%
Total runs:
58
Run Growth:
-28
Growth Rate:
-49.12%
Total runs:
51
Run Growth:
38
Growth Rate:
74.51%
Total runs:
48
Run Growth:
34
Growth Rate:
70.83%
Total runs:
45
Run Growth:
6
Growth Rate:
13.33%
Total runs:
40
Run Growth:
-51
Growth Rate:
-127.50%
Total runs:
31
Run Growth:
22
Growth Rate:
70.97%
Total runs:
26
Run Growth:
-13
Growth Rate:
-50.00%
Total runs:
25
Run Growth:
-7
Growth Rate:
-28.00%
Total runs:
24
Run Growth:
0
Growth Rate:
0.00%
Total runs:
24
Run Growth:
22
Growth Rate:
91.67%
Total runs:
23
Run Growth:
11
Growth Rate:
47.83%
Total runs:
23
Run Growth:
12
Growth Rate:
52.17%
Total runs:
22
Run Growth:
3
Growth Rate:
13.64%
Total runs:
22
Run Growth:
12
Growth Rate:
54.55%
Total runs:
21
Run Growth:
-2
Growth Rate:
-9.52%
Total runs:
19
Run Growth:
5
Growth Rate:
26.32%
Total runs:
17
Run Growth:
8
Growth Rate:
47.06%
Total runs:
16
Run Growth:
10
Growth Rate:
62.50%
Total runs:
14
Run Growth:
0
Growth Rate:
0.00%
Total runs:
14
Run Growth:
-10
Growth Rate:
-71.43%
Total runs:
14
Run Growth:
-3
Growth Rate:
-21.43%
Total runs:
14
Run Growth:
2
Growth Rate:
14.29%
Total runs:
14
Run Growth:
4
Growth Rate:
28.57%
Total runs:
13
Run Growth:
0
Growth Rate:
0.00%
Total runs:
13
Run Growth:
12
Growth Rate:
92.31%
Total runs:
13
Run Growth:
2
Growth Rate:
15.38%
Total runs:
12
Run Growth:
6
Growth Rate:
50.00%
Total runs:
12
Run Growth:
1
Growth Rate:
8.33%
Total runs:
12
Run Growth:
1
Growth Rate:
8.33%
Total runs:
11
Run Growth:
1
Growth Rate:
9.09%
Total runs:
11
Run Growth:
10
Growth Rate:
90.91%
Total runs:
10
Run Growth:
5
Growth Rate:
50.00%
Total runs:
10
Run Growth:
10
Growth Rate:
100.00%
Total runs:
10
Run Growth:
3
Growth Rate:
30.00%
Total runs:
10
Run Growth:
9
Growth Rate:
90.00%
Total runs:
10
Run Growth:
-5
Growth Rate:
-50.00%
Total runs:
10
Run Growth:
9
Growth Rate:
90.00%
Total runs:
10
Run Growth:
6
Growth Rate:
60.00%
Total runs:
9
Run Growth:
8
Growth Rate:
88.89%
Total runs:
9
Run Growth:
6
Growth Rate:
66.67%
Total runs:
9
Run Growth:
-3
Growth Rate:
-33.33%
Total runs:
8
Run Growth:
7
Growth Rate:
87.50%
Total runs:
8
Run Growth:
5
Growth Rate:
62.50%
Total runs:
7
Run Growth:
-1
Growth Rate:
-14.29%
Total runs:
7
Run Growth:
6
Growth Rate:
85.71%
Total runs:
7
Run Growth:
5
Growth Rate:
71.43%
Total runs:
7
Run Growth:
6
Growth Rate:
85.71%
Total runs:
7
Run Growth:
-2
Growth Rate:
-28.57%
Total runs:
6
Run Growth:
-2
Growth Rate:
-33.33%
Total runs:
6
Run Growth:
1
Growth Rate:
16.67%
Total runs:
6
Run Growth:
6
Growth Rate:
100.00%
Total runs:
6
Run Growth:
6
Growth Rate:
100.00%
Total runs:
5
Run Growth:
2
Growth Rate:
40.00%
Total runs:
5
Run Growth:
-4
Growth Rate:
-80.00%
Total runs:
5
Run Growth:
0
Growth Rate:
0.00%
Total runs:
5
Run Growth:
3
Growth Rate:
60.00%
Total runs:
5
Run Growth:
3
Growth Rate:
60.00%
Total runs:
5
Run Growth:
3
Growth Rate:
60.00%
Total runs:
5
Run Growth:
3
Growth Rate:
60.00%
Total runs:
5
Run Growth:
2
Growth Rate:
40.00%
Total runs:
5
Run Growth:
4
Growth Rate:
80.00%
Total runs:
5
Run Growth:
4
Growth Rate:
80.00%
Total runs:
4
Run Growth:
0
Growth Rate:
0.00%
Total runs:
4
Run Growth:
4
Growth Rate:
100.00%
Total runs:
4
Run Growth:
4
Growth Rate:
100.00%
Total runs:
4
Run Growth:
3
Growth Rate:
75.00%
Total runs:
4
Run Growth:
4
Growth Rate:
100.00%
Total runs:
4
Run Growth:
0
Growth Rate:
0.00%
Total runs:
3
Run Growth:
2
Growth Rate:
66.67%
Total runs:
3
Run Growth:
1
Growth Rate:
33.33%
Total runs:
3
Run Growth:
3
Growth Rate:
100.00%