Total runs:
9.0K
Run Growth:
304
Growth Rate:
3.36%
Sponsored by
Vmake Labs Video Enhancer
abacusai / Llama-3-Giraffe-70B
Model's Last Updated: May 01 2024
text-generation
Introduction of Llama-3-Giraffe-70B
Model Details of Llama-3-Giraffe-70B
Llama-3-Giraffe-70B
Abacus.AI presents our longer-necked variant of Llama 3 70B!
This model has an effective context length of approximately 128k.
We have currently trained on ~1B tokens. This is an initial release and we are hoping to improve the heatmap below further as we continue training.
Training Methodology
The methodology for training uses PoSE and dynamic-NTK interpolation.
NTK-scaling
The scale factor for NTK is 4. Note that we also tried theta-scaling but this did not work as well as NTK scaling in our experiments.
PoSE
We utilise Positional Skip-wise Training (PoSE) with the following parameters:
- Number of Chunks : 5
- Max position ID : 32768
Data
We use on average ~8K long samples from RedPajama .
Hardware
We train on 8xH100 GPUs with Deepspeed Zero Stage 3.
Evaluation Methodology
We use the EasyContext implementation of Needle-in-a-Haystack to evaluate Llama-3-Giraffe-70B.
We evaluate with the following parameters:
- Min context length : 2000
- Max context length : 128000
- Context interval : 4000
- Depth interval : 0.1
- Num samples : 2
- Rnd number digits : 7
- Haystack dir : PaulGrahamEssays
Runs of abacusai Llama-3-Giraffe-70B on huggingface.co
37
Total runs
0
24-hour runs
2
3-day runs
3
7-day runs
6
30-day runs
More Information About Llama-3-Giraffe-70B huggingface.co Model
Url of Llama-3-Giraffe-70B
Llama-3-Giraffe-70B huggingface.co Url
Provider of Llama-3-Giraffe-70B huggingface.co
Other API from abacusai
Updated:September 09 2024
Total runs:
8.8K
Run Growth:
-73
Growth Rate:
-0.83%
Total runs:
8.7K
Run Growth:
-47
Growth Rate:
-0.54%
Total runs:
8.7K
Run Growth:
-40
Growth Rate:
-0.46%
Total runs:
8.6K
Run Growth:
100
Growth Rate:
1.16%
Total runs:
8.6K
Run Growth:
32
Growth Rate:
0.37%
Total runs:
2.9K
Run Growth:
-12.8K
Growth Rate:
-444.20%
Total runs:
932
Run Growth:
32
Growth Rate:
80.00%
Total runs:
98
Run Growth:
-36
Growth Rate:
-39.56%
Total runs:
97
Run Growth:
2
Growth Rate:
2.06%
Total runs:
96
Run Growth:
12
Growth Rate:
12.50%
Total runs:
96
Run Growth:
16
Growth Rate:
16.67%
Total runs:
90
Run Growth:
-14
Growth Rate:
-18.42%
Total runs:
90
Run Growth:
-36
Growth Rate:
-40.00%
Total runs:
82
Run Growth:
-43
Growth Rate:
-52.44%
Total runs:
79
Run Growth:
-75
Growth Rate:
-108.70%
Total runs:
74
Run Growth:
2
Growth Rate:
2.70%
Total runs:
71
Run Growth:
0
Growth Rate:
0.00%
Total runs:
64
Run Growth:
-7
Growth Rate:
-10.94%
Total runs:
61
Run Growth:
28
Growth Rate:
45.90%
Total runs:
59
Run Growth:
-4
Growth Rate:
-6.78%
Total runs:
59
Run Growth:
-9
Growth Rate:
-15.25%
Total runs:
54
Run Growth:
19
Growth Rate:
35.19%
Total runs:
38
Run Growth:
-8
Growth Rate:
-21.62%
Total runs:
34
Run Growth:
-2
Growth Rate:
-7.14%
Total runs:
23
Run Growth:
-6
Growth Rate:
-26.09%
Total runs:
20
Run Growth:
-3
Growth Rate:
-15.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%