Total runs:
8.7K
Run Growth:
496
Growth Rate:
5.68%
Sponsored by
APIMart
abacusai / Llama-3-Giraffe-70B-Instruct
Model's Last Updated: May 05 2024
text-generation
Introduction of Llama-3-Giraffe-70B-Instruct
Model Details of Llama-3-Giraffe-70B-Instruct
Llama-3-Giraffe-70B-Instruct
Abacus.AI presents our longer-necked variant of Llama 3 70B - now with the instruct variant!
This model has an effective context length of approximately 128k.
We have currently trained on ~1.5B tokens.
There are our Needle-in-a-Haystack heatmap results. We are conducting further evals of model efficacy and will update our model card as these come in:
MT-Bench Evaluation
We also measured performance on MT-Bench to verify that the context extension did not significantly impact performance on instruct tasks:
####### 1st turn:
Meta-Llama-3-70B-Instruct 9.21
Llama-3-Giraffe-70B-Instruct 9.19
####### 2nd turn:
Meta-Llama-3-70B-Instruct 2 8.80
Llama-3-Giraffe-70B-Instruct 2 8.54
####### average:
Meta-Llama-3-70B-Instruct 9.00
Llama-3-Giraffe-70B-Instruct 8.87
Training Methodology
The methodology for training uses PoSE and dynamic-NTK interpolation.
NTK-scaling
The scale factor for NTK is 4. Note that we also tried theta-scaling but this did not work as well as NTK scaling in our experiments.
PoSE
We utilise Positional Skip-wise Training (PoSE) with the following parameters:
- Number of Chunks : 5
- Max position ID : 32768
Data
We use on average ~8K long samples from RedPajama .
Hardware
We train on 8xH100 GPUs with Deepspeed Zero Stage 3.
Evaluation Methodology
We use the EasyContext implementation of Needle-in-a-Haystack to evaluate Llama-3-Giraffe-70B.
We evaluate with the following parameters:
- Min context length : 2000
- Max context length : 128000
- Context interval : 4000
- Depth interval : 0.1
- Num samples : 2
- Rnd number digits : 7
- Haystack dir : PaulGrahamEssays
Adapter Transfer
We apply the above techniques first to Llama-3-70B-Base, using LoRA on the Q and K weights only. This adapter is then applied to Llama-3-70B-Instruct, and we release the merged version here.
Runs of abacusai Llama-3-Giraffe-70B-Instruct on huggingface.co
25
Total runs
0
24-hour runs
1
3-day runs
5
7-day runs
15
30-day runs
More Information About Llama-3-Giraffe-70B-Instruct huggingface.co Model
Url of Llama-3-Giraffe-70B-Instruct
Llama-3-Giraffe-70B-Instruct huggingface.co Url
Provider of Llama-3-Giraffe-70B-Instruct huggingface.co
Other API from abacusai
Updated:February 22 2024
Total runs:
8.7K
Run Growth:
225
Growth Rate:
2.58%
Total runs:
8.6K
Run Growth:
519
Growth Rate:
6.02%
Total runs:
8.6K
Run Growth:
323
Growth Rate:
3.75%
Total runs:
8.5K
Run Growth:
372
Growth Rate:
4.38%
Total runs:
8.4K
Run Growth:
315
Growth Rate:
3.73%
Total runs:
300
Run Growth:
183
Growth Rate:
61.00%
Total runs:
154
Run Growth:
67
Growth Rate:
43.51%
Total runs:
125
Run Growth:
35
Growth Rate:
28.00%
Total runs:
123
Run Growth:
93
Growth Rate:
75.61%
Total runs:
123
Run Growth:
56
Growth Rate:
45.53%
Total runs:
103
Run Growth:
24
Growth Rate:
23.30%
Total runs:
86
Run Growth:
24
Growth Rate:
27.91%
Total runs:
86
Run Growth:
20
Growth Rate:
23.26%
Total runs:
81
Run Growth:
29
Growth Rate:
37.18%
Total runs:
75
Run Growth:
25
Growth Rate:
33.33%
Total runs:
74
Run Growth:
23
Growth Rate:
31.08%
Total runs:
71
Run Growth:
24
Growth Rate:
33.80%
Total runs:
69
Run Growth:
22
Growth Rate:
31.88%
Total runs:
68
Run Growth:
19
Growth Rate:
27.94%
Total runs:
44
Run Growth:
15
Growth Rate:
34.09%
Total runs:
37
Run Growth:
18
Growth Rate:
48.65%
Total runs:
34
Run Growth:
20
Growth Rate:
58.82%
Total runs:
30
Run Growth:
15
Growth Rate:
50.00%
Total runs:
25
Run Growth:
17
Growth Rate:
68.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%