Total runs:
367.7K
Run Growth:
-156.0K
Growth Rate:
-41.48%
Sponsored by
YouArt
NousResearch / Yarn-Mistral-7b-64k
Model's Last Updated: November 03 2023
text-generation
Introduction of Yarn-Mistral-7b-64k
Model Details of Yarn-Mistral-7b-64k
Model Card: Nous-Yarn-Mistral-7b-64k
Model Description
Nous-Yarn-Mistral-7b-64k is a state-of-the-art language model for long context, further pretrained on long context data for 1000 steps using the YaRN extension method. It is an extension of Mistral-7B-v0.1 and supports a 64k token context window.
To use, pass
trust_remote_code=True
when loading the model, for example
model = AutoModelForCausalLM.from_pretrained("NousResearch/Yarn-Mistral-7b-64k",
use_flash_attention_2=True,
torch_dtype=torch.bfloat16,
device_map="auto",
trust_remote_code=True)
In addition you will need to use the latest version of
transformers
(until 4.35 comes out)
pip install git+https://github.com/huggingface/transformers
Benchmarks
Long context benchmarks:
| Model | Context Window | 8k PPL | 16k PPL | 32k PPL | 64k PPL | 128k PPL |
|---|---|---|---|---|---|---|
| Mistral-7B-v0.1 | 8k | 2.96 | - | - | - | - |
| Yarn-Mistral-7b-64k | 64k | 3.04 | 2.65 | 2.44 | 2.20 | - |
| Yarn-Mistral-7b-128k | 128k | 3.08 | 2.68 | 2.47 | 2.24 | 2.19 |
Short context benchmarks showing that quality degradation is minimal:
| Model | Context Window | ARC-c | Hellaswag | MMLU | Truthful QA |
|---|---|---|---|---|---|
| Mistral-7B-v0.1 | 8k | 59.98 | 83.31 | 64.16 | 42.15 |
| Yarn-Mistral-7b-64k | 64k | 59.38 | 81.21 | 61.32 | 42.50 |
| Yarn-Mistral-7b-128k | 128k | 58.87 | 80.58 | 60.64 | 42.46 |
Collaborators
- bloc97 : Methods, paper and evals
- @theemozilla : Methods, paper, model training, and evals
- @EnricoShippole : Model training
- honglu2875 : Paper and evals
The authors would like to thank LAION AI for their support of compute for this model. It was trained on the JUWELS supercomputer.
Runs of NousResearch Yarn-Mistral-7b-64k on huggingface.co
1.8K
Total runs
0
24-hour runs
0
3-day runs
65
7-day runs
446
30-day runs
More Information About Yarn-Mistral-7b-64k huggingface.co Model
Url of Yarn-Mistral-7b-64k
Yarn-Mistral-7b-64k huggingface.co Url
Provider of Yarn-Mistral-7b-64k huggingface.co
Other API from NousResearch
Updated:September 08 2024
Total runs:
227.4K
Run Growth:
20.5K
Growth Rate:
9.02%
Total runs:
157.9K
Run Growth:
67.6K
Growth Rate:
42.80%
Total runs:
134.2K
Run Growth:
98.1K
Growth Rate:
73.10%
Total runs:
49.1K
Run Growth:
6.6K
Growth Rate:
13.52%
Total runs:
44.1K
Run Growth:
794
Growth Rate:
1.77%
Total runs:
40.3K
Run Growth:
16.0K
Growth Rate:
39.83%
Total runs:
30.1K
Run Growth:
-1.8K
Growth Rate:
-6.02%
Total runs:
26.9K
Run Growth:
17.3K
Growth Rate:
62.30%
Total runs:
16.0K
Run Growth:
1.6K
Growth Rate:
9.76%
Total runs:
13.8K
Run Growth:
-23.8K
Growth Rate:
-186.94%
Total runs:
10.7K
Run Growth:
-2.0K
Growth Rate:
-18.93%
Total runs:
10.5K
Run Growth:
-1.0K
Growth Rate:
-9.61%
Total runs:
9.5K
Run Growth:
-181
Growth Rate:
-1.90%
Total runs:
8.9K
Run Growth:
317
Growth Rate:
3.56%
Total runs:
7.5K
Run Growth:
4.2K
Growth Rate:
55.33%
Total runs:
7.1K
Run Growth:
2.5K
Growth Rate:
43.75%
Total runs:
7.0K
Run Growth:
3.4K
Growth Rate:
48.20%
Total runs:
6.5K
Run Growth:
-4.2K
Growth Rate:
-64.55%
Total runs:
6.0K
Run Growth:
1.5K
Growth Rate:
25.24%
Total runs:
5.1K
Run Growth:
1.3K
Growth Rate:
25.51%
Total runs:
5.1K
Run Growth:
-2.6K
Growth Rate:
-51.69%
Total runs:
4.6K
Run Growth:
354
Growth Rate:
7.69%
Total runs:
4.2K
Run Growth:
3.0K
Growth Rate:
69.83%
Total runs:
3.6K
Run Growth:
2.0K
Growth Rate:
55.06%
Total runs:
3.5K
Run Growth:
-1.5K
Growth Rate:
-43.63%
Total runs:
3.5K
Run Growth:
1.0K
Growth Rate:
29.24%
Total runs:
3.4K
Run Growth:
-3.4K
Growth Rate:
-99.56%
Total runs:
3.3K
Run Growth:
-14.8K
Growth Rate:
-444.81%
Total runs:
3.0K
Run Growth:
-684
Growth Rate:
-22.96%
Total runs:
2.9K
Run Growth:
-6.1K
Growth Rate:
-204.18%
Total runs:
2.6K
Run Growth:
-470
Growth Rate:
-17.29%
Total runs:
2.6K
Run Growth:
-2.7K
Growth Rate:
-105.86%
Total runs:
2.5K
Run Growth:
947
Growth Rate:
37.86%
Total runs:
2.3K
Run Growth:
1.3K
Growth Rate:
56.85%
Total runs:
2.2K
Run Growth:
594
Growth Rate:
26.23%
Total runs:
2.2K
Run Growth:
922
Growth Rate:
42.57%
Total runs:
2.1K
Run Growth:
-1.5K
Growth Rate:
-72.53%
Total runs:
2.1K
Run Growth:
-475
Growth Rate:
-20.53%
Total runs:
2.0K
Run Growth:
467
Growth Rate:
23.80%
Total runs:
1.7K
Run Growth:
-1.3K
Growth Rate:
-75.96%
Total runs:
1.7K
Run Growth:
-174
Growth Rate:
-10.31%
Total runs:
1.7K
Run Growth:
176
Growth Rate:
10.54%
Total runs:
1.6K
Run Growth:
-915
Growth Rate:
-56.52%
Total runs:
1.5K
Run Growth:
640
Growth Rate:
41.34%
Total runs:
1.5K
Run Growth:
777
Growth Rate:
50.78%
Total runs:
1.3K
Run Growth:
577
Growth Rate:
42.77%
Total runs:
1.2K
Run Growth:
485
Growth Rate:
42.14%
Total runs:
1.1K
Run Growth:
155
Growth Rate:
14.38%
Total runs:
1.1K
Run Growth:
-603
Growth Rate:
-54.37%
Total runs:
1.1K
Run Growth:
-207
Growth Rate:
-19.55%
Total runs:
923
Run Growth:
121
Growth Rate:
13.11%
Total runs:
864
Run Growth:
8
Growth Rate:
0.91%
Total runs:
847
Run Growth:
180
Growth Rate:
21.25%
Total runs:
803
Run Growth:
126
Growth Rate:
15.69%
Total runs:
686
Run Growth:
-2.3K
Growth Rate:
-340.32%
Total runs:
616
Run Growth:
37
Growth Rate:
6.01%
Total runs:
536
Run Growth:
233
Growth Rate:
44.81%
Total runs:
462
Run Growth:
-634
Growth Rate:
-137.23%
Total runs:
436
Run Growth:
31
Growth Rate:
7.11%
Total runs:
419
Run Growth:
100
Growth Rate:
23.87%
Total runs:
410
Run Growth:
-234
Growth Rate:
-57.07%
Total runs:
356
Run Growth:
62
Growth Rate:
19.44%
Total runs:
354
Run Growth:
-52
Growth Rate:
-18.06%
Total runs:
352
Run Growth:
-6.4K
Growth Rate:
-1797.18%
Total runs:
332
Run Growth:
79
Growth Rate:
23.80%
Total runs:
332
Run Growth:
46
Growth Rate:
13.86%
Total runs:
315
Run Growth:
101
Growth Rate:
32.06%
Total runs:
312
Run Growth:
143
Growth Rate:
45.83%
Total runs:
309
Run Growth:
65
Growth Rate:
21.04%
Total runs:
309
Run Growth:
287
Growth Rate:
92.88%
Total runs:
307
Run Growth:
-25
Growth Rate:
-8.14%
Total runs:
303
Run Growth:
103
Growth Rate:
33.99%
Total runs:
293
Run Growth:
62
Growth Rate:
21.16%
Total runs:
278
Run Growth:
-82
Growth Rate:
-29.50%
Total runs:
223
Run Growth:
-42
Growth Rate:
-18.83%
Total runs:
210
Run Growth:
47
Growth Rate:
22.82%
Total runs:
190
Run Growth:
31
Growth Rate:
16.32%
Total runs:
188
Run Growth:
-162
Growth Rate:
-88.52%
Total runs:
185
Run Growth:
-16
Growth Rate:
-8.74%
Total runs:
177
Run Growth:
43
Growth Rate:
24.29%
Total runs:
155
Run Growth:
-27
Growth Rate:
-17.53%
Total runs:
153
Run Growth:
21
Growth Rate:
13.73%
Total runs:
151
Run Growth:
26
Growth Rate:
17.22%
Total runs:
148
Run Growth:
52
Growth Rate:
35.14%
Total runs:
145
Run Growth:
-11
Growth Rate:
-7.59%
Total runs:
137
Run Growth:
25
Growth Rate:
18.25%
Total runs:
133
Run Growth:
20
Growth Rate:
15.04%
Total runs:
133
Run Growth:
1
Growth Rate:
0.78%
Total runs:
132
Run Growth:
37
Growth Rate:
28.91%
Total runs:
128
Run Growth:
32
Growth Rate:
25.00%
Total runs:
125
Run Growth:
37
Growth Rate:
30.33%
Total runs:
122
Run Growth:
25
Growth Rate:
21.37%
Total runs:
111
Run Growth:
-58
Growth Rate:
-52.25%
Total runs:
110
Run Growth:
2
Growth Rate:
1.82%
Total runs:
89
Run Growth:
0
Growth Rate:
0.00%
Total runs:
88
Run Growth:
6
Growth Rate:
6.82%