Total runs:
3.4M
Run Growth:
-154.9K
Growth Rate:
-4.59%
Sponsored by
APIMaster
EleutherAI / llemma_7b
Model's Last Updated: February 19 2024
text-generation
Introduction of llemma_7b
Model Details of llemma_7b
ArXiv | Models | Data | Code | Blog | Sample Explorer
Zhangir Azerbayev , Hailey Schoelkopf , Keiran Paster , Marco Dos Santos , Stephen McAleer , Albert Q. Jiang , Jia Deng , Stella Biderman , Sean Welleck
Llemma 7B is a language model for mathematics. It was initialized with Code Llama 7B weights, and trained on the Proof-Pile-2 for 200B tokens.
This model also comes in a 34B parameter version: Llemma 34B .
Evaluations
Llemma models are particularly strong at chain-of-thought mathematical reasoning and using computational tools for mathematics, such as Python and formal theorem provers.
Chain-of-thought Math
On chain-of-thought mathematics tasks, Llemma models outperform Llama-2, Code Llama, and when controlled for model size, outperform Minerva.
| Model | Size | GSM8k | OCW | MMLU-STEM | SAT | MATH |
|---|---|---|---|---|---|---|
| Llama 2 | 7B | 11.8% | 3.7% | 29.9% | 25% | 3.2% |
| Code Llama | 7B | 10.5% | 4.4% | 25.1% | 9.4% | 4.5% |
| LLEMMA | 7B | 36.4% | 7.7% | 37.7% | 53.1% | 18.0% |
| Minerva | 8B | 16.2% | 7.7% | 35.6% | - | 14.1% |
| ------------ | ------ | -------- | ------- | ----------- | ------- | ------- |
| Code Llama | 34B | 29.6% | 7.0% | 40.5% | 40.6% | 12.2% |
| LLEMMA | 34B | 51.5% | 11.8% | 49.0% | 71.9% | 25.0% |
| ------------ | ------ | -------- | ------- | ----------- | ------- | ------- |
| Minerva | 62B | 52.4% | 12.0% | 53.9% | - | 27.6% |
| Minerva | 540B | 58.8% | 17.6% | 63.9% | - | 33.6% |
Further performance can be extracted by using majority voting:
| Model | Size | GSM8k maj@100 | OCW maj@100 | MMLU-STEM maj@16 | SAT maj@16 | MATH maj@256 |
|---|---|---|---|---|---|---|
| LLEMMA | 7B | 54.0% | 14.3% | 49.9% | 78.1% | 33.5 |
| Minerva | 8B | 28.4% | 12.5% | 43.4% | - | 25.4% |
| --------- | ------ | ------------- | ----------- | ----------------- | ----------- | ------------ |
| LLEMMA | 34B | 69.3% | 18.4% | 59.7% | 81.3% | 43.1% |
| --------- | ------ | ------------- | ----------- | ----------------- | ----------- | ------------ |
| Minerva | 62B | 68.5% | 23.5% | 63.5% | - | 43.4% |
| Minerva | 540B | 78.5% | 30.8% | 75.0% | - | 50.3% |
Tool Use and Theorem Proving
In addition to chain-of-thought reasoning, Llemma has strong capabilities in computational mathematics tasks. For tool use and formal theorem proving evaluations, see our paper .
Citation
@misc{azerbayev2023llemma,
title={Llemma: An Open Language Model For Mathematics},
author={Zhangir Azerbayev and Hailey Schoelkopf and Keiran Paster and Marco Dos Santos and Stephen McAleer and Albert Q. Jiang and Jia Deng and Stella Biderman and Sean Welleck},
year={2023},
eprint={2310.10631},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
Runs of EleutherAI llemma_7b on huggingface.co
1.1K
Total runs
0
24-hour runs
12
3-day runs
90
7-day runs
251
30-day runs
More Information About llemma_7b huggingface.co Model
Url of llemma_7b
llemma_7b huggingface.co Url
Provider of llemma_7b huggingface.co
Other API from EleutherAI
Updated:July 09 2023
Total runs:
1.3M
Run Growth:
453.0K
Growth Rate:
34.70%
Total runs:
740.1K
Run Growth:
678.3K
Growth Rate:
91.66%
Total runs:
597.8K
Run Growth:
111.3K
Growth Rate:
18.90%
Total runs:
523.7K
Run Growth:
12.0K
Growth Rate:
2.28%
Total runs:
504.9K
Run Growth:
-267.2K
Growth Rate:
-51.52%
Total runs:
252.2K
Run Growth:
55.8K
Growth Rate:
21.56%
Total runs:
142.5K
Run Growth:
56.9K
Growth Rate:
41.04%
Total runs:
137.5K
Run Growth:
-589.5K
Growth Rate:
-428.65%
Total runs:
60.5K
Run Growth:
7.0K
Growth Rate:
11.49%
Total runs:
53.3K
Run Growth:
-10.3K
Growth Rate:
-19.43%
Total runs:
45.7K
Run Growth:
-813
Growth Rate:
-1.79%
Total runs:
35.6K
Run Growth:
-8.8K
Growth Rate:
-25.17%
Total runs:
27.4K
Run Growth:
10.3K
Growth Rate:
37.66%
Total runs:
22.5K
Run Growth:
-8.4K
Growth Rate:
-37.54%
Total runs:
20.9K
Run Growth:
-2.5K
Growth Rate:
-11.82%
Total runs:
17.1K
Run Growth:
-8.0K
Growth Rate:
-46.67%
Total runs:
15.7K
Run Growth:
-6.6K
Growth Rate:
-40.81%
Total runs:
15.3K
Run Growth:
-19.1K
Growth Rate:
-120.99%
Total runs:
9.5K
Run Growth:
4.7K
Growth Rate:
49.16%
Total runs:
9.1K
Run Growth:
-31.5K
Growth Rate:
-341.57%
Total runs:
8.9K
Run Growth:
-54.9K
Growth Rate:
-625.00%
Total runs:
7.8K
Run Growth:
-498
Growth Rate:
-6.35%
Total runs:
7.0K
Run Growth:
-31.2K
Growth Rate:
-448.61%
Total runs:
6.4K
Run Growth:
5.7K
Growth Rate:
90.50%
Total runs:
5.6K
Run Growth:
4.0K
Growth Rate:
26.38%
Total runs:
4.8K
Run Growth:
-5.7K
Growth Rate:
-121.64%
Total runs:
4.1K
Run Growth:
-2.2K
Growth Rate:
-55.20%
Total runs:
3.7K
Run Growth:
-1.3K
Growth Rate:
-34.37%
Total runs:
3.5K
Run Growth:
-11.9K
Growth Rate:
-346.73%
Total runs:
3.2K
Run Growth:
-12.2K
Growth Rate:
-385.40%
Total runs:
3.0K
Run Growth:
416
Growth Rate:
14.55%
Total runs:
3.0K
Run Growth:
-2.2K
Growth Rate:
-77.47%
Total runs:
3.0K
Run Growth:
-1.1K
Growth Rate:
-37.40%
Total runs:
2.8K
Run Growth:
-13.0K
Growth Rate:
-465.52%
Total runs:
2.8K
Run Growth:
2.4K
Growth Rate:
86.78%
Total runs:
2.6K
Run Growth:
642
Growth Rate:
25.78%
Total runs:
2.5K
Run Growth:
-2.6K
Growth Rate:
-104.84%
Total runs:
2.4K
Run Growth:
649
Growth Rate:
27.52%
Total runs:
2.3K
Run Growth:
1.0K
Growth Rate:
46.26%
Total runs:
2.3K
Run Growth:
-3.0K
Growth Rate:
-130.16%
Total runs:
2.3K
Run Growth:
-3.1K
Growth Rate:
-135.19%
Total runs:
2.2K
Run Growth:
-2.8K
Growth Rate:
-136.19%
Total runs:
2.2K
Run Growth:
-2.0K
Growth Rate:
-92.77%
Total runs:
2.1K
Run Growth:
-2.8K
Growth Rate:
-138.97%
Total runs:
2.1K
Run Growth:
904
Growth Rate:
43.73%
Total runs:
2.1K
Run Growth:
767
Growth Rate:
38.37%
Total runs:
2.0K
Run Growth:
-454
Growth Rate:
-23.44%
Total runs:
1.9K
Run Growth:
-3.9K
Growth Rate:
-209.07%
Total runs:
1.8K
Run Growth:
610
Growth Rate:
35.69%
Total runs:
1.8K
Run Growth:
657
Growth Rate:
38.58%
Total runs:
1.7K
Run Growth:
-59
Growth Rate:
-3.45%
Total runs:
1.6K
Run Growth:
780
Growth Rate:
47.47%
Total runs:
1.6K
Run Growth:
-2.7K
Growth Rate:
-168.04%
Total runs:
1.5K
Run Growth:
-55.4K
Growth Rate:
-3546.38%
Total runs:
1.5K
Run Growth:
606
Growth Rate:
39.30%
Total runs:
1.5K
Run Growth:
575
Growth Rate:
37.98%
Total runs:
1.5K
Run Growth:
-133
Growth Rate:
-9.00%
Total runs:
1.5K
Run Growth:
-53
Growth Rate:
-3.60%
Total runs:
1.5K
Run Growth:
302
Growth Rate:
21.62%
Total runs:
1.4K
Run Growth:
283
Growth Rate:
19.85%
Total runs:
1.4K
Run Growth:
-4.9K
Growth Rate:
-372.68%
Total runs:
1.4K
Run Growth:
470
Growth Rate:
34.21%
Total runs:
1.3K
Run Growth:
556
Growth Rate:
41.31%
Total runs:
1.3K
Run Growth:
528
Growth Rate:
39.31%
Total runs:
1.3K
Run Growth:
-22.6K
Growth Rate:
-1804.47%
Total runs:
1.3K
Run Growth:
978
Growth Rate:
77.56%
Total runs:
1.2K
Run Growth:
259
Growth Rate:
21.42%
Total runs:
1.2K
Run Growth:
-768
Growth Rate:
-64.70%
Total runs:
1.1K
Run Growth:
-134
Growth Rate:
-11.85%
Total runs:
1.1K
Run Growth:
-349
Growth Rate:
-31.11%
Total runs:
1.1K
Run Growth:
-1.1K
Growth Rate:
-97.34%
Total runs:
1.0K
Run Growth:
273
Growth Rate:
26.61%
Total runs:
1.0K
Run Growth:
253
Growth Rate:
25.53%
Total runs:
990
Run Growth:
-11.7K
Growth Rate:
-1124.90%
Total runs:
963
Run Growth:
921
Growth Rate:
96.54%
Total runs:
877
Run Growth:
-7.2K
Growth Rate:
-776.75%
Total runs:
852
Run Growth:
104
Growth Rate:
12.21%
Total runs:
807
Run Growth:
-11.9K
Growth Rate:
-1548.70%
Total runs:
804
Run Growth:
388
Growth Rate:
47.49%
Total runs:
733
Run Growth:
-648
Growth Rate:
-87.80%
Total runs:
709
Run Growth:
-7.9K
Growth Rate:
-1156.24%
Total runs:
686
Run Growth:
-2.3K
Growth Rate:
-333.09%
Total runs:
684
Run Growth:
-88
Growth Rate:
-12.79%
Total runs:
673
Run Growth:
-129
Growth Rate:
-19.17%
Total runs:
657
Run Growth:
-43
Growth Rate:
-7.04%
Total runs:
645
Run Growth:
645
Growth Rate:
100.00%
Total runs:
608
Run Growth:
-66
Growth Rate:
-10.86%
Total runs:
607
Run Growth:
-365
Growth Rate:
-60.13%
Total runs:
606
Run Growth:
-34
Growth Rate:
-5.61%
Total runs:
603
Run Growth:
-132
Growth Rate:
-21.02%
Total runs:
595
Run Growth:
-89
Growth Rate:
-14.45%
Total runs:
591
Run Growth:
-32
Growth Rate:
-5.41%
Total runs:
589
Run Growth:
-19
Growth Rate:
-3.23%