Total runs:
341
Run Growth:
-435
Growth Rate:
-127.57%
Sponsored by
ZenMux
fblgit / UNA-SimpleSmaug-34b-v1beta
Model's Last Updated: August 13 2024
text-generation
Introduction of UNA-SimpleSmaug-34b-v1beta
Model Details of UNA-SimpleSmaug-34b-v1beta
UNA-SimpleSmaug-34b-v1beta
Scoring 04-February-2024 #1 34B model, outperforming its original base model Smaug-34B-v0.1 with
77.41
😎
Oh, btw.. this one went thru SFT so the abacus inside Smaug is back to normal.. so you can further train/dpo him .. RESET!..
UPDATES March : Stills undisputed 34B King Smaug 70B stills undisputed 70B King
==== And people wonders.. why there is no UNA of Hermes or Smaug 70B? << i dont think is worth the time to spend on a model that is widely known for not being too useful, likely UNA can fix some of the internal mess.. for Hermes, we spoke chitchat quick a couple times but nothing solid, but we would like to make a reborn of excellent models using UNA, just liek we did with UNA-Dolphin where we saw relevant performance is short time.
Applied UNA only on the Attention, not on the MLP's
- Is based on Smaug
- SimpleMath dataset
- It was trained on Axolotl
Experiment
The thing here is to understand whats the impact of SimpleMath applied at the attention layer during a SFT session and how it impacts on the neural network overall.
Results: Improving mathematican and reasoning capabilities without degrading and presserving previous training sessions.
And enjoy our ModelSimilarities tool detector https://github.com/fblgit/model-similarity where we confirmed numerically the bloodties of the model.
Evals
| Metric | Value |
|---|---|
| Avg. | 77.41 |
| AI2 Reasoning Challenge (25-Shot) | 74.57 |
| HellaSwag (10-Shot) | 86.74 |
| MMLU (5-Shot) | 76.68 |
| TruthfulQA (0-shot) | 70.17 |
| Winogrande (5-shot) | 83.82 |
| GSM8k (5-shot) | 72.48 |
| Task |Version| Metric |Value |
|-------------|------:|--------|----------------:|
|arc_challenge| HF|acc_norm| 0.7457337883959 |
|gsm8k | HF|acc | 0.7247915087187 |
|mmlu | HF|acc | 0.7649553475572 |
|mmlu | HF|acc_norm| 0.7681713551647 |
|hellaswag | HF|acc_norm| 0.8673571001792 |
|truthfulqa | HF|mc2 | 0.7016557407771 |
|winogrande | HF|acc | 0.8382004735595 |
|------------------------------------------------|
Increasing GSM, MMLU, ARC, WINO.
Citations
To abacusai for making Smaug-34B, the Bagel, and all the magic behind the base model.
If you use the model, provide citation even for merges or anything.
Open LLM Leaderboard Evaluation Results
Detailed results can be found here
Runs of fblgit UNA-SimpleSmaug-34b-v1beta on huggingface.co
7.8K
Total runs
13
24-hour runs
20
3-day runs
107
7-day runs
9
30-day runs
More Information About UNA-SimpleSmaug-34b-v1beta huggingface.co Model
Url of UNA-SimpleSmaug-34b-v1beta
UNA-SimpleSmaug-34b-v1beta huggingface.co Url
Provider of UNA-SimpleSmaug-34b-v1beta huggingface.co
Other API from fblgit
Updated:March 08 2024
Total runs:
301
Run Growth:
-472
Growth Rate:
-156.81%
Total runs:
298
Run Growth:
-496
Growth Rate:
-166.44%
Total runs:
292
Run Growth:
-396
Growth Rate:
-135.62%
Total runs:
284
Run Growth:
-498
Growth Rate:
-175.35%
Total runs:
276
Run Growth:
-501
Growth Rate:
-181.52%
Total runs:
274
Run Growth:
-466
Growth Rate:
-170.07%
Total runs:
245
Run Growth:
-494
Growth Rate:
-201.63%
Total runs:
239
Run Growth:
-500
Growth Rate:
-209.21%
Total runs:
173
Run Growth:
107
Growth Rate:
61.85%
Total runs:
137
Run Growth:
26
Growth Rate:
18.98%
Total runs:
50
Run Growth:
16
Growth Rate:
32.00%
Total runs:
36
Run Growth:
-16
Growth Rate:
-44.44%
Total runs:
22
Run Growth:
18
Growth Rate:
81.82%
Total runs:
22
Run Growth:
15
Growth Rate:
68.18%
Total runs:
20
Run Growth:
9
Growth Rate:
45.00%
Total runs:
17
Run Growth:
-2
Growth Rate:
-11.76%
Total runs:
6
Run Growth:
3
Growth Rate:
50.00%
Total runs:
2
Run Growth:
2
Growth Rate:
100.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%