Sponsored by
Vmake Labs Video Enhancer
Undi95 / Mistral-11B-OmniMix
Model's Last Updated: November 18 2023
text-generation
Introduction of Mistral-11B-OmniMix
Model Details of Mistral-11B-OmniMix
I FUCKED UP, THIS MODEL IS MEANT TO BE A BFLOAT16 MODEL, I'M CURRENTLY REDOING IT IN THE CORRECT WAY (look at the recipe, it end in float16, i'm so dumb lmao). It SHOULD be even better, I saw the problem after finetuning it, something was off. It's usable, it rank the best, but it's not even on the right float...KEK
Fixed model should be here: NeverSleep/Mistral-11B-OmniMix-bf16
Don't mind this one at the moment, I need to finetune it for RP, it's just a test.
Description
This repo contains fp16 files of Mistral-11B-OmniMix.
My goal for this model was only to make it score the highest possible with merge and layer toying, proving that:
- Benchmark are objective
- You should try a model yourself and don't go blindly to the highest rated one
- Merge/Layer toying CAN be usable to do better model (maybe?)
Model used
- Mistral-7B-OpenOrca
- Mistral-7B-v0.1-Open-Platypus
- CollectiveCognition-v1.1-Mistral-7B
- zephyr-7b-alpha
Prompt template
The best one after further testing is this one:
<|system|>
Below is an instruction that describes a task. Write a response that appropriately completes the request.
<|user|>
{prompt}
<|assistant|>
But these one work too:
Below is an instruction that describes a task. Write a response that appropriately completes the request.
### Instruction:
{prompt}
### Response:
USER: <prompt>
ASSISTANT:
Or use any prompting system from one of the 4 source model, should work.
The secret sauce
Mistral-11B-OpenOrcaPlatypus :
slices:
- sources:
- model: Open-Orca/Mistral-7B-OpenOrca
layer_range: [0, 24]
- sources:
- model: akjindal53244/Mistral-7B-v0.1-Open-Platypus
layer_range: [8, 32]
merge_method: passthrough
dtype: bfloat16
Mistral-11B-CC-Zephyr :
slices:
- sources:
- model: "/content/drive/MyDrive/CC-v1.1-7B-bf16"
layer_range: [0, 24]
- sources:
- model: "/content/drive/MyDrive/Zephyr-7B"
layer_range: [8, 32]
merge_method: passthrough
dtype: bfloat16
Mistral-11B-OmniMix :
slices:
- sources:
- model: Mistral-11B-OpenOrcaPlatypus
layer_range: [0, 48]
- model: Mistral-11B-CC-Zephyr
layer_range: [0, 48]
merge_method: slerp
base_model: Undi95/Mistral-11B-OpenOrcaPlatypus
parameters:
t:
- filter: lm_head
value: [0.75]
- filter: embed_tokens
value: [0.75]
- filter: self_attn
value: [0.75, 0.25]
- filter: mlp
value: [0.25, 0.75]
- filter: layernorm
value: [0.5, 0.5]
- filter: modelnorm
value: [0.75]
- value: 0.5 # fallback for rest of tensors
dtype: float16
I use mergekit for all the manipulation told here.
Some scoring I done myself
This was named "Mistral-11B-TestBench11", keep that in mind while looking trough this.
hf-causal-experimental (pretrained=/content/drive/MyDrive/Mistral-11B-Test), limit: None, provide_description: False, num_fewshot: 0, batch_size: 4
| Task | Version | Metric | Value | Stderr | |
|---|---|---|---|---|---|
| arc_challenge | 0 | acc | 0.5597 | ± | 0.0145 |
| acc_norm | 0.5819 | ± | 0.0144 | ||
| arc_easy | 0 | acc | 0.8308 | ± | 0.0077 |
| acc_norm | 0.8215 | ± | 0.0079 | ||
| hellaswag | 0 | acc | 0.6371 | ± | 0.0048 |
| acc_norm | 0.8213 | ± | 0.0038 | ||
| piqa | 0 | acc | 0.8134 | ± | 0.0091 |
| acc_norm | 0.8275 | ± | 0.0088 | ||
| truthfulqa_mc | 1 | mc1 | 0.3990 | ± | 0.0171 |
| mc2 | 0.5685 | ± | 0.0155 | ||
| winogrande | 0 | acc | 0.7474 | ± | 0.0122 |
This model seem to be the best out of my 3 latest try:
You can find all the work I have done trying on this Pastebin .
Others
Special thanks to Sushi, Henky for the machine he give me for big task, and Charles Goddard for his amazing tool.
If you want to support me, you can here .
Open LLM Leaderboard Evaluation Results
Detailed results can be found here
| Metric | Value |
|---|---|
| Avg. | 53.01 |
| ARC (25-shot) | 64.42 |
| HellaSwag (10-shot) | 83.93 |
| MMLU (5-shot) | 63.82 |
| TruthfulQA (0-shot) | 56.68 |
| Winogrande (5-shot) | 77.74 |
| GSM8K (5-shot) | 14.94 |
| DROP (3-shot) | 9.57 |
Runs of Undi95 Mistral-11B-OmniMix on huggingface.co
70
Total runs
0
24-hour runs
0
3-day runs
13
7-day runs
32
30-day runs
More Information About Mistral-11B-OmniMix huggingface.co Model
Url of Mistral-11B-OmniMix
Mistral-11B-OmniMix huggingface.co Url
Provider of Mistral-11B-OmniMix huggingface.co
Other API from Undi95
Total runs:
876
Run Growth:
268
Growth Rate:
30.59%
Total runs:
804
Run Growth:
-245
Growth Rate:
-30.47%
Total runs:
802
Run Growth:
-251
Growth Rate:
-31.30%
Total runs:
800
Run Growth:
-229
Growth Rate:
-28.63%
Total runs:
729
Run Growth:
-253
Growth Rate:
-34.71%
Total runs:
719
Run Growth:
-277
Growth Rate:
-38.53%
Total runs:
717
Run Growth:
-225
Growth Rate:
-31.38%
Total runs:
713
Run Growth:
-252
Growth Rate:
-35.34%
Total runs:
704
Run Growth:
-258
Growth Rate:
-36.65%
Total runs:
691
Run Growth:
69
Growth Rate:
9.99%
Total runs:
532
Run Growth:
502
Growth Rate:
94.36%
Total runs:
386
Run Growth:
-11
Growth Rate:
-2.85%
Total runs:
310
Run Growth:
-63
Growth Rate:
-20.32%
Total runs:
284
Run Growth:
-27.0K
Growth Rate:
-9501.06%
Total runs:
231
Run Growth:
172
Growth Rate:
69.08%
Total runs:
197
Run Growth:
-825
Growth Rate:
-418.78%
Total runs:
190
Run Growth:
-155
Growth Rate:
-81.58%
Total runs:
185
Run Growth:
92
Growth Rate:
48.68%
Total runs:
182
Run Growth:
116
Growth Rate:
63.74%
Total runs:
157
Run Growth:
18
Growth Rate:
11.46%
Total runs:
157
Run Growth:
91
Growth Rate:
57.96%
Total runs:
146
Run Growth:
-449
Growth Rate:
-307.53%
Total runs:
142
Run Growth:
86
Growth Rate:
58.90%
Total runs:
141
Run Growth:
-469
Growth Rate:
-332.62%
Total runs:
141
Run Growth:
-53
Growth Rate:
-42.06%
Total runs:
141
Run Growth:
-477
Growth Rate:
-338.30%
Total runs:
138
Run Growth:
-60
Growth Rate:
-43.48%
Total runs:
138
Run Growth:
-96
Growth Rate:
-69.57%
Total runs:
136
Run Growth:
-28
Growth Rate:
-19.86%
Total runs:
134
Run Growth:
-322
Growth Rate:
-240.30%
Total runs:
133
Run Growth:
90
Growth Rate:
69.23%
Total runs:
123
Run Growth:
-314
Growth Rate:
-255.28%
Total runs:
122
Run Growth:
-477
Growth Rate:
-390.98%
Total runs:
120
Run Growth:
-485
Growth Rate:
-404.17%
Total runs:
113
Run Growth:
-97
Growth Rate:
-81.51%
Total runs:
106
Run Growth:
0
Growth Rate:
0.00%
Total runs:
104
Run Growth:
22
Growth Rate:
21.15%
Total runs:
97
Run Growth:
-66
Growth Rate:
-68.04%
Total runs:
97
Run Growth:
17
Growth Rate:
17.53%
Total runs:
95
Run Growth:
61
Growth Rate:
62.24%
Total runs:
95
Run Growth:
-354
Growth Rate:
-372.63%
Total runs:
90
Run Growth:
-66
Growth Rate:
-69.47%
Total runs:
85
Run Growth:
-41
Growth Rate:
-47.67%
Total runs:
85
Run Growth:
77
Growth Rate:
90.59%
Total runs:
84
Run Growth:
63
Growth Rate:
75.00%
Total runs:
83
Run Growth:
-4
Growth Rate:
-4.76%
Total runs:
83
Run Growth:
-5
Growth Rate:
-5.75%
Total runs:
82
Run Growth:
3
Growth Rate:
3.66%
Total runs:
73
Run Growth:
-134
Growth Rate:
-176.32%
Total runs:
72
Run Growth:
-1
Growth Rate:
-1.39%
Total runs:
69
Run Growth:
0
Growth Rate:
0.00%
Total runs:
69
Run Growth:
-33
Growth Rate:
-44.59%
Total runs:
67
Run Growth:
-52
Growth Rate:
-61.90%
Total runs:
64
Run Growth:
18
Growth Rate:
29.03%
Total runs:
63
Run Growth:
-39
Growth Rate:
-61.90%
Total runs:
62
Run Growth:
8
Growth Rate:
12.12%
Total runs:
59
Run Growth:
-13
Growth Rate:
-21.31%
Total runs:
58
Run Growth:
-36
Growth Rate:
-57.14%
Total runs:
57
Run Growth:
41
Growth Rate:
71.93%
Total runs:
57
Run Growth:
47
Growth Rate:
88.68%
Total runs:
56
Run Growth:
-165
Growth Rate:
-294.64%
Total runs:
55
Run Growth:
-31
Growth Rate:
-56.36%
Total runs:
55
Run Growth:
-9
Growth Rate:
-16.36%
Total runs:
54
Run Growth:
-33
Growth Rate:
-61.11%
Total runs:
54
Run Growth:
5
Growth Rate:
9.26%
Total runs:
53
Run Growth:
-13
Growth Rate:
-24.53%
Total runs:
52
Run Growth:
-1.1K
Growth Rate:
-2084.62%
Total runs:
50
Run Growth:
12
Growth Rate:
24.00%
Total runs:
49
Run Growth:
-28
Growth Rate:
-57.14%
Total runs:
49
Run Growth:
16
Growth Rate:
32.65%