I have a theory! But, I have to go to bed, so I'm setting this to upload while I sleep.
The 13Bs struggled because they were inherently lopsided. So, with this layout, I not only free up more parameters for further finetuning, I also address the imbalance. Crazy? Maybe.
Results
Unsurprisingly, it is totally demented. It was worth a shot for science's sake, but watching the per-token perplexity and seeing WHERE it fails... I've just come to the conclusion this line of experimentation is indeed a dead end.
7b models are just too small per layer to have the kind of redundancy needed for multiple slices like this, leaving 11b merges as the only real viable enlarged Mistral. Even then, the problems seen here are scaled down but still apparent in 11b, right down to the pattern of what sequences cause massive perplexity spikes.
Perhaps, if one toyed with the layer placement just right, you could get a “solid” >7b Mistral merge. Even then, it would be smaller than I really want to work with. 70 billions and merges like Venus and Goliath prove what seems intuitive, higher parameter count models (when executed sanely) will outperform a smaller model at certain tasks.
My last foray into this will be a single-join merge that eats a little more into the layers at the beginning and end, hopefully my hypothesis that you can bleed into the last few layers more with Mistral is correct. But multiple joins is a dead-end.
Recipe
slices:
sources:
model: mistralai/Mistral-7B-v0.1
layer_range: [0, 25]
sources:
model: mistralai/Mistral-7B-v0.1
layer_range: [7, 25]
sources:
model: mistralai/Mistral-7B-v0.1
layer_range: [7, 25]
sources:
model: mistralai/Mistral-7B-v0.1
layer_range: [7, 32]
merge_method: passthrough
dtype: bfloat16
Runs of athirdpath Thicc-Mistral-19b-FAIL on huggingface.co
15
Total runs
0
24-hour runs
-2
3-day runs
-1
7-day runs
6
30-day runs
More Information About Thicc-Mistral-19b-FAIL huggingface.co Model
Thicc-Mistral-19b-FAIL huggingface.co
Thicc-Mistral-19b-FAIL huggingface.co is an AI model on huggingface.co that provides Thicc-Mistral-19b-FAIL's model effect (), which can be used instantly with this athirdpath Thicc-Mistral-19b-FAIL model. huggingface.co supports a free trial of the Thicc-Mistral-19b-FAIL model, and also provides paid use of the Thicc-Mistral-19b-FAIL. Support call Thicc-Mistral-19b-FAIL model through api, including Node.js, Python, http.
Thicc-Mistral-19b-FAIL huggingface.co is an online trial and call api platform, which integrates Thicc-Mistral-19b-FAIL's modeling effects, including api services, and provides a free online trial of Thicc-Mistral-19b-FAIL, you can try Thicc-Mistral-19b-FAIL online for free by clicking the link below.
athirdpath Thicc-Mistral-19b-FAIL online free url in huggingface.co:
Thicc-Mistral-19b-FAIL is an open source model from GitHub that offers a free installation service, and any user can find Thicc-Mistral-19b-FAIL on GitHub to install. At the same time, huggingface.co provides the effect of Thicc-Mistral-19b-FAIL install, users can directly use Thicc-Mistral-19b-FAIL installed effect in huggingface.co for debugging and trial. It also supports api for free installation.
Thicc-Mistral-19b-FAIL install url in huggingface.co: