athirdpath / Thicc-Mistral-19b-FAIL

huggingface.co
Total runs: 15
24-hour runs: 0
7-day runs: -1
30-day runs: 6
Model's Last Updated: December 03 2023
text-generation

Introduction of Thicc-Mistral-19b-FAIL

Model Details of Thicc-Mistral-19b-FAIL

What the hell is going on here?

I have a theory! But, I have to go to bed, so I'm setting this to upload while I sleep.

The 13Bs struggled because they were inherently lopsided. So, with this layout, I not only free up more parameters for further finetuning, I also address the imbalance. Crazy? Maybe.

Results

Unsurprisingly, it is totally demented. It was worth a shot for science's sake, but watching the per-token perplexity and seeing WHERE it fails... I've just come to the conclusion this line of experimentation is indeed a dead end.

7b models are just too small per layer to have the kind of redundancy needed for multiple slices like this, leaving 11b merges as the only real viable enlarged Mistral. Even then, the problems seen here are scaled down but still apparent in 11b, right down to the pattern of what sequences cause massive perplexity spikes.

Perhaps, if one toyed with the layer placement just right, you could get a “solid” >7b Mistral merge. Even then, it would be smaller than I really want to work with. 70 billions and merges like Venus and Goliath prove what seems intuitive, higher parameter count models (when executed sanely) will outperform a smaller model at certain tasks.

My last foray into this will be a single-join merge that eats a little more into the layers at the beginning and end, hopefully my hypothesis that you can bleed into the last few layers more with Mistral is correct. But multiple joins is a dead-end.

Recipe

slices:

  • sources:

    • model: mistralai/Mistral-7B-v0.1

      layer_range: [0, 25]

  • sources:

    • model: mistralai/Mistral-7B-v0.1

      layer_range: [7, 25]

  • sources:

    • model: mistralai/Mistral-7B-v0.1

      layer_range: [7, 25]

  • sources:

    • model: mistralai/Mistral-7B-v0.1

      layer_range: [7, 32]

merge_method: passthrough

dtype: bfloat16

Runs of athirdpath Thicc-Mistral-19b-FAIL on huggingface.co

15
Total runs
0
24-hour runs
-2
3-day runs
-1
7-day runs
6
30-day runs

More Information About Thicc-Mistral-19b-FAIL huggingface.co Model

Thicc-Mistral-19b-FAIL huggingface.co

Thicc-Mistral-19b-FAIL huggingface.co is an AI model on huggingface.co that provides Thicc-Mistral-19b-FAIL's model effect (), which can be used instantly with this athirdpath Thicc-Mistral-19b-FAIL model. huggingface.co supports a free trial of the Thicc-Mistral-19b-FAIL model, and also provides paid use of the Thicc-Mistral-19b-FAIL. Support call Thicc-Mistral-19b-FAIL model through api, including Node.js, Python, http.

Thicc-Mistral-19b-FAIL huggingface.co Url

https://huggingface.co/athirdpath/Thicc-Mistral-19b-FAIL

athirdpath Thicc-Mistral-19b-FAIL online free

Thicc-Mistral-19b-FAIL huggingface.co is an online trial and call api platform, which integrates Thicc-Mistral-19b-FAIL's modeling effects, including api services, and provides a free online trial of Thicc-Mistral-19b-FAIL, you can try Thicc-Mistral-19b-FAIL online for free by clicking the link below.

athirdpath Thicc-Mistral-19b-FAIL online free url in huggingface.co:

https://huggingface.co/athirdpath/Thicc-Mistral-19b-FAIL

Thicc-Mistral-19b-FAIL install

Thicc-Mistral-19b-FAIL is an open source model from GitHub that offers a free installation service, and any user can find Thicc-Mistral-19b-FAIL on GitHub to install. At the same time, huggingface.co provides the effect of Thicc-Mistral-19b-FAIL install, users can directly use Thicc-Mistral-19b-FAIL installed effect in huggingface.co for debugging and trial. It also supports api for free installation.

Thicc-Mistral-19b-FAIL install url in huggingface.co:

https://huggingface.co/athirdpath/Thicc-Mistral-19b-FAIL

Url of Thicc-Mistral-19b-FAIL

Thicc-Mistral-19b-FAIL huggingface.co Url

Provider of Thicc-Mistral-19b-FAIL huggingface.co

athirdpath
ORGANIZATIONS

Other API from athirdpath

huggingface.co

Total runs: 63
Run Growth: -48
Growth Rate: -76.19%
Updated:November 28 2023
huggingface.co

Total runs: 18
Run Growth: 0
Growth Rate: 0.00%
Updated:August 16 2024
huggingface.co

Total runs: 16
Run Growth: 0
Growth Rate: 0.00%
Updated:September 28 2024
huggingface.co

Total runs: 10
Run Growth: 4
Growth Rate: 40.00%
Updated:December 13 2023
huggingface.co

Total runs: 9
Run Growth: 0
Growth Rate: 0.00%
Updated:August 17 2024