I have been wanting a better model for RP on a 24GB Nvidia card. And there are some great models out there, but I wanted something that I knew I could quantize to Q4, have a great context size, have a very fast response, and would provide some dynamic content. The total is around 30B, but since there are 2/3 models active, the response is quite fast!
so enter velvet eclipse. This is using 3x Mistral Nemo finetunes, each with a separate purpose.
-
A reasoning model as the base
-
A RP fine tune
-
an ERP fine tune.
Many of the mergekit MoEs I have found, frequently combine several experts that excel in the same areas, I felt ensuring three models with 3 different purposes would be a better fit.
I was hoping to make the reasoning mistral fine tune always active, but I had trouble getting that working with mergekit and Mistral models. If I can get that working I'll be very happy.
I'll update the repo with more specifics soon!