arshiaafshani / Arsh-llm-gpt

Arsh LLM

This model is based on gpt2 architecture. checkout the main model. Arsh llm is the newest project developed to be helpful for researches. This version is the pretrained on olmo mix dataset using T4 GPU (This proccess took about 4-5 days, with the limitations, I slpitted it into 8 parts and then I did each in one or two day)!

Model Details

Model Description

Arsh llm is my newest research project. It needs more training and this is just one of the first version of it. My goal is to prepare and prove big models don't need big hardwares, at least by these all tools to make everything so fast! First, I created an architecture using Gpt neox and llama documents and ai to have the best optimization (not sure it's done or not). Then, I created a model and I calculated initial weights using phi-4. then I trained it on a part of PILE dataset to make the model stable to use! Then, I trained a model names arshGpt using olmo-mix-1124 dataset. My goal was to convert datas more easily into my big new model. This is the merged model finetuned using some small conversational open source datasets, I guess it's performance is better that what it was.

License

I choosed the MIT license for this model, the best license to smell the freedom 😅!