FlameF0X / Snowflake-G0-Release

I know that you dont see any differences in the README.md, but I've trained the model to val loss 1.6836 than the previous version. I aslo try to fix all the previous errors when you tired to laod the model. Please tell me if you find any bugs or anything useful!

Snowflake-G0-Release

This is the initial release of the Snowflake series language models, trained on the DialogMLM-50K dataset with optimized memory usage.

Model details

• Architecture: SnowflakeCore
• Hidden size: 384
• Number of attention heads: 6
• Number of layers: 4
• Feed-forward dimension: 768
• Maximum sequence length: 384
• Vocabulary size: 30522

HuggingFace Transformers Compatibility

This model is fully compatible with the HuggingFace Transformers library. You can load it using:

from transformers import AutoConfig, AutoModel, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("path/to/snowflake_g0_release")
config = AutoConfig.from_pretrained("path/to/snowflake_g0_release")
model = AutoModel.from_pretrained("path/to/snowflake_g0_release")

Memory Optimization Techniques

• Mixed precision training
• Gradient accumulation (4 steps)
• Fused QKV projection
• Pre-norm architecture
• Weight tying between embedding and output layers
• Half-precision model storage

The model weights are stored in both PyTorch (.bin) and safetensors format for improved security, loading efficiency, and compatibility.