FlameF0X / SnowflakeCore-G0-Release-2

SnowflakeCore-G0-Release-2

This is the initial release of the SnowflakeCore series language models, trained on the DialogMLM-50K dataset with optimized memory usage.

SUPPORT ME

You can support me via https://ko-fi.com/flamef0x

Model details

• Architecture: SnowflakeCore
• Hidden size: 768
• Number of attention heads: 12
• Number of layers: 8
• Feed-forward dimension: 1536
• Maximum sequence length: 768
• Vocabulary size: 30522

Flowchart

HuggingFace Transformers Compatibility

This model is fully compatible with the HuggingFace Transformers library. You can load it using:

from transformers import AutoConfig, AutoModel, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("FlameF0X/SnowflakeCore-G0-Release-2")
config = AutoConfig.from_pretrained("FlameF0X/SnowflakeCore-G0-Release-2")
model = AutoModel.from_pretrained("FlameF0X/SnowflakeCore-G0-Release-2")

Memory Optimization Techniques

• Mixed precision training
• Gradient accumulation (8 steps)
• Fused QKV projection
• Pre-norm architecture
• Weight tying between embedding and output layers
• Half-precision model storage

The model weights are stored in both PyTorch (.bin) and safetensors format for improved security, loading efficiency, and compatibility.

Training

Complexity

The overall time complexity of training SnowflakeCore-G0-Release-2 falls under the O(n²) class due to the self-attention mechanism used in the transformer architecture. Here's a breakdown of the major computational costs:

• Self-Attention : O(n² · d), where n is the sequence length (768) and d is the hidden size (768). This term dominates because each token attends to every other token.

• Feedforward Layers : O(n · d²), with two projection layers per block.

• Stacked Layers : Multiplied by the number of layers L = 8 .

Overall per step complexity:

O(L · (n² · d + n · d²)) ≈ O(n² · d · L)

Training over the dataset for E = 40 epochs with B = 8 batch size leads to total training complexity:

O(E · (N / B) · n² · d · L) Where N is the number of training samples.

This puts SnowflakeCore-G0-Release-2 in the O(n²) class with respect to sequence length, which is a key scaling bottleneck. Optimizations such as fused QKV, gradient accumulation, and mixed-precision help reduce practical training cost.