FlameF0X / SnowflakeCore-G0-Release-3-1B

🔬 SnowflakeCore-G0-Release-3 Architecture Size Report

Summary

This document provides a detailed breakdown of the parameter count and structural design of the SnowflakeCore-G0-Release-3 model. SnowflakeCore-G0-Release-3 is a custom decoder-only transformer model built from scratch, designed for autoregressive language modeling with rotary positional embeddings (RoPE).

📐 Model Architecture Overview

🧮 Parameter Count Breakdown

1. Embedding Layers

• Token Embedding : V × d = 50,000 × 1536 = 76.8M
• Output Projection : d × V = 1536 × 50,000 = 76.8M

Total :

P_embedding = 2 · 1536 · 50,000 = 153.6M

2. Transformer Blocks

Each of the 32 layers contains:

• Multi-Head Attention (Q, K, V, Out) :
  4 · d² = 4 · 1536² = 9.44M

• Feedforward Network (MLP) :
  2 · d · d_ff = 2 · 1536 · 6144 = 18.87M

• Total per Layer :

9.44M + 18.87M = 28.31M

• Total across 32 layers :

32 · 28.31M = 905.97M

3. Positional Embedding

• Type : Rotary Positional Embeddings (RoPE)
• Parameter Count : 0 (non-learned, sinusoidal basis)

📊 Final Parameter Estimate

Total Parameters ≈ P_embedding + P_transformer = 153.6M + 905.97M = 1,059.6M

🧠 Training Regime (Contextual)

Approximate number of tokens:

2M × avg_tokens_per_row ≤ 4B tokens

🧾 Notes

• SnowflakeCore-G0-Release-3 exceeds the size of GPT-2 Large (~774M parameters).
• With RoPE and 32 layers, the model is well-positioned for long-range generalization.
• This parameter size is consistent with the compute-optimal design frontier for mid-scale language models.

📦 Conclusion

SnowflakeCore-G0-Release-3 is a rigorously engineered, 1.06B parameter language model with modern architectural choices (RoPE, deep stack, wide FFN) that position it as a strong open foundation model for further research, deployment, and extension.