mahwizzzz / tinygemma-Urdu

tinyGemma Urdu

Trained a 0.96 million parameters Urdu Gemma.

• Gemma Paper : https://arxiv.org/abs/2503.19786 - Core architecture and design principles
• RMSNorm : https://arxiv.org/abs/1910.07467 - Root Mean Square Layer Normalization
• RoPE : https://arxiv.org/abs/2104.09864 - Rotary Position Embedding methodology
• Grouped Query Attention : https://arxiv.org/abs/2305.13245 - Memory efficient attention mechanism
• SwiGLU/GELU : https://arxiv.org/abs/2002.05202 - Gated linear unit activations

Architecture

A version of Google's Gemma architecture with the following components as defined in GemmaConfig :

• GemmaAttention : Multi-head attention with grouped query attention (num_queries_per_kv), RoPE positional embeddings via apply_rotary_emb() , and causal masking using pre-computed triangular mask
• GemmaMLP : Feed-forward network with GELU activation implementing gate_proj * up_proj gating mechanism through down_proj
• GemmaDecoderLayer : Transformer block combining self_attn and mlp with pre-normalization using RMSNorm
• RMSNorm : Root Mean Square Layer Normalization with optional unit offset (add_unit_offset=True) and learnable weight parameter
• tinyGemma : Complete model with embedder scaled by sqrt(hidden_size) and tied weights for language modeling head

Training Results

Achieved convergence on Urdu corpus with the following performance metrics:

Final Training Metrics (5000 iterations):
- Training Loss: 2.7668
- Validation Loss: 2.9250  
- Validation Perplexity: 18.6348
- Learning Rate: 3e-4 with AdamW optimizer
- Batch Size: 16 with 2 gradient accumulation steps

Loss Curves

License

MIT License