Lyon28 / caca-6M-untrained

🚀 CACA-6M

Model Transformer Modern dengan Arsitektur Canggih

6,698,016 parameters • 6.70M • 4 layers

📖 Dokumentasi • 🚀 Quick Start • 💡 Fitur • 🔧 Training • 📊 Spesifikasi

---

⚠️ PENTING: Model Belum Dilatih (Untrained)

PERHATIAN : Ini adalah model yang belum melalui proses training . Bobot model masih dalam kondisi random initialization. Output yang dihasilkan akan tidak bermakna dan acak .

Status Model:

• 🔴 Belum dilatih - Bobot masih random
• 🟡 Hanya untuk riset - Eksperimen arsitektur & training
• 🟢 Siap dilatih - Arsitektur sudah teruji

Widget di atas hanya menunjukkan format input yang diharapkan . Setelah model dilatih dengan dataset yang tepat, format yang sama akan menghasilkan output berkualitas.

📋 Deskripsi

Caca adalah arsitektur Large Language Model (LLM) generasi terbaru yang menggabungkan berbagai teknik state-of-the-art dalam deep learning. Model ini dirancang dengan fokus pada efisiensi , skalabilitas , dan performa tinggi .

Caca itu eksperimen open-source Indonesian LLM yang dibuat dari nol secara individual dan bertahap. Bukan kompetitor siapa-siapa, cuma pengen eksplorasi apa yang bisa dilakukan dengan budget terbatas, passion unlimited, dan mindset collaborative. Kalau berguna buat orang lain, alhamdulillah. Kalau enggak, ya tetap fun kok.

Ini proyek eksplorasi, jadi kalau gagal ya bagian dari proses belajar. Kalau berhasil, itu bonus.

🎯 Keunggulan Utama

• 🇮🇩 Bilingual Support : Optimized untuk Bahasa Indonesia & English
• ⚡ Ultra Fast : Flash Attention 2 untuk inferensi 3x lebih cepat
• 💾 Memory Efficient : Grouped Query Attention menghemat 75% KV cache
• 🎯 Long Context : Support hingga 2,048 token
• 🔧 Modular : Arsitektur fleksibel dengan berbagai opsi konfigurasi

✨ Fitur Utama

🎯 Core Features

• ✅ Grouped Query Attention (GQA) - Efisiensi memori dan komputasi superior

  • Query heads: 4
  • KV heads: 2
  • Ratio: 2:1 (hemat 75% KV cache)

• ✅ Rotary Position Embeddings (RoPE) - Generalisasi konteks panjang lebih baik

  • Theta: 10000
  • Support extrapolation untuk konteks > training length

• ✅ RMSNorm - Normalisasi lebih stabil dan 50% lebih cepat dari LayerNorm

  • Epsilon: 1e-06

• ✅ SwiGLU Activation - Performa 10-15% lebih baik dari ReLU/GELU

  • Intermediate size: 384

• ✅ Flash Attention 2 - Akselerasi hingga 3x dengan memory efficiency

  • Otomatis aktif jika tersedia CUDA

🔥 Advanced Features

🎯 Attention Mechanisms

• ⚡ Flash Attention v2 - 3x faster with IO-aware algorithm
• 🔑 Grouped Query Attention (GQA) - 4Q : 2KV heads
• 🚀 xFormers Support - Memory efficient attention fallback
• 🎯 PyTorch SDPA - Native scaled dot product attention

📍 Position Encodings

• 🔄 RoPE - Rotary embeddings (θ=10000)

🪟 Long Context Features

🎓 Training Optimizations

• 💾 Gradient Checkpointing - Memory efficient training
• 🎯 Mixed Precision - BF16 & FP16 support

📦 Quantization Support

• 4️⃣ 4-bit Quantization - NF4, FP4 via bitsandbytes
• 8️⃣ 8-bit Quantization - LLM.int8() support
• 🔄 Double Quantization - Further compression

🛠️ Optimization Features

• 💾 KV Cache - Generasi autoregressive 5-10x lebih cepat
• 🔧 Gradient Checkpointing - Training model besar dengan memory terbatas
• 📦 Quantization Ready - Support 4-bit & 8-bit quantization
• 🎯 Mixed Precision Training - BF16 & FP16 support

📊 Spesifikasi Teknis

📐 Arsitektur Detail

CacaForCausalLM (6.70M)
│
├─ Embedding Layer
│  └─ Token Embeddings: 32,000 × 96
│     └─ Parameters: 3,072,000
│
├─ Transformer Layers (4x)
│  │
│  ├─ Layer {i} (repeated 4 times)
│  │  │
│  │  ├─ Input LayerNorm (RMSNorm)
│  │  │  └─ Params: 96
│  │  │
│  │  ├─ Self-Attention (Grouped Query Attention)
│  │  │  ├─ Q Projection: 96 → 96
│  │  │  ├─ K Projection: 96 → 48
│  │  │  ├─ V Projection: 96 → 48
│  │  │  ├─ O Projection: 96 → 96
│  │  │  ├─ RoPE Embeddings: θ=10000
│  │  │  └─ Flash Attention 2 (if available)
│  │  │
│  │  ├─ Post-Attention LayerNorm (RMSNorm)
│  │  │  └─ Params: 96
│  │  │
│  │  ├─ MLP (SwiGLU)
│  │  │  ├─ Gate: 96 → 384
│  │  │  ├─ Up: 96 → 384
│  │  │  ├─ Activation: SiLU (Swish)
│  │  │  └─ Down: 384 → 96
│  │  │
│  │  └─ Residual Connections (2x per layer)
│  │
│  └─ Total Layer Params: ~0M per layer
│
├─ Final LayerNorm (RMSNorm)
│  └─ Params: 96
│
└─ LM Head (Output Projection)
   └─ Linear: 96 → 32,000
      └─ Parameters: 3,072,000

🚀 Quick Start

📦 Instalasi

# Dependencies dasar
pip install torch>=2.0.0 transformers>=4.35.0 accelerate safetensors

# Optional: Untuk performa maksimal
pip install flash-attn --no-build-isolation  # Flash Attention 2
pip install xformers                          # Memory efficient attention
pip install bitsandbytes                      # Quantization support

💻 Penggunaan Dasar

1️⃣ Load Model

from transformers import AutoModelForCausalLM, AutoConfig
import torch

# Load configuration
config = AutoConfig.from_pretrained(
    "Lyon28/caca-6M-untrained",
    trust_remote_code=True
)

print(f"Model: {config.model_type}")
print(f"Parameters: 6,698,016")
print(f"Hidden size: {config.hidden_size}")
print(f"Layers: {config.num_hidden_layers}")

# Load model
model = AutoModelForCausalLM.from_pretrained(
    "Lyon28/caca-6M-untrained",
    config=config,
    torch_dtype=torch.bfloat16,  # Gunakan BF16 untuk efisiensi
    device_map="auto",            # Otomatis distribusi ke GPU
    trust_remote_code=True
)

print(f"Model loaded! Device: {model.device}")

2️⃣ Verifikasi Model

# Hitung total parameter
total_params = sum(p.numel() for p in model.parameters())
trainable_params = sum(p.numel() for p in model.parameters() if p.requires_grad)

print(f"Total parameters: {total_params:,}")
print(f"Trainable parameters: {trainable_params:,}")
print(f"Model size: {total_params * 2 / 1e9:.2f} GB (BF16)")

# Test forward pass
batch_size, seq_len = 2, 10
input_ids = torch.randint(0, config.vocab_size, (batch_size, seq_len))
input_ids = input_ids.to(model.device)

with torch.no_grad():
    outputs = model(input_ids)

print(f"Output shape: {outputs.logits.shape}")
print("✅ Model berfungsi dengan baik!")

3️⃣ Generate Text (Setelah Training)

from transformers import AutoTokenizer

# Load tokenizer (gunakan tokenizer yang sesuai)
tokenizer = AutoTokenizer.from_pretrained("your-tokenizer-here")

# Prepare input
text = "Jelaskan tentang kecerdasan buatan"
inputs = tokenizer(text, return_tensors="pt").to(model.device)

# Generate
outputs = model.generate(
    **inputs,
    max_new_tokens=100,
    temperature=0.7,
    top_p=0.9,
    top_k=50,
    do_sample=True,
    repetition_penalty=1.1,
    pad_token_id=tokenizer.eos_token_id
)

# Decode
generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(generated_text)

🔧 Training Guide

📚 Persiapan Dataset

from datasets import load_dataset

# Load dataset (contoh)
dataset = load_dataset("indonesian-nlp/id-wikipedia")

# Atau load dari file lokal
from datasets import Dataset
import pandas as pd

df = pd.read_csv("your_data.csv")
dataset = Dataset.from_pandas(df)

print(f"Dataset size: {len(dataset)}")

🎯 Training Configuration

from transformers import Trainer, TrainingArguments
from transformers import DataCollatorForLanguageModeling

# Training arguments
training_args = TrainingArguments(
    # Output
    output_dir="./caca-6M-trained",
    run_name="caca-6M-v1",
    
    # Training
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=8,  # Effective batch size = 32
    learning_rate=2e-4,
    weight_decay=0.1,
    warmup_steps=2000,
    
    # Optimization
    bf16=True,                      # Mixed precision training
    gradient_checkpointing=True,     # Hemat memory
    optim="adamw_torch_fused",      # Optimizer tercepat
    max_grad_norm=1.0,
    
    # Logging & Evaluation
    logging_steps=10,
    logging_first_step=True,
    eval_strategy="steps",
    eval_steps=500,
    save_steps=1000,
    save_total_limit=3,
    
    # Hub integration
    push_to_hub=True,
    hub_model_id="your-username/caca-6M-trained",
    hub_strategy="every_save",
    
    # Distributed training
    ddp_find_unused_parameters=False,
    dataloader_num_workers=4,
)

# Data collator
data_collator = DataCollatorForLanguageModeling(
    tokenizer=tokenizer,
    mlm=False  # Causal LM, bukan Masked LM
)

# Trainer
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=dataset["train"],
    eval_dataset=dataset["validation"],
    data_collator=data_collator,
)

# Train!
print("🚀 Starting training...")
trainer.train()

# Save final model
print("💾 Saving model...")
trainer.save_model("./caca-6M-final")
trainer.push_to_hub()

print("✅ Training complete!")

📊 Estimasi Resource

💬 Format Chat

Model ini mendukung format chat standar:

# Single-turn
messages = [
    {"role": "user", "content": "Halo! Siapa kamu?"},
]

# Multi-turn conversation
messages = [
    {"role": "system", "content": "Kamu adalah asisten AI yang membantu."},
    {"role": "user", "content": "Jelaskan tentang fotosintesis"},
    {"role": "assistant", "content": "Fotosintesis adalah proses..."},
    {"role": "user", "content": "Apa manfaatnya bagi manusia?"},
]

# Apply chat template
formatted = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)

print(formatted)
# Output:
# System: Kamu adalah asisten AI yang membantu.
#
# User: Jelaskan tentang fotosintesis
# Assistant: Fotosintesis adalah proses...
# User: Apa manfaatnya bagi manusia?
# Assistant:

🎯 Use Cases

✅ Cocok Untuk:

• 🔬 Penelitian : Eksperimen arsitektur LLM modern
• 📚 Edukasi : Belajar tentang transformer & training
• 🎓 Akademis : Paper, thesis, project
• 🚀 Base Model : Fine-tuning untuk task spesifik
• 💡 Proof of Concept : Test ide sebelum scale up

❌ Tidak Cocok Untuk:

• 🚫 Production : Model belum dilatih
• 🚫 Real-world apps : Output masih random
• 🚫 Safety-critical : Belum ada safety alignment
• 🚫 Direct deployment : Perlu training dulu

📖 Dokumentasi

🔗 Links Penting

• 📚 Hugging Face Docs : transformers.github.io
• 💻 GitHub : Lyon-28/caca-transformers
• 💬 Discussions : Model discussions
• 🐛 Issues : Report bugs

🤝 Contributing

Kami sangat terbuka untuk kontribusi! Beberapa cara untuk berkontribusi:

• 🐛 Report bugs : Temukan bug? Buka issue
• 💡 Suggest features : Punya ide? Share di discussions
• 📝 Improve docs : PR welcome untuk dokumentasi
• 🎓 Share results : Training hasil? Share di model card
• ⭐ Star & Share : Bantu project ini berkembang

📜 License & Citation

📄 License

Model ini dirilis di bawah Apache License 2.0 :

• ✅ Gratis untuk penggunaan komersial
• ✅ Gratis untuk penggunaan riset
• ✅ Boleh modifikasi & distribusi
• ✅ Tidak ada garansi

📚 Citation

Jika Anda menggunakan model ini dalam penelitian atau project, mohon cite:

@misc{caca-2025,
  author = {Lyon},
  title = {Caca: Modern Transformer Architecture with GQA and Advanced Features},
  year = {2025},
  publisher = {Hugging Face},
  journal = {Hugging Face Model Hub},
  howpublished = {\url{https://huggingface.co/Lyon28/caca-6M-untrained}},
}

🙏 Acknowledgments

Model ini terinspirasi dan mengimplementasikan berbagai penelitian terkini:

🏗️ Core Architecture

• LLaMA (Meta AI, 2023) - Base decoder-only architecture, RMSNorm, SwiGLU
  • Paper: LLaMA: Open and Efficient Foundation Language Models
• GPT-3 (OpenAI, 2020) - Transformer language modeling paradigm
• PaLM (Google, 2022) - SwiGLU activation function

🎯 Attention Mechanisms

• Flash Attention v2 (Tri Dao et al., 2023) - Efficient attention with IO-awareness
  • Paper: FlashAttention-2: Faster Attention with Better Parallelism
• Grouped Query Attention (GQA) (Ainslie et al., Google, 2023) - Memory-efficient attention
  • Paper: GQA: Training Generalized Multi-Query Transformer
• Multi-Query Attention (MQA) (Shazeer, Google, 2019) - Fast decoding
• xFormers (Meta AI, 2022) - Memory efficient attention implementations
• PyTorch SDPA (PyTorch Team, 2023) - Built-in scaled dot product attention

📍 Position Encodings

• RoPE (Su et al., EleutherAI, 2021) - Rotary Position Embeddings
  • Paper: RoFormer: Enhanced Transformer with Rotary Position Embedding
• ALiBI (Press et al., 2022) - Attention with Linear Biases for extrapolation
  • Paper: Train Short, Test Long: Attention with Linear Biases
• YaRN (Peng et al., 2023) - Yet another RoPE extensioN for long context
  • Paper: YaRN: Efficient Context Window Extension

🪟 Long Context & Efficiency

• Sliding Window Attention (Mistral AI, 2023) - Local attention patterns
  • Paper: Mistral 7B
• StreamingLLM / Attention Sink (Xiao et al., MIT, 2023) - Infinite sequence lengths
  • Paper: Efficient Streaming Language Models with Attention Sinks
• Logit Softcapping (Google Gemma, 2024) - Prevent attention overflow
  • Paper: Gemma: Open Models Based on Gemini

🧠 Mixture of Experts (MoE)

• Mixtral 8x7B (Mistral AI, 2024) - Sparse MoE architecture
  • Paper: Mixtral of Experts
• Switch Transformers (Fedus et al., Google, 2021) - Scaling with expert choice
  • Paper: Switch Transformers: Scaling to Trillion Parameter Models
• GLaM (Du et al., Google, 2021) - Generalist Language Model with MoE
• Expert Choice Routing (Zhou et al., Google, 2022) - Improved load balancing

🎓 Training Optimizations

• Layer Scale (Touvron et al., Meta, 2021) - Training stability for deep networks
  • Paper: Going Deeper with Image Transformers (CaiT)
• Stochastic Depth (Huang et al., 2016) - Regularization via random layer dropping
  • Paper: Deep Networks with Stochastic Depth
• Mixture of Depths (MoD) (Raposo et al., Google DeepMind, 2024) - Dynamic compute allocation
  • Paper: Mixture-of-Depths: Dynamically allocating compute in transformer-based models
• Gradient Checkpointing (Chen et al., 2016) - Memory-efficient training

📦 Quantization

• LLM.int8() (Dettmers et al., 2022) - 8-bit matrix multiplication
  • Paper: LLM.int8(): 8-bit Matrix Multiplication for Transformers
• QLoRA (Dettmers et al., 2023) - 4-bit quantized LoRA fine-tuning
  • Paper: QLoRA: Efficient Finetuning of Quantized LLMs
• GPTQ (Frantar et al., 2022) - Post-training quantization
• bitsandbytes (Dettmers) - Efficient quantization library

🎨 Multimodal Components

• Vision Transformer (ViT) (Dosovitskiy et al., Google, 2020) - Image encoding
  • Paper: An Image is Worth 16x16 Words
• Perceiver Resampler (Alayrac et al., DeepMind, 2022) - Multimodal fusion
  • Paper: Flamingo: a Visual Language Model
• Q-Former (Li et al., Salesforce, 2023) - Query-based multimodal alignment
  • Paper: BLIP-2: Bootstrapping Language-Image Pre-training
• Whisper (Radford et al., OpenAI, 2022) - Audio encoding inspiration

🛠️ Normalization & Activations

• RMSNorm (Zhang & Sennrich, 2019) - Root Mean Square Layer Normalization
  • Paper: Root Mean Square Layer Normalization
• SwiGLU (Shazeer, Google, 2020) - GLU activation variant
  • Paper: GLU Variants Improve Transformer

🔧 Implementation & Tools

• Hugging Face Transformers - Model implementation framework
• PyTorch - Deep learning framework
• Safetensors - Secure tensor serialization format
• Accelerate - Distributed training utilities

Special Thanks to:

• 🇮🇩 Indonesian NLP Community
• 🤗 Hugging Face Team
• 🔬 Open source AI research community

⚠️ Limitations & Bias

Keterbatasan

• 🔴 Untrained : Model belum dilatih, output random
• 🟡 No Tokenizer : Perlu prepare tokenizer sendiri
• 🟡 No Safety : Belum ada content filtering/alignment
• 🟠 Memory Intensive : Training butuh GPU besar

Potential Biases

Model ini akan mewarisi bias dari data training yang digunakan. Mohon perhatikan:

• Bahasa : Bias terhadap bahasa mayoritas di dataset
• Kultur : Bias terhadap perspektif kultur tertentu
• Gender & Demografis : Potential stereotypes
• Faktual : Bisa generate informasi tidak akurat

Rekomendasi : Lakukan evaluation & filtering sebelum deployment.

📞 Support & Contact

💬 Community

• Discussions : HF Discussions

📧 Contact

Untuk pertanyaan atau kolaborasi:

• Email: [email protected]
• HF Profile: @Lyon28

Quote Dari caca