Lyon28 / caca-500M-untrained

🚀 CACA-500M - Untrained Model

Size : 654.4M parameters Status : 🟡 Untrained (random weights)

---

📋 Deskripsi

Caca adalah arsitektur Large Language Model (LLM) modern yang menggabungkan teknik-teknik terkini dalam deep learning. Model ini didesain dengan fleksibilitas tinggi dan mendukung berbagai fitur advanced seperti Mixture of Experts, Multimodal Processing, dan optimisasi performa tingkat lanjut.

⚠️ PENTING : Ini adalah model yang belum dilatih (untrained). Model ini hanya berisi arsitektur dan bobot acak. Anda perlu melatih model ini terlebih dahulu sebelum digunakan untuk inferensi.

✨ Fitur Utama

🎯 Core Features

• ✅ Grouped Query Attention (GQA) - Efisiensi memori dan komputasi lebih baik
• ✅ Flash Attention - Akselerasi hingga 3x lebih cepat
• ✅ Rotary Position Embeddings (RoPE) - Generalisasi panjang konteks lebih baik
• ✅ RMSNorm - Normalisasi lebih stabil dan cepat
• ✅ SwiGLU Activation - Performa lebih baik dari ReLU/GELU

🔥 Advanced Features

• 🧠 Mixture of Experts (MoE) - Skalabilitas model dengan efisiensi tinggi
• 🪟 Sliding Window Attention - Efisiensi untuk konteks panjang
• 🌊 Attention Sink (StreamingLLM) - Inferensi stream untuk konteks unlimited
• 🎨 Multimodal Support - Vision + Audio processing
• 📊 Mixture of Depths - Dynamic compute allocation per token
• ⚡ Cross Attention - Untuk encoder-decoder tasks

🛠️ Optimization Features

• 💾 KV Cache - Generasi lebih cepat
• 🔧 Gradient Checkpointing - Training model besar dengan memory terbatas
• 📦 4-bit/8-bit Quantization - Deploy model besar di hardware terbatas
• 🎯 Layer Scale & Stochastic Depth - Training stability yang lebih baik

📊 Spesifikasi Model

✨ Features

• ⚡ Flash Attention : 3x faster inference

🔧 Instalasi

Requirements

# Requirements dasar
pip install torch>=2.0.0
pip install transformers>=4.35.0
pip install accelerate
pip install safetensors

# Optional (untuk performa maksimal)
pip install flash-attn  # Flash Attention
pip install xformers    # Memory efficient attention
pip install bitsandbytes  # Quantization

💻 Penggunaan

1. Load Model (Untrained)

from transformers import AutoModelForCausalLM, AutoConfig
import torch

# Load configuration
config = AutoConfig.from_pretrained("Lyon28/caca-500M-untrained")
print(f"Hidden size: {config.hidden_size}")
print(f"Num layers: {config.num_hidden_layers}")

# Load model
model = AutoModelForCausalLM.from_pretrained(
    "Lyon28/caca-500M-untrained",
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

# Cek jumlah parameter
total_params = sum(p.numel() for p in model.parameters())
print(f"Total parameters: {total_params:,}")

2. Kustomisasi Konfigurasi

from transformers import AutoConfig, AutoModelForCausalLM

# Load base config
config = AutoConfig.from_pretrained("Lyon28/caca-500M-untrained")

# Modifikasi config
config.use_moe = True  # Aktifkan MoE
config.num_experts = 16  # 16 experts
config.sliding_window = 1024  # Sliding window attention
config.use_flash_attn = True  # Flash attention

# Create model dengan config custom
model = AutoModelForCausalLM.from_config(config)

3. Training (Contoh Sederhana)

from transformers import Trainer, TrainingArguments
from datasets import load_dataset

# Load dataset
dataset = load_dataset("your-dataset")

# Training arguments
training_args = TrainingArguments(
    output_dir="./caca-finetuned",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=8,
    learning_rate=2e-4,
    bf16=True,
    gradient_checkpointing=True,
    logging_steps=10,
    save_steps=500,
)

# Trainer
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=dataset["train"],
    eval_dataset=dataset["validation"],
)

# Train!
trainer.train()

🏗️ Arsitektur Detail

Struktur Model

CacaForCausalLM
├── CacaModel
│   ├── Embedding Layer (32,000 vocab)
│   ├── 26x CacaDecoderLayer
│   │   ├── Self Attention (GQA)
│   │   ├── [Optional] Cross Attention
│   │   ├── MLP / MoE
│   │   └── Residual Connections
│   └── Final LayerNorm
└── LM Head

Grouped Query Attention (GQA)

Q heads: 20
KV heads: 4
Ratio: 5:1

Benefits:
- 75% smaller KV cache
- 1.5-2x faster inference
- Nearly identical quality to MHA

🎯 Use Cases

1. Text Generation

• Chatbot dan conversational AI
• Content generation (artikel, cerita, kode)
• Text summarization
• Translation

2. Multimodal Tasks (jika diaktifkan)

• Image captioning
• Visual question answering (VQA)
• Audio transcription + understanding

3. Long Context Tasks

• Document understanding
• Code completion dengan repository context
• Long-form content analysis

⚠️ Limitations

1. Untrained Model : Model ini belum dilatih, output adalah random
2. Resource Intensive : Training membutuhkan GPU dengan memory besar
3. No Tokenizer : Anda perlu prepare tokenizer sendiri
4. Experimental : Beberapa fitur masih dalam tahap eksperimen

📌 Citation

Jika Anda menggunakan arsitektur Caca dalam penelitian atau project, kami sangat menghargai jika Anda cite:

@misc{caca2025,
  author = {Lyon},
  title = {Caca: Modern Transformer Architecture with GQA, RoPE, and SwiGLU},
  year = {2025},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/Lyon28/caca-500m}},
  note = {Apache License 2.0}
}

📄 License

Apache License 2.0

🙏 Acknowledgments

Terinspirasi dari:

• LLaMA (Meta): Base architecture
• Mistral : Sliding Window Attention
• Mixtral : Mixture of Experts
• Flash Attention : Efficient attention

---