Lyon28 / caca-250m

🤖 CACA-250M

Tiny/Small - Arsitektur Transformer Modern

📊 Spesifikasi Model

🎯 Fitur Utama

Arsitektur Caca menggabungkan teknik-teknik modern terbaik dari berbagai model state-of-the-art:

🚀 Quick Start

💻 Instalasi

# Install library utama
pip install caca-transformers

# Dengan xFormers untuk 3x speedup (recommended)
pip install caca-transformers[xformers]

# Atau install dari source
git clone https://github.com/Lyon-28/caca-transformers
cd caca-transformers
pip install -e .

⚡ Optimisasi Performance

Model mendukung berbagai optimization backends dengan auto-fallback :

Model akan otomatis memilih backend terbaik yang tersedia.

# Minimum (2x speedup)
pip install torch>=2.0.0

# Recommended (3x speedup) ⭐
pip install xformers

# Maximum (4x speedup) - opsional
pip install flash-attn --no-build-isolation

🧪 Test Cepat (Tanpa Training)

from caca_transformers import CacaForCausalLM
import torch

# Load model
model = CacaForCausalLM.from_pretrained("Lyon28/caca-250m").eval()

# Test forward pass
input_ids = torch.randint(0, 50000, (2, 128))
with torch.no_grad():
    outputs = model(input_ids)
    
print(f"✅ Output shape: {outputs['logits'].shape}")
print(f"✅ Model berfungsi! Siap untuk training.")

# Test KV cache
with torch.no_grad():
    outputs = model(input_ids, use_cache=True)
    past_kv = outputs['past_key_values']
    print(f"✅ KV cache: {len(past_kv)} layers")

⚠️ Common Errors & Solutions

🔍 Input Validation

Model melakukan validasi ketat pada input untuk mencegah error yang sulit di-debug:

# ❌ SALAH
input_ids = torch.tensor([[1, 2, -1, 4]])
model(input_ids)

# ✅ BENAR
input_ids = torch.tensor([[1, 2, 3, 4]])
model(input_ids)

# ❌ SALAH
input_ids = torch.tensor([[1, 2, 999999]])
model(input_ids)

# ✅ BENAR
# Pastikan < vocab_size
input_ids = torch.tensor([[1, 2, 50000]])
model(input_ids)

# ❌ SALAH - float dtype
input_ids = torch.tensor([[1.0, 2.0, 3.0]])
model(input_ids)

# ✅ BENAR - integer dtype
input_ids = torch.tensor([[1, 2, 3]], dtype=torch.long)
# Atau
input_ids = input_ids.long()

⚙️ Config Validation

Config juga divalidasi dengan warning untuk suboptimal settings :

# ⚠️ SUBOPTIMAL
config = CacaConfig(
    num_attention_heads=64,
    num_key_value_heads=2,  # Ratio 32:1
)

# ✅ OPTIMAL
config = CacaConfig(
    num_attention_heads=64,
    num_key_value_heads=8,  # Ratio 8:1
)

# ⚠️ SUBOPTIMAL
config = CacaConfig(
    hidden_size=512,
    num_attention_heads=32,  # head_dim = 16
)

# ✅ OPTIMAL
config = CacaConfig(
    hidden_size=2048,
    num_attention_heads=16,  # head_dim = 128
)

# ⚠️ SUBOPTIMAL
config = CacaConfig(
    hidden_size=2048,
    intermediate_size=4096,  # Terlalu kecil untuk SwiGLU
)

# ✅ OPTIMAL
config = CacaConfig(
    hidden_size=2048,
    intermediate_size=5461,  # ~2.67x hidden_size
)

💡 Helpful Error Messages

Semua error dilengkapi dengan actionable solutions :

# Error dengan solution
config = CacaConfig(
    hidden_size=1000,
    num_attention_heads=16,
)

# ValueError: hidden_size (1000) harus habis dibagi num_attention_heads (16)
# Sisa bagi: 8
# Hint: Gunakan hidden_size yang merupakan kelipatan dari num_attention_heads

⚡ Attention Backend (Advanced)

🔄 Automatic Backend Selection

Model secara otomatis memilih backend terbaik yang tersedia dengan fallback chain :

📋 Fallback Logic

🎛️ Force Specific Backend

# Force disable Flash Attention
config = CacaConfig(use_flash_attn=False)
model = CacaForCausalLM(config)
# Model akan skip Flash dan langsung ke xFormers/SDPA

# Check backend yang digunakan
import logging
logging.basicConfig(level=logging.INFO)

# Model akan log backend selection saat pertama kali digunakan
dummy_input = torch.randint(0, 1000, (1, 10)).cuda()
_ = model(dummy_input)
# Log: "✅ Using Flash Attention backend" atau fallback

📊 Backend Compatibility

🔧 Troubleshooting Backend Issues

# Flash Attention gagal dengan error CUDA
# Model akan otomatis fallback dan log warning:
# "⚠️ Flash Attention gagal: <error>, fallback ke backend lain"

# Untuk debug, set logging level:
import logging
logging.basicConfig(level=logging.INFO)

# Model akan log:
# "✅ Using Flash Attention backend"
# atau
# "⚠️ Fallback to xFormers backend"
# atau
# "ℹ️ Using SDPA backend"
# atau
# "ℹ️ Using Standard attention backend"

💾 Memory Requirements (Detailed)

📐 Training Memory Formula

Memory = Model_Params × Precision_Bytes × Training_Overhead

Training_Overhead:
- FP32: 4 bytes × 4 (weights + gradients + optimizer states) = 16 bytes/param
- FP16 Mixed: 2 bytes × 6 (FP16 + FP32 copy + gradients + optimizer) = 12 bytes/param
- BF16 Mixed: 2 bytes × 6 = 12 bytes/param
- 8-bit: ~6 bytes/param (with 8-bit optimizer)

Activations: ~Batch_Size × Seq_Length × Hidden_Size × Num_Layers × 8 bytes

🧮 Estimation untuk Model Ini

✂️ Dengan Gradient Checkpointing

Gradient checkpointing mengurangi activation memory ~50-70% tapi menambah compute ~30%

# Enable gradient checkpointing
model.gradient_checkpointing_enable()

# Training dengan GC bisa pakai batch size lebih besar!

🎯 Inference Memory

💡 Tip: GQA mengurangi KV cache 7x dibanding Multi-Head Attention!
MHA KV cache: ~0.6 GB vs GQA: 0.1 GB

📊 Batch Size Recommendations

GC = Gradient Checkpointing

🎓 Complete Training Example

📦 1. Prepare Dataset

from datasets import load_dataset
from transformers import AutoTokenizer

# Load tokenizer (gunakan tokenizer existing)
tokenizer = AutoTokenizer.from_pretrained("gpt2")
tokenizer.pad_token = tokenizer.eos_token

# Load dataset - Indonesian Wikipedia
dataset = load_dataset("wikipedia", "20220301.id")

# Tokenize function
def tokenize_function(examples):
    return tokenizer(
        examples["text"],
        truncation=True,
        max_length=2048,
        padding="max_length",
        return_tensors="pt",
    )

# Tokenize dataset
tokenized_dataset = dataset.map(
    tokenize_function,
    batched=True,
    remove_columns=dataset["train"].column_names,
    desc="Tokenizing dataset",
)

print(f"✅ Dataset ready: {len(tokenized_dataset['train']):,} samples")

🏗️ 2. Initialize Model

from caca_transformers import CacaForCausalLM, CacaConfig

# Create config
config = CacaConfig(
    vocab_size=tokenizer.vocab_size,
    hidden_size=896,
    num_hidden_layers=20,
    num_attention_heads=14,
    num_key_value_heads=2,
    intermediate_size=2432,
    max_position_embeddings=8192,
    sliding_window=2048,
    use_flash_attn=True,  # Auto fallback jika tidak available
)

# Initialize model dengan random weights
model = CacaForCausalLM(config)

# Enable gradient checkpointing untuk hemat memory
model.gradient_checkpointing_enable()

# Model info
print(f"✅ Model created")
print(f"   Parameters: {model.num_parameters():,}")
print(f"   Size (FP16): {model.num_parameters() * 2 / 1e9:.2f} GB")
print(f"   Architecture: 20 layers, 14 heads")

⚙️ 3. Setup Training

from transformers import Trainer, TrainingArguments, DataCollatorForLanguageModeling

# Data collator untuk causal LM
data_collator = DataCollatorForLanguageModeling(
    tokenizer=tokenizer,
    mlm=False,  # Causal LM, bukan masked LM
)

# Training arguments
training_args = TrainingArguments(
    output_dir="./caca-indo-caca-250m",
    
    # === Training Schedule ===
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=32,  # Effective BS = 4 × 32 = 128
    
    # === Learning Rate ===
    learning_rate=3e-4,              # Standard untuk pretrain
    lr_scheduler_type="cosine",      # Cosine decay
    warmup_steps=2000,               # 5-10% dari total steps
    
    # === Optimization ===
    bf16=True,                       # Mixed precision (recommended)
    gradient_checkpointing=True,     # Hemat memory
    max_grad_norm=1.0,               # Gradient clipping
    weight_decay=0.1,                # Regularization
    optim="adamw_torch",             # Optimizer
    
    # === Logging & Saving ===
    logging_steps=10,
    logging_first_step=True,
    save_steps=1000,
    save_total_limit=3,              # Keep 3 latest checkpoints
    
    # === Evaluation ===
    evaluation_strategy="steps",
    eval_steps=1000,
    
    # === Performance ===
    dataloader_num_workers=4,
    dataloader_pin_memory=True,
    remove_unused_columns=True,
    
    # === Reporting (optional) ===
    # report_to="wandb",             # Untuk Weights & Biases
    # run_name="caca-caca-250m-pretrain",
)

# Initialize trainer
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_dataset["train"],
    eval_dataset=tokenized_dataset["validation"],
    data_collator=data_collator,
    tokenizer=tokenizer,
)

print("✅ Trainer initialized")
print(f"   Total steps: {trainer.state.max_steps:,}")
print(f"   Warmup steps: {training_args.warmup_steps:,}")
print(f"   Effective batch size: {training_args.per_device_train_batch_size * training_args.gradient_accumulation_steps}")

🚀 4. Start Training

# Start training
print("🚀 Starting training...")
trainer.train()

# Save final model
print("💾 Saving final model...")
trainer.save_model("./caca-indo-caca-250m-final")
tokenizer.save_pretrained("./caca-indo-caca-250m-final")

print("✅ Training complete!")

📊 5. Monitor Training

# Install & login
pip install wandb
wandb login

# Di training_args
training_args = TrainingArguments(
    ...
    report_to="wandb",
    run_name="caca-pretrain",
)

# Launch TensorBoard
tensorboard --logdir ./caca-indo-caca-250m/runs

# Di training_args
training_args = TrainingArguments(
    ...
    report_to="tensorboard",
)

🔄 6. Resume Training

# Resume dari checkpoint
trainer.train(resume_from_checkpoint="./caca-indo-caca-250m/checkpoint-1000")

# Atau auto-resume dari last checkpoint
trainer.train(resume_from_checkpoint=True)

☁️ 7. Push to Hub

# Login ke Hugging Face
from huggingface_hub import login
login()

# Push model
model.push_to_hub("your-username/caca-indo-caca-250m")
tokenizer.push_to_hub("your-username/caca-indo-caca-250m")

print("✅ Model uploaded to Hugging Face Hub!")

🎨 Text Generation Examples

🔤 Basic Generation

from caca_transformers import CacaForCausalLM
from transformers import AutoTokenizer
import torch

# Load model & tokenizer
model = CacaForCausalLM.from_pretrained("your-username/caca-indo-caca-250m")
tokenizer = AutoTokenizer.from_pretrained("your-username/caca-indo-caca-250m")

model.eval()
model.cuda()

# Prepare input
prompt = "Ibu kota Indonesia adalah"
input_ids = tokenizer(prompt, return_tensors="pt").input_ids.cuda()

# Generate
with torch.no_grad():
    outputs = model.generate(
        input_ids,
        max_length=100,
        temperature=0.7,
        top_p=0.9,
        do_sample=True,
        use_cache=True,  # KV cache untuk speedup
    )

# Decode
generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(generated_text)

🎯 Advanced Generation Parameters

outputs = model.generate(
    input_ids,
    max_length=100,
    do_sample=False,
    use_cache=True,
)

outputs = model.generate(
    input_ids,
    max_length=100,
    num_beams=5,
    early_stopping=True,
    use_cache=True,
)

outputs = model.generate(
    input_ids,
    max_length=100,
    do_sample=True,
    top_k=50,
    top_p=0.95,
    temperature=0.8,
    use_cache=True,
)

outputs = model.generate(
    input_ids,
    max_length=100,
    do_sample=True,
    temperature=0.7,
    top_p=0.9,
    repetition_penalty=1.2,
    use_cache=True,
)

📦 Batch Generation

prompts = [
    "Ibu kota Indonesia adalah",
    "Presiden pertama Indonesia adalah",
    "Bahasa pemrograman Python adalah",
]

# Tokenize batch
inputs = tokenizer(prompts, return_tensors="pt", padding=True).to("cuda")

# Generate batch
with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_length=100,
        temperature=0.7,
        top_p=0.9,
        do_sample=True,
        use_cache=True,
        pad_token_id=tokenizer.pad_token_id,
    )

# Decode
generated_texts = tokenizer.batch_decode(outputs, skip_special_tokens=True)
for i, text in enumerate(generated_texts):
    print(f"=== Prompt {i+1} ===")
    print(text)
    print("-" * 50)

📡 Streaming Generation

from transformers import TextIteratorStreamer
from threading import Thread

streamer = TextIteratorStreamer(tokenizer, skip_special_tokens=True)

generation_kwargs = dict(
    input_ids=input_ids,
    max_length=100,
    temperature=0.7,
    top_p=0.9,
    do_sample=True,
    streamer=streamer,
    use_cache=True,
)

# Generate di thread terpisah
thread = Thread(target=model.generate, kwargs=generation_kwargs)
thread.start()

# Stream output token by token
print("Generating: ", end="", flush=True)
for text in streamer:
    print(text, end="", flush=True)
print()

thread.join()

🏗️ Arsitektur Detail

📐 Konfigurasi Teknis

🔍 Struktur Layer

Input
  ├─ Token Embedding [100,000 → 896]
  ├─ RoPE Position Encoding (θ=10,000.0)
  │
  ├─ Transformer Block ×20
  │   ├─ RMSNorm (ε=1e-06)
  │   ├─ Grouped Query Attention
  │   │   ├─ Query: 14 heads × 64 dim
  │   │   ├─ Key/Value: 2 heads × 64 dim
  │   │   ├─ Sliding Window: 2,048 tokens
  │   │   └─ Multi-backend: Flash/xFormers/SDPA/Standard
  │   ├─ Residual Connection
  │   ├─ RMSNorm (ε=1e-06)
  │   ├─ SwiGLU FFN
  │   │   ├─ Gate: [896 → 2432]
  │   │   ├─ Up: [896 → 2432]
  │   │   ├─ SiLU(gate) × up
  │   │   └─ Down: [2432 → 896]
  │   └─ Residual Connection
  │
  ├─ Final RMSNorm
  └─ LM Head [896 → 100,000]

💡 Key Components Explained

# Query heads: 14
# KV heads: 2
# Ratio: 7:1


	
		
	
	
		Setiap KV head di-share oleh 7 query heads
	

key_states = key_states.repeat_interleave(7, dim=1)
value_states = value_states.repeat_interleave(7, dim=1)

q' = q * cos(mθ) + rotate(q) * sin(mθ)
k' = k * cos(mθ) + rotate(k) * sin(mθ)

θ = 10,000.0
m = position index

SwiGLU(x) = Swish(W₁·x) ⊗ (W₂·x)
Swish(x) = x · sigmoid(βx)

Gate projection: W₁ [896 → 2432]
Up projection: W₂ [896 → 2432]
Down projection: W₃ [2432 → 896]

RMSNorm(x) = x / RMS(x) * γ
RMS(x) = sqrt(mean(x²) + ε)

ε = 1e-06

# Layer 1: 2,048 tokens
# Layer 2: 4,096 tokens
# Layer 20: 40,960 tokens


	
		
	
	
		Model bisa "melihat" sampai 40,960 tokens ke belakang!
	

💻 Kebutuhan Hardware

🎓 Training Requirements

GC = Gradient Checkpointing

🚀 Inference Requirements

💡 Tip: Dengan gradient checkpointing, memory training bisa turun 30-50% tapi compute time naik ~30%

📊 GPU Recommendations by Use Case

📚 Model Family

Tersedia 30 ukuran model dari 10M hingga 1T parameters:

🔬 Model Introspection & Debugging

📊 Check Model Info

from caca_transformers import CacaForCausalLM

model = CacaForCausalLM.from_pretrained("Lyon28/caca-250m")

# Parameter count
total_params = model.num_parameters()
trainable_params = model.num_parameters(only_trainable=True)

print(f"✅ Total parameters: {total_params:,}")
print(f"✅ Trainable parameters: {trainable_params:,}")

# Model size
model_size_fp16 = total_params * 2 / 1e9
model_size_fp32 = total_params * 4 / 1e9

print(f"✅ Model size (FP16): {model_size_fp16:.2f} GB")
print(f"✅ Model size (FP32): {model_size_fp32:.2f} GB")

# Architecture info
config = model.config
print(f"\n📐 Architecture:")
print(f"   Hidden size: {config.hidden_size:,}")
print(f"   Num layers: {config.num_hidden_layers}")
print(f"   Attention heads: {config.num_attention_heads}")
print(f"   KV heads: {config.num_key_value_heads}")
print(f"   GQA ratio: {config.num_attention_heads // config.num_key_value_heads}:1")
print(f"   Head dim: {config.head_dim}")
print(f"   Context length: {config.max_position_embeddings:,}")
print(f"   Sliding window: {config.sliding_window:,}")

⚡ Check Attention Backend

import logging
logging.basicConfig(level=logging.INFO)

# Check available backends
attn_layer = model.model.layers[0].self_attn

print("🔍 Available backends:")
print(f"   Flash Attention: {attn_layer.has_flash_attn}")
print(f"   xFormers: {attn_layer.has_xformers}")
print(f"   SDPA: {attn_layer.has_sdpa}")

# Run dummy forward untuk lihat backend yang digunakan
dummy_input = torch.randint(0, 1000, (1, 10)).cuda()
model = model.cuda()

print("\n🚀 Running forward pass...")
_ = model(dummy_input)
# Log akan muncul: "✅ Using Flash Attention backend" atau fallback

💾 Memory Profiling

import torch

model = model.cuda().eval()

# Warmup
print("🔥 Warming up...")
for _ in range(5):
    with torch.no_grad():
        _ = model(torch.randint(0, 1000, (1, 512)).cuda())

# Profile
print("📊 Profiling memory...")
torch.cuda.reset_peak_memory_stats()
torch.cuda.empty_cache()

with torch.no_grad():
    _ = model(torch.randint(0, 1000, (1, 2048)).cuda())

peak_memory = torch.cuda.max_memory_allocated() / 1e9
current_memory = torch.cuda.memory_allocated() / 1e9

print(f"✅ Peak memory: {peak_memory:.2f} GB")
print(f"✅ Current memory: {current_memory:.2f} GB")

⏱️ Speed Benchmark

import time
import torch

model = model.cuda().eval()
batch_size = 1
seq_length = 512
input_ids = torch.randint(0, 1000, (batch_size, seq_length)).cuda()

# Warmup
print("🔥 Warming up...")
for _ in range(10):
    with torch.no_grad():
        _ = model(input_ids)

# Benchmark
print("⏱️ Benchmarking...")
torch.cuda.synchronize()
start = time.time()

num_runs = 100
for _ in range(num_runs):
    with torch.no_grad():
        _ = model(input_ids)

torch.cuda.synchronize()
elapsed = time.time() - start

# Calculate metrics
total_tokens = num_runs * seq_length
tokens_per_sec = total_tokens / elapsed
ms_per_token = (elapsed * 1000) / total_tokens
samples_per_sec = num_runs / elapsed

print(f"✅ Throughput: {tokens_per_sec:.0f} tokens/sec")
print(f"✅ Latency: {ms_per_token:.2f} ms/token")
print(f"✅ Samples/sec: {samples_per_sec:.1f}")
print(f"✅ Total time: {elapsed:.2f}s for {num_runs} runs")

🔍 Layer-wise Analysis

# Analyze each layer
print("📊 Layer-wise parameter count:\n")

layer_total = 0  # ✅ Ganti nama variable biar gak bentrok
for name, module in model.named_modules():
    if isinstance(module, (torch.nn.Linear, torch.nn.Embedding)):
        params = sum(p.numel() for p in module.parameters())
        layer_total += params
        print(f"{name:50s}: {params:>12,} params")

print(f"\n{'='*65}")
print(f"{'Total':50s}: {layer_total:>12,} params")  # ✅ Use layer_total

# Check gradient status
print("\n🎯 Gradient status:")

print(f"   Trainable: 257,081,216 (100.0%)")
print(f"   Frozen: 0 (0.0%)")

❓ FAQ (Frequently Asked Questions)

Flash Attention (4x) → xFormers (3x) → SDPA (2x) → Standard (1x)

# Model akan jalan dengan/tanpa Flash Attention
config = CacaConfig(use_flash_attn=True)  # Auto fallback
model = CacaForCausalLM(config)

# Gunakan xFormers (masih 3x speedup)
pip install xformers


	
		
	
	
		Atau gunakan SDPA built-in (2x speedup)
	


	
		
	
	
		Tidak perlu install apapun, PyTorch 2.0+ sudah include
	

pip install xformers
# Masih dapat 3x speedup!

config = CacaConfig(
    use_flash_attn=False
)
model = CacaForCausalLM(config)
# Fallback ke SDPA/Standard

# Automatic dengan .generate()
outputs = model.generate(
    input_ids,
    max_length=100,
    use_cache=True,  # Enable KV cache
)

# Manual forward pass
outputs = model(input_ids, use_cache=True)
past_kv = outputs['past_key_values']

# Next token dengan cache
next_token_ids = torch.tensor([[next_token]])
next_outputs = model(
    next_token_ids,
    past_key_values=past_kv,
    use_cache=True,
)

from transformers import Trainer, TrainingArguments

# Load pretrained model
model = CacaForCausalLM.from_pretrained("Lyon28/caca-250m")

# Fine-tuning args (LR lebih kecil dari pretrain)
training_args = TrainingArguments(
    output_dir="./caca-finetuned",
    learning_rate=1e-5,          # Lebih kecil dari pretrain (3e-4)
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=8,
    warmup_steps=100,
    bf16=True,
    gradient_checkpointing=True,
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=your_task_dataset,
)

trainer.train()

# Install Optimum
pip install optimum[onnxruntime]

# Export ke ONNX
from optimum.onnxruntime import ORTModelForCausalLM

model = ORTModelForCausalLM.from_pretrained(
    "Lyon28/caca-250m",
    export=True,
    provider="CUDAExecutionProvider",  # Untuk GPU
)

# Save ONNX model
model.save_pretrained("./caca-onnx")

💡 Use Cases

📊 Perbandingan dengan Arsitektur Lain

💡 Tips & Best Practices

🎓 Untuk Training

from transformers import TrainingArguments

# Recommended hyperparameters untuk pretraining
training_args = TrainingArguments(
    # Learning rate
    learning_rate=3e-4,              # Standard untuk pretrain
    lr_scheduler_type="cosine",      # Cosine decay recommended
    warmup_steps=2000,               # 5-10% dari total steps
    
    # Optimization
    max_grad_norm=1.0,               # Gradient clipping
    weight_decay=0.1,                # Regularization
    bf16=True,                       # Mixed precision (recommended)
    
    # Memory
    gradient_checkpointing=True,     # Hemat memory 30-50%
    per_device_train_batch_size=4,   # Adjust sesuai GPU
    gradient_accumulation_steps=32,  # Effective BS = 4 × 32 = 128
    
    # Logging
    logging_steps=10,
    logging_first_step=True,
    
    # Saving
    save_steps=1000,
    save_total_limit=3,              # Keep 3 latest checkpoints
)

🚀 Untuk Inference

# Enable eval mode
model.eval()

# Use FP16 untuk speedup
model = model.half()

# Generate dengan optimal settings
with torch.no_grad():
    outputs = model.generate(
        input_ids,
        max_length=512,
        temperature=0.7,              # Control randomness
        top_p=0.9,                    # Nucleus sampling
        do_sample=True,
        use_cache=True,               # Penting! KV cache speedup
        pad_token_id=tokenizer.pad_token_id,
    )

💾 Untuk Hemat Memory

# 1. Gradient checkpointing
model.gradient_checkpointing_enable()

# 2. 8-bit training
from transformers import BitsAndBytesConfig

config = BitsAndBytesConfig(
    load_in_8bit=True,
    llm_int8_threshold=6.0,
)

model = CacaForCausalLM.from_pretrained(
    "Lyon28/caca-250m",
    quantization_config=config,
    device_map="auto",
)

# 3. Clear cache periodically
import torch
torch.cuda.empty_cache()

⚡ Untuk Multi-GPU

# Dengan Accelerate
from accelerate import Accelerator

accelerator = Accelerator()
model, optimizer, train_dataloader = accelerator.prepare(
    model, optimizer, train_dataloader
)

# Dengan DeepSpeed
training_args = TrainingArguments(
    deepspeed="ds_config.json",
    ...
)

⚠️ Keterbatasan

# 1. Gunakan tokenizer dari model lain Atau Train Tokenize
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("gpt2")

# 2. Disable Flash Attention jika error
config = CacaConfig(use_flash_attn=False)

# 3. load Model
model = CacaForCausalLM.from_pretrained("Lyon28/caca-250m")

# 4. Gradient checkpointing untuk model besar
model.gradient_checkpointing_enable()

# 5. 8-bit training untuk hemat memory
from transformers import BitsAndBytesConfig
config = BitsAndBytesConfig(load_in_8bit=True)

📄 Lisensi

Apache License 2.0 - Bebas digunakan untuk keperluan komersial dan penelitian.

Lihat LICENSE untuk detail lengkap.

📧 Kontak & Support

🔗 Resources

📌 Citation

Jika Anda menggunakan arsitektur Caca dalam penelitian atau project, kami sangat menghargai jika Anda cite:

@misc{caca2025,
  author = {Lyon},
  title = {Caca: Modern Transformer Architecture with GQA, RoPE, and SwiGLU},
  year = {2025},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/Lyon28/caca-250m}},
  note = {Apache License 2.0}
}

🙏 Acknowledgments

Arsitektur Caca terinspirasi dari berbagai penelitian dan implementasi:

• LLaMA (Meta AI) - GQA dan arsitektur dasar
• Mistral (Mistral AI) - Sliding Window Attention
• GPT-NeoX (EleutherAI) - Implementasi RoPE
• PaLM (Google) - SwiGLU activation
• Flash Attention (Dao et al.) - Efficient attention implementation

📈 Changelog

v0.0.1 (2025)

• 🎉 Initial release
• ✅ 30 variants (10M - 1T parameters)
• ✅ GQA, RoPE, SwiGLU, RMSNorm
• ✅ Sliding Window Attention dengan enforcement
• ✅ KV cache support
• ✅ Flash Attention support (opsional)
• ✅ Multi-backend attention (Flash/xFormers/SDPA/Standard)
• ✅ Extensive input validation & error handling
• ✅ Helpful error messages dengan solutions
• ✅ Config validation dengan warnings
• ✅ PyPI package: caca-transformers
• ✅ Hugging Face Hub integration
• ✅ Gradient checkpointing support
• ✅ Auto-fallback mechanism

🌟 Star History

💖 Support Project

Jika project ini helpful untuk Anda:

Setiap kontribusi sangat berarti! 🙏

⬆️ Back to top