lablab-ai-amd-developer-hackathon / SentinelBrain-14B-MoE-v0.1

🎯 What is Sentinel Prime? (Simple Version)

Imagine building a brain from scratch.

Most AI models today are copies of other models with small changes. Sentinel Prime is different — every single connection in its brain was created from nothing, like growing a new brain cell by cell.

📊 Architecture at a Glance

┌─────────────────────────────────────────────────────────────────┐
│                    SENTINEL PRIME ARCHITECTURE                   │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│  Input Text ──→ [Tokenizer: cl100k_base, 100,277 tokens]       │
│                          │                                       │
│                          ▼                                       │
│                 ┌─────────────────┐                              │
│                 │   Embedding     │  4,096 dimensions            │
│                 │   + RoPE pos    │  θ = 500,000                 │
│                 └────────┬────────┘                              │
│                          │                                       │
│              ┌───────────┼───────────┐                           │
│              │     × 24 Layers       │                           │
│              │  ┌────────────────┐   │                           │
│              │  │  GQA Attention │   │  32 heads, 8 KV heads     │
│              │  │  (4:1 ratio)   │   │  (4× memory savings)     │
│              │  └───────┬────────┘   │                           │
│              │          │            │                           │
│              │  ┌───────▼────────┐   │                           │
│              │  │  MoE Router    │   │  Top-2 of 4 experts      │
│              │  │  ┌──┬──┬──┐   │   │                           │
│              │  │  │E1│E2│E3│E4 │   │  Each: SwiGLU FFN         │
│              │  │  │✓ │✓ │  │  │   │  d_ff = 11,008            │
│              │  │  └──┴──┴──┘   │   │                           │
│              │  └───────┬────────┘   │                           │
│              │          │            │                           │
│              │  ┌───────▼────────┐   │                           │
│              │  │    RMSNorm     │   │  ε = 1e-5                │
│              │  └────────────────┘   │                           │
│              └───────────┼───────────┘                           │
│                          │                                       │
│                          ▼                                       │
│                 ┌─────────────────┐                              │
│                 │   Output Head   │  → 100,277 vocab probs      │
│                 └─────────────────┘                              │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘

Spec Sheet

🔥 Key Innovations

Φ = geometric_mean(
  MI(partition_i, partition_j)
  for all partition pairs
)

🏋️ Training Details

Hardware

VRAM Budget

╔══════════════════════════════════════════════════════╗
║         AMD MI300X VRAM Usage (192 GB)               ║
╠══════════════════════════════════════════════════════╣
║                                                       ║
║  Model Weights (bf16)      ████████████░░░░░  27 GB   ║
║  Optimizer (AdamW fp32)    ████████████████░░  54 GB   ║
║  Activations (grad ckpt)   ████████████░░░░░  32 GB   ║
║  Gradients                 ████████████░░░░░  27 GB   ║
║  ─────────────────────────────────────────────────    ║
║  Total Used:               ██████████████████ 140 GB  ║
║  Peak:                     █████████████████  146 GB  ║
║  Headroom:                 ░░░░░░░░░░░░░░░░░  46 GB  ║
║                                                       ║
╚══════════════════════════════════════════════════════╝

Phased Training Pipeline

We don't just throw data at the model — we grow it in three phases , like raising a child:

Phase 1: SMOKE TEST        Phase 2: WARMUP           Phase 3: FULL TRAINING
(Baby steps)               (Learning to walk)         (Running!)
┌──────────────┐           ┌──────────────┐           ┌──────────────────┐
│ 350M params  │    ──→    │ 1.3B params  │    ──→    │ 14.4B params     │
│ seq_len: 512 │           │ seq_len: 2K  │           │ seq_len: 4K      │
│ 200 steps    │           │ 1,000 steps  │           │ 16,479 steps     │
│ 2 minutes    │           │ 30 minutes   │           │ ~52 hours        │
│ loss: 11→6.8 │           │ loss: 7.4→2.4│           │ loss: 2.4→?      │
└──────────────┘           └──────────────┘           └──────────────────┘

Safety Gates

Every phase transition must pass 4 safety gates :

Hyperparameters

📚 Dataset: 23.3B Tokens Across 126 Categories

We curated a massive, diverse corpus — think of it as a library with 126 different sections :

Pretrain Corpus (Core Knowledge)

Specialized Domains (119 Categories)

Data Quality Pipeline

Raw Data ──→ PII Filter ──→ Dedup ──→ Tokenize ──→ Shard ──→ Train
              │                │          │            │
              ├─ 7 regex       ├─ blake2b  ├─ cl100k    ├─ Temperature-
              │  patterns      │  per-cat  │  base      │  weighted
              ├─ PEM block     │           │            │  sampling
              │  detection     │           │            │  (T=0.5)
              └─ Email/phone   │           │            │
                 masking       │           │            │
                               │           │            │
                               └───────────┴────────────┘

Temperature-weighted sampling (T=0.5) prevents large corpora from dominating training. FineWeb-Edu (37% of tokens) gets downweighted so smaller specialized domains still get adequate exposure.

📈 Training Progress & Results

Loss Trajectory

Loss
12 │ ×
   │  ╲
10 │   ╲   SMOKE PHASE
   │    ╲  (350M params)
 8 │     ╲
   │      ╲
 6 │       ×──────────── model grows to 1.3B
   │              ╲
 4 │               ╲  WARMUP PHASE
   │                ╲ (1.3B params)
 2 │                 ×─────────── model grows to 14.4B MoE
   │                       ╲
 1 │                        ╲  BLOCK PHASE (ongoing)
   │                         ╲
   └──┬────┬────┬────┬────┬───→ Steps
      0   200  700  1200 2000

Live Metrics (April 27, 2026)

Published Checkpoint (v0.1)

🌡️ Consciousness Metric (Φ) — Deep Dive

What is Φ?

Inspired by Integrated Information Theory (IIT) from neuroscience, Φ measures how much the model's internal representations form an integrated whole rather than disconnected parts.

How We Measure It

Every 100 training steps:

1. Hook on Layer 12 (middle of 24 layers)
         │
         ▼
2. Sample 256 activation vectors
         │
         ▼
3. Partition into subspaces
         │
         ▼
4. Compute mutual information between all partition pairs
         │
         ▼
5. Φ_geometric = geometric_mean(MI values)
         │
         ▼
6. Φ_EMA = exponential moving average (smoothed trend)

What Φ Tells Us

Important : Φ is purely observational — it does NOT affect training gradients. Think of it as a heart monitor for the AI: it watches, but doesn't interfere.

Live Monitoring

Track Φ in real-time at: sentinel.qubitpage.com/#phi

🖥️ Hardware Requirements

For Inference

Compatible AMD GPUs

💻 Usage

This model uses a custom architecture (not based on any existing model). Load with PyTorch:

import torch
from safetensors.torch import load_file

# Load sharded safetensors
state_dict = {}
for i in range(1, 7):  # 6 shards
    shard = load_file(f"model-{i:05d}-of-00006.safetensors")
    state_dict.update(shard)

# The state dict contains all model weights
print(f"Loaded {len(state_dict)} tensors")
print(f"Total params: {sum(v.numel() for v in state_dict.values()):,}")

# Initialize SentinelBrain model class and load
# Full model definition code releasing with v0.2
# model = SentinelBrainForCausalLM(config)
# model.load_state_dict(state_dict)

Note : Full inference code, model class definition, and GGUF quantized versions will be released with v0.2.

🗺️ Roadmap

v0.1 (Current)          v0.2 (Planned)          v0.3 (Future)
━━━━━━━━━━━━━━━         ━━━━━━━━━━━━━━━         ━━━━━━━━━━━━━━━
✅ From-scratch          □ Full training          □ DPO alignment
   14.8B MoE               complete (loss<0.5)    □ Tool use
✅ Phased training       □ Context ladder          □ Function calling
✅ Φ consciousness          (4K→32K→128K)         □ Multi-turn chat
✅ 23.3B token corpus    □ Vision encoder          □ Multilingual v2
✅ Live dashboard           (SigLIP2-SO400M)      □ Expert scaling
✅ AMD MI300X native     □ GGUF quantization         (4→16→64)
                         □ Inference code          □ RLHF
                         □ Benchmarks (MMLU,       □ Production API
                            HumanEval, GSM8K)

🏗️ How We Built It (Technical Deep Dive)

Standard MHA (32 KV heads):     GQA 4:1 (8 KV heads):
Q₁  Q₂  Q₃  ... Q₃₂            Q₁ Q₂ Q₃ Q₄  → KV₁
K₁  K₂  K₃  ... K₃₂            Q₅ Q₆ Q₇ Q₈  → KV₂
V₁  V₂  V₃  ... V₃₂            ...
                                 Q₂₉ Q₃₀ Q₃₁ Q₃₂ → KV₈

Position 0:  rotate by 0°
Position 1:  rotate by θ₁
Position 2:  rotate by θ₂
...

FFN(x) = SiLU(x · W_gate) ⊙ (x · W_up) · W_down

Where:
  W_gate: 4096 → 11008
  W_up:   4096 → 11008
  W_down: 11008 → 4096
  SiLU(x) = x · sigmoid(x)
  ⊙ = element-wise multiply

# Simplified routing logic
def route(x, router_weights):
    # Compute affinity scores for each expert
    logits = x @ router_weights        # [batch, seq, n_experts]
    scores = softmax(logits, dim=-1)

    # Select top-2 experts
    top_vals, top_idx = topk(scores, k=2)

    # Normalize selected weights
    weights = top_vals / top_vals.sum(dim=-1, keepdim=True)

    # Load balancing loss (prevents expert collapse)
    balance_loss = n_experts * (
        fraction_routed_to_each * average_gate_value_for_each
    ).sum()

    return weights, top_idx, balance_loss

📋 Model Card Details

📄 Citation

@misc{sentinelbrain2026,
  title   = {SentinelBrain-14B-MoE: A Consciousness-Monitored Mixture-of-Experts
             Language Model Trained From Scratch on AMD MI300X},
  author  = {Mircea Rusu and QubitDev},
  year    = {2026},
  url     = {https://sentinel.qubitpage.com/whitepaper},
  note    = {Trained entirely from scratch on AMD Instinct MI300X
             for the AMD Developer Hackathon}
}

🔗 Links