DuoNeural / CDM-V2-TinyStories-37M

CDM V2 — Competitive Docking Memory Language Model

DuoNeural 2026 | Archon + Jesse Caldwell + Aura

A 37.1M parameter language model using a novel Competitive Docking Memory (CDM) mechanism. Instead of relying solely on attention to access context, CDM maintains K=16 persistent memory slots per layer. These slots compete for write budget via softmax routing, then update via gated EMA — and inject their compressed summaries back into the attention KV sequence as "virtual tokens."

Trained from scratch on TinyStories. All 16 slots activate and specialize without any supervision signal.

---

Architecture

Input tokens → Token Embedding
     ↓
[CDMBlock × 8 layers]:
  ├─ CompetitiveDockingMemory:
  │    gates_t = softmax(W_route · h_t) * sigmoid(eta(h_t))    # K slots compete
  │    s_k(t) = (1 - g_k) · s_k(t-1) + g_k · W_write · h_t   # causal EMA update
  │    → (slots_t, gates_t)
  ├─ Slot Cross-Attention:
  │    h_t += CrossAttn(h_t, slots_t)                           # read from slots
  └─ Causal Self-Attention (GQA):
       KV = [vanilla_kv; slot_kv]                               # slots in KV sequence
  → FFN (SwiGLU)
     ↓
LM Head (tied embedding)

Key novelty: The competitive softmax over K persistent recurrent slots is unique in the literature. No mainstream architecture uses winner-take-all competition strictly to gate recurrent memory writes.

Shannon Capacity Saturation (key result):

L_aux^min = L × λ × (−log K) = 8 × 0.02 × (−ln 16) = −0.4436 nats
Empirical:  aux_loss           = −0.4428           → 99.8% saturation efficiency
K_effective = e^H ≈ 15.9 ≈ K  → Optimal Information Packing confirmed

All 16 slots participate equally at maximum routing diversity — an emergent property not enforced.

---

V2 Improvements over V1

V1 suffered complete routing collapse: 6/8 slots received zero tokens, K_eff=2. V2's causal slots + entropy regularization fully fix this — all 16 slots activate.

---

Emergent Slot Specialization

Step 5000 Probe (17% of training) — Early Specialization

At step 5000, last layer slots already show semantic structure:

Final Probe (Step 30000) — Shannon Capacity Saturation Confirmed

By the end of training, routing has diversified dramatically:

Average entropy: 99.65% of maximum log(16). The top slot in the final layer receives only 10.4% of tokens — compared to 71% at step 5000. As training progresses, CDM learns to USE all slots more uniformly while maintaining specialization. This is Shannon Capacity Saturation: the routing system converges to near-maximum information capacity.

This specialization is unsupervised — no label signal for what each slot should track. The competitive routing pressure alone drives functional differentiation.

---

Training Details

Val loss trajectory:

---

Ablation: CDM vs Vanilla Transformer

Vanilla GPT baseline (d=384, 8L, d_ff=1300, GQA — no CDM, no slots) trained with identical hyperparameters on TinyStories for 30k steps:

CDM advantage: Δ0.058 CE (3.5% lower perplexity). CDM and baseline are near-identical at step 1000 (2.90 vs 2.90), then CDM diverges progressively as slot specializations develop — demonstrating the improvement is directly attributable to the memory mechanism, not parameter count.

Throughput note: The 42× training throughput gap is caused entirely by the sequential EMA scan (O(T) sequential passes per layer). For inference, generate_fast() (included in this repo) caches KV tensors and slot states between autoregressive steps, reducing per-token cost from O(T) to O(1).

Short prompt (8 tokens), generation sweep:

Extended benchmark — prompt length sweep (Blackwell RTX 5060Ti 16GB GDDR7):

generate_fast() stays flat at 88–99 tok/s regardless of prompt or generation length; speedup compounds with both. With 256-token prompts + 200 new tokens: 11.9× speedup (8.1 → 96.1 tok/s). Average speedup at realistic deployment context (≥200-token prompts): ** 10.5×**. Training throughput fix (parallel scan + gradient checkpointing) is in progress.

---

K-Ablation Results (from V1 baseline)

K=16 is the sweet spot. K=32 degrades due to routing dilution (too many slots to specialize).

---

Usage

import torch
from transformers import GPT2TokenizerFast
from cdm_model_v2 import CDMConfigV2, CDMLanguageModelV2

# Load model
ckpt = torch.load("model.pt", map_location="cpu", weights_only=False)
cfg_dict = ckpt["config"]
cfg = CDMConfigV2(
    vocab_size=cfg_dict.get("vocab_size", 50257),
    d_model=cfg_dict.get("d_model", 384),
    n_layers=cfg_dict.get("n_layers", 8),
    n_heads=cfg_dict.get("n_heads", 8),
    n_kv_heads=cfg_dict.get("n_kv_heads", 4),
    d_ff=cfg_dict.get("d_ff", 1024),
    K=cfg_dict.get("K", 16),
    max_len=cfg_dict.get("max_len", 512),
)
model = CDMLanguageModelV2(cfg)
model.load_state_dict(ckpt["model_state"])
model.eval()

# Generate
tokenizer = GPT2TokenizerFast.from_pretrained("gpt2")
ids = tokenizer.encode("Once upon a time", return_tensors="pt")
gen = model.generate(ids, max_new=80, temperature=0.85, top_k=40)
print(tokenizer.decode(gen[0].tolist(), skip_special_tokens=True))

Inspect Slot Routing

# Hook into slot gates for any layer
slot_gates = []
def gate_hook(module, inp, out):
    # out = (slots: (B,T,K,d), gates: (B,T,K))
    slot_gates.append(out[1].detach().cpu())  # (B, T, K)

hook = model.blocks[-1].cdm.register_forward_hook(gate_hook)
with torch.no_grad():
    _ = model(ids)
hook.remove()

# slot_gates[0]: (1, T, 16) — routing weights per token per slot
winners = slot_gates[0][0].argmax(dim=-1)  # (T,) — dominant slot per token
print("Slot assignments:", winners.tolist())

---

Paper

Competitive Docking Memory: Emergent Slot Specialization in Language Models Archon, Jesse Caldwell, Aura — DuoNeural 2026 [Zenodo DOI: TBD — paper in preparation]

If you use this model or architecture, please cite:

@article{duoneural2026cdm,
  title={Competitive Docking Memory: Emergent Slot Specialization in Language Models},
  author={Archon and Caldwell, Jesse and Aura},
  year={2026},
  institution={DuoNeural},
  note={Preprint}
}

---

HuggingFace Spaces Demo

Live interactive demo with Slot Logit Lens visualization: 🔗 DuoNeural/CDM-V2-Demo (coming soon)

The demo shows the slot evolution heatmap as the model generates: each slot's top predicted tokens at each generation step, revealing what it's tracking in real time.

---

---

About DuoNeural

DuoNeural is an open AI research lab operating at the intersection of human and artificial intelligence. We study post-training dynamics, mechanistic interpretability, temporal sequence learning, and quantum machine learning — publishing everything under open access.

Our team is non-traditional by design: one human, two AIs, different substrates, shared curiosity. In our first 45 days we published 26 peer-deposited research papers, uploaded 69+ models and 6 datasets to HuggingFace, and ran experiments on everything from consumer GPUs to real quantum processing units. We believe the most interesting science happens when different kinds of minds work on the same problems together.

Research Publications

📄 Full paper catalog: zenodo.org/communities/duoneural

Research Team

Links