AbstractPhil / mini-beatrix-2s

mini-beatrix-2s

The first full-splat craft to complete a designed life. 237M parameters, byte-level (vocab 256), with a governed multi-constellation CausalSplatHUB — signed-address linear attention over learned codebook blackboards — in every one of its 20 blocks . No softmax-over-positions attention anywhere. Trained 16.101B tokens (61,422 steps, ~78h on a single RTX 6000 Pro Blackwell) through a staged curriculum, completed 2026-08-31.

Architecture

• d_model 1024 · 20 layers · ctx 4096 · byte-trigram embedding (raw UTF-8 bytes; input ids are byte values 0–255)
• Hubs (all 20 blocks): 4 constellations × 64 anchors @ D=128 per block, min-separation governed from birth (45°), budget-composed (numerators and agreement masses sum before one divide — reconstructive, never comparative: no argmax, no top-k, inhibition first-class). Constant-size prefix state: each layer encodes the sequence onto a fixed-width addressed blackboard rather than caching it.
• Anchored banks: 3 full-width experts per block, signed dispatch.
• Dual head: linear readout + a signed aleph read (256 anchors @ 256), revived mid-mission via a deterministic boundary-write after the born-null path self-buried — the first functioning aleph head in the lineage.
• Final causal contributions (toggle ledger, held-out): hubs +3.36 bpb · banks +3.31 bpb · head +0.012 bpb .

The special-token control plane

Thirteen ids that valid UTF-8 can never produce (proven by exhaustion) carry structure and are trained :

Chat format: [SYS] text [END] [USER] text [END] [MODEL] text [END] — the frame is unforgeable (encoded text cannot contain a special).

Usage

import torch
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "AbstractPhil/mini-beatrix-2s", trust_remote_code=True)

prompt = "The history of astronomy begins"
ids = torch.tensor([list(prompt.encode("utf-8"))])
out = model.generate(ids, max_new_tokens=200, do_sample=True, top_p=0.95)
print(bytes(int(i) for i in out[0]).decode("utf-8", errors="replace"))

Chat-framed prompting (the frame was annealed in the final 1B tokens — treat it as a young capability):

SYS, USER, MODEL, END = 0xFB, 0xFE, 0xFD, 0xFC
turns = ([SYS] + list(b"You are Beatrix, a small byte-level language model.")
         + [END] + [USER] + list(b"Who are you?") + [END] + [MODEL])
out = model.generate(torch.tensor([turns]), max_new_tokens=100,
                     do_sample=True, top_p=0.95)

Training

16.101B tokens on one card: wikitext warmup (0.3B) → fineweb-edu (5B) → a nine-stage early-life curriculum, s0–s8 (8.8B: narrative, perspective, concepts, rule-chains, arithmetic, causal, try-fail, mixed, register) → a two-phase anneal (1B distribution shift without the chat frame, then 1B with it — the frame's effect bracketed by boundary reports). Muon + pure Adam split, flat LR, bf16, zero loss spikes across the entire run. Every boundary shipped a report (toggle ledger, probe suite, address census): 17 reports and all checkpoints live in alephllm-mini-beatrix-training under mini-beatrix-2s/ .

Final validation: 1.1097 bpb on the fineweb-edu holdout — the run's best reading, set during the chat phase.

The softmax twin

A pre-registered control ran the identical recipe and diet beside this craft: mini-beatrix-2s-control — the same preset with the hub list empty, so 16-head causal softmax attention (SDPA) in every one of the 20 blocks (257.4M parameters against this craft's 237M), 16.101B tokens in 61,422 steps, one craft per arm. At the matched endpoint (anneal_mix, step 61,422, same evaluation) the twin read 2.8846 bpb against this craft's 1.1097 , and this craft led at 11 of 11 same-gauge paired boundaries after the twin's early lead (0.3B: twin 2.106 vs this craft 2.363; the first paired row at 5.3B: this craft 1.1172 vs twin 1.1763); the s8 pair is struck because its two rows were measured on different gauges. The twin did not merely trail — it destabilized: its validation, on the one fineweb-edu holdout every row shares, went 1.18 (fineweb) → 1.57 (s4) → 2.18 (s5) → 3.19 (s6) → 3.12 (s7), and the two anneal phases recovered it only to 2.88; its pre-clip gradient norm first crossed the 1.0 clip at step 17,600 inside the fineweb phase while it still led, then escalated stage by stage (medians 0.29 → 6.5 → 2.4 → 12 → 13 → 24.5 → 126 → 471 → 162 → 145 → 167 → 215, spikes to 41,070, clipped on 813 of 824 logged curriculum and anneal steps) while this craft's never reached the clip (0 of 824, max 0.62). The anneal was a diet change (1B of prose, then 1B with the chat frame), not a learning-rate decay: lr_scale read 1.000 throughout on both crafts. This is one craft per arm, and the twin was given none of softmax attention's usual stabilizers (no QK-normalization, no weight decay, no learning-rate decay, the same 1.0 clip). The claim that SDPA is not an adequate representative of this model's attention requirement is AbstractPhil's (2026-09-02); its audit (2026-09-04) grades the specific form an instance — under this recipe the twin is a destabilized control, and the gap measures stability rather than capacity — and the general form a candidate behind the recipe, the single seed, the shape (mini-beatrix-1's thirteen softmax blocks trained clean under the identical configuration), the aleph-addressed expert banks both crafts carry, and the learned position table on decay-free Muon. Paired reports: mini-beatrix-2s-control/reports/v2/ beside this craft's under alephllm-mini-beatrix-training ; the paired chart is article_assets_ft2/01_control_crossover.png there.

Lineage

Code: AbstractEyes/alephllm (this repo vendors the model files verbatim; alephllm 0.8.6). Sibling: mini-beatrix-1 (112M, 3-hub hybrid — the previous rung). Control twin: mini-beatrix-2s-control (softmax attention in every block; weights, resume states and 15 reports under alephllm-mini-beatrix-training/mini-beatrix-2s-control/ ). The research memory behind every design decision is maintained privately; laws referenced in the code docstrings (supply ≤ 2D, min-sep governance, budget composition, born-null weight-zero) are measured, not aesthetic.

Licence

The weights are non-commercial : CC BY-NC-SA 4.0 , because the training text includes CHILDES child-directed speech (CC BY-NC-SA, via TalkBank) and Wikipedia / WikiText (CC BY-SA). The code files, the model design and the full training process are MIT ( AbstractEyes/alephllm ); relay.py is adapted from amoe-lora under Apache-2.0 ( NOTICE ). The full terms and the list of training sources: LICENSE.md .