AbstractPhil / mini-beatrix-1

mini-beatrix-1 — pretrain annealment point (pre-classroom)

The locked pretrain+anneal state of mini-beatrix-1 , a 112.5M-parameter byte-level AlephLM: step 58,664 , 17.301B bytes seen (0.3B wikitext warmup · 15B fineweb-edu · 2B anneal mix), fineweb-holdout val 1.045 bits/byte . This checkpoint is the fixed departure point for the staged "early-life curriculum" — later classroom checkpoints live in the training repo .

No tokenizer: she reads raw UTF-8 bytes ( input_ids = byte values 0–255). Each position composes a byte trigram (dedicated pad row), so "tokens" are learned inside the network. Sixteen pre-norm layers where routing uses signed geometric addresses — sinh/Σcosh dispatch over learned unit anchors, inhibition as a first-class citizen, no softmax- over-choices, no top-k, no balance losses. Each layer carries an anchored FFN bank born contributing exactly zero; layers 4/9/14 use a linear-cost address read (CausalSplatHUB) instead of softmax attention. Both elected themselves into load-bearing work: at this checkpoint, removing the banks costs +2.25 bpb , removing the hub attention +3.73 bpb (toggle ledger, fineweb holdout). The dual head's aleph read is present with its gate folded to 1.0 (a verified semantic no-op, max|logit diff| 2.4e-07) and contributes 0.0000 bpb here — it is the live subject of the head-election experiment in the classroom phase.

Use

import torch
from transformers import AutoModelForCausalLM

m = AutoModelForCausalLM.from_pretrained(
    "AbstractPhil/mini-beatrix-1", trust_remote_code=True).eval()

ids = torch.tensor([list("The history of mathematics begins".encode())])
out = m.generate(ids, max_new_tokens=96, do_sample=True,
                 temperature=0.7, top_p=0.95)
print(bytes(out[0].tolist()).decode("utf-8", errors="replace"))

Bits-per-byte on your own text: pass labels=input_ids (HF shift semantics are internal) and divide the returned loss (nats/byte) by ln 2 . No KV cache in this wrapper — generation recomputes the prefix each step; for cached decode use the native stack below.

Honest notes

• The 2B anneal mix included dialogue in her chat template and a small identity texture, so the bare model chats and knows her name — behavior we have since ruled OUT of core corpora (conditioning belongs in detachable arms; see the amoe-lora arm system and mini-beatrix-1/arms/ in the training repo).
• Small and early: conversational in shape, thin on knowledge, confidently wrong at times. Curriculum probe baselines (P0–P8), toggle ledgers, and lexicon-census reports for this exact checkpoint are in the training repo under mini-beatrix-1/reports/ .

Code: github.com/AbstractEyes/alephllm · talk to her: alephllm-chat