AbstractPhil / alephlm-adopt-0

alephlm-adopt-0 — an AlephLM by adoption

A live experiment repository. The question: can the aleph — a closed-form signed addressing mechanism — carry the capability delta between a small frozen LLM and its larger sibling, as a fully detachable attachment? Here the trunk is Qwen2.5-0.5B-Instruct, frozen entirely ; the trained object is ~2.9M parameters of attachments distilled by per-token KL from Qwen2.5-1.5B-Instruct .

Two param-matched arms, reported as a contribution ledger (no winner declared — the arms turned out to be different mechanisms, not ranked ones):

Each relay is a gated residual write, born nearly silent (gate σ(−3) ≈ 0.047). The aleph relay reads its input against a codebook of unit anchors with a signed closed-form coefficient ( sinh/Σcosh ) — an anchor can contribute negatively — and writes back a reconstructive read of the codebook; the mlp adapter is the standard bottleneck-residual control.

Distillation verdict (20k steps, ~31M tokens each, wikitext stream)

Both arms cut KL-to-teacher ~25% with ~0.6% of the trunk's parameter count, and both are perfectly detachable — the cleanest toggle-law result in this program at LLM scale.

The mechanistic finding (why no winner is declared): at equal capability the two arms allocated oppositely . The aleph opened its pre-head relay 8× (gate σ .047 → .387, its codebook rotating 0.47 rad) — it elected the output-distribution placement as its main pathway. The mlp closed the same placement 10× (σ → .004) and did its work in the early blocks. One mechanism gravitates to where its lineage predicts its advantage lives; the other flees it.

Sign census (the readout that motivated this repo's name): the aleph relays run a balanced bipolar code — ~50% of coefficients negative, |w| ≈ 1/K, dense across all 64 slots — a reconstructive sign code, categorically different from softmax routing, and invisible to any gauge that takes an absolute value.

Standardized bench

0-shot via lm-eval-harness (acc_norm where defined, else acc):

The honest verdict : the wikitext-KL distillation did not convert to task capability — both arms trade ~.02 of 0-shot accuracy for their ~25% KL improvement, identically , closing none of the 0.5B→1.5B gap (.11). The finding indicts the distillation corpus/objective , not either mechanism: matching the teacher's distribution on encyclopedic text is not the same thing as acquiring its task behavior. The mechanistic differences between the arms (placement election, signed regime, both fully detachable) survive as the real result of this round; an instruct-domain distillation corpus is the named next revision.

Artifacts

• adopt0/adopt0_aleph_s0.pt — aleph relays + head relay (state dict keyed block{i}.* / head.* ), with config and final metrics inside
• adopt0/adopt0_mlp_s0.pt — the param-matched control, same layout
• adopt0/adopt0_results.json — the distillation verdict table
• adopt0/adopt0_bench.json — the standardized bench (all four rows)

Use : load Qwen/Qwen2.5-0.5B-Instruct , wrap each decoder layer with its block{i} relay as a residual on the block's hidden-state output, wrap the final norm's output with the head relay, or load nothing to recover stock behavior exactly. Training code ships in a future revision.

Lineage

The relay construction and its signed addressing come from a long-running research program on geometric routing (AMOE); the placement law motivating the pre-head relay — the signed address's advantage concentrates where it parameterizes the output distribution — was established on from-scratch LM substrates before this adoption test. Sibling experiment: alephlm-0 , the same addressing family trained as the trunk rather than bolted onto one.

Maintained as a live research log. Numbers are measured, not projected; anything not yet measured is marked as such.