AbstractPhil / captionbert-8192-v2

captionbert-8192-v2

A 58.3M standalone sentence encoder distilled from the geometric consensus of five BERT-family teachers. No expert models at inference: tokenizer + this model, 768-d L2-normalized output.

12 layers, 512-d, 8 heads, FFN 2048, 8192 position capacity. 0.53x bert-base.

from transformers import AutoModel, AutoTokenizer
model = AutoModel.from_pretrained("AbstractPhil/captionbert-8192-v2", trust_remote_code=True)
tok   = AutoTokenizer.from_pretrained("google-bert/bert-base-uncased")

emb = model.encode(["a cat on a windowsill", "a feline by the window"])   # (2, 768)
(emb[0] @ emb[1]).item()

Results

Measured in one harness; every model mean-pooled and L2-normalized, no task tuning. erank is the participation ratio of the embedding spectrum -- how many of the 768 directions are actually used.

It edges every teacher it was distilled from , at 13% of their combined parameters, having never seen a similarity label. The bare trunk does not reach all-MiniLM-L6-v2 (1B+ curated pairs, a different comparison class); with the 3-arm AMOE collective below, STS-B closes to .7684 vs .8203 and SICK-R to .7391 vs .7758 .

Isotropy is the mechanism. Mean-pooled BERT-family embeddings sit in a narrow cone (self_cos .58-.98); this model reads +.129 , and cosine discriminates far better in a space that is not collapsed.

With AMOE arms ( amoe/ )

The trunk is frozen; each anchor is 1.6M params across 12 sites. Anchors toggle bit-exact -- all disabled reproduces the bare trunk exactly -- so one artifact serves both the unsupervised baseline and the adapted model.

Best measured roster: equiv + simplify + paraphrase under a trained dispatch, across the full 8-task STS suite:

Eight for eight over every single member , +.1210 mean over the trunk, from 1.6M x 3 adapter params on a model that never moves. Note the BIOSSES column: the strongest member alone reads .5845, below the bare trunk -- the mixture turns a liability into a gain.

Solo anchor scores (trained alone, not masked inside a dispatch):

How many arms, and which

Arms do not add capacity -- they split a roughly conserved amplitude budget. w_k/z = sinh(u_k) / SUM_j cosh(u_j) sums over all anchors, so measured sums run .850 at A=2 to .757 at A=6 while each arm's share collapses. And sharper routing does not rescue it: tau .10/.05/.02 gave STS-B .7520 / .7514 / .7468 while key separation rose . The value is graded blending, not selection.

So every candidate arm is judged against an untrained arm at the same arm count , not against the smaller base. An empty arm gains +.0045 here -- real generic capacity. Greedy forward selection, bar = the measured seed spread .0054:

Every A=4 candidate -- including the control -- came in below .7684. Selection stopped at three arms. Zero starvation alarms; per-block routing spread .348, so the 12 dispatches learned different decisions rather than 12 copies of one.

Loading arms

Arms attach through the same AutoModel object. amoe-lora is imported lazily, so the base model still loads on a machine that has never heard of it.

from transformers import AutoModel
model = AutoModel.from_pretrained("AbstractPhil/captionbert-8192-v2",
                                  trust_remote_code=True)

model.attach_amoe()                       # the shipped 3-arm collective
emb = model.encode(["a cat on a windowsill"])

with model.amoe_off():                    # the unsupervised baseline
    base = model.encode(["a cat on a windowsill"])

model.set_amoe(["equiv"])                 # one arm, inside the dispatch
model.detach_amoe()                       # bit-exact restore, asserted

attach_amoe() with no arguments reads the roster and its order from the dispatch checkpoint -- routing keys are per-arm and positional, so order is not cosmetic. Both attach_amoe and detach_amoe assert the toggle law and raise rather than return a silently-wrong model.

pip install git+https://github.com/AbstractEyes/amoe-lora for the arm methods.

Arm library

amoe/arms/ holds one canonical copy of every anchor with ARMS.json (provenance, training spec, solo scores, capacity verdict). Resolve from there; do not retrain what is already listed. The campaign folders ( amoe/{sts,sts-combo,moe,moe-v2,collective} ) remain as the records behind the published numbers.

Scope worth stating: these anchors are solo-trained and always-on, which the aleph line identifies as blend regime -- the dispatch blends them rather than containing them. That is survivable here only because every arm serves one task. On a multi-task trunk, expect the cross-task interference that record documents. See amoe-lora .

How it was built

1. Five teachers embedded 33M CC12M llava-next captions (mean-pooled, 768-d).
2. One global whitened Procrustes map per teacher into bert-base 's frame, fit on a stratified random sample and reported out-of-sample .
3. Consensus = normalized centroid of the aligned teachers, per chunk.
4. Student trained from scratch: InfoNCE(T=0.07) + per-sample MSE against the consensus. Pure Adam, no weight decay. 26.9M rows, 52,548 steps at batch 2048, ~5.4 h on one RTX 6000 Pro.

Known limits -- read before using

• Consensus rank is ~28.7 of 768. Five BERT-family teachers only agree on about 29 directions. The student uses ~103 in domain but falls back to ~33 on out-of-domain text: the structure it builds on captions does not transfer. This is the model's ceiling and it is a property of the consensus, not the student.
• Alignment quality varies by teacher. Out-of-sample cosine to the bert frame: distil .625, roberta .372, albert .331, modern .327. The ordering tracks architectural distance from bert-base.
• 10 of 66 source chunks lacked ModernBERT , so 54 chunks (~27M rows) were used. No 4-expert fallback: that would change the target definition mid-dataset.
• Trained on image captions. Expect caption-like text to be its strongest domain.
• The trunk is single-seed. The AMOE results are 2-seed with a measured spread of .003-.005, against margins of +.012 to +.019.
• amoe/sts-combo stopped at step 500 of a planned 4000; its card and config describe the full run. amoe/moe-v2 retrained its anchors instead of reusing them, so the moe-vs-moe-v2 comparison moved two variables, not one. Both are recorded rather than quietly corrected.

Output convention (differs from v1)

geolip-captionbert-8192 (v1) returned the pooled embedding as last_hidden_state . If porting v1 code, use pooler_output . v1 also shipped an AlignmentBank ; v2 does not -- measured on v1, its expert-consistency features varied 0.2% across samples because a rotation round-trip carries no data.

Citation

@misc{abstractphil2026captionbertv2,
  title  = {captionbert-8192-v2: consensus distillation at CC12M scale},
  author = {AbstractPhil},
  year   = {2026},
  url    = {https://huggingface.co/AbstractPhil/captionbert-8192-v2}
}

MIT.