bench-labs / cagliostro-v2

cagliostro-v2

A 150M parameter language model trained from scratch on 50B tokens.

It reaches an Open SLM Intelligence Index of 19.99 , ahead of the previous release cagliostro-v1 by 0.84 when both are measured the same way.

Results

All three columns below were scored identically: lm-eval at 0-shot acc_norm for HellaSwag, ARC and PIQA, and AxiomicLabs' bencharithmark-3 for ArithMark. The new model was scored twice on different GPUs at different batch sizes and the two runs agreed to within 0.12 on the Index.

The Index was scored at five checkpoints. It is flat through the constant learning rate phase and moves once the decay starts, which is the shape this schedule produces: downstream metrics do not resolve small improvements while the weights are still in a high noise regime.

Second scoring run of cagliostro-v2: 19.93. Reported Index is 19.99, the mean.

The Index is (N(HellaSwag,25) + N(CombinedARC,25) + N(PIQA,50) + 0.65*N(ArithMark,25)) / 3.65 where N(v,c) = 100(v-c)/(100-c) .

Note on the v1 figures

The cagliostro-v1 model card reports 17.85 and 19.13. Those came from an in-house harness that was later retired: checked against lm-eval on a model whose leaderboard row was already known, it read ARC-Challenge 3.9 points high, and the error changed sign between models so no constant correction recovers it. The 19.22 above is v1 rescored with lm-eval, which is higher than its own card claims. The comparison here is deliberately against the corrected, more favourable number.

The internal 70B run is an unreleased checkpoint trained on 70B tokens with a staged curriculum that included synthetic arithmetic. It is listed for context because it is the only other model of this family measured on this harness.

How to read this

Against the published v1, this model is ahead on four of six components and on the Index by 0.84, roughly seven times the spread between repeat measurements. ARC-Challenge is the largest single gain at +2.39, moving well clear of the 25.0 chance floor.

Against the internal 70B run the Index margin is only +0.26, which is not a claim worth making on its own. What that comparison does show is token efficiency: a comparable score on 50B tokens where the other run needed 70B, at identical parameter count and without the synthetic arithmetic that run used.

ARC-Easy is the one component where v1 remains ahead, 48.95 against 47.43.

Architecture

150,038,400 parameters, 129,066,880 excluding embeddings.

Two additions beyond a standard decoder, both adding zero parameters:

Exclusive Self Attention ( arXiv:2603.09078 ). After attention, each head removes the component of its output that lies along its own value vector, so the block stops re-encoding what the residual stream already carries. Measured at -0.0785 nats over 123 paired readings against an otherwise identical baseline.

Logit soft-capping. Output logits pass through cap * tanh(logits / cap) with cap 15. Measured at -0.027 nats, flat across four quartiles of a 5B token ablation rather than decaying like every rejected variant did.

Because neither adds tensors, nothing in the state dict reveals them, and a checkpoint loaded without them scores at chance while every shape check passes. They are recorded in the config and the bundled modeling code implements them, so trust_remote_code loads the model correctly.

Ablations

Each arm is a 5B token run against a byte-identical baseline on the same pre-tokenized shards with the same seed, so a loss gap is attributable to the one flag that differs.

Three of the five ideas tried did not survive. Canon layers improved loss but lost on a compute-matched basis: they cost 9 percent throughput, and spending that 9 percent on more tokens instead was measured at roughly -0.008 nats against Canon's -0.0027.

Training

Data mixture: FineWeb-Edu 57.6 percent, DCLM-baseline 38.4 percent, Cosmopedia 4 percent. Pre-tokenized to uint16 memmapped shards, sampled by weight per sequence. 63.89B unique tokens backed the 50B token run, so repetition is near zero on every source.

A staged curriculum with code, arithmetic and dialogue was tried in earlier versions and abandoned: it scored Index 11.73 at 9.14B tokens, below GPT-2's 13.58, because the components that dominate the Index need web text rather than the specialised data the curriculum spent tokens on.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "bench-labs/cagliostro-v2", trust_remote_code=True)
tok = AutoTokenizer.from_pretrained("bench-labs/cagliostro-v2")

ids = tok("The capital of France is", return_tensors="pt")
out = model.generate(**ids, max_new_tokens=32)
print(tok.decode(out[0]))

trust_remote_code=True is required. The architecture is not in transformers and travels with the weights.

Limitations

This is a 150M parameter base model trained on 50B tokens. It is not instruction tuned, not aligned, and not filtered for safety. ARC-Challenge at 28.58 is meaningfully above chance but still low in absolute terms. It will produce confident nonsense on anything requiring real world knowledge.

The training data was decontaminated against the evaluation sets with 13-gram exact matching. Measured removal was 0.00 percent of FineWeb-Edu and 0.01 percent of DCLM, so decontamination is not doing meaningful work at this scale and the scores should not be read as more trustworthy because of it.