bench-labs / pulvis-v1

pulvis-v1

A 2.96M parameter decoder-only language model pretrained from scratch on 20B tokens. On the Open SLM Leaderboard Index it scores 7.85 , which is the highest score among models under 3M parameters. It is not yet listed on the board.

The name is Latin for dust, after the alchemists' pulvis projectionis, the pinch of powder that was supposed to transmute a whole crucible.

Results

Zero-shot, measured on the public weights in this repository with exactly the commands under Reproducing the evaluation: the lm-evaluation-harness hf backend and the leaderboard's official ArithMark-3 script, both in float32.

The Index is the leaderboard's own formula, (N(HellaSwag,25) + N(CombinedARC,25) + N(PIQA,50) + 0.65*N(ArithMark,25)) / 3.65 where N(v,c) = 100(v-c)/(100-c) and CombinedARC is the mean of ARC-Easy and ARC-Challenge.

The top of the sub-3M field, using the leaderboard's published figures on 30 September 2026:

How big the lead really is

At this size every benchmark sits only a few points above chance, and a single evaluation carries about 0.8 Index of sampling noise, most of it from PIQA's 1,838 questions. The 0.64 gap to Ember-2 is inside that noise, and it is worth saying where it comes from.

To get a steadier comparison, both models were scored on the training splits of the same benchmarks, which lm-evaluation-harness does not report and which neither model trained on: 39,905 HellaSwag items, 16,113 PIQA items, 4,239 ARC items, plus ArithMark-2's 2,500. Same code, same prompts, same scoring for both.

On general knowledge the two are level within noise, with Ember-2 slightly ahead. On arithmetic pulvis-v1 is far ahead: 48 against 28 on ArithMark-2, where chance is 25. Part of pulvis-v1's 7.85 is a favorable PIQA draw. It scores 55.93 on the PIQA test set and 54.51 on the much larger training split, and at 54.5 the Index would be about 7.1. Ember-2's ARC-Easy shows the same effect in the other direction, 33.42 on test against 31.87 on the training split.

What made the difference

The architecture work and the first full run got to 6.60, below Ember-2. The step that moved it past was the data in the final 25% of training.

The run uses a warmup-stable-decay schedule. Through the stable phase the benchmark scores barely move, and HellaSwag in particular sits at about 27.6 from the first checkpoint to the last. Almost everything happens when the learning rate decays. The first full run spent its decay on a mathematics-heavy mixture drawn from FineMath and OpenMathInstruct-2 and reached 6.60 on the public evaluation.

The decay phase was then run again from the same pre-decay weights, with the mathematics share moved to grade-school word problems: the GSM8K-derived half of OpenMathInstruct-2 and Microsoft's Orca-Math. ArithMark-2 went from 32.6 to 47.9 on the training-time evaluation while the general benchmarks stayed level. On the public evaluation the released model scores 7.85.

The loss in the second decay phase is lower mostly because worked arithmetic is easier to predict than web text. It is not a like-for-like comparison.

How the model was chosen

Everything that picked a direction was measured on held-out splits, not on the reported test sets, with two disclosed exceptions.

1. Pilot ablations. About 30 runs of 0.5B to 3B tokens, most of them on one RTX 5080, compared tokenizers, widths, layer reuse, optimizers, attention variants and data mixtures, scored by validation bits per byte and the training-split benchmarks above. Run-to-run noise was measured with repeat seeds and turned out to be about 0.01 bits per byte, so several apparent wins were treated as ties.
2. Mathematics mixture (first exception). ArithMark-2's bare expressions proved a poor stand-in for ArithMark-3's word problems, so the official ArithMark-3 script was run on ten pilot models to choose how much mathematics to use and when. This is use of a test set for a design decision.
3. Decay-phase branches. Three decay phases were run from the same pre-decay weights: an exact repeat of the first run's decay as a control, a grade-school mathematics mixture, and a heavier version of it. The selection rule was written down before any result was seen: highest score on the training-split benchmarks plus ArithMark-2. The heavier mixture won, and only it was evaluated on the test sets.
4. Test evaluations of release candidates (second exception). Two models were scored on the full public evaluation: the first full run (6.60) and this one (7.85). Both numbers are reported here.

These are training-time evaluations on the training splits. The control landed 0.5 below the first run on the general benchmarks with an identical recipe, which gives a sense of how much of this is noise.

No data was generated for this model, and no benchmark items or benchmark generators were used in training. OpenMathInstruct-2 and Orca-Math are themselves synthetic datasets, released by NVIDIA and Microsoft and built from GSM8K-style seed problems. They were checked against ArithMark-3: none of its 1,000 items shares a single 10-word sequence with any of the 200,035 Orca-Math problems, or with a sample of 160,904 of the GSM8K-derived OpenMathInstruct-2 rows (2 of its 32 files).

Findings from the ablations

At 3M parameters the budget is dominated by what the embedding table costs, and compute is cheap, so the choices differ from larger models.

Model details

Each pass through the core blocks adds a small learned vector first, so the model can tell the passes apart. trust_remote_code=True is required because PulvisForCausalLM is not part of transformers.

Training data

FinePhrase is used through its rephrased text, not the original web page it was written from. OpenMathInstruct-2 and Orca-Math are used as problem followed by worked solution.

Training setup

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "bench-labs/pulvis-v1"
tok = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True, dtype=torch.float32)

ids = tok("The capital of France is", return_tensors="pt")
out = model.generate(**ids, max_new_tokens=32, do_sample=False)
print(tok.decode(out[0], skip_special_tokens=True))

This is a base model with no instruction tuning and no chat template. It completes text.

Samples

Twelve fixed prompts, one sample each, unedited. Sampling at temperature 0.7, top-p 0.9, repetition penalty 1.1, 48 new tokens, seed 0 before every prompt. These are the same prompts used for tinctura-v1 and cagliostro-v3 . At 3M parameters it strings together plausible phrasing, often with a mathematical flavor from the decay phase, and very little of it is true.

The water cycle begins when the water is moving at a constant rate.
Speed = 10 km/hr
Average speed of the water = Speed * Time = 10 * 30^(-6)

Photosynthesis is the process by which your product will be produced in a single product. If you want to know how to choose from the manufacturer, use the formula for the product of two products, and use the formula for the product of the product of two products.

To solve 47 + 68, first finding the value of x which is 337. This means that 337 = 47 and 48 = 27 which implies that x = 90.
So the number of

The Roman Empire fell because of the struggle between the Trojanes and the Apollo of Hague.

2. **Growth**
   - The Catholic Church of Tori took control of the Roman

A prime number is a number that can be either 1 or 2.
Since 100 is an integer and 1 is 50, we can choose 50 that is an integer.
This means that the number 5

The three states of matter are:
1. Silver: 60% of the original material is being added.
2. Temperature: 20% of the original material is being added.
3. Magnetic Field

Gravity is the force that can be applied to a person's body.

### 4. How does an intelligentness work?

The intelligentness of a person is a process where one's body is able to use a number of

In 1969, astronauts had discovered the oldest atom.
Seven years later, the gravitational field of star formed by the sun and a star formed by a new stars forming the same dichotomy

The heart pumps blood through the blood. This blood supply is a part of the body. This process can be transmitted to other body parts and can occur on other things.

**Step 6: Promote Your Career**

To find the area of a rectangle you can find by multiplying the length of each side of the rectangle by their width and then multiplying by their respective lengths.

The area of one rectangle is given by the product of

Volcanoes form when there is a shaded or southeast it is likely to have an event that occurs in a large area of 1000 feet.

What is the significance of this documentary on the historical significance of

The difference between weather and climate is 53.5%.
Since there are 400000 people in the city, and 40% of these people have a dry weather, the number of people who have a dry weather is

Reproducing the evaluation

pip install lm-eval
python -m lm_eval --model hf \
  --model_args pretrained=bench-labs/pulvis-v1,dtype=float32,trust_remote_code=True \
  --tasks hellaswag,arc_easy,arc_challenge,piqa \
  --num_fewshot 0 --batch_size 64 --device cuda:0

ArithMark-3 uses the leaderboard's official script from AxiomicLabs/ArithMark-3.0 . The script defaults to bfloat16, so pass --dtype float32 :

python bencharithmark-3.py --model bench-labs/pulvis-v1 --data-path arithmark-3.jsonl --device cuda --dtype float32

ArithMark-2 uses the official script from AxiomicLabs/ArithMark-2.0 . It runs in bfloat16 whenever a GPU is visible and has no dtype option, so the number above comes from a CPU run, which is float32:

CUDA_VISIBLE_DEVICES="" python benchmark_arithmark-2.0.py --model bench-labs/pulvis-v1 --data-path arithmark_2.0.jsonl

Evaluate in float32 throughout. The model uses a logit cap of 15, and bfloat16 shifts the ArithMark scores of logit-capped models. The exported weights agree with the training checkpoint exactly: the largest logit difference between the two is zero.

Provenance

checkpoints/ holds the stable-phase weights every 2.5B tokens, including the weights the released decay phase started from (step 57,222, three steps after the first run began its own decay), and the end of the first full run that scored 6.60. Each subfolder loads with subfolder= in from_pretrained .

Limitations

English only. 1,024 token context. No instruction tuning, no safety tuning, no RLHF. At 3M parameters it confabulates freely, as the samples show, and should not be relied on for anything factual. Its arithmetic ability is two-digit arithmetic in short word problems, not mathematical reasoning.

License

Apache-2.0. The training data is drawn from FinePhrase, FineWeb-Edu and FineMath (ODC-By), DCLM-Baseline and OpenMathInstruct-2 (CC-BY-4.0), and Orca-Math (MIT).