chutesai / parallax-8b-theta

Parallax 8B theta: mid-training checkpoints

These are research checkpoints from theta , a training run of Parallax. Training was stopped at step 6387 (156.2B tokens), before the planned end of the run; the final checkpoint is marked in the table below. Parallax is a mixture-of-experts language model trained on consumer GPUs spread over several countries. The hosts have no direct connections to each other and synchronize over the public internet.

• Base model. It is pretrained on web text only. It is not instruction-tuned, chat-tuned or safety-tuned, and it will continue text rather than follow instructions.
• Mid-training. Every export, the final one included, is a snapshot taken well before the planned ~973B tokens. Quality between exports is not monotonic.
• Research artifact. It is published so the training can be followed and inspected. It is not intended for production use.

Live training dashboard: parallax.chutes.ai .

Model

Training

• Data: FineWeb-Edu, planned as a single pass over about 973B tokens. The published validation number ( val_mix ) is measured on held-out FineWeb-Edu files that are not in the training stream.
• Optimization: the dense trunk is synchronized with a decoupled DiLoCo scheme over the internet. Each routed expert has one owner GPU that folds low-rank updates from the nodes that trained it into its full-precision master and publishes new ternary versions. The tech report describes the details.
• The learning rate was lowered by hand during the run, to 0.5x and later 0.25x of the planned peak, so later exports do not follow the planned schedule exactly.

Exports

One folder per export under exports/ , named by training tokens (rounded down to whole billions) and fleet step. Exports were taken about once an hour; training has stopped and no new exports will be added. Published exports are never modified or removed.

• val_mix : mean cross-entropy (nats per token, lower is better) on the held-out FineWeb-Edu validation set.
• 0-shot macro : mean over 11 tasks (ARC-Challenge, ARC-Easy, BoolQ, COPA, HellaSwag, LAMBADA, OpenBookQA, PIQA, SciQ, SIQA, WinoGrande). 5-shot macro : the same tasks without LAMBADA (10 tasks). Per task, acc_norm is used for ARC, HellaSwag, OpenBookQA, PIQA and SciQ, and acc for the rest. Values are percentages.
• The scores come from the project's own scorer, which runs the native ternary experts. They track progress within this run. Compare them with numbers from other evaluation harnesses with care.
• A - means the export has not been scored yet. The table fills in as scores arrive.

Latest export

exports/156B-tokens_step6387 : 156.2B training tokens, step 6387.

hf download chutesai/parallax-8b-theta --include "exports/156B-tokens_step6387/*" --local-dir parallax-8b-theta
cd parallax-8b-theta/exports/156B-tokens_step6387
tar -xf packed_experts.tar
sha256sum -c --quiet SHA256SUMS   # every file of the original export, byte for byte

Files and format

The files are in Parallax's native compact export format (inference only, no optimizer state). They are byte-identical to the export the training system produced:

The only change from the original export is packaging. The 4096 expert files are stored in one uncompressed tar to keep the repository's file count manageable. Extract it and check SHA256SUMS as shown above. Every upload was checked against the training system's own digests before and after it was published.

Running it. Standard transformers cannot load this format. A llama.cpp-based runtime for Parallax and GGUF conversions of these checkpoints are in preparation (link to follow).

Tech report

The Parallax tech report: https://parallax.chutes.ai/tech-report.pdf . It is AI-generated from the project's measurements and logs, and it is a living document that changes as the run progresses. It describes the previous run (eta). Theta uses the same architecture with changes to the synchronization recipe.

Limitations

This is an early base model trained on educational web text. It can produce incorrect, biased or nonsensical output. It has no alignment or safety tuning, and it is small and far from converged.

License

MIT.

_{Table updated 2026-09-28 15:14 UTC.}