Gensyn / open-1b-midtrained-93B

open-1b-midtrained-93B

open-1b is a 1.6 billion parameter decoder-only language model pretrained on 400B tokens. It is the first language model whose training can be independently verified.

Every open model release to date has asked that its users trust the account of how it was trained — open-1b allows you to check. It is released with its complete pretraining dataset, training and evaluation code, intermediate checkpoints at 100-step intervals, and a canonical state hash for every one of the 80,957 optimizer steps that produced it. Anyone can load the checkpoint before a given step, replay that step on their own hardware, hash the result, and confirm it matches the published fingerprint.

As organizations become more and more dependent on swarms of agents, we're reaching a point where AI verification is about to stop being a theory and start being a necessity. The only durable defense against models you cannot trust is models you can — models whose entire history is on the public record and can be replayed by anyone. That property cannot be added afterward; it has to be designed into the run from the first step. open-1b is our proof that this can be done in practice, and our attempt to set the standard for how.

This repository holds the midtrained checkpoint : the base model ( Gensyn/open-1b-base ) after continued pretraining on 93B additional tokens of the Dolma 3 Dolmino annealing mixture ( allenai/dolma3_dolmino_mix-100B-1125 , ingredient 1) — 23 sources spanning synthetic math, code, QA, reasoning and instruction data plus high-quality web and PDF text — with the learning rate annealed linearly to zero (OLMo 2-style midtraining recipe).

Model details

Model description

• Developed by: Gensyn
• Model type: decoder-only transformer, 1.61B total parameters (1.08B non-embedding)
• Language: English, with code and STEM text
• Licence: Apache 2.0
• Pretraining tokens: 400 billion, 80,957 steps
• Context length: 4,096 tokens
• Training cluster: 6 nodes, 48× NVIDIA H100
• Verification record: the record shows which training steps have accepted audits and which segments are confirmed

The open-1b family

Model sources

• Verification record: https://open1b.gensyn.ai
• Audit tool: https://github.com/gensyn-ai/pretraining-audit-cli
• Tech report: open1b.gensyn.ai/open1b-tech-report.pdf (arXiv soon)
• Pretraining data: gs://gensyn-open-1b/data — browse and search it at open1b.gensyn.ai/#/data/search
• Checkpoint trajectory (every 100 steps, full optimizer state): gs://gensyn-open-1b/ckpt
• Training code, configs and RepOps: https://github.com/gensyn-ai/open-transformers
• Per-step state hashes: served by the verification record

What auditable means here

The unit. One training step. The run is divided into segments of 100 steps between published checkpoints (810 segments; the last is short). Every step has a committed state hash. An auditor replays one step from the checkpoint the previous step produced and compares the hash their machine computes with the one committed before the run finished.

The hash. After each step's weight and optimizer update, the trainer hashed five things together: the previous step's hash, a digest of the batch, the model weights, the gradients and the optimizer state. Chaining to the previous hash is what makes the sequence tamper-evident. Matching it requires exact reproduction of all five.

The commitments. Hashes were committed for every step before auditing opened. Each segment's hashes are also summarised as a Merkle root, and the record serves an inclusion proof for every step so a third party can verify that a hash belongs to its segment without trusting the record's backend.

Reproducibility, and why a replay lands on the same bytes

Determinism means the same machine gives the same answer twice. Reproducibility means a different machine gives the same bits. open-1b was trained with RepOps, Gensyn's library of reproducible operations, which targets zero difference across devices rather than agreement within a tolerance.

Inference notes

The repo bundles its own modeling code ( modeling_open1b.py ), loaded with trust_remote_code=True — the architecture (gain-free QK-norm, embedding RMSNorm, block-aligned hybrid sliding-window attention) matches no stock transformers class. Parameter names and bytes in model.safetensors are identical to the training checkpoint the published state hashes commit to.

The model was trained with int8 W8A8 quantization-aware training (LSQ), and the learned per-channel weight_scale tensors ship in the checkpoint. By default the forward emulates the training int8 grid using those scales ( config.quantized_forward=True ); set it to False for plain bf16 GEMMs on the master weights. Inference logits are numerically close to, but not bit-identical with, the training stack (fp32 GEMM accumulation, bf16 attention vs the training int8 P·V flash kernel) — bit-exact replay of training steps is the job of the audit tool .

Load the tokenizer as-is. In particular, ignore transformers' suggestion to pass fix_mistral_regex=True : the "fix" changes the pre-tokenizer's behaviour, and tokenization bit-identical to training is part of this release's reproducibility contract.

Intended use

open-1b-midtrained is the recommended starting point for further training (e.g. supervised fine-tuning). Like the base model it is not tuned for chat or instruction following; use Gensyn/open-1b-sft for a chat-capable model.

How to use

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "Gensyn/open-1b-midtrained-93B", torch_dtype="bfloat16", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("Gensyn/open-1b-midtrained-93B")