open-1b
is a 1.6 billion parameter decoder-only language model pretrained on 400B tokens. It is the first language model whose training can be independently verified.
Every open model release to date has asked that its users trust the account of how it was trained —
open-1b
allows you to check. It is released with its complete pretraining dataset, training and evaluation code, intermediate checkpoints at 100-step intervals, and a canonical state hash for every one of the 80,957 optimizer steps that produced it. Anyone can load the checkpoint before a given step, replay that step on their own hardware, hash the result, and confirm it matches the published fingerprint.
As organizations become more and more dependent on swarms of agents, we're reaching a point where AI verification is about to stop being a theory and start being a necessity. The only durable defense against models you cannot trust is models you can — models whose entire history is on the public record and can be replayed by anyone. That property cannot be added afterward; it has to be designed into the run from the first step.
open-1b
is our proof that this can be done in practice, and our attempt to set the standard for how.
This repository holds the supervised fine-tune (SFT)
: the midtrained model (
Gensyn/open-1b-midtrained-93B
) fine-tuned on
allenai/tulu-3-sft-olmo-2-mixture-0225
. This is an SFT-only model — no RLHF, DPO or other preference tuning was applied. It uses the chat template shipped in
tokenizer_config.json
(apply it with
tokenizer.apply_chat_template
).
Model details
Model description
Developed by:
Gensyn
Model type:
decoder-only transformer, 1.61B total parameters (1.08B non-embedding)
Language:
English, with code and STEM text
Licence:
Apache 2.0
Pretraining tokens:
400 billion, 80,957 steps
Context length:
4,096 tokens
Training cluster:
6 nodes, 48× NVIDIA H100
Verification record:
the record shows which training steps have accepted audits and which segments are confirmed
Per-step state hashes:
served by the verification record
What auditable means here
The unit.
One training step. The run is divided into segments of 100 steps between published checkpoints (810 segments; the last is short). Every step has a committed state hash. An auditor replays one step from the checkpoint the previous step produced and compares the hash their machine computes with the one committed before the run finished.
The hash.
After each step's weight and optimizer update, the trainer hashed five things together: the previous step's hash, a digest of the batch, the model weights, the gradients and the optimizer state. Chaining to the previous hash is what makes the sequence tamper-evident. Matching it requires exact reproduction of all five.
The commitments.
Hashes were committed for every step before auditing opened. Each segment's hashes are also summarised as a Merkle root, and the record serves an inclusion proof for every step so a third party can verify that a hash belongs to its segment without trusting the record's backend.
Reproducibility, and why a replay lands on the same bytes
Determinism means the same machine gives the same answer twice. Reproducibility means a different machine gives the same bits. open-1b was trained with RepOps, Gensyn's library of reproducible operations, which targets zero difference across devices rather than agreement within a tolerance.
Inference notes
The repo bundles its own modeling code (
modeling_open1b.py
), loaded with
trust_remote_code=True
— the architecture (gain-free QK-norm, embedding
RMSNorm, block-aligned hybrid sliding-window attention) matches no stock
transformers
class. Parameter names and bytes in
model.safetensors
are
identical to the training checkpoint the published state hashes commit to.
The model was trained with int8 W8A8 quantization-aware training (LSQ), and
the learned per-channel
weight_scale
tensors ship in the checkpoint. By
default the forward emulates the training int8 grid using those scales
(
config.quantized_forward=True
); set it to
False
for plain bf16 GEMMs on
the master weights. Inference logits are numerically close to, but not
bit-identical with, the training stack (fp32 GEMM accumulation, bf16
attention vs the training int8 P·V flash kernel) — bit-exact replay of
training steps is the job of the
audit tool
.
Load the tokenizer as-is. In particular, ignore transformers' suggestion to
pass
fix_mistral_regex=True
: the "fix" changes the pre-tokenizer's
behaviour, and tokenization bit-identical to training is part of this
release's reproducibility contract.
Chat format
The chat template (shipped in
tokenizer_config.json
and
chat_template.jinja
) renders every turn as
using the tokenizer's reserved special tokens (
<|start_header|>
id 2,
<|end_header|>
id 3,
<|eot|>
id 4 — all atomic single-id encodes). There
is no sequence-level BOS prefix.
<|eot|>
terminates every turn and is the
model's eos for generation, so
generate()
stops at the end of the
assistant turn.
The generation prompt deliberately ends at
<|end_header|>
(no trailing
newlines) and the model generates the
\n\n
itself — that transition
follows every header in its training data. Completions therefore start with
two newline characters; strip leading whitespace from the decoded text. Do
not append the two newlines to the prompt yourself: encoded at the end of a
prompt they merge into a single token the model never saw at that position
(assistant content always followed in training, splitting them), and the
model responds to it with an immediate
<|eot|>
— an empty generation. If
you must reproduce the exact training-style prompt (e.g. for scoring),
append a sentinel character before encoding and strip its tokens afterwards.
Intended use
open-1b-sft is the chat-capable member of the family, intended for research use and as a demonstration that the verifiable base model fine-tunes cleanly. It has had supervised fine-tuning only: expect a helpful but lightly-tuned assistant, not a production chat model. Always prompt it through the chat template.
How to use
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
"Gensyn/open-1b-sft", torch_dtype="bfloat16", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("Gensyn/open-1b-sft")
messages = [{"role": "user", "content": "Explain verifiable training in one paragraph."}]
inputs = tokenizer.apply_chat_template(
messages, add_generation_prompt=True, return_tensors="pt", return_dict=True
).to(model.device)
out = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True).strip())
Runs of Gensyn open-1b-sft on huggingface.co
1.2K
Total runs
0
24-hour runs
0
3-day runs
548
7-day runs
1.2K
30-day runs
More Information About open-1b-sft huggingface.co Model
open-1b-sft huggingface.co is an AI model on huggingface.co that provides open-1b-sft's model effect (), which can be used instantly with this Gensyn open-1b-sft model. huggingface.co supports a free trial of the open-1b-sft model, and also provides paid use of the open-1b-sft. Support call open-1b-sft model through api, including Node.js, Python, http.
open-1b-sft huggingface.co is an online trial and call api platform, which integrates open-1b-sft's modeling effects, including api services, and provides a free online trial of open-1b-sft, you can try open-1b-sft online for free by clicking the link below.
Gensyn open-1b-sft online free url in huggingface.co:
open-1b-sft is an open source model from GitHub that offers a free installation service, and any user can find open-1b-sft on GitHub to install. At the same time, huggingface.co provides the effect of open-1b-sft install, users can directly use open-1b-sft installed effect in huggingface.co for debugging and trial. It also supports api for free installation.
