blab-jhu / KYS-1.5B-Quality-Base

KYS-1.5B-Quality-Base

QUALITY-BASE — the non-rewritten baseline.

1.5B-parameter Llama-2-architecture language models pretrained from scratch for the paper Know Your Sources: Data Selection Matters when Rewriting for Data-Constrained Pretraining . This repo holds all 9 checkpoints for this setting: 3 seeds × 3 epoch boundaries.

⚠️ These are base models trained on 30B tokens. They are research artifacts for studying data selection, not instruction-tuned assistants.

What this setting does

The next 5B tokens down the DCLM fastText ranking after the shared anchor is removed from the pool, used verbatim . No rewriting anywhere in this mixture.

This is the control arm every other setting is compared against.

Every one of the six settings trains on a 10B-token mixture built the same way: a shared 5B-token anchor of top-ranked DCLM fastText documents, identical across all six settings and never rewritten, plus 5B tokens contributed by this setting's selection strategy . Only that second half differs between settings, so downstream differences isolate source selection alone.

Quick start

The repo root is a copy of seed 42 at the end of epoch 3 , so from_pretrained works with no subfolder argument:

from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("blab-jhu/KYS-1.5B-Quality-Base")      # = seed42/epoch3
tok   = AutoTokenizer.from_pretrained("blab-jhu/KYS-1.5B-Quality-Base")

Any other checkpoint is addressed by subfolder:

model = AutoModelForCausalLM.from_pretrained("blab-jhu/KYS-1.5B-Quality-Base", subfolder="seed43/epoch1")

Layout

.                      <- copy of seed42/epoch3 (seed 42, end of epoch 3)
seed42/epoch1  epoch2  epoch3
seed43/epoch1  epoch2  epoch3
seed44/epoch1  epoch2  epoch3

Each directory is a complete HF checkpoint: config.json , generation_config.json , model.safetensors (bf16, 3,008,627,352 B), tokenizer.json , tokenizer_config.json , special_tokens_map.json .

Optimizer state is not included. Nanotron wrote an 18.05 GB AdamW state next to every checkpoint (fp32 master weights + both moments, 974.80 GB across the 54 released checkpoints); it is omitted from the release. These checkpoints are for inference and evaluation, not for resuming training.

Step → epoch mapping

One optimizer step consumes 4 (micro) × 64 (accum) × 4 (DP) × 2048 = 2,097,152 tokens .

Checkpoints were written every 477 steps; 4770 = 477 × 10 and 9540 = 477 × 20.

Training configuration

Identical across all 18 runs (6 settings × 3 seeds).

Seeds 42/43/44 control initialization only — there is no dropout and the data order is fixed. All six settings at a given seed start from bit-identical weights, released once as KYS-Configs/nanotron/init/ .

How these were produced

1. A 100M-document pool was reservoir-sampled from DCLM-RefinedWeb and annotated with three quality scorers and a 24-way WebOrganizer topic classifier → KYS-DCLM-Refinedweb-100M-Scored .
2. This setting's strategy selected source documents from the pool → KYS-1.5B-Pretraining-Corpora .
3. The corpus was tokenized and trained with Nanotron ; configs are in KYS-Configs .
4. Checkpoints were converted from Nanotron to HF format with the same converter used for the paper's own evaluations. The conversion was verified by re-running LightEval on a converted checkpoint and reproducing the paper's stored numbers exactly ( rw_piqa acc_norm 0.710555, rw_hellaswag acc_norm 0.486158, delta 0.000e+00).

Evaluation

LightEval, 0-shot, acc_norm with continuation-token-length-normalized log-likelihood, on ARC-Easy, HellaSwag, PIQA, SIQA, OpenBookQA, CommonsenseQA and MMLU (57 subsets). Task definitions, launchers and all 276 raw result JSONs are in KYS-Configs/eval/ .

The rest of the release

Citation

@misc{kys2026,
  title  = {Know Your Sources: Data Selection Matters when Rewriting for Data-Constrained Pretraining},
  author = {TODO},
  year   = {2026},
  note   = {TODO: fill in venue / arXiv id / URL}
}