Compactbot / tinystories-50m

tinystories-50m

A 56,902,144-parameter transformer language model trained from scratch on TinyStories , a corpus of simple, repetitive children's stories. It is the 50M scale-up in the tinystories-24m → tinystories-50m lineage.

v2 (2026-09-25): retrained with a larger 12288-vocab BPE tokenizer (was 8192). The 8192-vocab v1 is fully superseded — same repo, same loader, better weights. v1's held-out val loss was 1.6566; v2's is 1.3837 .

It writes fluent, on-domain children's stories. It is not a general language model — out-of-domain generation degrades, and it should not be used for anything beyond the story domain it was trained on.

Architecture

Parameter breakdown (sums exactly to 56,902,144):

• token embedding: 12288 × 512 = 6,291,456
• position embedding: 512 × 512 = 262,144
• 16 blocks × 3,146,752 = 50,348,032
  • 2 × RMSNorm (512) + qkv (512×1536) + proj (512×512) + fc1 (512×2048) + fc2 (2048×512)
• final RMSNorm: 512

Training

• Data: TinyStories (ronendagan/TinyStories), 523,389,481 tokens after BPE-12288 re-tokenization (2,119,489 stories, ~9.19 tokens/param), with a 2M-token held-out tail for validation.
• Optimizer: AdamW, cosine LR decay with warmup (peak 6e-4), grad clip 1.0.
• Batch: 64, seq 512 → 32,768 tokens/step.
• Steps: 15,910 (one full epoch). Best checkpoint at step 13,500.
• Hardware: single NVIDIA RTX 5090 (32 GB).
• Final val loss: 1.3924; best val loss 1.3837 (step 13,500). The shipped weights are the end-of-run checkpoint (val 1.3924), within 0.009 of the best.

Evaluated numbers

• Held-out perplexity (TinyStories val split): exp(1.3837) ≈ 3.99 (best ckpt). This is the honest primary metric for a narrow-domain model.

• General zero-shot log-likelihood accuracy (the 12288-vocab tokenizer can read these datasets, so we report them — v1's 8192-vocab tokenizer could not):

  Task	Accuracy	n
  BLiMP	64.00%	200
  ARC-Easy	51.09%	599
  PIQA	45.50%	200
  HellaSwag	54.83%	600

  These are single-shot, zero-shot, no-few-shot, on a 57M model trained on one narrow domain — treat them as a scale reference, not a competitive result.

• Coherence: seeded generations are fluent, on-domain, with consistent characters and correct punctuation. Minor artifacts expected at this scale (occasional garbled quote char, a couple of logical slips).

Files

Usage

from load_model import load
model, tok = load()
ids = tok.encode("Once upon a time,")
out = model.generate(torch.tensor([ids]).cuda(), 100, temp=0.8, top_k=40)
print(tok.decode(out[0].tolist(), skip_special_tokens=True))

What it is and is not

• Is: a small, from-scratch, on-domain story generator. Good for studying how a ~57M transformer learns a narrow, repetitive domain.
• Is not: a general-purpose LM. Do not expect coherent output on code, math, or open-domain text. The low perplexity is domain-specific.