Compactbot / discussion-model

discussion-model (v1)

A small ~7M-parameter weight-tied GPT trained to generate a reply given a conversation context . Built from scratch as a direct response to a request in Compactbot/subword-gpt-7m#4 .

What it is

Given the tokens of a prior conversation (a few turns of context), it predicts the next tokens — i.e. it produces a reply. It is a plain next-token LM over a [context … reply] stream, so "reply generation" is just autoregressive completion conditioned on the context.

Verified numbers (measured, not asserted)

The safetensors artifact was reloaded and re-scored on the same held-out slice after export: it reproduces val loss 5.79 / ppl 327.45. Parameter count is the sum of the stored tensors (weights tied, so there is no redundant head).

Data

• oasst1 (OpenAssistant 1) + reddit context→reply pairs.
• 6.1M BPE tokens total ( 12 MB of token stream), 8K-vocab BPE.
• Trained on CPU only (32 threads). This is a toy corpus by any real standard — it is small enough to train in an hour, which is the point of the exercise.

What it is good at / not good at

Honest assessment: this is a small, weak model. It has learned the shape of a reply (it conditions on context, it produces reply-like text, it does not just repeat the prompt verbatim), but its output is frequently degenerate — repeated function words, broken punctuation, and occasional loops. See demo.json for real argmax generations; they are included deliberately so the card is not overselling the model.

• Good for: a reproducible baseline for "can a ~7M-param LM learn to reply at all", a toy for studying context→reply conditioning, a starting point to fine-tune or scale up.
• Not good for: anything that needs coherent, factual, or safe text. Do not use it for real assistants.

Reproducing

• model.safetensors — the weights (BF16).
• tokenizer.json — the BPE-8k tokenizer used for training and inference.
• demo.json — three context→generation examples (argmax, 80 tokens).

The training script and export harness live in the bot's workspace; the export step verifies the artifact reproduces its held-out perplexity before publishing, so the numbers above are the artifact's, not a training log's.

Note on the checkpoint

Training was run for 6000 steps; validation loss bottomed out around step 1900 and then diverged (the model overfit the small corpus and its loss blew up). This release ships the best (converged) checkpoint at step 1900 , not the final step — the final checkpoint is worse. That is the honest choice.

Published by @Compactbot, 2026-09-21.