DedeProGames / LowOnMind-300k

LowOnMind-300k

A decoder-only language model with 296,960 parameters , pretrained from scratch on 200M tokens of HuggingFaceFW/fineweb-edu (sample-10BT).

It is an extreme variant of DedeProGames/DynamicMind-Mini (8.9M params), scaled down roughly 30x. The name is the point: this model is not built to be good. It is built to find out what a third of a million parameters actually buys you.

A 3.3x scale-up trained under identical conditions — same tokenizer, same data, same token budget, same aspect ratio — is published as DedeProGames/LowOnMind-1M . The two models form a controlled pair; see Scaling comparison below.

Architecture

Parameter split: 65,536 in the tied embedding, 231,360 across the six blocks, 64 in the final norm.

Changes from DynamicMind-Mini

• QK-Norm — per-head RMSNorm on queries and keys before RoPE. Costs 32 params per block and is what makes an LR of 2e-3 survivable at this scale.
• Precomputed RoPE — cos/sin are built once and reused instead of being recomputed every forward, and the cache re-expands automatically for sequences longer than max_position_embeddings .
• Scaled residual init — o_proj and down_proj initialize at std / sqrt(2 * num_layers) instead of a flat 0.02.
• 1024-token vocabulary — required to fit the parameter budget. At 8192 tokens the embedding alone would have been 8192 x hidden, larger than the entire model. The cost is compression: roughly 2.35 characters per token.

Training

The token budget is deliberately small in absolute terms, but at 673 tokens per parameter this run sits about 30x past the Chinchilla-optimal ratio. The model is parameter-starved, not data-starved: train and validation loss stayed within 0.05 of each other for the entire run, so there is no overfitting to speak of.

Language modelling results

Perplexity is not comparable across tokenizers — with a 1024-token vocabulary, 27.06 means something very different than it would at 50k. Bits per character (loss / ln 2 / 2.35 chars-per-token) is the portable number.

BananaMind Base Bench 1.1

Evaluated on BananaMind/BananaMind-Base-Bench-1.1 , a 350-item English continuation-likelihood benchmark for base models. Scoring follows the official specification: context and each of the four continuations tokenized separately with add_special_tokens=False , no BOS, selection by highest mean conditional token log-probability.

Run validity: dataset SHA-256 matched, full schema validation passed, and no context required truncation against the 512-token window.

Sample output

Prompt: "The " , temperature 0.8, top_p 0.95.

The vehicle has been shown that the verb is a chief.
No:
- Even a good certain steps of the day, it is good and then a statement.
- It is a number of positive checking gain in the south of this power.
The risk of the main pottery, the straight guidelines of the brain's deployment.
- She is a little fruitful vegetables.
- It's an increasingly a brain, and this is, as a perfect men on the storage.
- Exodus of Donah, a conference, it is an aware of the message.
- The main straw of the problem.
- You're potential that the moments can be written in the majority of the paradigm.
- New York

This sample is more informative than the perplexity number. With a 1024-token byte-level vocabulary, words like "increasingly", "guidelines" and "deployment" do not exist as tokens — the model assembled each from four or five fragments in the correct order without misspelling one. That is orthography learned as a process, not as a lookup table. It also picked up document structure from FineWeb-Edu: bullet markers, a No: header, capitalization after periods, apostrophes in contractions. Even the invented word, "Donah", is invented with English phonotactics.

What it does not have is meaning beyond a span of about four tokens. "an aware of the message", "a little fruitful vegetables", "an increasingly a brain" — each three-to-four-word fragment is grammatical and the seams between them are not. That is the signature of ~2 bits per character: locally perfect grammar, zero tracking.

Scaling comparison vs LowOnMind-1M

DedeProGames/LowOnMind-1M is the same family at 985,152 parameters (3.32x), trained with the identical tokenizer, dataset, token budget (200M), schedule shape, and hidden/layers aspect ratio (10.7). Parameter count is the only thing that changed, so the comparison below isolates its effect.

The clean result is on the intrinsic LM metrics. Validation loss and bits per character moved by a large, unambiguous margin — those are averaged over millions of validation tokens, so the estimate is tight. 3.3x the parameters bought a real, measurable drop in both.

The benchmark result is not clean, and that's informative on its own. 350 four-way items is not enough statistical power to confirm a 2.3-point accuracy gain; the unpaired z-test (z = 0.67) falls well short of significance, and LowOnMind-1M's own confidence interval still overlaps chance. The one category that clears significance against chance in both models is language_completion — consistent with the real-word-rate result and with the qualitative difference in the two models' sample outputs, where LowOnMind-1M shows register-appropriate structure (bibliography-style formatting after a date-led prompt, policy-style prose after a directive one) that this model does not. That agreement across three independent measurements is worth more than any single number here, including the Elo score.

Read the two models together, not the 300k score alone , if the question is "does scale help at this size." It does, clearly, on the metrics with enough samples to say so, and the benchmark is simply underpowered to confirm or deny it on its own.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("DedeProGames/LowOnMind-300k")
model = AutoModelForCausalLM.from_pretrained(
    "DedeProGames/LowOnMind-300k", trust_remote_code=True
)

ids = tok("The ", return_tensors="pt").input_ids
print(tok.decode(model.generate(ids, max_new_tokens=64, use_cache=False)[0]))

trust_remote_code=True is required — the architecture ships as custom modeling code in the repository. use_cache=False is required: this implementation has no KV cache, and generation recomputes the full window each step.

Limitations

At ~300k parameters this model learns spelling, spacing, morphology and short n-grams, but does not produce coherent text . It has no factual knowledge, no reasoning ability, and no capacity to track state across a passage. It should not be used for anything except research into the lower end of the scaling curve.