DedeProGames / LowOnMind-1M

LowOnMind-1M

A decoder-only language model with 985,152 parameters , pretrained from scratch on 200M tokens of HuggingFaceFW/fineweb-edu (sample-10BT).

This is the 3.3x scale-up of DedeProGames/LowOnMind-300k , run as a controlled experiment : identical tokenizer, identical dataset, identical token budget, identical schedule, and the same hidden/layers aspect ratio. Parameter count is the only variable, so the two models are directly comparable on validation loss and on downstream benchmarks.

Architecture

Architecture details, inherited from DynamicMind-Mini with modifications: GQA, SwiGLU, RMSNorm, tied embeddings, QK-Norm per head, precomputed RoPE with automatic re-expansion, and residual projections initialized at std / sqrt(2 * num_layers) .

Training

At 203 tokens per parameter this run still sits far past the Chinchilla-optimal ratio. Train and validation loss tracked each other for the entire run — this model is parameter-limited, not data-limited.

Results

Perplexity is not comparable across tokenizers, but it is comparable between these two models because they share one. Bits per character (loss / ln 2 / 2.35 chars-per-token) is the portable figure.

Real-word rate

With a 1024-token byte-level vocabulary, no long word exists as a single token — the model has to assemble every one of them from fragments. The fraction of emitted words that are real English words measures this directly, and neither validation loss nor a multiple-choice benchmark captures it.

Measured over 64 unconditional samples (5,647 words). The reference lexicon is every lowercase word appearing at least 5 times in a 20k-document sample of the training corpus, so the corpus row is the practical ceiling rather than 100%. At 98.0% against a 98.4% ceiling, the lexicon is essentially saturated — the 300k → 1M scale-up bought almost all of its capacity in spelling and word formation, not in anything downstream of it.

Most frequent non-words: sculieness , lockholm , scul , purpled , paradigmar , prefection , frushing , fullly

BananaMind Base Bench 1.1

Evaluated on BananaMind/BananaMind-Base-Bench-1.1 , the same 350-item English continuation-likelihood benchmark used for LowOnMind-300k, with identical scoring: context and each of the four continuations tokenized separately with add_special_tokens=False , no BOS, selection by highest mean conditional token log-probability.

Run validity: dataset SHA-256 matched, full schema validation passed, and no context required truncation against the 512-token window.

Sample output

Prompts sampled at temperature 0.8, top_p 0.95. Unlike the 300k model, output now shifts register with the prompt — a date-led prompt produces bibliography-style formatting, a directive prompt produces policy-style prose — and invented non-words (see the real-word-rate section) are markedly rarer than at 300k parameters. Meaning and referential continuity are still absent.

prompt: "The "
-> The veil-group, where is used to solve solar passwords and vegetables. It
is often very important to reduce their treatments, including checkouts,
labs, maintenance, mental groups, chemicals and maintenance costs. These
literally confirmed elements are the most commonly used in the elementary
front, solar energy, a

prompt: "Photosynthesis is "
-> Photosynthesis is essential to prevent disease infection. These types of
disease include:
• Transformation of Brook Scientific World Organization
• Effectiveness of a diagnosis is commonly diagnosed with diagnosis. The
Cultural Disorder is an important factor in diagnosis. Analysis of this
purpose can be considered.

prompt: "In 1969, "
-> In 1969, 8, 1999, Michael Kerzin, Anderson, 1997, Scotland, 1993, 1,359,
p. 547-414
Our war was chosen in the West of Europe, Papua Party, Ottoman Economy,
Pluto Mountains, and Evans, 1854, 1946, Maine, 1888-81, p

prompt: "Students should "
-> Students should identify their learners and adults.
The United States has free quotations in the world that can help them with
this curriculum and finding problem concerning these funding.
One major solutions for malnutrition in South East and Africa is a way to
limit half its own loop. This is the best way to ensure that the drills of
the state and

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("DedeProGames/LowOnMind-1M")
model = AutoModelForCausalLM.from_pretrained("DedeProGames/LowOnMind-1M", trust_remote_code=True)

ids = tok("The ", return_tensors="pt").input_ids
print(tok.decode(model.generate(ids, max_new_tokens=64, use_cache=False)[0]))

trust_remote_code=True is required — the architecture ships as custom modeling code in the repository. use_cache=False is required: this implementation has no KV cache and recomputes the full window at each generation step.

Limitations

At ~1M parameters this model learns spelling, morphology, and short-range syntax. It does not produce coherent text , has no reliable factual knowledge, and cannot track state across a passage. It exists to measure the lower end of the scaling curve, not to be used.

The 512-token context, 1024-token vocabulary, and absent KV cache make it unsuitable for any real workload.