AtomixLabs / AtomixS2-5M-v1.0

AtomixS2-5M-v1.0

At AtomixLabs , our research often focuses on the physical constraints of neural architectures. With AtomixS2-5M-v1.0, we wanted to explore the absolute floor of language acquisition: What happens when you restrict a model to just under 6 million parameters?

At this extreme scale, a model does not have the capacity to memorize the internet, store vast encyclopedias of trivia, or comprehend deep physical-world mechanics. Instead, it becomes a pure engine of syntax and structure. This model is our attempt to build a highly active, fluent, and structurally sound micro-model that runs easily on almost any hardware—from older consumer GPUs to standard laptop processors. It is an exploration of parameter density and careful data curation over sheer scale.

Architectural Constraints and Vocabulary Design

AtomixS2-5M is a standard decoder-only transformer, built with a very tight structural configuration:

• Parameter Count: 5.98 Million
• Layers: 4
• Attention Heads: 8
• Context Window: 512 Tokens

When working with a parameter budget this constrained, traditional methods of tokenization become a major liability. Standard vocabularies often leak capacity by assigning precious embedding parameters to unused symbols, broken formatting fragments, or redundant uppercase and lowercase variations of the exact same word.

To resolve this, we engineered a custom, highly dense 3,584-token vocabulary. We utilized custom token mapping to ensure the model doesn't waste parameters relearning basic capitalization patterns or storing empty placeholder slots. Every single parameter in the embedding matrix is designed to be an active, high-yield node. By structuring the vocabulary this way, the model is able to direct more of its limited capacity toward understanding grammatical rules and logical transitions.

The Training Mixture: Syntax over Trivia

To get a model this small to output cohesive text, the training diet has to be incredibly deliberate. If you feed a micro-model nothing but raw, unfiltered web data, it tends to become noisy and chaotic. We trained AtomixS2 over a heavily curated, multi-domain mixture designed specifically to teach structure, narrative flow, and procedural formatting rather than just isolated facts.

Our final pre-training corpus was constructed using carefully balanced subsets from the following open-source datasets:

• openbmb/Ultra-FineWeb-L3 (~54%)
• HuggingFaceTB/smollm-corpus (~22%)
• Aarushhh/finemath-refined (~14%)
• openbmb/UltraData-Math (~10%)

By blending foundational web text with conversational narratives and a heavy dose of structured mathematics, we provided the model with a strong cognitive anchor. The math data, for instance, isn't there to teach a 6M parameter model advanced calculus. Instead, it forces the model's limited attention heads to learn how to track states, format markdown lists, close LaTeX brackets, and follow a strict sequential chain of thought, which cleanly transfers over to its general English syntax.

Recommended Generation Settings

Because AtomixS2-5M has a narrow parameter space, its probability distributions can behave differently than those of massive models. To get the cleanest, most coherent text generation, we strongly recommend the following sampling parameters:

• temperature: 0.5 — A slightly lower temperature helps keep the model grounded. It prevents the network from wandering into the noisy "tail" of its vocabulary.
• min_p: 0.1 — This dynamically truncates the lowest-probability tokens. It acts as a great filter against sudden hallucinations or spelling breaks.
• repetition_penalty: 1.05 to 1.1 — Small models can occasionally get caught in structural formatting loops (like generating continuous markdown tables). A light repetition penalty gently nudges the model to keep moving forward without destroying its natural vocabulary flow.

Benchmark Performance

We evaluated AtomixS2-5M-v1.0 using the standard EleutherAI evaluation harness. The scores reported below are length-normalized accuracies ( acc_norm ), which provide the fairest assessment of a model's true reasoning ability by neutralizing length bias.

What It's Like to Use

Interacting with AtomixS2-5M is a unique experience. It leans heavily into a formal, academic, and highly structured tone. It spells words with high accuracy and reliably outputs clean punctuation, markdown steps, and logical clauses.

However, users should keep in mind that its factual grounding is incredibly thin. It will confidently hallucinate an entirely fabricated historical event or scientific theory, but it will do so using flawless grammar and excellent paragraph structure. It is a model that has mastered the shape of human language, but relies entirely on you to provide the factual context. It serves as an excellent, lightweight foundation for local research, educational testing, or syntax-parsing experiments.