AtomixLabs / Photon-1.0-1M

⚛️ Photon-1.0-1M

Welcome to Photon-1.0-1M , an ultra-compact, hyper-dense Small Language Model (SLM) developed by AtomixLabs . Built to explore performance efficiency under tight resource constraints, Photon-1.0-1M packs just 1 million parameters and is designed to deliver steady reasoning and retrieval capabilities within a highly compact footprint.

Instead of relying on sprawling parameter counts to capture general knowledge, AtomixLabs designed Photon to be streamlined. By allocating its parameter budget directly into dense, high-utility transformer layers rather than larger vocabulary tables, this model functions as a sharp, fast, and capable logic engine for its size.

🛠️ Architectural Configuration

While many micro-models in this parameter range rely on highly customized, non-standard scripts that layer on experimental attention mechanisms, exotic gating structures, or non-native dependencies, AtomixLabs opted to stick to a clean, standard LLaMA architecture. The team believes that if a core architecture requires heavy modifications to function effectively, the challenge often lies in the training and optimization process rather than the standard transformer block itself.

By utilizing native, standard LLaMA layers, Photon remains compatible, stable, and highly dense:

# Photon-1.0-1M Structural Specification
vocab_size: int = 1536              # Optimized vocabulary allocation
hidden_size: int = 128              # High-density representations
intermediate_size: int = 384        # Symmetric 3.0x SwiGLU ratio
num_hidden_layers: int = 4          # The standard "Markov Floor" limit
num_attention_heads: int = 4        # Head dimension = 32
num_key_value_heads: int = 4        # 4:4 Multi-Head Attention (MHA)
max_position_embeddings: int = 512  # Native context length
rope_theta: float = 1110.0          # Tuned angular frequency for context scaling

By keeping the vocabulary tightly constrained and utilizing a full 4:4 Multi-Head Attention (MHA) scheme, the model retains reliable contextual memory and precise information-retrieval characteristics without needing external, non-native code dependencies.

📚 Dataset & Training Mix

To encourage collaboration, open science, and reproducibility in the micro-SLM space, AtomixLabs is sharing the training recipe.

Photon-1.0-1M was trained on approximately 2.56 billion tokens in total. To ensure licensing compliance and proper attribution, the training datasets listed below are custom, curated subsets extracted directly from the main openbmb/UltraX-Preview dataset. While the overarching UltraX repository is distributed under an Apache-2.0 license, users must abide by the specific underlying licenses of the original source datasets:

The Math Boost

To round out the 2.56-billion-token run, AtomixLabs augmented the subsets above with approximately 160,000,000 tokens of highly curated, synthetically generated arithmetic .

While the team is keeping the exact generation recipe and structural constraints of this synthetic data proprietary, this targeted math scaffolding was helpful in teaching the 1M-parameter model how to process logical relationships and numerical consistency.

📊 Benchmark Evaluation Results

All evaluations were executed in a strict zero-shot ( num_fewshot=0 ) environment to ensure genuine model capability and prevent few-shot prompt bias.

We evaluate the overall score using a standard 4-way average formula that aggregates the two ARC benchmarks first: $\text{ARC Combined}=\frac{\text{ARC-Easy}+\text{ARC-Challenge}}{2}\backslash text\{ARC\; Combined\}\; =\; \backslash frac\{\backslash text\{ARC-Easy\}\; +\; \backslash text\{ARC-Challenge\}\}\{2\}$ ARC Combined = 2 ARC-Easy + ARC-Challenge ​ $\text{Overall Average}=\frac{\text{Hellaswag}+\text{ARC Combined}+\text{PIQA}+\text{ArithMark 2.0}}{4}\backslash text\{Overall\; Average\}\; =\; \backslash frac\{\backslash text\{Hellaswag\}\; +\; \backslash text\{ARC\; Combined\}\; +\; \backslash text\{PIQA\}\; +\; \backslash text\{ArithMark\; 2.0\}\}\{4\}$ Overall Average = 4 Hellaswag + ARC Combined + PIQA + ArithMark 2.0 ​

Zero-Shot Results

🖥️ Reproducibility & Environment Context

To ensure you can replicate these exact scores in your own environment, we have extracted a snippet of our hardware and library diagnostic report.

OS Platform              : Windows (11)
CPU Architecture         : AMD64
Target GPU Name          : NVIDIA GeForce RTX 5070 (11.94 GB VRAM)
PyTorch Version          : 2.13.0+cu132
transformers Library     : 5.12.1
lm-eval-harness Library  : 0.4.13.dev0