FlameF0X / i3-12m

i3 Model - Ultra-Efficient Pretraining Language Model

Model Description

The i3 Model is designed to optimize pretraining efficiency while retaining core language modeling capabilities.
Its architecture allows training on memory-constrained hardware , including CPU-only setups, without sacrificing sequence modeling performance.

The i3 architecture is present within the model for highly efficient pretraining. It is designed to reduce memory usage , speed up training , and allow pretraining from scratch on tiny hardware. Internal details are abstracted for simplicity.

Use

from transformers import pipeline

pipe = pipeline("text-generation", model="FlameF0X/i3-12m")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

Model Statistics

• Vocabulary Size: 4,466 (variable-length chunks)

• Hidden Dimension: 512

• Number of Layers: 12

• Max Sequence Length: 256

• Total Parameters: 12,691,186

• Tokenization: Memory-efficient variable-length chunking (2–3 characters)

  • Total tokens: 334,524,736

Key Features

1. Memory-Optimized: Designed to train on tiny hardware with minimal RAM usage
2. Pretraining-Focused Architecture: i3 layers provide efficient sequence modeling, low-rank linear updates, and factorized attention
3. Variable-Length Tokenization: 2–3 character chunks for compact embeddings
4. Conversational Readiness: Optimized for dialogue and text generation

i3 Architecture (Abstract Overview)

Design Philosophy

The i3 model targets CPU-friendly, memory-constrained pretraining , emphasizing:

• Long-range sequence modeling
• Low-rank weight updates for memory savings
• Efficient factorized attention
• 4-bit weights and microbatching for minimal memory footprint

Technologies used in the i3 Architecture that are open-sourced by me:

• Low-Rank Pre-training - LoRa for pre-training.

Conceptual Layout

Input Tokens
    │
+-----------------+
| Embedding Layer |
+-----------------+
    │
+-----------------+
| i3 Architecture |
+-----------------+
    │
+------------------------+
| KQV Low-Rank Attention |
+------------------------+
    │
+-----------------------+
| LayerNorm + Residuals |
+-----------------------+
    │
+-------------------+
| Output Projection |
+-------------------+
    │
Predicted Tokens

Key idea: Every component is optimized for memory efficiency and pretraining speed on small hardware, while preserving essential transformer dynamics.

Training Details

• Sequence length: 128–512 tokens
• Model size: ~12M parameters (CPU-friendly)
• Optimizer: AdamW or Lion (4-bit / mixed precision)
• Dataset: TinyChat (~50–200 MB)
• Training loop: gradient checkpointing + recomputation
• Objective: token prediction / text generation

Citation

@software{lorpt2025,
  title={LoRPt: Low-Rank Pretraining for Resource-Efficient Language Models},
  author={[FlameF0X]},
  year={2025},
  url={https://github.com/FlameF0X/Low-Rank-Pretraining}
}