anthonym21 / Eve-2-MoE-272M

Eve-2-MoE-272M

A custom 272M-parameter Mixture-of-Experts language model trained from scratch on 8× NVIDIA H200 GPUs. Implements a DeepSeek-V3 style architecture with a shared expert, top-k routed experts, RoPE positional encoding, and SwiGLU activations.

Eve-2 is a base model for specialized fine-tuning — not a chatbot. Fine-tune it in ~20 minutes on consumer hardware for narrow tasks like PII redaction, text classification, semantic compression cleanup, or lightweight routing in multi-agent pipelines. Runs on a Raspberry Pi.

Author: Anthony Maio / Making Minds AI (Independent) https://www.github.com/anthony-maio https://www.linkedin.com/in/anthony-maio

Architecture

Design Rationale

MoE at this scale is a deliberate choice. With 8 experts but only 2 active per token, inference cost is roughly equivalent to a 80M dense model while the total parameter budget gives each expert room to specialize. The shared expert handles common patterns across all tokens; the routed experts develop narrow competencies during fine-tuning.

This makes Eve-2 a natural base for nano-LM swarms — fine-tune copies for specific tasks, deploy at the edge, coordinate through lightweight protocols.

Training

Convergence

Final Perplexity (WikiText-2): ~30

Training logs: Weights & Biases

Quick Start

This is a custom architecture — you need the model class to load it. Download modeling_eve.py from this repo.

import torch
import tiktoken
from modeling_eve import ModelConfig, DeepSeekMoE
from huggingface_hub import hf_hub_download

# Load
device = "cuda" if torch.cuda.is_available() else "cpu"
config = ModelConfig()
model = DeepSeekMoE(config)

weights = hf_hub_download(repo_id="anthonym21/Eve-2-MoE-272M", filename="pytorch_model.bin")
model.load_state_dict(torch.load(weights, map_location=device))
model.to(device).eval()

# Generate
enc = tiktoken.get_encoding("gpt2")
tokens = torch.tensor(enc.encode("The future of artificial intelligence is"),
                       dtype=torch.long, device=device).unsqueeze(0)

output = model.generate(tokens, max_new_tokens=100, temperature=0.8, top_k=50)
print(enc.decode(output[0].tolist()))

CPU / Raspberry Pi

The model runs on CPU at ~272M parameters. Inference is slower but functional — memory footprint is under 1 GB.

device = "cpu"
# Everything else stays the same

Intended Use

Eve-2 is a fine-tuning base , not a finished product. Out of the box it produces coherent English but has no instruction-following capability. The workflow:

1. Take this base model
2. Fine-tune on a narrow task (~20 min on consumer GPU)
3. Deploy at the edge as part of a specialized nano-LM swarm

Target applications: Data cleaning, PII redaction, text classification, semantic compression repair, lightweight routing/triage in multi-agent pipelines.

Limitations

This is a 272M model. It will not write essays, follow complex instructions, or compete with larger models on general benchmarks. That's by design — it's a small, fast, cheap-to-tune specialist base.

The train/val gap of ~0.62 at convergence suggests the model could benefit from additional data diversity beyond FineWeb-Edu for downstream generalization.

Files

├── pytorch_model.bin     # Model weights
├── config.json           # Architecture config
├── modeling_eve.py       # Model class definitions (required to load)
├── generate.py           # Standalone inference script
├── train.py              # DDP training script
└── requirements.txt      # Dependencies

Citation

@misc{anthony_maio_2026_eve2,
    author       = { Anthony Maio },
    title        = { Eve-2-MoE-272M (Revision ee90542) },
    year         = 2026,
    url          = { https://huggingface.co/anthonym21/Eve-2-MoE-272M },
    doi          = { 10.57967/hf/7731 },
    publisher    = { Hugging Face }
}

License

MIT — free for research and commercial use.