COPA-AI / arm-gemma-e4b

arm-gemma-e4b

arm-gemma-e4b is Gemma-4-E4B adapted to Armenian by continued pretraining. To our knowledge, it is the first open Armenian LLM released together with its complete training corpus and recipe — every training token is either public ( ArmWeb , ArmSTEM , FineWeb-Edu, Stack-smol) or reproducible from the released pipeline.

This is a base model : no instruction tuning, no chat template. Use it for Armenian text continuation, likelihood scoring, or as a starting point for Armenian SFT.

Recipe

• Base : Gemma-4-E4B, stock tokenizer (best Armenian fertility among CPT candidates: 4.15 tokens/word; vocabulary extension was sharply harmful in our ablation at this budget).
• Continued pretraining : 10B tokens, sequence-packed at length 4096, LR 3×10⁻⁵ cosine. The architectural context window is unchanged from the base (128K, max_position_embeddings=131072 ); CPT only exercised positions up to 4096, and long-context behavior beyond that is inherited from the base and not specifically evaluated.
• Mixture (epoch-capped) : 69% ArmWeb, 4% ArmSTEM-HY, 2% ArmSTEM-EN, 20% English web replay (FineWeb-Edu), 5% code (Stack-smol) — every ArmWeb token seen 2× and every ArmSTEM token 7–9× (Gemma-tokenizer accounting). The ArmSTEM stream is a 109,885-item subset of the corpus ( 30%, sampled at random with balanced stratification across the math and science pools); its manifest ships with ArmSTEM ( stats/cpt_training_subset_ids.txt , 104,630 items verbatim in the release, 5,255 superseded before release). A control run trained on the full 373K corpus (~1.3 epochs instead of 7–9) scores within noise of the released model (paper §5), so the repetition is costless.
• Details : global batch 512 sequences (~2.1M tokens/step), 4,770 steps, 100 warmup steps, AdamW (β₂=0.95), weight decay 0.1, bf16, seed 42.

The mixture is the headline finding of the accompanying paper: news-only CPT catastrophically forgets (−21.2pp Belebele at LR 10⁻⁴); a gentler LR recovers two-thirds; 6% verified translated STEM data reverses forgetting entirely , ending +2.2pp above the unadapted base while keeping the fluency gains.

Results

Six-task Armenian likelihood suite (accuracy; harness: lm-eval):

This is the highest six-task mean among all open Armenian models we evaluated — the best prior models score 0.471 (ArmenianGPT-1.0-3B) and 0.436 (HyGPT-10b), both below the unadapted base.

ArmBench-LLM generative tasks (base-model-appropriate metrics; full results in the paper appendix):

*Audited: zero shared 8-grams between the benchmark items and the ArmSTEM training corpus (paper, Appendix).

Honest negatives : POS tagging regresses under CPT (0.18→0.01); exam mathematics is flat for the released model (1.75 points), and the full-corpus control run reaching 2.75 suggests data diversity rather than difficulty is the binding factor; instruction-dependent ArmBench tasks (judged generation, BLEU QA) are low for all base-style models including this one — they measure formatting, and we defer them to an instruction-tuned variant.

Training data & contamination

Training data was decontaminated by 13-gram overlap against ten Armenian evaluation sets and all ArmBench items, on both the English and Armenian side for translated data. The two remaining reported benchmarks, m-MMLU-hy and ARC-hy, were scanned post hoc: zero of 4.31M ArmWeb training documents and 4 of 372,907 ArmSTEM pairs share any 13-gram with their items. Existing public Armenian corpora overlap these benchmarks at 7.9–17.4% (see the ArmWeb card ).

Limitations

Inherits Gemma-4's biases and terms of use. News-domain-concentrated Armenian exposure. Machine-translated STEM data verified for answer preservation and language identity, not stylistic fluency (though a two-annotator native-speaker audit rated 299 of 300 sampled problems valid, Cohen's κ = 1.0). No safety tuning.

Citation

@article{arakelyan2026armweb,
  title  = {From Zero to Hero: An Open LLM Ecosystem for Armenian},
  author = {Arakelyan, Erik and Avetisyan, Khatun and Davtyan, Meri and Grigoryan, Heghine and Khachatryan, Nane and Shahsuvaryan, Hayk and Sergoyan, Henrik and Martirosyan, Vahan},
  year   = {2026},
  note   = {arXiv, forthcoming}
}