birgermoell / oellm-9b-yarn-multilingual-32k

OpenEuroLLM 9B — YaRN Multilingual 32K

A long-context extension of OpenEuroLLM 9B, trained via continued pre-training using YaRN to extend the context window from 2 048 → 32 768 tokens .

This is a base language model — not instruction-tuned. Intended for research into multilingual long-context language modelling.

Model details

RoPE scaling config

Add to config.json before loading:

"rope_scaling": {
  "factor": 16.0,
  "original_max_position_embeddings": 2048,
  "type": "yarn"
},
"rope_theta": 10000

Languages

Evaluation — Base-LM NIAH

Forced-choice log-likelihood needle-in-a-haystack across 5 context lengths × 5 depths × 4 languages. Random-chance baseline: 25%.

Accuracy by language × context length (all depths averaged)

† NL 32K: depths 0%/25%/50% only.

Near-perfect across 2K–16K at all depths and languages. The only failure is 32K depth=0% (needle at document start, maximum retrieval distance):

All other 32K depths score 1.00. Effective reliable retrieval range: ~24K tokens .

Root cause: missing mscale = 0.1 × ln(16) + 1.277 in v1 training. A corrected v2 model covering 35 European languages is in preparation.

Known limitations

• Missing mscale → retrieval failure at 32K depth=0%. Fixed in v2.
• 8 languages (v2 will cover 35).
• Base model only — needs instruction tuning for assistant use.

Citation

@misc{peng2023yarn,
  title={YaRN: Efficient Context Window Extension of Large Language Models},
  author={Bowen Peng and Jeffrey Quesnelle and Honglu Fan and Enrico Shippole},
  year={2023}, eprint={2309.00071}, archivePrefix={arXiv}
}