birgermoell / oellm-9b-256k-sft

OELLM 9B 256K SFT

Experimental instruction-tuned checkpoint derived from openeurollm/oellm-9b-256k-theta64m-prelude . It is the first SFT stage of an OpenEuroLLM post-training experiment on LUMI. The base is a dense Qwen3 9B model using the independent OpenEuroLLM 256K tokenizer and a 262,144-token context configuration.

This repository contains the SFT model, not the subsequent SimPO model.

Intended use and interpretation

The purpose of this checkpoint is to test whether short-sequence instruction tuning can add useful chat behavior while retaining the base model's long-context capability. It is not presented as a reasoning-specialized model. The 262,144-token configuration describes the maximum input length accepted by the architecture; it does not, by itself, establish accurate retrieval at that length.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "birgermoell/oellm-9b-256k-sft"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="auto",
).eval()

messages = [{"role": "user", "content": "Förklara allemansrätten kort på svenska."}]
inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt",
    return_dict=True,
).to(model.device)

with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=256,
        eos_token_id=tokenizer.eos_token_id,
        pad_token_id=tokenizer.pad_token_id,
    )
print(tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Long-context inference, especially near 256K, requires multiple GPUs for the KV cache. Keep max_position_embeddings=262144 and rope_theta=64000000 unchanged.

Training

The published weights are an unquantized BF16 export of the consolidated training checkpoint.

Export validation

Before publication, the BF16 export was required to pass a Transformers GPU smoke test covering:

• model and tokenizer loading;
• finite forward-pass logits;
• the native OpenEuroLLM turn-token chat template;
• generation in Swedish, English, and German;
• BF16 weight dtypes;
• 262,144-token context metadata and RoPE theta 64M.

Long-context retrieval

The main question for this checkpoint is whether it can still retrieve information after SFT on 4K sequences. We ran deterministic needle-in-a-haystack (NIAH) probes on the exact published BF16 artifact. One passphrase was inserted at each of five token depths (0%, 25%, 50%, 75%, and 100%), then the model was asked to return it with greedy decoding.

The primary metric is full retrieval : the complete correct passphrase appears anywhere in the answer. Exact format is secondary: the answer contains only the requested passphrase. A verbose answer containing the correct value is therefore a retrieval success and an instruction-format miss.

Natural word passphrases (primary retrieval test)

Outcome: 40/40 full-passphrase retrieval across all eight tested lengths and all five depths. The ~262K prompts contained 262,047 to 262,052 actual input tokens, leaving room for generation inside the model's 262,144-token limit. This verifies full-span single-needle retrieval for this checkpoint; it does not imply 256K reasoning or synthesis quality.

All 40 natural-word cases retained the complete passphrase. Exact-output compliance fell at longer lengths because the model often wrapped the correct passphrase in a sentence.

The 512-to-8K result gives no evidence of a short-context retrieval collapse from RoPE theta 64M. This probe cannot rule out subtler short-context quality or perplexity effects; attributing those to theta would require an otherwise identical lower-theta control checkpoint.

Random hexadecimal codes (hard copying stress test)

Random codes are unusually difficult for this tokenizer because much of each code becomes character/byte-fallback tokens. The model retrieved only 11/25 codes from 512 through 8K, sometimes returning the record ID rather than its passkey. The same harder test retrieved the full correct code at 5/5 depths at 32K, 5/5 at 128K, and 5/5 at ~262K . This non-monotonic pattern points to synthetic-code copying/entity confusion, not loss of access to distant context.

Natural word passphrases were stable at every length. Random hexadecimal copying was unreliable at 512-8K but succeeded at every tested depth from 32K onward; with five cases per point, this should be read as a diagnostic pattern rather than evidence that longer inputs improve the model.

Representative outputs

The inputs below reproduce the evaluated prompt structure, with repeated filler omitted. Outputs are shown verbatim.

Success: exact retrieval at 32K, needle at 50%

Input (truncated from 32,765 tokens):

Read the following archive carefully. One important record is hidden among routine background
material. Remember that record and answer the question at the end.
[routine archive filler omitted]
IMPORTANT RECORD: The passkey for record-524f75bb is sunrise-wren-raven.
[routine archive filler omitted]
Question: What is the passkey for record-524f75bb? Answer with only the exact passkey and no other words.

Model output:

sunrise-wren-raven

Result: full retrieval pass; exact-format pass .

Success: retrieval at ~262K, needle at 50%, with a format miss

Input (truncated from 262,048 tokens):

Read the following archive carefully. One important record is hidden among routine background material.
[about half of the routine archive filler omitted]
IMPORTANT RECORD: The passkey for record-b52bade9 is stone-willow-falcon.
[remaining routine archive filler omitted]
Question: What is the passkey for record-b52bade9? Answer with only the exact passkey and no other words.

Model output:

The passkey for record-b52bade9 is "stone-willow-falcon".

Result: full retrieval pass; exact-format fail . The distant value was recalled, but the model did not follow the requested output format.

Success: retrieval at ~262K, needle near the end

Input (truncated from 262,047 tokens):

Read the following archive carefully. One important record is hidden among routine background material.
[almost the entire routine archive omitted]
IMPORTANT RECORD: The passkey for record-f3bbed67 is north-falcon-river.
Question: What is the passkey for record-f3bbed67? Answer with only the exact passkey and no other words.

Model output:

The passkey for record-f3bbed67 is north-falcon-river.

Result: full retrieval pass; exact-format fail .

Short-context format error: altered prefix at 2K

Input (truncated from 2,012 tokens; needle at 75%):

[routine archive filler omitted]
IMPORTANT RECORD: The passkey for record-f70c7b1f is anchor-willow-wren.
[remaining filler omitted]
Question: What is the passkey for record-f70c7b1f? Answer with only the exact passkey and no other words.

Model output:

passkey-anchor-willow-wren

Result: full retrieval pass; exact-format fail . This is a formatting/instruction-following error, not a memory failure, because the complete target string is present.

Failure: copied the record ID instead of the passkey at 8K

Input (truncated from 8,153 tokens; needle at 50%):

[routine archive filler omitted]
IMPORTANT RECORD: The passkey for record-42efa78b is NIAH-4859184CAD75.
[remaining filler omitted]
Question: What is the passkey for record-42efa78b? Answer with only the exact passkey and no other words.

Model output:

42efa78b

Result: retrieval fail . The model copied the identifier from the question rather than its associated value, consistent with entity-role confusion on token-fragmented synthetic strings.

Failure: returned no answer for a near-end 8K needle

Input (truncated from 8,154 tokens; needle at 99.4%):

[almost the entire routine archive omitted]
IMPORTANT RECORD: The passkey for record-c8da86af is NIAH-0A78C2D44DC2.
Question: What is the passkey for record-c8da86af? Answer with only the exact passkey and no other words.

Model output:

N/A

Result: retrieval fail . Because the needle was adjacent to the question, distance alone does not explain this error.

Hard-code retrieval success at ~262K, needle at 50%

Input (truncated from 262,056 tokens):

[about half of the routine archive filler omitted]
IMPORTANT RECORD: The passkey for record-b52bade9 is NIAH-7AFD87186345.
[remaining routine archive filler omitted]
Question: What is the passkey for record-b52bade9? Answer with only the exact passkey and no other words.

Model output:

The passkey for record-b52bade9 is NIAH-7AFD87186345.

Result: full retrieval pass; exact-format fail .

The ~262K hex cells averaged about 269 seconds each using Transformers, BF16, FlashAttention 2, and a model/KV-cache split over eight LUMI MI250X GCDs. This is an evaluator latency observation, not a serving-throughput benchmark.

These are small synthetic probes (five cases per length), not comprehensive evidence of long-document reasoning, multi-document synthesis, or robustness to distractors unlike the repeated filler used here. Raw reports: word passphrases, 512-8K , word passphrases, 32K-262K , and hex codes at 512-2K , hex codes at 4K , and hex codes at 8K-262K .

Secondary general behavior

This model is not reasoning-tuned. On a 1,368-example multilingual development suite it scored 23.2% overall, including 63.2% instruction following, 58.6% grounded QA, and 5.9% reasoning/math. A 24-prompt Swedish generation probe had mean language-detection probability 0.96, but manual review found factual, arithmetic, grammar, and exact-constraint errors. These results are included to prevent the long-context retrieval result from being mistaken for a broad reasoning-quality claim. Raw reports: multilingual holdouts and Swedish generations .

Limitations

• Experimental SFT-stage research checkpoint; not a final production assistant.
• No completed safety evaluation.
• Preference optimization (SimPO) is a separate subsequent stage.
• Long-context evidence is currently limited to single-needle retrieval; broader long-document comprehension and reasoning remain unestablished.
• It often retrieves the correct value but ignores exact output-format constraints.
• Random alphanumeric copying is unreliable at some short lengths despite strong word-passphrase retrieval and successful full-code retrieval at 32K through ~262K.
• General factuality, reasoning, grammatical correction, and structured-output reliability are weak.
• The multilingual instruction mix includes synthetic and translated data and may contain translationese.
• Inherits limitations, biases, and knowledge gaps from the base model and training data.

Reproduction

Training and evaluation code: https://github.com/BirgerMoell/qwen35-posttrain , especially docs/RUNBOOK_oellm9b_256k_posttrain.md .

License

Apache 2.0, inherited from the base model. Consult the component dataset cards for training-data terms.