sajalregmi4 / arkios-1b-base

Arkios-1B-base

A 1.04B-parameter bilingual English–Nepali base model , pretrained fully from scratch — architecture, tokenizer, and training loop all built for this project — on 150B tokens (150 tokens/parameter, well past Chinchilla-optimal, which is the standard recipe for a model intended for deployment, where inference cost recurs on every request, rather than for a one-time benchmark run).

This is the base checkpoint: next-token pretraining only, no instruction-tuning. Use it as a fine-tuning foundation, or as a completion / autocomplete model. For a ready-to-chat model, see arkios-1b-chat .

Why this model is different

Most small bilingual claims are a Latin-script tokenizer stretched over a second language. Arkios uses a tokenizer purpose-built for Devanagari : the standard GPT-2-style byte-level BPE regex defines a word as \p{L}+ (Unicode letters ), but Nepali vowel signs and virama are Unicode marks ( \p{M} ), not letters — so that regex shatters every Nepali word at every matra, and no amount of data or vocabulary fixes a structural ceiling. Arkios's tokenizer uses [\p{L}\p{M}]+ as the word class instead, at 65,536 vocab entries, reaching 1.69 tokens/word on Nepali . Full analysis: arXiv:2608.26449 and arkios-tokenizer .

Architecture

Qwen3-compatible dense transformer (GQA + RMSNorm + SwiGLU + RoPE + QK-norm):

Training data

150B tokens total. Pretraining mixture, by measured token count against the frozen tokenizer:

The entire practical public Nepali corpus available at training time was ~4.18B unique tokens. At the mixture used, Nepali text is repeated across several epochs; English is not. This is the single largest constraint on this model's Nepali ability, and it is a data-availability ceiling, not a modeling one — see Limitations.

Held-out validation bits-per-byte (bpb; the only cross-tokenizer-comparable loss metric): English 0.746, Nepali 0.314. Do not read the lower Nepali number as "the model is better at Nepali" — Devanagari is ~3 bytes/character in UTF-8, so bpb is compressed by script alone. Use the benchmark numbers below for a language comparison instead.

Training infrastructure

Pretrained on a single 8×H100 SXM5 node (~79 wall-clock hours, ~42.95% measured MFU) using a custom single-file C/CUDA trainer built for this project — no PyTorch, no framework dependency in the training hot path. That trainer and the rest of the training pipeline are not released with these weights; this repository ships weights, tokenizer, and config only.

Evaluation

Full test-split results (not subsampled). Reproduction harness and raw JSON: see the companion technical report.

English — ARC (acc_norm, character-normalized)

For orientation only (published acc_norm, other models): ARC-Easy — Pythia-1.4B (300B tok) 0.57, TinyLlama-1.1B (3T tok) 0.55, OLMo-1B (3T tok) 0.57. ARC-Challenge — Pythia-1.4B 0.26, TinyLlama-1.1B 0.30, OLMo-1B 0.31. Arkios's strong ARC result is likely helped by fineweb-edu 's educational-text composition being unusually well-matched to ARC's grade-school-science format; treat it as a real but domain-favorable number, not a general capability claim (see Limitations).

Other English zero-shot tasks (500-item partial sample; see technical report for caveats on this subset): HellaSwag 0.54, PIQA 0.726, COPA 0.710, OpenBookQA 0.402, LAMBADA 0.484, CommonsenseQA 0.252.

Nepali — Belebele (900-item reading comprehension, 4-way, chance = 0.250)

The same 900 questions exist in Nepali and English, so the English row below is a built-in control, not a separate benchmark — only the language changes.

Read this carefully before drawing conclusions. In the standard multiple-choice-letter format used by most evaluation harnesses, this model scores at chance on Nepali — and also at chance on English, in the same format. That is not a language limitation; it is this small base model's inability to reliably follow the A/B/C/D convention, in either language. Scored on the answer text directly, Nepali comprehension is real (0.306, 3.6 standard errors above chance) though clearly behind English (0.387). A model card, benchmark, or leaderboard entry that reports only the letter format will understate this model's Nepali ability and should not be taken as evidence the model "cannot do Nepali."

Translation (en↔ne, chrF++; 5-shot completion prompting — this is the

base model, not the chat model)

The base model, prompted with few-shot completion, frequently continues in the source language rather than switching to the target — most acutely when asked to produce Nepali. This is substantially a prompting effect, not solely a property of the weights: the same underlying weights, prompted as a zero-shot chat instruction (see arkios-1b-chat), reach 58% target-script rate en→ne and 81% ne→en. If you are building on the base model directly, expect to need instruction-formatted or few-shot prompts tuned for language-locking, not the plain completion format used above.

Passages are the FLORES-200 paragraphs embedded in Belebele (488 aligned by source link ), not FLORES's own sentence-level devtest split — do not compare these numbers to published FLORES-200 devtest scores.

Limitations

• Not instruction-tuned. This model completes text; it does not follow instructions or hold a conversation. Use arkios-1b-chat for that.
• Nepali is behind English on every measured axis, proportional to a ~82:1 English:Nepali unique-token ratio in the pretraining pool.
• No arithmetic, no unit conversion. A 1B parameter budget does not hold this reliably; needs a tool or calculator in any downstream application.
• The ARC result is likely domain-favorable (see above) and should not be generalized into a broad capability claim; HellaSwag and LAMBADA are ordinary for this model class.
• Belebele letter-format score understates Nepali ability — see above.

Citation

@misc{arkios2026,
  title  = {Arkios: An Open Bilingual English-Nepali Language Model Trained
            From Scratch, with a Devanagari-Aware Tokenizer},
  author = {Regmi, Sajal and Pudasaini, Siddhartha and Phakami Pun, Chetan},
  year   = {2026},
  eprint = {2608.30092},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL}
}

Tokenizer paper:

@misc{arkiostokenizer2026,
  title  = {Vowel Signs Are Not Letters: A Pre-tokenization Ceiling on
            Multilingual Tokenizer Fertility},
  author = {Regmi, Sajal and Pudasaini, Siddhartha and Phakami Pun, Chetan},
  year   = {2026},
  eprint = {2608.26449},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL}
}

License

Apache-2.0. Data sources (FineWeb-Edu, OpenWebMath, GitHub-code-clean, FineWeb-2, Sangraha, IndicCorpV2, Wikipedia) retain their own licenses/terms; see the technical report for the per-source attribution list.