CohereLabs / tiny-aya-base-32K

Model Card for Tiny Aya Base 32K

Model Summary

Cohere Labs Tiny Aya Base 32K is an open-weights research release of a pretrained 3.35 billion parameter model optimized for efficient, strong, and balanced multilingual representation across 70+ languages, including many lower-resourced ones. This variant provides a 32K context window and is designed to support downstream adaptation, instruction tuning and long-context research.

This is a base pretrained model and has not been instruction-tuned or preference-aligned.

This checkpoint is used as the base model for Tiny Aya L2-Thinker and Tiny Aya En-Thinker .

Developed by: Cohere and Cohere Labs

• Point of Contact: Cohere Labs
• License: CC-BY-NC , requires also adhering to Cohere Lab's Acceptable Use Policy
• Model: tiny-aya-base-32K
• Model Size: 3.35B
• Context Length: 32K (input + output)

For the 8K release, see tiny-aya-base . For the broader family, see tiny-aya-global and the Tiny Aya collection .

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "CohereLabs/tiny-aya-base-32K"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    torch_dtype="auto",
)

prompt = "The capital of Spain is"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=128,
    do_sample=True,
    temperature=0.1,
    top_p=0.9,
    top_k=50,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

The model can also be used with the Transformers pipeline abstraction:

from transformers import pipeline

generator = pipeline(
    "text-generation",
    model="CohereLabs/tiny-aya-base-32K",
    torch_dtype="auto",
    device_map="auto",
)

output = generator(
    "The capital of Spain is",
    max_new_tokens=128,
    do_sample=True,
    temperature=0.1,
    top_p=0.9,
    repetition_penalty=1.1,
)
print(output[0]["generated_text"])

The combined prompt and generated sequence must fit within the 32K context window.

Model Details

Input: Text only.

Output: Model generates text.

Model Architecture: Auto-regressive transformer in the Tiny Aya / Cohere family, using interleaved sliding-window and global-attention layers with rotary positional embeddings.

Languages covered: The model was trained on 70+ languages, with a focus on English, Dutch, French, Italian, Portuguese, Romanian, Spanish, Czech, Polish, Ukrainian, Russian, Greek, German, Danish, Swedish, Norwegian, Catalan, Galician, Welsh, Irish, Basque, Croatian, Latvian, Lithuanian, Slovak, Slovenian, Estonian, Finnish, Hungarian, Serbian, Bulgarian, Arabic, Persian, Urdu, Turkish, Maltese, Hebrew, Hindi, Marathi, Bengali, Gujarati, Punjabi, Tamil, Telugu, Nepali, Tagalog, Malay, Indonesian, Vietnamese, Javanese, Khmer, Thai, Lao, Chinese, Burmese, Japanese, Korean, Amharic, Hausa, Igbo, Malagasy, Shona, Swahili, Wolof, Xhosa, Yoruba, and Zulu.

Context Length: Tiny Aya Base 32K supports a context length of 32K tokens across input and output.

Usage and Limitations

Intended Usage

Tiny Aya Base 32K is intended for continued pretraining, instruction tuning, and research on multilingual and long-context language modeling. Potential downstream applications include multilingual text generation, summarization, translation, cross-lingual tasks, and adaptation for lower-resource languages.

Because this is a base model, prompts should use text completion rather than a chat template. Additional post-training is recommended before deploying it as a conversational assistant.

Strengths

Tiny Aya provides strong open-ended generation across broad language coverage, including many lower-resource languages. Shared training signals across language families and scripts support multilingual generation, translation, summarization, and cross-lingual adaptation. This release supports substantially longer sequences than the 8K base release.

Limitations

Base-model behavior. This checkpoint is not instruction-tuned or preference-aligned and may not reliably follow instructions or behave like a conversational assistant.

Reasoning tasks. The model's strongest performance is on open-ended generation. Chain-of-thought reasoning tasks such as multilingual math are comparatively weaker.

Factual knowledge. Outputs may contain incorrect or outdated statements, particularly in lower-resource languages with thinner training data coverage.

Uneven resource distribution. High-resource languages benefit from richer training signals and tend to show more consistent quality. The lowest-resource languages may show greater variability, and culturally specific nuance, sarcasm, or figurative language may be less reliable.

Long-context use. A 32K context window does not guarantee uniform recall or reasoning quality at every position. Long inputs also require substantially more memory and compute.

Model Card Contact

For errors or additional questions about details in this model card, contact [email protected] .

Terms of Use

This model is governed by a CC-BY-NC License (Non-Commercial) and also requires adhering to Cohere Lab's Acceptable Use Policy . If you are interested in commercial use, please contact Cohere’s Sales team.