BananaMind / BananaMind-2-Micro

BananaMind 2 Micro

BananaMind 2 Micro is a 2,933,193-parameter base causal language model trained from scratch on 74,998,349,824 tokens. It is a pretrained completion model, not an instruction-tuned chat model.

Highlights

• 2.93M parameters with tied input and output embeddings
• 4,096-token context window
• 2,048-token custom tokenizer
• GQA, pre-RoPE QK normalization, SwiGLU, and RMSNorm
• XSA refresh path with detached attention readout, causal depthwise gating, and token-embedding reinjection
• Trained with stock PyTorch Muon for matrix parameters and AdamW for embeddings and one-dimensional parameters

Benchmarks

All results below use the final checkpoint. BananaMind Base Bench 1.1 and ArithMark use raw continuation likelihood rather than generated answers.

Base Bench category Elo

Small-model comparison

ARC, PIQA, HellaSwag, and ArithMark 3 values are acc_norm . ArithMark 2 uses raw accuracy. CMA-1M-Mini was not run on either ArithMark version, so those cells are intentionally blank.

Base Bench values for GPT-S2-5M and CMA-1M-Mini come from the BananaMind Base Bench leaderboard. GPT-S-5M was not listed there when this card was prepared, so its complete official local Base Bench 1.1 result is shown.

Parameter efficiency

The comparison chart uses raw BananaMind Base Bench 1.1 accuracy and removes the four-choice random baseline before normalizing by model size:

efficiency = (accuracy - 25.00) / (parameters / 100,000)

The result is accuracy percentage points above random per 100K parameters . For example, 30% raw accuracy becomes 5 points above random before the size normalization. Higher is better.

This is a narrow parameter-efficiency measurement, not a complete model quality score. The chart can be regenerated with:

python make_parameter_efficiency_chart.py

Architecture

The refresh path reads the detached attention output as a signal, combines a learned projection with a strictly causal depthwise convolution, and gates a projection of the original token embedding. Its learned residual scalar was initialized to zero. During cached generation, convolution history is carried alongside each layer's key/value cache.

Training

Curriculum

The exact source revisions and observed source-token totals are preserved in checkpoint_metadata.json . The tokenizer corpus and revisions are preserved in tokenizer_training_manifest.json .

Usage

This repository contains custom Transformers architecture code. Review configuration_bananamind2micro.py and modeling_bananamind2micro.py before enabling trust_remote_code .

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "BananaMind/BananaMind-2-Micro"

tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    trust_remote_code=True,
)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype="auto",
)

inputs = tokenizer("The future of small language models is", return_tensors="pt")
outputs = model.generate(
    **inputs,
    max_new_tokens=80,
    do_sample=True,
    temperature=0.8,
    top_p=0.95,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Limitations

• This is a very small base model and can produce repetitive, incorrect, or incoherent text.
• It is not instruction aligned and does not reliably follow chat prompts.
• Training was primarily English, so other languages are not a target use.
• The model has no deployment safety guarantees and requires application-level evaluation and safeguards.
• Benchmark results describe fixed datasets and do not guarantee downstream performance.

Reproducibility

• Final preview revision: f45a0e949494c0e75892f65e0e5245f6060ae303
• Final training step: 35,762 / 35,762
• Weight SHA-256: f7a410921a02cd2617cd72c7cfe694da0501edcc6e4ca0747e2cdcdb5d2d14d9
• Weights are stored in float32 Safetensors format.

License

Released under the Apache License 2.0.