living-models / Botanic1-L

BOTANIC-1 — Botanic1-L

Botanic1 is a family of plant genome foundation models from Living Models : bidirectional Mamba-2 ( BiMamba2 ) encoders pre-trained with masked language modeling on single-nucleotide tokens. Training windows come from 320 embryophyte species , 314.6B tokens , in 8,192-token sequences ( <cls> + 8,191 bases).

If you use Botanic1 in your research, please cite the technical report on bioRxiv :

@article{Barozet2026.09.04.749355,
    author = {Barozet, Am{\'e}lie and Cabeli, Vincent and Ogier du Terrail, Jean
              and Rukhovich, Alexey and Janssoone, Thomas and Klajer, Gary
              and Sheikhitarghi, Zeinab and Andrews, Gregory and Veran, Cyril
              and Strouk, L{\'e}onard},
    title = {BOTANIC-1: a series of long-context plant genomic foundation
             models in the agentic era},
    journal = {bioRxiv},
    year = {2026},
    elocation-id = {2026.09.04.749355},
    doi = {10.64898/2026.09.04.749355},
    publisher = {Cold Spring Harbor Laboratory},
    URL = {https://www.biorxiv.org/content/early/2026/09/09/2026.09.04.749355},
    eprint = {https://www.biorxiv.org/content/early/2026/09/09/2026.09.04.749355.full.pdf},
}

---

⚠️ Research use only

These models are intended for research use only. They must not be used in production, clinical, or diagnostic contexts, or for any purpose other than non-commercial research and experimentation. Living Models and the model providers disclaim any liability for use outside this scope.

---

Model variants

S_bal^train is the species-balanced composite of the technical report (mean over 9 task families of per-family species means) computed on the benchmark's train split . The report's leaderboard reports S_bal^test on the held-out test split. The two are not interchangeable; take leaderboard numbers from the report, not from this table.

This card: Botanic1-L (2.1B, hidden size 1536, 72 BiMamba2 blocks, d_state 128, headdim 64, expand 2).

• Tokenizer : single-nucleotide ( A/T/C/G/N , one token per base) plus <cls> , <mask> , <pad> , <unk> . A <cls> token is prepended; no BOS/EOS. The model embeds 9 token ids. Lower-case input is upper-cased.
• Architecture : each block runs two independent Mamba-2 scans (forward and time-reversed) and averages them — bidirectional without attention, no positional embeddings.
• Context : pre-training sequences are 8,192 tokens ( <cls> + 8,191 bases). max_seqlen in config.json is 8,194 tokens; embed_sequences truncates to it and score_variant_zero_shot rejects longer inputs. The SSM has no positional embeddings, so longer inputs run when called directly but are untested.
• Pre-training : MLM (15% mask probability), 314.6B tokens, WSD schedule, fully decayed at the released checkpoint, reverse-complement augmentation with probability 0.5.

---

Installation

Access to living-models/Botanic1-L is gated: while logged in to Hugging Face, accept the terms at the top of the model page (your username and e-mail address are shared with Living Models), then authenticate locally so the download can use your token.

pip install "torch>=2.8" "transformers>=4.57" "huggingface_hub>=0.36"
hf auth login

For the tutorials (2–4 below):

pip install "datasets>=3" "xgboost>=3" "peft>=0.20" scikit-learn pandas tqdm
brew install libomp   # macOS only: the xgboost wheel links Homebrew's OpenMP

datasets<3 fails against current pyarrow with AttributeError: module 'pyarrow' has no attribute 'PyExtensionType' ; keep the >=3 pin.

Tested from scratch in two fresh virtual environments on macOS 15 (Apple M4 Pro, 24 GB) and, for the CUDA numbers, on one H200:

The tutorial scripts live in this repo's Files tab. Download them (they share tutorial_common.py ):

hf download living-models/Botanic1-L tutorial_common.py zero_shot_llr.py \
    frozen_probe_tis.py finetune_tis.py benchmark_paths.py --local-dir botanic1_tutorials
cd botanic1_tutorials

Tutorials 3 and 4 also download the PlantCAD2 TIS split ( kuleshov-group/cross-species-single-nucleotide-annotation , TIS/train.tsv

• TIS/valid.tsv , ~130 MB) on first run.

Devices, memory and batch sizes

The implementation is self-contained pure PyTorch and runs on CPU, Apple Silicon (MPS) and CUDA in fp32. Every tutorial picks cuda , then mps , then cpu , and takes --device to override ( finetune_tis.py uses the Hugging Face Trainer's device selection and --cpu ).

On CUDA, the mamba_ssm + causal_conv1d kernels are used automatically when installed ( pip install mamba-ssm causal-conv1d ). Set BOTANIC1_FORCE_PURE_TORCH=1 to disable them. Parity of the two paths, measured on H200 for all four sizes: argmax agreement 1.000000, Pearson ≥ 0.9999993 on logits, max |Δ log-prob| ≤ 0.06. MPS vs CPU (S and M): max |Δ logit| ≤ 3e-5, argmax agreement 1.0. Forward-pass throughput of this size, measured with benchmark_paths.py (fp32 weights, random 513-token windows, 5 timed iterations after a warm-up):

Measure it on your own hardware:

python benchmark_paths.py --model living-models/Botanic1-L --batch-size 8 --out bench.json

Memory: loading needs the fp32 weights (8.4 GB for this size) plus the pure-torch scan's transient, which grows linearly with batch × sequence length. For 513-token windows the tutorials default to batch 32 on CUDA and 8 on CPU/MPS; raise --batch-size until memory runs out. Botanic1-S at batch 32 × 513 tokens peaks at 3.5 GiB of transient memory per scan on CPU.

Padding behavior

Like at training time, the SSM scans do not mask pad tokens: pads participate in the recurrence. attention_mask is accepted for API compatibility but ignored by the backbone. Prefer batches of equal-length sequences.

---

Examples of use

1. Generating embeddings

from transformers import AutoModel, AutoTokenizer

model_name = "living-models/Botanic1-L"
model = AutoModel.from_pretrained(model_name, trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model.to("cuda")  # or "mps" / "cpu"

embeddings = model.embed_sequences(
    ["ACGTACGTNNACGT", "ACGTACGTACGTAC"],
    tokenizer,
    average_with_reverse_complement=True,
)["post_encoder_norm"]          # (batch, seq_len, 1536)
sequence_embedding = embeddings.mean(dim=1)   # mean-pool -> (batch, 1536)

average_with_reverse_complement=True embeds the reverse complement as a second pass and averages the two outputs position by position . Position i of the reverse-complement pass describes base L−1−i of the input, so the averaged per-position vectors are not nucleotide-aligned . Use the option only ahead of pooling (as above) to get a strand-agnostic sequence-level vector. For per-nucleotide features, embed each strand separately and flip the reverse-complement output along the sequence axis yourself.

Other layers: layer_names=[0, 11, "post_encoder_norm"] (block indices or submodule names).

2. Zero-shot variant effect scoring (LLR)

LLR = log P(alt) − log P(ref) from the masked logits at the variant position (one nucleotide = one token, so exactly one token is masked). Deleterious variants score negative. On the whole-suite zero-shot LLR benchmark of the technical report, Botanic1-L reaches AUROC 0.718 .

# PHYB gene (AT2G18790) in Arabidopsis thaliana
# 500 bp around the ATG start codon: 5'UTR (pos 0-249) | CDS (pos 250+)
sequence = (
    "TTTTTTTTTGTTATCTCTCTCTATCTGAGAGGCACACATTTTGCTTCGTCTTCTTCAATTTATTTTATTGGTTTCTC"
    "CACTTATCTCCGATCTCAATTCTCCCCATTTTCTTCTTCCTCAAGTTCAAAATTCTTGAGAATTTAGCTCTACCAGA"
    "ATTCGTCTCCGATAACTAGTGGATGATGATTCACCCTAAATCCTTCCTTGTCTCAAGGTAATTCTGAGAAATTTCTC"
    "AAATTCAAAATCAAACGGCATGGTTTCCGGAGTCGGGGGTAGTGGCGGTGGCCGTGGCGGTGGCCGTGGCGGAGAA"
    "GAAGAACCGTCGTCAAGTCACACTCCTAATAACCGAAGAGGAGGAGAACAAGCTCAATCGTCGGGAACGAAATCTC"
    "TCAGACCAAGAAGCAACACTGAATCAATGAGCAAAGCAATTCAACAGTACACCGTCGACGCAAGACTCCACGCCGT"
    "TTTCGAACAATCCGGCGAATCAGGGAAATCATTCGACTACT"
)

# Start codon: mutate the T of ATG
llr_atg = model.score_variant_zero_shot(
    sequence=sequence, tokenizer=tokenizer, variant_pos=251, variant_char="C",
)
# 5'UTR position
llr_utr = model.score_variant_zero_shot(
    sequence=sequence, tokenizer=tokenizer, variant_pos=155, variant_char="A",
)
print(f"ATG T->C: {llr_atg:+.4f}   5'UTR T->A: {llr_utr:+.4f}")

Full script with device selection and a results JSON: zero_shot_llr.py ( python zero_shot_llr.py --model living-models/Botanic1-L ). Values it produced:

A strongly negative LLR flags the start-codon change as disruptive. A near-zero LLR means the model finds both alleles about equally likely in this 500 bp context; it is not evidence that the variant is neutral. The model sees no expression, phenotype, or population data, and LLR magnitudes are not calibrated across positions or species.

3. XGBoost probing on frozen embeddings (PlantCAD-TIS)

Frozen-probe protocol on the "TIS" (translation initiation site) binary task from the PlantCAD2 dataset suite (Zhai et al. 2025): embed each 512 bp window, take the middle-nucleotide embedding (token index 1 + 256), train XGBoost on the train split, score the valid split. Full script: frozen_probe_tis.py .

python frozen_probe_tis.py --model living-models/Botanic1-L --num-samples 1000   # -1 = full dataset

The script saves the embeddings ( embeddings_*.npy , labels_*.npy ), the classifier ( xgb_model.json ) and results.json under botanic1_tis_probe/Botanic1-L/n<num-samples>/ . Retrain the classifier without re-embedding with --reuse-embeddings ; reload the classifier with xgb.XGBClassifier().load_model(path) .

Values it produced (strand-aware: minus-strand windows are reverse-complemented):

For reference, the technical report's PlantCAD-TIS score on Arabidopsis thaliana (stratified, species-balanced, different protocol) is 0.912 for this model.

4. Fine-tuning (LoRA or full)

Botanic1ForSequenceClassification adds a CLS-pooled classification head. Full script: finetune_tis.py (LoRA via peft targeting the SSM in_proj / out_proj with the head trained fully, or full fine-tuning).

python finetune_tis.py --model living-models/Botanic1-L --mode lora --num-samples 2000 --epochs 3

Memory for training is set by the per-device batch: backward through the pure-torch scan keeps the chunk intermediates of every layer. Botanic1-S trains at --batch-size 16 on one H200 (kernel path) but needs --batch-size 2 --grad-accum 8 (same effective batch of 16) on a 24 GB Apple M4 Pro, where batch 8 exhausts MPS memory; Botanic1-XL needs --batch-size 8 --grad-accum 2 even on an H200. The batch × grad-accum column below records what each measurement used.

The seed ( --seed , default 0) is applied before the data subsample, the adapter and the classifier head are created. The trained weights (adapter + head for LoRA, full model otherwise) and the tokenizer land in botanic1_tis_ft/Botanic1-L/lora_n2000/final/ ; the script reloads them and checks the logits before exiting. Reload a LoRA run with PeftModel.from_pretrained(base_model, final_dir) , a full run with AutoModelForSequenceClassification.from_pretrained(final_dir, trust_remote_code=True) .

Values it produced (LoRA, N=2000, 3 epochs, seed 0):

Where the numbers come from

Every value in this card is read from a JSON written by the scripts above ( --json-out ), which record the device, library versions and timestamp of the run; Living Models keeps these files with the release tooling. Rows labelled "scripts of this card" and the Apple M4 Pro rows were produced by the scripts shipped in this repo (H200 runs on 2026-09-08, one GPU per size). Rows labelled "2026-09-01 scripts" come from the release-candidate scripts run on the same hardware on 2026-09-01; those seeded the fine-tuning run after creating the adapter and the classifier head, so their LoRA numbers are not seed-reproducible with the current script. A metric re-measured with the current scripts shows only the new row.

---

License

• Model : see the LICENSE for terms of use and citation.

---

Model card maintained by Living Models. For questions or issues, please contact Living Models.