EphAsad / BactoTiny-Beta

BactoTiny-Beta

A from-scratch BERT-style encoder trained on bacterial 16S rRNA sequences for sequence representation learning and genus-group classification.

⚠️ Research use only. This model is not a diagnostic device. Predictions must not be used for clinical, food-safety, or regulatory decisions without reference confirmation by an accredited laboratory.

---

Overview

BactoTiny-Beta is a 86M parameter BERT encoder trained from random initialisation on bacterial 16S rRNA sequences from SILVA 138.1 NR99. No weights from any existing language model were transferred. All sequence knowledge was learned during masked language modelling (MLM) pretraining on a curated, fragment-augmented corpus of ~62,000 bacterial 16S sequences.

The model produces fixed-length sequence embeddings (768-dimensional CLS vectors) that can be used for downstream bacterial group classification, sequence similarity search, or transfer learning.

---

Architecture

Tokenizer: Character-level. Each nucleotide is one token. Vocabulary size of 10 means that nearly all model parameters reside in the transformer layers rather than embeddings, forcing the model to learn sequence structure directly.

---

Training

Fragment augmentation rationale: Full-length 16S sequences (~1500 bp) are truncated to 512 tokens at tokenisation. Without augmentation, the model never sees short-context inputs during pretraining, causing CLS embedding collapse on partial fragments at inference. Augmentation ensures the model trains on the same length distribution it will encounter downstream.

---

Target Genus Groups (18)

Note: Clostridium was excluded from training due to insufficient SILVA representation (2 sequences) in this filtered corpus.

---

Benchmark Results

Classification accuracy on held-out test sequences (20% stratified split). Classifier: logistic regression linear probe on frozen CLS embeddings.

Full benchmark — all fragment lengths

Per-class report — full-length sequences (linear probe)

---

Honest Assessment of Results

What the model does well: Listeria (F1 0.95), Salmonella (0.94), Campylobacter (0.91), Staphylococcus (0.91), and Yersinia (0.90) are all strongly represented. These genera have genuinely distinctive 16S signatures and the model has learned them effectively. Full-length classification accuracy of 0.80 is meaningful for a from-scratch model with a 10-token vocabulary.

Where the model struggles: The five Enterobacteriaceae genera with low F1 — Cronobacter (0.32), Citrobacter (0.51), Enterobacter (0.55), Serratia (0.57), Proteus (0.61) — are genuinely difficult to separate by 16S rRNA. This is a biological limitation, not only a model limitation. The 16S gene is not sufficiently variable within Enterobacteriaceae to reliably discriminate at genus level. Biochemical confirmation is required regardless of any sequence-based prediction.

Fragment-length performance: Short-fragment accuracy (0.185–0.368 at 150–800 bp) is substantially below the XGBoost k-mer baseline at equivalent lengths. XGBoost on 4-mer frequencies is a strong baseline for 16S classification and outperforms this model at all fragment lengths in direct comparison. The primary value of BactoTiny-Seq is not competitive accuracy against k-mer methods, but rather:

• Learned sequence representations that capture structural and evolutionary patterns beyond frequency statistics
• A domain-specific pretraining foundation for transfer learning to other bacterial genomics tasks
• A proof of concept that a character-level transformer can be trained from scratch on bacterial sequence data and produce biologically meaningful embeddings

The short-fragment gap relative to XGBoost reflects that (a) the model is trained on 512-token windows of full-length sequences and may benefit from longer pretraining with more fragment diversity, and (b) 4-mer frequency profiles are a very efficient encoding for short sequences where there is little structural context for a transformer to exploit.

---

Limitations

• Context window is 512 tokens (~510 bp). Sequences longer than this are truncated at the 3′ end.
• 16S rRNA cannot reliably distinguish Escherichia from Shigella — these are always reported as a combined group.
• Short fragments (<300 bp) produce low-confidence predictions. Minimum recommended input: 300 bp.
• Trained on 18 target genera only. Organisms outside this set will produce low-confidence, unreliable predictions.
• Cronobacter and Proteus have low test support (45 and 68 sequences respectively). Per-class metrics for these groups should be interpreted with caution.
• The Enterobacteriaceae confusion cluster (Enterobacter, Citrobacter, Serratia, Cronobacter, Proteus) reflects the fundamental limitation of 16S-based genus-level identification within this family.
• This model has not been validated against clinical, environmental, or food-safety isolates. It is a research prototype.

---

Intended Use

• Bacterial group prediction from 16S rRNA sequences as a research tool
• Sequence embedding extraction for downstream microbiological ML tasks
• Benchmarking and comparison of learned versus frequency-based 16S representations
• Educational demonstration of from-scratch biological sequence modelling

Out-of-Scope Use

• Clinical or diagnostic microbiology without reference confirmation
• Species-level identification (model operates at family/genus-group level only)
• Regulatory, food-safety, or public health decision-making
• Organisms outside the 18 target genera

---

Usage

from transformers import BertModel, PreTrainedTokenizerFast
import torch
import numpy as np

tokenizer = PreTrainedTokenizerFast.from_pretrained("EphAsad/BactoTiny-Beta")
model     = BertModel.from_pretrained("EphAsad/BactoTiny-Beta",
                                       add_pooling_layer=False)
model.eval()

def embed(sequence: str) -> np.ndarray:
    sequence = sequence.upper().replace("U", "T")
    # Dynamic padding: pad to sequence length, not global max
    seq_len  = min(len(sequence) + 2, 512)
    pad_len  = ((seq_len + 7) // 8) * 8
    enc = tokenizer(sequence, max_length=pad_len,
                    truncation=True, padding="max_length",
                    return_tensors="pt")
    with torch.no_grad():
        out = model(**enc)
    return out.last_hidden_state[:, 0, :].squeeze().numpy()  # CLS vector

emb = embed("AGAGTTTGATCCTGGCTCAGAACGAACGCTGGCGGCAGGCCTAACACATGCAAGTCGAGCGG")
print(emb.shape)  # (768,)

---

Citation

@misc{bactotiny2025,
  author    = {Asad, Zain},
  title     = {BactoTiny-Beta: From-scratch bacterial 16S rRNA transformer},
  year      = {2025},
  publisher = {HuggingFace},
  url       = {https://huggingface.co/EphAsad/BactoTiny-Beta}
}

---

Project Context

BactoTiny-Seq is part of a portfolio of applied microbiology AI projects including BactAID (hybrid bacterial identification, XGBoost + LoRA FLAN-T5), DomainEmbedder (domain-adaptive embeddings with RL routing), and FireSOP (BM25 + FAISS hybrid SOP retrieval). The unifying design philosophy across all projects is deterministic fallback at every model integration point, confidence-aware outputs, and human-in-the-loop design.