dnagpt / OmniGene-4-CPT-v2-merged

OmniGene-4-CPT-v2-Merged

Full BF16 model with CPT (Continued Pretraining) merged

This is the complete merged model (not LoRA adapter). You can load and use it directly without needing the base Gemma-4 model.

Model Description

OmniGene-4-CPT-v2-Merged is a biological foundation model with:

• Base : Gemma-4-26B-A4B-Instruct (MoE, 128 experts, top-8 routing)
• Vocabulary : 290,048 tokens (262,020 original + 28,028 bio tokens)
• CPT data : 32.5 GB mixed corpus (DNA, Protein, OpenWebText, Structure)
• Training : 0.6 epoch, 2,806 steps, 8×H20 GPUs
• Precision : BF16 (~50 GB)

Quick Start

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

# Load model (requires ~50GB GPU memory)
model = AutoModelForCausalLM.from_pretrained(
    "dnagpt/OmniGene-4-CPT-v2-merged",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("dnagpt/OmniGene-4-CPT-v2-merged")

# Generate
prompt = "MKTAYIAKQRQISFVKSHFSRQLEERLGLIEVQAPILSRVGDGTQDNLSGAEKAVQVKVKALPDAQFEVVHSLAKWKRQTLGQHDFSAGEGLYTHMKALRPDEDRLSPLHSVYVDQWDWERVMGDGERQFSTLKSTVEAIWAGIKATEAAVSEEFGLAPFLPDQIHFVHSQELLSRYPDLDAKGRERAIAKDLGAVFLVGIGGKLSDGHRHDVRAPDYDDWSTPSELGHAGLNGDILVWNPVLEDAFELSSMGIRVDADTLKHQLALTGDEDRLELEWHQALLRGEMPQTIGGGIGQSRLTMLLLQLPHIGQVQAGVWPAAVRESVPSLL"

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

For Lower Memory Usage

If you have limited GPU memory, use 4-bit quantization:

from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
import torch

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

model = AutoModelForCausalLM.from_pretrained(
    "dnagpt/OmniGene-4-CPT-v2-merged",
    quantization_config=bnb_config,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("dnagpt/OmniGene-4-CPT-v2-merged")

This reduces memory usage to ~13GB.

Model Architecture

• Layers : 30 transformer layers
• Experts : 128 experts per layer (top-8 routing)
• Hidden size : 2816
• Attention heads : 22
• Active parameters : ~3.8B per token
• Total parameters : ~26B

Biological Tokens

The model includes 28,028 additional biological tokens:

• DNA BPE : 20,000 tokens (optimized for genomic sequences)
• Protein BPE : 8,000 tokens (optimized for amino acid sequences)
• 3Di alphabet : 20 tokens (Foldseek structural alphabet)
• DSSP : 8 tokens (secondary structure: H, E, C, etc.)

Training Data

Differences from LoRA Version

• LoRA version ( dnagpt/OmniGene-4-CPT-v2 ): 1.9 GB, requires base Gemma-4 model
• Merged version (this repo): ~50 GB, standalone, no base model needed

Next Steps

For instruction-tuned version, see:

• OmniGene-4-SFT-v3-Merged : https://huggingface.co/dnagpt/OmniGene-4-SFT-v3-merged

Citation

@article{wang2026omnigene4,
  title={OmniGene-4: A Unified Bio-Language MoE Model with Router-Level Interpretability},
  author={Wang, Liang},
  journal={bioRxiv},
  year={2026}
}

Paper

Full paper: https://github.com/maris205/omnigene4

License

Apache 2.0

Contact

Liang Wang ( [email protected] )
School of Artificial Intelligence and Automation
Huazhong University of Science and Technology