dsfsi / simcse-dna

Project Description

This repository contains the trained model for our paper: Fine-tuning a Sentence Transformer for DNA & Protein tasks that is currently under review at BMC Bioinformatics. This model, called simcse-dna ; is based on the original implementation of SimCSE [1] . The original model was adapted for DNA downstream tasks by training it on a small sample size k-mer tokens generated from the human reference genome, and can be used to generate sentence embeddings for DNA tasks.

Prerequisites

Please see the original SimCSE for installation details. The model will als be hosted on Zenodo (DOI: 10.5281/zenodo.11046580).

Usage

Run the following code to get the sentence embeddings:

import torch
from transformers import AutoModel, AutoTokenizer

# Import trained model and tokenizer
tokenizer = AutoTokenizer.from_pretrained("dsfsi/simcse-dna")
model = AutoModel.from_pretrained("dsfsi/simcse-dna")


#sentences is your list of n DNA tokens of size 6 
inputs = tokenizer(sentences, padding=True, truncation=True, return_tensors="pt")

# Get the embeddings
with torch.no_grad():
    embeddings = model(**inputs, output_hidden_states=True, return_dict=True).pooler_output

The retrieved embeddings can be utilized as input for a machine learning classifier to perform classification.

Performance on evaluation tasks

Find out more about the datasets and access in the paper (TBA)

Task 1: Detection of colorectal cancer cases (after oversampling)

Task 2: Prediction of the Gleason grade group (after oversampling)

Task 3: Detection of human TATA sequences (after oversampling)

Authors

• Mpho Mokoatle, Vukosi Marivate, Darlington Mapiye, Riana Bornman, Vanessa M. Hayes
• Contact details : [email protected]

Citation

Bibtex Reference TBA

References

[1] Gao, Tianyu, Xingcheng Yao, and Danqi Chen. "Simcse: Simple contrastive learning of sentence embeddings." arXiv preprint arXiv:2104.08821 (2021).