SIRIS-Lab / erc-classifiers

ERC Classifiers

This repository contains a model trained for multi-label classification of scientific papers in the ERC (European Research Council) context. The model predicts multiple categories for a paper, such as its research domain or topic, based on the abstract and title.

Model Description

The model is based on SPECTER (a transformer-based model pre-trained on scientific literature), fine-tuned for multi-label classification on a dataset of scientific papers. The model classifies papers into several categories, which are defined by the ERC categories . The fine-tuned model is trained to predict these categories given the title and abstract of each paper.

Preprocessing

The preprocessing pipeline involves:

1. Data Loading : Papers are loaded from a Parquet file containing the title, abstract, and their respective categories.
2. Label Cleaning : Labels (categories) are processed to remove any unnecessary information (like content within parentheses).
3. Label Encoding : Categories are transformed into a binary matrix using the MultiLabelBinarizer from scikit-learn. Each category corresponds to a column, and the value is 1 if the paper belongs to that category, 0 otherwise.
4. Statistics and Visualization : Basic statistics and visualizations, such as label distributions, are generated to help understand the dataset better.

Training

The model is fine-tuned on the preprocessed dataset using the following setup:

• Base Model : The model uses the allenai/specter transformer as the base model for sequence classification.
• Optimizer : AdamW optimizer with a learning rate of 5e-5 is used.
• Loss Function : Binary Cross-Entropy with logits ( BCEWithLogitsLoss ) is employed, as the task is multi-label classification.
• Epochs : The model is trained for 5 epochs with a batch size of 4.
• Training Data : The model is trained on a processed dataset stored in train_ready.parquet .

Evaluation

The model is evaluated using both single-label and multi-label metrics:

Single-Label Evaluation

• Accuracy : The accuracy is measured by checking how often the true label appears in the predicted labels.
• Precision, Recall, F1 : These metrics are calculated for each class and averaged for the entire dataset.

Multi-Label Evaluation

• Micro and Macro Metrics : Precision, recall, and F1 scores are computed using both micro-averaging (overall performance) and macro-averaging (performance per label).
• Label Frequency Plot : A plot showing the frequency distribution of labels in the test set.
• Top and Bottom F1 Plot : A plot visualizing the top and bottom labels based on their F1 scores.

Dataset

The dataset consists of scientific papers, each with the following columns:

• title : The title of the paper.
• abstract : The abstract of the paper.
• label : A list of categories (labels) assigned to the paper.

The dataset is preprocessed and stored in a train_ready.parquet file.

Files

• config.json : Model configuration file.
• model.safetensors : Saved fine-tuned model weights.
• tokenizer.json : Tokenizer configuration for the fine-tuned model.
• tokenizer_config.json : Tokenizer settings.
• special_tokens_map.json : Special tokens used by the tokenizer.
• vocab.txt : Vocabulary file for the fine-tuned tokenizer.

Usage

To use the model, follow these steps:

1. Install Dependencies :

   pip install transformers torch datasets
   

2. Load the Model and Tokenizer :

   from transformers import AutoModelForSequenceClassification, AutoTokenizer
   
   model_name = "SIRIS-Lab/erc-classifiers"
   
   # Load fine-tuned model and tokenizer
   model = AutoModelForSequenceClassification.from_pretrained(model_name)
   tokenizer = AutoTokenizer.from_pretrained(model_name)
   

3. Use the Model for Prediction :

   # Example paper title and abstract
   text = "Example title and abstract of a scientific paper."
   
   # Tokenize the input text
   inputs = tokenizer(text, return_tensors="pt", padding=True, truncation=True)
   
   # Make predictions
   with torch.no_grad():
       logits = model(**inputs).logits
   
   # Apply sigmoid activation to get probabilities
   probabilities = torch.sigmoid(logits)
   
   # Get predicted labels (threshold at 0.5)
   predicted_labels = (probabilities >= 0.5).long().cpu().numpy()
   print(predicted_labels)
   

Conclusion

This model provides an efficient solution for classifying scientific papers into multiple categories based on their content. It uses state-of-the-art transformer-based techniques and is fine-tuned on a real-world dataset of ERC-related scientific papers.