SetFit: Efficient Few-Shot Learning for Text Classification

In the ever-evolving world of Natural Language Processing (NLP), the challenge of training effective models with limited labeled data remains a significant hurdle. Traditional machine learning methods often require extensive datasets to achieve acceptable accuracy. SetFit emerges as a powerful solution, offering an efficient and intuitive approach to few-shot learning for text classification tasks. It is a technique that helps you use just a very few labeled examples per class for better and simpler text classification. This article explores SetFit, delving into its workings, advantages, and practical applications.

The Challenge of Labeled Data in Text Classification

Text classification, the task of assigning predefined categories to text documents, is a cornerstone of NLP. From sentiment analysis to topic categorization, its applications are vast and varied. However, achieving high accuracy often necessitates training models on large, meticulously labeled datasets. This process is time-consuming, resource-intensive, and sometimes impractical, especially when dealing with specialized domains or emerging topics where labeled data is scarce.

The state-of-the-art way for doing text classification, is to bring your text in, use a large language model, something like BERT. That's going to give you some embeddings, which you then put into a classifier or classifier head of a network which will then give you your classification outcome.

• Fine-tuning challenges: Fine-tuning a classifier for a neural network often requires lots of labeled data. Hundreds or thousands of examples and this amount of labeled data goes up dramatically when you start thinking about more classes because all of a sudden, you're going to want plenty of examples for each class to cover that. Fine-tuning, while powerful, demands substantial computational resources and time.
• Prompt engineering limitations: Manually crafted prompts are difficult to employ since they are subject to high variability from manually crafted prompts, and typically require billion-parameter language models to achieve high accuracy.

Introducing SetFit: Few-Shot Learning with Sentence Transformers

SetFit offers an alternative approach that addresses the limitations of traditional methods by leveraging few-shot learning. It uses as little as 10 labeled examples per class.

This technique enables you to achieve comparable or even superior accuracy with significantly less labeled data.

Sentence Transformers: SetFit hinges on the power of Sentence Transformers, pre-trained language models designed to generate meaningful sentence embeddings. Embeddings capture the semantic information of text, allowing models to understand the relationships between words and sentences. Sentence Transformers excel at generating high-quality embeddings, making them ideal for few-shot learning scenarios.

The Core Idea: SetFit uses contrastive learning, widely used in NLP. The pre-trained model itself, as you can see on the left has some idea of what businesses versus sport versus politics but you can see it's a little bit scattered compared to if we could finely tune the sentence transformer so it really understood these classes. Now you can see these clusters are much tighter and the classification performance is going to improve. The core idea is to leverage these pre-trained embeddings and fine-tune them with a small set of labeled examples. This fine-tuning process aligns the embeddings with the specific classification task, enhancing the model's ability to discriminate between categories. The key is to measure the distance between the two, literally the Euclidean distance and what we'd want to do is have great reception, be very close to I love the Family Plan because they are in the same category and very and you want the two that are dissimilar very far apart from each other and that's actually the way the loss functions work when we're using contrast of learning where they put similar things together, try to spread apart the differences.

This approach enables the model to learn from a limited number of examples and generalize effectively to unseen data.

Contrasting Learning

At the heart of SetFit lies the concept of contrasting learning, a technique designed to train embeddings by contrasting similar and dissimilar examples. The process involves creating pairs of text examples and labeling them as either similar (belonging to the same category) or dissimilar (belonging to different categories).

During training, the model learns to minimize the distance between similar pairs and maximize the distance between dissimilar pairs. This process effectively shapes the embedding space, clustering examples from the same category together and separating examples from different categories. In this way similar sample pairs stay close to each other while dissimilar ones.

Benefits of contrasting learning:

• Improved discrimination: It enhances the model's ability to distinguish between categories, even with limited data.
• Robustness to noise: It reduces the impact of noisy or ambiguous examples, improving the model's overall performance.

SetFit Efficiency

The implementation of the sentence pairs is efficient and Prompt-free fine-tuning. To address these shortcomings, the framework is efficient for the fine-tuning of Sentence Transformers.

As a result, SetFit works by first fine-tuning a pre-trained ST on a small number of labeled examples. Using Customer Reviews dataset, SetFit can use a small number of labeled examples to exceed the performance of a large model. But this one should just give you an intuitive understanding of why we want to use set fit and when it works the best.

This efficiency makes SetFit an attractive option for resource-constrained environments, such as mobile devices or edge computing platforms.

Generating Sentence Pairs

The initial step involves generating sentence pairs from the labeled data. For each example, the process creates a similar example (belonging to the same category) and a dissimilar example (belonging to a different category).

The model then can start with a churn dataset with just eight rows, so not enough to do any fine tuning but should be enough to give you an idea of what's going on. To make this work, what you want to do is, you're going to use something called contrastive learning, this is why they used an NLP.

Random sampling or data augmentation techniques can be employed to create these pairs, ensuring diversity and robustness.

Fine-Tuning Sentence Transformers

The generated sentence pairs are then fed into a pre-trained Sentence Transformer, which is fine-tuned using a contrastive loss function. This function encourages the model to generate embeddings that are closer for similar pairs and further apart for dissimilar pairs.

The fine-tuning process adapts the Sentence Transformer to the specific classification task, improving the quality of the generated embeddings and enhancing the model's ability to discriminate between categories.

Training a Classifier

Once the Sentence Transformer is fine-tuned, the next step involves training a classifier on top of the generated embeddings. A simple linear classifier, such as logistic regression, is often sufficient to achieve high accuracy in few-shot scenarios.

The classifier learns to map the embeddings to the corresponding categories, enabling the model to perform text classification on unseen data.

Step 1: Installation

Before diving into the code, you'll need to install the SetFit package. This can be done using pip, Python's package installer. I want to now walk through a quick notebook demonstration of using set fit . I'm going to go through this pretty fast because it's actually pretty simple to use set fit. Here a quick notebook demonstration of using set fit . I'm going to go through this pretty fast because it's actually pretty simple to use set fit.```bash pip install setfit

Step 2: Loading the Dataset

The next step is to load the dataset that you want to use for text classification. SetFit seamlessly integrates with the Hugging Face Datasets library, providing access to a wide range of pre-existing datasets. However, you can also use a custom dataset in a Pandas format.

I'm going to use the same customer reviews data set that was used um in the paper one of the blog posts there. So you'll see we're just loading the sentence transformers that we're going to be the ones that we're going to fine tune as well as the data set from the hugging face hub.

Step 3: Training SetFit

SetFit makes training new test classification models easier than ever. Here we use a standard model from hugging face:```python import numpy as np from setfit import SetFitModel, SetFitTrainer

model = SetFitModel.from_pretrained( "sentence-transformers/all-mpnet-base-v2", )

trainer = SetFitTrainer( model = model, train_dataset = train_dataset, eval_dataset = test_dataset, loss_class = CosineSimilarityLoss )

trainer.train() metrics = trainer.evaluate() print(metrics)

License

SetFit is open-source.

Minimal Labeled Data

SetFit has demonstrated that with just 8 examples per class it can fine tune different types of sentence transformers for text classification. It is where few shot learning comes into place.

Integration with Hugging Face Hub

The data can load a SetFit model from Hub and log into Hugging Face Hub to continue to work. You can download the model for local inference.

High-Growth Business: Customer Reviews

One use case is with using customer reviews dataset to use a small number of labeled examples to exceed the performance of a large model which is well known for a few shot learning. With state of the art performance and model, it is easy to do integration with customer reviews.