dleemiller / finecat-nli-s

FineCat-NLI Small

This model is a fine tune of the 68M parameter jhu-clsp/ettin-encoder-68m (ModernBERT architecture), using the dleemiller/FineCat-NLI dataset. This dataset is a compilation of several high quality sources of NLI data, with quality screening and reduction of easy samples in the train split.

This model and dataset specifically targets improving NLI, through high quality sources. The tasksource models are the best checkpoints to start from, although training from ModernBERT is also competitive.

This model is distilled using teacher logits from dleemiller/finecat-nli-l .

NLI Evaluation Results

F1-Micro scores (equivalent to accuracy) for each dataset.

Usage

Label Map:

• entailment : 0
• neutral : 1
• contradiction : 2

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import CrossEncoder

# Download from the 🤗 Hub
model = CrossEncoder("dleemiller/finecat-nli-s")
# Get scores for pairs of texts
pairs = [
    ['She was quite young, not more than eighteen.', 'The girls was at least eighteen years old, but not much older. '],
    ['Two men working on something in a shop.', 'Two boys are milking a milk cart.'],
    ['A man in goggle swimming.', 'A man swimming in the water with goggles.'],
    ['The Argentine Grand Prix (Spanish: "Gran Premio de Argentina") was a round of the Formula One championship, held intermittently from to , all at the same autodrome in the Argentine national capital of Buenos Aires. Argentine president Juan Perón was the driving force behind the creation of the circuit, after seeing the success of the country\'s own Juan Manuel Fangio.', 'The Argentine Grand Prix involves the racing of automobiles with four wheels'],
    ['Because of limited resources, local legal services programs are forced to turn away tens of thousands of people with critical legal problems.', 'If the resources were there, it would be unthinkable for these programs to be turning so many people away.'],
]
scores = model.predict(pairs)
print(scores.shape)
# (5, 3)