dleemiller / finecat-nli-m

FineCat-NLI Medium

Overview

This model is a fine-tune of the excellent tasksource/ModernBERT-base-nli , trained on the dleemiller/FineCat-NLI dataset—a compilation of several high-quality NLI data sources with quality screening and reduction of easy samples in the training split. The training also incorporates logit distillation from dleemiller/finecat-nli-l .

Distillation loss looks like this: $\begin{array}{cccc}& L=\alpha \cdot L_{\text{CE}}(z^{\left(s\right)},y)+\beta \cdot L_{\text{MSE}}(z^{\left(s\right)},z^{\left(t\right)})& & \end{array}\backslash begin\{equation\}\; \backslash mathcal\{L\}\; =\; \backslash alpha\; \backslash cdot\; \backslash mathcal\{L\}\_\{\backslash text\{CE\}\}(z^\{(s)\},\; y)\; +\; \backslash beta\; \backslash cdot\; \backslash mathcal\{L\}\_\{\backslash text\{MSE\}\}(z^\{(s)\},\; z^\{(t)\})\; \backslash end\{equation\}$ L = α ⋅ L CE ​ ( z ( s ) , y ) + β ⋅ L MSE ​ ( z ( s ) , z ( t ) ) ​ ​

where $z^{\left(s\right)}z^\{(s)\}$ z ( s ) and $z^{\left(t\right)}z^\{(t)\}$ z ( t ) are the student and teacher logits, $yy$ y are the ground truth labels, and $\alpha \backslash alpha$ α and $\beta \backslash beta$ β are equally weighted at 0.5.

This model and dataset specifically targets improving NLI, through high quality sources. The tasksource models are the best checkpoints to start from, although training from ModernBERT is also competitive.

NLI Evaluation Results

F1-Micro scores (equivalent to accuracy) for each dataset. Performance was measured at bs=32 using a Nvidia Blackwell PRO 6000 Max-Q.

Usage

Label Map:

• entailment : 0
• neutral : 1
• contradiction : 2

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load the model and run inference.

from sentence_transformers import CrossEncoder
import numpy as np

model = CrossEncoder("dleemiller/finecat-nli-l")
id2label = model.model.config.id2label  # {0:'entailment', 1:'neutral', 2:'contradiction'}

pairs = [
    ("The glass fell off the counter and shattered on the tile.",
     "The glass broke when it hit the floor."),          # E
    ("The store opens at 9 a.m. every day.",
     "The store opens at 7 a.m. on weekdays."),          # C
    ("A researcher presented results at the conference.",
     "The presentation won the best paper award."),       # N
    ("It started raining heavily, so the match was postponed.",
     "The game was delayed due to weather."),             # E
    ("Every seat on the flight was taken.",
     "There were several empty seats on the plane."),     # C
]

logits = model.predict(pairs)  # shape: (5, 3)

for (prem, hyp), row in zip(pairs, logits):
    pred_idx = int(np.argmax(row))
    pred = id2label[pred_idx]
    print(f"[{pred}]  Premise: {prem}  |  Hypothesis: {hyp}")

Acknowledgments

We thank the creators and contributors of tasksource and MoritzLaurer for making their work available. This model would not be possible without their efforts and open source contributions.

Citation

@misc{nli-compiled-2025,
  title = {FineCat NLI Dataset},
  author = {Lee Miller},
  year = {2025},
  howpublished = {Refined compilation of 6 major NLI datasets}
}