potsawee / deberta-v3-large-mnli

DeBERTa-v3 (large) fine-tuned to Multi-NLI (MNLI)

This model is for Textual Entailment (aka NLI), i.e., predict whether textA is supported by textB . More specifically, it's a 2-way classification where the relationship between textA and textB can be entail, neutral, contradict .

• Input: ( textA , textB )
• Output: prob(entail), prob(contradict)

Note that during training, all 3 labels (entail, neural, contradict) were used. But for this model, the neural output head has been removed.

Model Details

• Base model: deberta-v3-large
• Training data: MNLI
• Training details: num_epochs = 3, batch_size = 16, textA=hypothesis , textB=premise

Example

from transformers import AutoTokenizer, AutoModelForSequenceClassification
tokenizer = AutoTokenizer.from_pretrained("potsawee/deberta-v3-large-mnli")
model = AutoModelForSequenceClassification.from_pretrained("potsawee/deberta-v3-large-mnli")

textA = "Kyle Walker has a personal issue"
textB = "Kyle Walker will remain Manchester City captain following reports about his private life, says boss Pep Guardiola."

inputs = tokenizer.batch_encode_plus(
    batch_text_or_text_pairs=[(textA, textB)],
    add_special_tokens=True, return_tensors="pt",
)
logits = model(**inputs).logits # neutral is already removed
probs = torch.softmax(logits, dim=-1)[0]
# probs = [0.7080, 0.2920], meaning that prob(entail) = 0.708, prob(contradict) = 0.292

Citation

@article{manakul2023selfcheckgpt,
  title={Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models},
  author={Manakul, Potsawee and Liusie, Adian and Gales, Mark JF},
  journal={arXiv preprint arXiv:2303.08896},
  year={2023}
}