tau / t5-v1_1-large-rss

T5-V1.1-large-rss

This model is T5-v1.1-large finetuned on RSS dataset. The model was finetuned as part of "How Optimal is Greedy Decoding for Extractive Question Answering?" , while the RSS pretraining method was introduced in this paper .

Model description

The original T5-v1.1-large was only pre-trained on C4 excluding any supervised training. Our version is further trained on Rucurrent Span Selection scheme (RSS), using a sample from the dataset used to pretrain Splinter :

• contexts with a span occurring more than once are detected
• a single instance of the recurring span is maked
• the model is trained (teacher forcing) to predict the masked span This training scheme naturally matches the extractive question answering task.

During training time, the masked span is replaced with <extra_id_0> and the labels are formatted as <extra_id_0>span<extra_id_0> . Unlike Splinter , only one span is mask at a time.

Intended uses & limitations

This model naturally fits tasks where a span from a context is intended to be copied, like extractive question answering. This checkpoint is primarily aimed to be used in zero-shot setting - further fine-tuning it on an annotated dataset gives equal results to those of the original T5-v1.1-large.

How to use

You can use this model directly but it is recommended to format the input to be aligned with that of the training scheme and as a text-question context:

from transformers import  AutoModelForSeq2SeqLM, AutoTokenizer
model = AutoModelForSeq2SeqLM.from_pretrained('tau/t5-v1_1-large-rss')
tokenizer = AutoTokenizer.from_pretrained('tau/t5-v1_1-large-rss')

passage = 'Barack Hussein Obama II is an American politician and attorney who served as the 44th president of the United States from 2009 to 2017. '
question = 'When was Obama inaugurated?'
text = f'Text: {passage}.\nQuestion: {question}\nAnswer:{tokenizer.additional_special_tokens[0]}.'
encoded_input = tokenizer(text, return_tensors='pt')
output_ids = model.generate(input_ids=encoded_input.input_ids, attention_mask=encoded_input.attention_mask,
               eos_token_id=tokenizer.additional_special_tokens_ids[1], num_beams=1, max_length=512, min_length=3)
tokenizer.decode(output_ids[0])

The generated answer is then "<pad><extra_id_0> 2009<extra_id_1>" , while the one generated by the original T5-v1.1-large is "<pad><extra_id_0> On January 20, 2009<extra_id_1>" - a correct yet non-extractive answer.

Limitations and bias

Although using the model with greedy decoding tends toward extracted outputs, is may sometimes produce non-extracted ones - may it be different casing or a whole different string (or substring) that may bear another semantic meaning.

Pretraining

The model was finetuned with 100,000 rss-examples for 3 epochs using Adafactor optimizer with constant learning rate of 5e-5.

Evaluation results

Evaluated over few-shot QA in a zero-shot setting (no finetuning on annotated examples):

The gap between the two models diminishes as more training examples are introduced, for additional result see the paper .

BibTeX entry and citation info

@inproceedings{ram-etal-2021-shot,
    title = "Few-Shot Question Answering by Pretraining Span Selection",
    author = "Ram, Ori  and
      Kirstain, Yuval  and
      Berant, Jonathan  and
      Globerson, Amir  and
      Levy, Omer",
    booktitle = "Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)",
    month = aug,
    year = "2021",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2021.acl-long.239",
    doi = "10.18653/v1/2021.acl-long.239",
    pages = "3066--3079",
},
@misc{castel2021optimal,
      title={How Optimal is Greedy Decoding for Extractive Question Answering?}, 
      author={Or Castel and Ori Ram and Avia Efrat and Omer Levy},
      year={2021},
      eprint={2108.05857},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}