cointegrated / rut5-small-normalizer

This is a small Russian denoising autoencoder. It can be used for restoring corrupted sentences.

This model was produced by fine-tuning the rut5-small model on the task of reconstructing a sentence:

• restoring word positions (after slightly shuffling them)
• restoring dropped words and punctuation marks (after dropping some of them randomly)
• restoring inflection of words (after changing their inflection randomly using natasha and pymorphy2 packages)

The fine-tuning was performed on a Leipzig web corpus of Russian sentences.

The model can be applied as follows:

# !pip install transformers sentencepiece
import torch
from transformers import T5ForConditionalGeneration, T5Tokenizer
tokenizer = T5Tokenizer.from_pretrained("cointegrated/rut5-small-normalizer")
model = T5ForConditionalGeneration.from_pretrained("cointegrated/rut5-small-normalizer")

text = 'меня тобой не понимать'
inputs = tokenizer(text, return_tensors='pt')
with torch.no_grad():
    hypotheses = model.generate(
        **inputs, 
        do_sample=True, top_p=0.95, 
        num_return_sequences=5, 
        repetition_penalty=2.5,
        max_length=32,
    )
for h in hypotheses:
    print(tokenizer.decode(h, skip_special_tokens=True))

A possible output is:

# Мне тебя не понимать.
# Если бы ты понимаешь меня?
# Я с тобой не понимаю.
# Я тебя не понимаю.
# Я не понимаю о чем ты.