readerbench / RoGPT2-base

Model card for RoGPT2-base

language: - ro

RoGPT2: Romanian GPT2 for text generation

All models are available:

• RoGPT2-base
• RoGPT2-medium
• RoGPT2-large

For code and evaluation check out GitHub .

How to use

# TensorFlow
from transformers import AutoTokenizer, TFAutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained('readerbench/RoGPT2-base')
model = TFAutoModelForCausalLM.from_pretrained('readerbench/RoGPT2-base')
inputs = tokenizer.encode("Este o zi de vara", return_tensors='tf')
text = model.generate(inputs, max_length=1024,  no_repeat_ngram_size=2)
print(tokenizer.decode(text[0]))

# PyTorch
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained('readerbench/RoGPT2-base')
model = AutoModelForCausalLM.from_pretrained('readerbench/RoGPT2-base')
inputs = tokenizer.encode("Este o zi de vara", return_tensors='pt')
text = model.generate(inputs, max_length=1024,  no_repeat_ngram_size=2)
print(tokenizer.decode(text[0]))

Training

Corpus Statistics

Training Statistics

Evaluation

1. MOROCO

2. LaRoSeDa

3. RoSTS

4. WMT16

5. XQuAD

6. Wiki-Ro: LM

7. RoGEC

Note : * the models were trained using the dataset of 3,000,000 artificially generated pairs

Acknowledgments

Research supported with Cloud TPUs from Google's TensorFlow Research Cloud (TFRC)

How to cite

@inproceedings{niculescu2021rogpt2,
  title={RoGPT2: Romanian GPT2 for Text Generation},
  author={Niculescu, Mihai Alexandru and Ruseti, Stefan and Dascalu, Mihai},
  booktitle={2021 IEEE 33rd International Conference on Tools with Artificial Intelligence (ICTAI)},
  pages={1154--1161},
  year={2021},
  organization={IEEE}
}