QCRI / AraDiCE-msa-to-lev

AraDiCE-msa-to-lev: An MSA to Levantine Machine Translation Model Based on NLLB-3.3B

This repository includes an MSA to Levantine machine translation model. This model was used to curate dialectal benchmarks for the AraDiCE paper (citation below). The human post-edited benchmarks can be found here.

Sample Usage

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("QCRI/AraDiCE-msa-to-lev")
model = AutoModelForSeq2SeqLM.from_pretrained("QCRI/AraDiCE-msa-to-lev")

article = "من مصلحتك أن ترحل من كازابلانكا لفترة. هناك موقع لفرنسا الحرة بالقرب من برازفيل. قد أسهل لك العبور."
inputs = tokenizer(article, return_tensors="pt")

translated_tokens = model.generate(
    **inputs, forced_bos_token_id=tokenizer.convert_tokens_to_ids("ajp_Arab"), max_length=30
)
translation = tokenizer.batch_decode(translated_tokens, skip_special_tokens=True)[0]
print(translation)

License

The model is distributed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License (CC BY-NC-SA 4.0) . The full license text can be found in the accompanying licenses_by-nc-sa_4.0_legalcode.txt file.

Citation

Please find the paper here.

@article{mousi2024aradicebenchmarksdialectalcultural,
      title={{AraDiCE}: Benchmarks for Dialectal and Cultural Capabilities in LLMs},
      author={Basel Mousi and Nadir Durrani and Fatema Ahmad and Md. Arid Hasan and Maram Hasanain and Tameem Kabbani and Fahim Dalvi and Shammur Absar Chowdhury and Firoj Alam},
      year={2024},
      publisher={arXiv:2409.11404},
      url={https://arxiv.org/abs/2409.11404},
}