eolang / sw-fillmask

SW-FillMask - Swahili Masked Language Model

Model description

SW-FillMask is a BERT-based transformer model pre-trained on a large corpus of Swahili text in a self-supervised fashion. The model was trained specifically to handle real-world Kenyan Swahili, which frequently involves code-mixing with English - a limitation of most existing Swahili NLP models trained on formal, clean text.

It was pre-trained on raw texts only, with no human labeling, using an automatic process to generate inputs and labels. More precisely, it was pre-trained with one objective:

• Masked language modeling (MLM): taking a sentence, the model randomly masks 15% of the words in the input then runs the entire masked sentence through the model and has to predict the masked words. This is different from traditional recurrent neural networks (RNNs) that usually see the terms one after the other, or from autoregressive models like GPT which internally masks the future tokens. It allows the model to learn a bidirectional representation of the sentence.

This way, the model learns an inner representation of the Swahili language that can then be used to extract features useful for downstream tasks e.g.

• Named Entity Recognition (Token Classification)
• Text Classification
• Sentiment Analysis

The model is based on the original BERT UNCASED which can be found on google-research/bert readme

Training data

The model was pre-trained on a curated corpus of Swahili text scraped from publicly available sources, with particular attention to Kenyan Swahili usage patterns including natural code-switching between Swahili and English. Standard preprocessing was applied including tokenisation and text cleaning.

Intended uses & limitations

You can use the raw model for masked language modeling, but it is primarily intended to be fine-tuned on a downstream task.

Best suited for:

• Swahili NLP research and applications
• Low-resource language modelling
• Downstream fine-tuning for NER, text classification, sentiment analysis

Not recommended for:

• Direct use in production without fine-tuning
• Tasks requiring formal or standardised Swahili only

How to use

You can use this model directly with a pipeline for masked language modeling:

Fill Mask Pipeline

from transformers import pipeline

fill_mask = pipeline("fill-mask", model="eolang/sw-fillmask")

sample_text = "Tumefanya mabadiliko muhimu [MASK] sera zetu za faragha na vidakuzi"

for prediction in fill_mask(sample_text):
    print(f"{prediction['sequence']}, confidence: {prediction['score']:.4f}")

Tokenizer

from transformers import AutoTokenizer, AutoModelForMaskedLM

tokenizer = AutoTokenizer.from_pretrained("eolang/sw-fillmask")
model = AutoModelForMaskedLM.from_pretrained("eolang/sw-fillmask")

text = "Hii ni tovuti ya idhaa ya Kiswahili ya BBC ambayo hukuletea habari na makala kutoka Afrika na kote duniani kwa lugha ya Kiswahili."
encoded_input = tokenizer(text, return_tensors='pt')
output = model(**encoded_input)
print(output)

Fine-tuning for Token Classification (NER)

from transformers import AutoTokenizer, AutoModelForTokenClassification

tokenizer = AutoTokenizer.from_pretrained("eolang/sw-fillmask")
model = AutoModelForTokenClassification.from_pretrained("eolang/sw-fillmask", num_labels=NUM_LABELS)

Limitations and Bias

Even if the training data used for this model could be reasonably neutral, this model can have biased predictions. The model's performance on formal or standardised Swahili may also differ from its performance on informal or code-mixed text. This is something I'm still working on improving. Feel free to share suggestions/comments via Discussions

Citation

If you use this model in your research, please cite:

@misc{olang2026swfillmask,
  author = {Olang', Emmanuel},
  title = {SW-FillMask: A Swahili Masked Language Model},
  year = {2026},
  publisher = {HuggingFace},
  url = {https://huggingface.co/eolang/sw-fillmask}
}

Model developed by Emmanuel Olang' | GitHub | Website