DerivedFunction / polyglot-tagger-60L-Experimental

Polyglot Tagger: 60L (Experimental)

This model is a fine-tuned version of xlm-roberta-base . It achieves the following results on the evaluation set:

• Loss: 0.0404
• Precision: 0.8848
• Recall: 0.9012
• F1: 0.8929
• Accuracy: 0.9909

Model description

Introducing Polyglot Tagger 60L, a new way to classify multi-lingual documents. By training specifically on token classification on individual sentences, the model generalizes well on a variety of languages, while also behaves as a multi-label classifier, and extracts sentences based on its language.

Intended uses & limitations

This model can be treated as a base model for further fine-tuning on specific language identification extraction tasks. Note that as a general language tagging model, it can potentially get confused from shared language families or from short texts. For example, English and German, Spanish and Portuguese, and Russian and Ukrainian.

The model is trained on a sentence with a minimum of four tokens, so it may not accurately classify very short and ambigous statements. Note that this model is experimental and may produce unexpected results compared to generic text classifiers. It is trained on cleaned text, therefore, "messy" text may unexpectedly produce different results.

Training and Evaluation Data

The model was trained on a synthetic dataset of roughly 2.5 million samples , covering 60 languages across diverse script families (Latin, Cyrillic, Indic, Arabic, Han, etc.), from wikimedia/wikipedia (up to 200,000 individual sentences, 120,000 reserve from up to 100,000 unique articles, by taking the first half of Wikipedia after filtering for stubs), google/smol (up to 1000 individual sentences), and HuggingFaceFW/finetranslations (up to 50,000 sentences, 30,000 reserve from up to 50,000 unique rows), in which it is split into a reserve set for pure documents, and a main set for everything else.

A synthetic training row consists of 1-4 individual and mostly independent sentences extracted from various sources.

The data composition follows a strategic curriculum:

• 60% Pure Documents: Single-language sequences to establish strong baseline profiles for each language.
• 30% Homogenous Mixed: Documents containing one main language, and clear transitions between two or more languages to train boundary detection.
• 10% Mixed with Noise: Integration of "neutral" spans including code snippets, mathematical notation, emojis, symbols, and rot_13 text tagged as O or their respective source to reduce hallucination.

Supported Languages and Limitations (60)

The model supports the following ISO-coded languages: af, am, ar, as, be, bg, bn, cs, da, de, el, en, es, fa, fi, fr, gu, he, hi, hu, hy, id, is, it, ja, ka, kk, km, kn, ko, la, lo, ml, mk, mn, mr, ms, my, nl, no, or, pa, pl, ps, pt, ro, ru, sd, sq, sr, sv, ta, te, th, tr, ug, uk, ur, vi, zh

Note that Romanized versions of any language is not included in the training set, such as Romanized Russian, and Hindi.

The coverage is as follows from a sample:

Per-group coverage (examples / tokens):

Top token languages: ml 8197 it 5178 ta 4903 he 4873 es 4756 de 4625 kn 4613 pa 4457 ja 4188 zh 4131 uk 4007 ko 3958

Evaluation

Please note that these results are not indicative that token classification can substitute for sequence classification.

The model scored the following on papulca/language-identification 's test set

As the training data is slightly biased toward English text, it may produce tokens for English rather than the target language in the Latin family.

The model scored the following on mikaberidze/lid200 's test set, which is derived from Davlan/sib200

Caution: training data include text from Wikipedia and Finetranslations, which may skew the results.

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 72
• eval_batch_size: 36
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 144
• optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• num_epochs: 2
• mixed_precision_training: Native AMP

Training results

Framework versions

• Transformers 5.0.0
• Pytorch 2.10.0+cu128
• Datasets 4.0.0
• Tokenizers 0.22.2