CrabInHoney / morphbert-large-morpheme-segmentation-ru

MorphBERT-Large: Russian Morpheme Segmentation

This repository contains the CrabInHoney/morphbert-large-morpheme-segmentation-ru model, a большая transformer-based system fine-tuned for morpheme segmentation of Russian words. The model classifies each character of a given word into one of 25 morpheme categories: ['END', 'END1', 'HYPH', 'HYPH1', 'LINK', 'LINK1', 'LINK2', 'LINK3', 'POSTFIX', 'PREF', 'PREF1', 'PREF2', 'ROOT', 'ROOT1', 'ROOT2', 'ROOT3', 'ROOT4', 'ROOT5', 'SUFF', 'SUFF1', 'SUFF2', 'SUFF3', 'SUFF4', 'SUFF5', 'SUFF6']

Model Description

morphbert-large-morpheme-segmentation-ru uses the powerful transformer architecture, aimed at more accurate prediction of morphological analysis at the character level. Due to its large size, the model demonstrates greater accuracy in determining the constituent morphemes in Russian words compared to the small version (CrabInHoney/morphbert-tiny-morpheme-segmentation-ru).

The model was obtained by learning from scratch, the architecture is comparable in complexity to bert-base.

Key Features:

• Task: Morpheme Segmentation (Token Classification at Character Level)
• Language: Russian (ru)
• Architecture: Transformer (BERT base -like)
• Labels: ['END', 'END1', 'HYPH', 'HYPH1', 'LINK', 'LINK1', 'LINK2', 'LINK3', 'POSTFIX', 'PREF', 'PREF1', 'PREF2', 'ROOT', 'ROOT1', 'ROOT2', 'ROOT3', 'ROOT4', 'ROOT5', 'SUFF', 'SUFF1', 'SUFF2', 'SUFF3', 'SUFF4', 'SUFF5', 'SUFF6']

Model Size & Specifications:

• Parameters: ~85.5 Million
• Tensor Type: F32
• Disk Footprint: ~342 MB

Usage

The model can be easily used with the Hugging Face transformers library. It processes words character by character.

from transformers import AutoTokenizer, AutoModelForTokenClassification
import torch

model_name = "CrabInHoney/morphbert-large-morpheme-segmentation-ru"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForTokenClassification.from_pretrained(model_name)
model.eval()

def analyze(word):
    tokens = list(word)
    encoded = tokenizer(tokens, is_split_into_words=True, return_tensors="pt", truncation=True, max_length=34)
    with torch.no_grad():
        logits = model(**encoded).logits
        predictions = logits.argmax(dim=-1)[0]
    
    word_ids = encoded.word_ids()
    output = []
    current_label = None
    current_chunk = []

    for i, word_idx in enumerate(word_ids):
        if word_idx is not None and word_idx < len(tokens):
            label_id = predictions[i].item()
            label = model.config.id2label[label_id]  
            token = tokens[word_idx]
            
            if label == current_label:
                current_chunk.append(token)
            else:
                if current_chunk:
                    chunk_str = "".join(current_chunk)
                    output.append(f"{chunk_str}:{current_label}")
                current_chunk = [token]
                current_label = label
    if current_chunk:
        chunk_str = "".join(current_chunk)
        output.append(f"{chunk_str}:{current_label}")

    return " / ".join(output)

# Примеры
for word in ["масляный", "предчувствий", "тарковский", "кот", "подгон", "сине-белый", "шторы", "абажур", "дедлайн", "веб-сайт", "адаптированная", "формообразующий"]:
    print(f"{word} → {analyze(word)}")

Example Predictions

масляный → масл:ROOT / ян:SUFF / ый:END
предчувствий → пред:PREF / чу:ROOT / в:SUFF / ств:SUFF1 / ий:END
тарковский → тарк:ROOT / ов:SUFF / ск:SUFF1 / ий:END
кот → кот:ROOT
подгон → под:PREF / гон:ROOT
сине-белый → син:ROOT / е:LINK / -:HYPH / бел:ROOT1 / ый:END
шторы → штор:ROOT / ы:END
абажур → абажур:ROOT
дедлайн → дедлайн:ROOT
веб-сайт → веб:ROOT / -:HYPH / сайт:ROOT1
адаптированная → адапт:ROOT / ир:SUFF / ова:SUFF1 / нн:SUFF2 / ая:END
формообразующий → форм:ROOT / о:LINK / образу:ROOT1 / ющ:SUFF / ий:END

Performance

The model achieves an approximate character-level accuracy of 0.99 on its evaluation dataset.

Limitations

• Performance may vary on out-of-vocabulary words, neologisms, or highly complex morphological structures not sufficiently represented in the training data.
• The model operates strictly at the character level; it does not incorporate broader lexical or syntactic context.
• Ambiguous cases in morpheme boundaries might be resolved based on patterns learned during training, which may not always align with linguistic conventions in edge cases.