aehrm / dtaec-type-normalizer

DTAEC Type Normalizer

This model is trained from scratch to normalize historic spelling of German to contemporary one. It is type-based, which means that it takes only a single token (without whitespace) as input, and generates the normalized variant. It achieves the following results on the evaluation set:

• Loss: 0.0308
• Wordacc: 0.9546
• Wordacc Oov: 0.9096

Note: This model is part of a larger system, which uses an additional GPT-based model to disambiguate different normalization forms by taking in the full context. See https://github.com/aehrm/hybrid_textnorm .

Training and evaluation data

The model has been trained on the DTA-EC Parallel Corpus Lexicon ( aehrm/dtaec-lexica ), which is from a parallel corpus of the Deutsche Textarchiv (German Text Archive), who aligned historic prints of documents with their moden editions in contemporary orthography.

Training was done on type-level, where, given the historic form of a type, the model must predict the corresponding normalized type that appeared most frequent in the parallel corpus .

Demo Usage

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained('aehrm/dtaec-type-normalizer')
model = AutoModelForSeq2SeqLM.from_pretrained('aehrm/dtaec-type-normalizer')

# Note: you CANNOT normalize full sentences, only word for word!
model_in = tokenizer(['Freyheit', 'seyn', 'ſelbstthätig'], return_tensors='pt', padding=True)
model_out = model.generate(**model_in)

print(tokenizer.batch_decode(model_out, skip_special_tokens=True))
# >>> ['Freiheit', 'sein', 'selbsttätig']

Or, more compact using the huggingface pipeline :

from transformers import pipeline

pipe = pipeline(model="aehrm/dtaec-type-normalizer")
out = pipe(['Freyheit', 'seyn', 'ſelbstthätig'])

print(out)
# >>> [{'generated_text': 'Freiheit'}, {'generated_text': 'sein'}, {'generated_text': 'selbsttätig'}]

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 8
• eval_batch_size: 64
• seed: 12345
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 20

Training results

Framework versions

• Transformers 4.41.2
• Pytorch 2.3.0+cu121
• Datasets 2.19.1
• Tokenizers 0.19.1

License

The model weights are marked with CC0 1.0 Universal .

NOTE: This model and its inferences or derivative works may be considered an Adaptation of

• the DTA EvalCorpus by Bryan Jurish, Henriette Ast, Marko Drotschmann, and Christian Thomas, licensed under the Creative Commons Attribution-NonCommercial 3.0 Unported License ,
• historical source text by the Deutsche Textarchiv, licensed under the Creative Commons Attribution-NonCommercial 3.0 Unported License ,
• contemporary target text by TextGrid, licensed under the Creative Commons Attribution 3.0 Unported License ,
• contemporary target text by Project Gutenberg, licensed under the Project Gutenberg License .

Conditions on attribution and/or restrictions to commercial use may apply.