Sponsored by APIMart
Toolify Toolify logo
AI Models
tokenizer-test

malteos / tokenizer-test

Run on huggingface.co
huggingface.co
Total runs: 0
24-hour runs: 0
7-day runs: 0
30-day runs: 0
Model's Last Updated: October 10 2022

Introduction of tokenizer-test

Model Details of tokenizer-test

A GPT2-tokenizer for English and German with a vocabulary size of 88,301.

This tokenizer is created by merging the original GPT2 tokenizer (English) with a German tokenizer .

Steps to reproduce 
from transformers import AutoTokenizer

a_tokenizer = AutoTokenizer.from_pretrained('gpt2')
b_tokenizer = AutoTokenizer.from_pretrained('malteos/gpt2-xl-wechsel-german')

a_vocab = set(a_tokenizer.vocab.keys())  # len(a_vocab)=50257
b_vocab = set(b_tokenizer.vocab.keys())  # len(b_vocab)=50257

missing_tokens_in_a = b_vocab - a_vocab  # len = 38044 

a_tokenizer.add_tokens(list(missing_tokens_in_a))

a_tokenizer.save('opengptx-en-de') # len = 88301

Runs of malteos tokenizer-test on huggingface.co

0
Total runs
0
24-hour runs
0
3-day runs
0
7-day runs
0
30-day runs

More Information About tokenizer-test huggingface.co Model

More tokenizer-test license Visit here:

https://choosealicense.com/licenses/mit

tokenizer-test huggingface.co

tokenizer-test huggingface.co is an AI model on huggingface.co that provides tokenizer-test's model effect (), which can be used instantly with this malteos tokenizer-test model. huggingface.co supports a free trial of the tokenizer-test model, and also provides paid use of the tokenizer-test. Support call tokenizer-test model through api, including Node.js, Python, http.

tokenizer-test huggingface.co Url

https://huggingface.co/malteos/tokenizer-test

malteos tokenizer-test online free

tokenizer-test huggingface.co is an online trial and call api platform, which integrates tokenizer-test's modeling effects, including api services, and provides a free online trial of tokenizer-test, you can try tokenizer-test online for free by clicking the link below.

malteos tokenizer-test online free url in huggingface.co:

https://huggingface.co/malteos/tokenizer-test

tokenizer-test install

tokenizer-test is an open source model from GitHub that offers a free installation service, and any user can find tokenizer-test on GitHub to install. At the same time, huggingface.co provides the effect of tokenizer-test install, users can directly use tokenizer-test installed effect in huggingface.co for debugging and trial. It also supports api for free installation.

tokenizer-test install url in huggingface.co:

https://huggingface.co/malteos/tokenizer-test

Url of tokenizer-test

tokenizer-test huggingface.co Url

Provider of tokenizer-test huggingface.co

malteos
ORGANIZATIONS
huggingface.co

Other API from malteos

 huggingface.co
malteos/scincl

Total runs: 13.3K
Run Growth: -19.8K
Growth Rate: -141.42%
Updated:June 05 2024
huggingface.co
malteos/gpt2-uk

Total runs: 2.4K
Run Growth: 2.1K
Growth Rate: 90.41%
Updated:December 10 2023
huggingface.co
malteos/scincl-wol

Total runs: 113
Run Growth: 1
Growth Rate: 0.89%
Updated:March 07 2022
huggingface.co
malteos/german-r1

Total runs: 34
Run Growth: 26
Growth Rate: 76.47%
Updated:February 12 2025
huggingface.co
malteos/PubMedNCL

Total runs: 11
Run Growth: -39
Growth Rate: -354.55%
Updated:October 12 2023
huggingface.co
malteos/hermeo-7b

Total runs: 7
Run Growth: -8
Growth Rate: -88.89%
Updated:February 13 2024
huggingface.co
malteos/specter-wol

Total runs: 2
Run Growth: 1
Growth Rate: 50.00%
Updated:April 11 2022