BEE-spoke-data / cl100k_base-mlm

huggingface.co
Total runs: 0
24-hour runs: 0
7-day runs: 0
30-day runs: 0
Model's Last Updated: December 29 2025

Introduction of cl100k_base-mlm

Model Details of cl100k_base-mlm

tiktoken cl100k_base : as HF MLM tokenizer

based on RobertaTokenizerFast

from pathlib import Path
from transformers import RobertaTokenizerFast, AutoTokenizer

repo_id = "BEE-spoke-data/cl100k_base-mlm"
tk = AutoTokenizer.from_pretrained(repo_id)
len(tk)
# 100266

testing that it does what it should:

input_text = "i love memes"
tokenized_ids = tk.encode(input_text)
decoded_tokens = tk.convert_ids_to_tokens(tokenized_ids)

print(f"for input '{input_text}' -> {tokenized_ids} -> {decoded_tokens}")
# for input 'i love memes' -> [100277, 72, 3021, 62277, 100278] -> ['<s>', 'i', 'Ġlove', 'Ġmemes', '</s>']

Runs of BEE-spoke-data cl100k_base-mlm on huggingface.co

0
Total runs
0
24-hour runs
0
3-day runs
0
7-day runs
0
30-day runs

More Information About cl100k_base-mlm huggingface.co Model

More cl100k_base-mlm license Visit here:

https://choosealicense.com/licenses/mit

cl100k_base-mlm huggingface.co

cl100k_base-mlm huggingface.co is an AI model on huggingface.co that provides cl100k_base-mlm's model effect (), which can be used instantly with this BEE-spoke-data cl100k_base-mlm model. huggingface.co supports a free trial of the cl100k_base-mlm model, and also provides paid use of the cl100k_base-mlm. Support call cl100k_base-mlm model through api, including Node.js, Python, http.

BEE-spoke-data cl100k_base-mlm online free

cl100k_base-mlm huggingface.co is an online trial and call api platform, which integrates cl100k_base-mlm's modeling effects, including api services, and provides a free online trial of cl100k_base-mlm, you can try cl100k_base-mlm online for free by clicking the link below.

BEE-spoke-data cl100k_base-mlm online free url in huggingface.co:

https://huggingface.co/BEE-spoke-data/cl100k_base-mlm

cl100k_base-mlm install

cl100k_base-mlm is an open source model from GitHub that offers a free installation service, and any user can find cl100k_base-mlm on GitHub to install. At the same time, huggingface.co provides the effect of cl100k_base-mlm install, users can directly use cl100k_base-mlm installed effect in huggingface.co for debugging and trial. It also supports api for free installation.

cl100k_base-mlm install url in huggingface.co:

https://huggingface.co/BEE-spoke-data/cl100k_base-mlm

Url of cl100k_base-mlm

Provider of cl100k_base-mlm huggingface.co

BEE-spoke-data
ORGANIZATIONS

Other API from BEE-spoke-data