You can load it with the transformers library like this:
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("CohereLabs/multilingual-22-12")
text = "Hellö World, this is my input string!"
enc = tokenizer(text)
print("Encoded input:")
print(enc)
inv_vocab = {v: k for k, v in tokenizer.vocab.items()}
tokens = [inv_vocab[token_id] for token_id in enc['input_ids']]
print("Tokens:")
print(tokens)
number_of_tokens = len(enc['input_ids'])
print("Number of tokens:", number_of_tokens)
Computing number of tokens
The following values can be used to approximate the number of tokens given the number input characters:
approx_number_of_tokens = len(input_text) / ratio
E.g. for English,
approx_number_of_tokens = len(input_text) / 4.8
.
Language
Avg. characters per token
ar
3.6
de
4.6
en
4.8
es
4.6
fr
4.4
hi
3.8
it
4.5
ja
1.3
ko
2.0
zh
1.1
These values have been computed on the first 10,000 paragraphs from
Wikipedia
. For other dataset, these values might change.
Runs of CohereLabs multilingual-22-12 on huggingface.co
0
Total runs
0
24-hour runs
0
3-day runs
0
7-day runs
0
30-day runs
More Information About multilingual-22-12 huggingface.co Model
multilingual-22-12 huggingface.co is an AI model on huggingface.co that provides multilingual-22-12's model effect (), which can be used instantly with this CohereLabs multilingual-22-12 model. huggingface.co supports a free trial of the multilingual-22-12 model, and also provides paid use of the multilingual-22-12. Support call multilingual-22-12 model through api, including Node.js, Python, http.
multilingual-22-12 huggingface.co is an online trial and call api platform, which integrates multilingual-22-12's modeling effects, including api services, and provides a free online trial of multilingual-22-12, you can try multilingual-22-12 online for free by clicking the link below.
CohereLabs multilingual-22-12 online free url in huggingface.co:
multilingual-22-12 is an open source model from GitHub that offers a free installation service, and any user can find multilingual-22-12 on GitHub to install. At the same time, huggingface.co provides the effect of multilingual-22-12 install, users can directly use multilingual-22-12 installed effect in huggingface.co for debugging and trial. It also supports api for free installation.