NeelNanda / gpt-neox-tokenizer-digits

huggingface.co
Total runs: 0
24-hour runs: 0
7-day runs: 0
30-day runs: 0
Model's Last Updated: November 29 2023

Introduction of gpt-neox-tokenizer-digits

Model Details of gpt-neox-tokenizer-digits

This is a fork of the GPT NeoX 20B tokenizer, edited to split every numerical digit into a separate token. This has the goal of making it easier for the model to learn arithmetic capabilities and to hopefully be more interpretable, and copies the idea from the PaLM tokenizer .

This was done, extremely hackily, by just removing every token that contained "\d\d" (eg "2013"). All remaining digit containing tokens are "0" ... "9" and " 0" ... " 9"

This comes at the cost of making modelling normal text harder, since eg dates like 2013 which naturally should be a single token are now 2|0|1|3.

This has a reduced vocab size of 48252 (several of the tokens towards the end are special whitespace tokens copied in from GPT-NeoX to make tokenizing code easier - some of these are duplicated in the vocabulary and thus may not actually show up at train time).

It includes a padding token (<|PAD|>) an End-Of-String token (<|EOS|>) and a Beginning-Of-String token (<|BOS|>)

Runs of NeelNanda gpt-neox-tokenizer-digits on huggingface.co

0
Total runs
0
24-hour runs
0
3-day runs
0
7-day runs
0
30-day runs

More Information About gpt-neox-tokenizer-digits huggingface.co Model

gpt-neox-tokenizer-digits huggingface.co

gpt-neox-tokenizer-digits huggingface.co is an AI model on huggingface.co that provides gpt-neox-tokenizer-digits's model effect (), which can be used instantly with this NeelNanda gpt-neox-tokenizer-digits model. huggingface.co supports a free trial of the gpt-neox-tokenizer-digits model, and also provides paid use of the gpt-neox-tokenizer-digits. Support call gpt-neox-tokenizer-digits model through api, including Node.js, Python, http.

gpt-neox-tokenizer-digits huggingface.co Url

https://huggingface.co/NeelNanda/gpt-neox-tokenizer-digits

NeelNanda gpt-neox-tokenizer-digits online free

gpt-neox-tokenizer-digits huggingface.co is an online trial and call api platform, which integrates gpt-neox-tokenizer-digits's modeling effects, including api services, and provides a free online trial of gpt-neox-tokenizer-digits, you can try gpt-neox-tokenizer-digits online for free by clicking the link below.

NeelNanda gpt-neox-tokenizer-digits online free url in huggingface.co:

https://huggingface.co/NeelNanda/gpt-neox-tokenizer-digits

gpt-neox-tokenizer-digits install

gpt-neox-tokenizer-digits is an open source model from GitHub that offers a free installation service, and any user can find gpt-neox-tokenizer-digits on GitHub to install. At the same time, huggingface.co provides the effect of gpt-neox-tokenizer-digits install, users can directly use gpt-neox-tokenizer-digits installed effect in huggingface.co for debugging and trial. It also supports api for free installation.

gpt-neox-tokenizer-digits install url in huggingface.co:

https://huggingface.co/NeelNanda/gpt-neox-tokenizer-digits

Url of gpt-neox-tokenizer-digits

gpt-neox-tokenizer-digits huggingface.co Url

Provider of gpt-neox-tokenizer-digits huggingface.co

NeelNanda
ORGANIZATIONS

Other API from NeelNanda

huggingface.co

Total runs: 0
Run Growth: 0
Growth Rate: 0.00%
Updated:October 15 2022
huggingface.co

Total runs: 0
Run Growth: 0
Growth Rate: 0.00%
Updated:October 03 2022
huggingface.co

Total runs: 0
Run Growth: 0
Growth Rate: 0.00%
Updated:October 03 2022