Sponsored by APIMart
Toolify Toolify logo
AI Models
Yi-Ko-DUS-9B

beomi / Yi-Ko-DUS-9B

Run on huggingface.co
huggingface.co
Total runs: 295
24-hour runs: 0
7-day runs: 67
30-day runs: 151
Model's Last Updated: Juli 08 2024
text-generation

Introduction of Yi-Ko-DUS-9B

Model Details of Yi-Ko-DUS-9B

Update @ 2024.01.29 Released Yi-Ko(KoEN)-DUS-9B model 🎉

beomi/Yi-Ko-DUS-9B

Yi-Ko-DUS model serves as DUS-applied and advanced iterations of beomi/Yi-Ko-6B model, benefiting from an expanded vocabulary and the inclusion of Korean/English corpus in its further pretraining.

Yi-Ko-DUS model operates with 9B billion parameters.

This repository focuses on the 9B pretrained version, which is tailored to fit the Hugging Face Transformers format, trained after DUS method applied.

Model Details

Model Developers Junbum Lee (Beomi), Taekyoon Choi (Taekyoon)

Variations Yi-Ko-DUS has 9B model only.

Input Models input text only.

Output Models generate text only.

Model Architecture

Yi-Ko-DUS series models are an auto-regressive language model that uses an optimized transformer architecture based on Llama-2*.

*Yi model architecture is based on Llama2, so it can be loaded via LlamaForCausalLM class on HF.

Model Name Training Data Params Context Length GQA Trained Tokens LR Batch Size(per step)
Yi-Ko-DUS-9B A mix of Korean + English online data 9B 4k O >120B 5e -5 2M tokens

Vocab Expansion

Model Name Vocabulary Size Description
Original Yi-Series 64000 Sentencepiece BPE
Expanded Yi-Ko(DUS) Series 78464 Sentencepiece BPE. Added Korean vocab and merges

Tokenizing "안녕하세요, 오늘은 날씨가 좋네요.ㅎㅎ"

Model # of tokens Tokens
Original Yi-Series 47 ['<0xEC>', '<0x95>', '<0x88>', '<0xEB>', '<0x85>', '<0x95>', '하', '<0xEC>', '<0x84>', '<0xB8>', '<0xEC>', '<0x9A>', '<0x94>', ',', '▁', '<0xEC>', '<0x98>', '<0xA4>', '<0xEB>', '<0x8A>', '<0x98>', '은', '▁', '<0xEB>', '<0x82>', '<0xA0>', '<0xEC>', '<0x94>', '<0xA8>', '가', '▁', '<0xEC>', '<0xA2>', '<0x8B>', '<0xEB>', '<0x84>', '<0xA4>', '<0xEC>', '<0x9A>', '<0x94>', '.', '<0xE3>', '<0x85>', '<0x8E>', '<0xE3>', '<0x85>', '<0x8E>']
Expanded Yi-Ko(DUS) Series 10 ['▁안녕', '하세요', ',', '▁오늘은', '▁날', '씨가', '▁좋네요', '.', 'ㅎ', 'ㅎ']
*Equal Korean vocab with Llama-2-Ko Series

Tokenizing "Llama 2: Open Foundation and Fine-Tuned Chat Models"

Model # of tokens Tokens
Original Yi-Series 21 ['The', '▁Y', 'i', '▁series', '▁models', '▁are', '▁large', '▁language', '▁models', '▁trained', '▁from', '▁scratch', '▁by', '▁developers', '▁at', '▁', '0', '1', '.', 'AI', '.']
Expanded Yi-Ko(DUS) Series 21 ['▁The', '▁Y', 'i', '▁series', '▁models', '▁are', '▁large', '▁language', '▁models', '▁trained', '▁from', '▁scratch', '▁by', '▁developers', '▁at', '▁', '0', '1', '.', 'AI', '.']
*Equal Korean vocab with Llama-2-Ko Series *Since Expanded Yi-Ko Series prepends _ at the beginning of the text(to ensure same tokenization for Korean sentences), it shows negilible difference for the first token on English tokenization.

Model Benchmark

5-shot Korean Dataset Evaluation

KMMLU : 43.3514 (exact_match, kmmlu_direct)

KorQuAD : 80.8798 (exact_match)

NSMC : 88.352 (acc)

KOBEST COPA : 84.4831 (macro_f1)

KOBEST HellaSwag : 52.6099 (macro_f1)

Apeach: Korean HateSpeech : 63.4723 (macro_f1)

LICENSE

Apache 2.0 (for research)

For commercial purpose, mailto: [email protected] to acquire Yi-Ko sereis commercial license.

Citation

Please use this bibtex below: 

@misc {lee_junbum_2024,
    author       = { {Lee Junbum, Choi Taekyoon} },
    title        = { Yi-Ko-DUS-9B },
    year         = 2024,
    url          = { https://huggingface.co/beomi/Yi-Ko-DUS-9B },
    doi          = { 10.57967/hf/1707 },
    publisher    = { Hugging Face }
}
Acknowledgement

The training is supported by TPU Research Cloud program.

Runs of beomi Yi-Ko-DUS-9B on huggingface.co

295
Total runs
0
24-hour runs
65
3-day runs
67
7-day runs
151
30-day runs

More Information About Yi-Ko-DUS-9B huggingface.co Model

More Yi-Ko-DUS-9B license Visit here:

https://choosealicense.com/licenses/apache-2.0

Yi-Ko-DUS-9B huggingface.co

Yi-Ko-DUS-9B huggingface.co is an AI model on huggingface.co that provides Yi-Ko-DUS-9B's model effect (), which can be used instantly with this beomi Yi-Ko-DUS-9B model. huggingface.co supports a free trial of the Yi-Ko-DUS-9B model, and also provides paid use of the Yi-Ko-DUS-9B. Support call Yi-Ko-DUS-9B model through api, including Node.js, Python, http.

Yi-Ko-DUS-9B huggingface.co Url

https://huggingface.co/beomi/Yi-Ko-DUS-9B

beomi Yi-Ko-DUS-9B online free

Yi-Ko-DUS-9B huggingface.co is an online trial and call api platform, which integrates Yi-Ko-DUS-9B's modeling effects, including api services, and provides a free online trial of Yi-Ko-DUS-9B, you can try Yi-Ko-DUS-9B online for free by clicking the link below.

beomi Yi-Ko-DUS-9B online free url in huggingface.co:

https://huggingface.co/beomi/Yi-Ko-DUS-9B

Yi-Ko-DUS-9B install

Yi-Ko-DUS-9B is an open source model from GitHub that offers a free installation service, and any user can find Yi-Ko-DUS-9B on GitHub to install. At the same time, huggingface.co provides the effect of Yi-Ko-DUS-9B install, users can directly use Yi-Ko-DUS-9B installed effect in huggingface.co for debugging and trial. It also supports api for free installation.

Yi-Ko-DUS-9B install url in huggingface.co:

https://huggingface.co/beomi/Yi-Ko-DUS-9B

Url of Yi-Ko-DUS-9B

Yi-Ko-DUS-9B huggingface.co Url

Provider of Yi-Ko-DUS-9B huggingface.co

beomi
ORGANIZATIONS
huggingface.co

Other API from beomi

 huggingface.co
beomi/kcbert-base

Total runs: 4.6K
Run Growth: 877
Growth Rate: 18.92%
Updated:März 30 2023
huggingface.co
beomi/kcbert-large

Total runs: 715
Run Growth: 431
Growth Rate: 60.28%
Updated:März 30 2023
huggingface.co
beomi/Yi-Ko-6B

Total runs: 671
Run Growth: 333
Growth Rate: 49.63%
Updated:Juli 08 2024
huggingface.co
beomi/llama-2-ko-7b

Total runs: 651
Run Growth: 158
Growth Rate: 24.27%
Updated:Dezember 27 2023
huggingface.co
beomi/kcgpt2

Total runs: 281
Run Growth: 258
Growth Rate: 91.81%
Updated:November 23 2021
huggingface.co
beomi/KoRWKV-6B

Total runs: 261
Run Growth: 123
Growth Rate: 47.13%
Updated:Juli 20 2023
huggingface.co
beomi/kollama-13b

Total runs: 250
Run Growth: 155
Growth Rate: 62.00%
Updated:Juni 28 2023
huggingface.co
beomi/gemma-ko-7b

Total runs: 131
Run Growth: 12
Growth Rate: 9.16%
Updated:März 26 2024
huggingface.co
beomi/KoRWKV-1.5B

Total runs: 93
Run Growth: 73
Growth Rate: 78.49%
Updated:November 13 2023
huggingface.co
beomi/gemma-ko-2b

Total runs: 77
Run Growth: -133
Growth Rate: -172.73%
Updated:März 26 2024
huggingface.co
beomi/kollama-7b

Total runs: 70
Run Growth: -22
Growth Rate: -31.43%
Updated:Juli 11 2023
huggingface.co
beomi/kobert

Total runs: 39
Run Growth: 22
Growth Rate: 56.41%
Updated:September 15 2023
huggingface.co
beomi/Yi-Ko-34B

Total runs: 33
Run Growth: 3
Growth Rate: 9.09%
Updated:Juli 08 2024
huggingface.co
beomi/KcT5-dev

Total runs: 11
Run Growth: -20
Growth Rate: -181.82%
Updated:März 01 2022
huggingface.co
beomi/kcgpt2-dev

Total runs: 11
Run Growth: -1
Growth Rate: -9.09%
Updated:Mai 21 2021
huggingface.co
beomi/KcBERT-v2023

Total runs: 10
Run Growth: 1
Growth Rate: 10.00%
Updated:Mai 07 2023
huggingface.co
beomi/kollama-33b

Total runs: 10
Run Growth: 3
Growth Rate: 30.00%
Updated:Juni 29 2023
huggingface.co
beomi/KcRoBERTa-dev

Total runs: 8
Run Growth: 0
Growth Rate: 0.00%
Updated:März 08 2023
huggingface.co
beomi/exKcBERT-paws

Total runs: 7
Run Growth: -2
Growth Rate: -28.57%
Updated:Juni 11 2021
huggingface.co
beomi/llama-2-ko-70b

Total runs: 1
Run Growth: 1
Growth Rate: 100.00%
Updated:November 03 2023
huggingface.co
beomi/KcT5

Total runs: 0
Run Growth: 0
Growth Rate: 0.00%
Updated:Februar 10 2022
huggingface.co
beomi/exKcBERT

Total runs: 0
Run Growth: 0
Growth Rate: 0.00%
Updated:Juni 10 2021