beomi / OPEN-SOLAR-KO-10.7B

Update Log

• 2024.01.08: Initial Test version Release of Solar-Ko

Open-Solar-Ko ⭐🇰🇷

Solar-Ko represents an advanced iteration of the upstage/SOLAR-10.7B-v1.0 model, featuring an expanded vocabulary and the inclusion of a Korean corpus for enhanced pretraining.

Open-Solar-Ko exclusively utilizes publicly accessible Korean corpora, including sources such as AI Hub , Modu Corpus, 모두의 말뭉치 , and Korean Wikipedia .

As training was conducted solely with publicly available corpora, this model is open for unrestricted use by everyone, adhering to the Apache2.0 open source License.

Model Details

Model Developers: Junbum Lee (Beomi)

Variations: Solar-Ko is available with one parameter sizes — 10B with Continual Pretrained version.

Input: The model accepts only text input.

Output: The model produces text output exclusively.

Model Architecture:

SOLAR-KO-10.7B is an auto-regressive language model that leverages an optimized transformer architecture derived from Llama-2.

Training Corpus

The model was trained using selected datasets from AIHub and Modu Corpus. Detailed information about the training datasets is available below:

• AI Hub: corpus/AI_HUB
  • Only the Training segment of the data was used.
  • The Validation and Test segments were deliberately excluded.
• Modu Corpus: corpus/MODU_CORPUS

The final JSONL dataset used to train this model is approximately 61GB in size.

Total token count: Approximately 15 billion tokens (*using the expanded tokenizer. With the original SOLAR tokenizer, >60 billion tokens.)

Vocab Expansion

Tokenizing "안녕하세요, 오늘은 날씨가 좋네요."

• SOLAR-10.7B: 26 tokens
• SOLAR-KO-10.7b: 8 tokens

Tokenizing "Meet 10.7B Solar: Elevating Performance with Upstage Depth UP Scaling!"

• SOLAR-10.7B: 22 tokens
• SOLAR-KO-10.7b: 22 tokens

LICENSE

Apache 2.0

Model Benchmark

LM Eval Harness - Korean (polyglot branch)

• Used EleutherAI's lm-evaluation-harness https://github.com/EleutherAI/lm-evaluation-harness/tree/polyglot

Citation

@misc {solar_ko_junbum_2023,
    author       = { {L. Junbum} },
    title        = { Solar-Ko-10.7b },
    year         = 2024,
    url          = { https://huggingface.co/beomi/SOLAR-KO-10.7B },
    publisher    = { Hugging Face }
}

Acknowledgements

• Training support was provided by the TPU Research Cloud program.
• The training corpus includes data from AI Hub , Modu Corpus , and Korean Wikipedia .