aisingapore / sealion-bert-large

huggingface.co
Total runs: 42
24-hour runs: -3
7-day runs: 3
30-day runs: 22
Model's Last Updated: January 10 2024
fill-mask

Introduction of sealion-bert-large

Model Details of sealion-bert-large

SEA-LION-BERT

SEA-LION stands for Southeast Asian Languages In One Network .

This is the card for the SEA-LION-BERT large model.

How To Use
from transformers import AutoModelForMaskedLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained('aisingapore/sealion-bert-large', trust_remote_code=True)
model = AutoModelForMaskedLM.from_pretrained('aisingapore/sealion-bert-large', trust_remote_code=True)

# prepare input
text = "Give me a <|mask|>!!!"
encoded_input = tokenizer(text, return_tensors='pt')
Model Details
Model Description

The SEA-LION-BERT model is built on the MosaicBERT architecture and has a vocabulary size of 256K.

For tokenization, the model employs our custom SEABPETokenizer, which is specially tailored for SEA languages, ensuring optimal model performance.

The training data for SEA-LION-BERT encompasses 980B tokens.

  • Developed by: Products Pillar, AI Singapore
  • Funded by: Singapore NRF
  • Model type: Encoder
  • Languages: English, Chinese, Indonesian, Malay, Thai, Vietnamese, Filipino, Tamil, Burmese, Khmer, Lao
  • License: MIT License
Training Details
Data

SEA-LION was trained on 790B tokens of the following data:

Data Source Tokens Percentage
RefinedWeb - English 571.3B 72.26%
mC4 - Chinese 91.2B 11.54%
mC4 - Indonesian 14.7B 1.86%
mC4 - Malay 2.9B 0.36%
mC4 - Filipino 5.3B 0.67%
mC4 - Burmese 4.9B 0.61%
mC4 - Vietnamese 63.4B 8.02%
mC4 - Thai 21.6B 2.74%
mC4 - Lao 1.1B 0.14%
mC4 - Khmer 3.9B 0.50%
mC4 - Tamil 10.2B 1.29%
Infrastructure

SEA-LION was trained using MosaicML Composer on the following hardware:

Training Details SEA-LION-BERT
Nvidia A100 40GB GPU 4
Training Duration 22 days
Configuration
HyperParameter SEA-LION-BERT
Precision bfloat16
Optimizer decoupled_adamw
Scheduler linear_decay_with_warmup
Learning Rate 1e-4
Global Batch Size 720
Micro Batch Size 36
Technical Specifications
Model Architecture and Objective

SEA-LION-BERT is an encoder model using the MosaicBERT architecture.

Parameter SEA-LION-BERT
Layers 24
d_model 1024
head_dim 16
Vocabulary 256000
Sequence Length 128
Tokenizer Details

We sample 20M lines from the training data to train the tokenizer.
The framework for training is SentencePiece .
The tokenizer type is Byte-Pair Encoding (BPE).

The Team

Montalan Jann Railey
Nguyen Thanh Ngan
Rengarajan Hamsawardhini
Teo Leslie
Tjhi William

Acknowledgements

AI Singapore is a national programme supported by the National Research Foundation, Singapore and hosted by the National University of Singapore. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore.

Contact

For more info, please contact us at [email protected]

Runs of aisingapore sealion-bert-large on huggingface.co

42
Total runs
-3
24-hour runs
-1
3-day runs
3
7-day runs
22
30-day runs

More Information About sealion-bert-large huggingface.co Model

More sealion-bert-large license Visit here:

https://choosealicense.com/licenses/mit

sealion-bert-large huggingface.co

sealion-bert-large huggingface.co is an AI model on huggingface.co that provides sealion-bert-large's model effect (), which can be used instantly with this aisingapore sealion-bert-large model. huggingface.co supports a free trial of the sealion-bert-large model, and also provides paid use of the sealion-bert-large. Support call sealion-bert-large model through api, including Node.js, Python, http.

sealion-bert-large huggingface.co Url

https://huggingface.co/aisingapore/sealion-bert-large

aisingapore sealion-bert-large online free

sealion-bert-large huggingface.co is an online trial and call api platform, which integrates sealion-bert-large's modeling effects, including api services, and provides a free online trial of sealion-bert-large, you can try sealion-bert-large online for free by clicking the link below.

aisingapore sealion-bert-large online free url in huggingface.co:

https://huggingface.co/aisingapore/sealion-bert-large

sealion-bert-large install

sealion-bert-large is an open source model from GitHub that offers a free installation service, and any user can find sealion-bert-large on GitHub to install. At the same time, huggingface.co provides the effect of sealion-bert-large install, users can directly use sealion-bert-large installed effect in huggingface.co for debugging and trial. It also supports api for free installation.

sealion-bert-large install url in huggingface.co:

https://huggingface.co/aisingapore/sealion-bert-large

Url of sealion-bert-large

sealion-bert-large huggingface.co Url

Provider of sealion-bert-large huggingface.co

aisingapore
ORGANIZATIONS

Other API from aisingapore

huggingface.co

Total runs: 11
Run Growth: 1
Growth Rate: 9.09%
Updated:March 02 2023