junnyu / structbert-large-zh

huggingface.co
Total runs: 24
24-hour runs: 0
7-day runs: -20
30-day runs: -31
Model's Last Updated: May 18 2022
feature-extraction

Introduction of structbert-large-zh

Model Details of structbert-large-zh

StructBERT: Un-Official Copy

Official Repository Link: https://github.com/alibaba/AliceMind/tree/main/StructBERT

Claimer

Reproduce HFHub models:

Download model/tokenizer vocab

wget https://raw.githubusercontent.com/alibaba/AliceMind/main/StructBERT/config/ch_large_bert_config.json && mv ch_large_bert_config.json config.json
wget https://raw.githubusercontent.com/alibaba/AliceMind/main/StructBERT/config/ch_vocab.txt
wget https://alice-open.oss-cn-zhangjiakou.aliyuncs.com/StructBERT/ch_model && mv ch_model pytorch_model.bin
from transformers import BertConfig, BertModel, BertTokenizer
config = BertConfig.from_pretrained("./config.json")
model = BertModel.from_pretrained("./", config=config)
tokenizer = BertTokenizer.from_pretrained("./")
model.push_to_hub("structbert-large-zh")
tokenizer.push_to_hub("structbert-large-zh")

https://arxiv.org/abs/1908.04577

StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding

Introduction

We extend BERT to a new model, StructBERT, by incorporating language structures into pre-training. Specifically, we pre-train StructBERT with two auxiliary tasks to make the most of the sequential order of words and sentences, which leverage language structures at the word and sentence levels, respectively.

Pre-trained models
Model Description #params Download
structbert.en.large StructBERT using the BERT-large architecture 340M structbert.en.large
structroberta.en.large StructRoBERTa continue training from RoBERTa 355M Coming soon
structbert.ch.large Chinese StructBERT; BERT-large architecture 330M structbert.ch.large
Results

The results of GLUE & CLUE tasks can be reproduced using the hyperparameters listed in the following "Example usage" section.

structbert.en.large

GLUE benchmark

Model MNLI QNLIv2 QQP SST-2 MRPC
structbert.en.large 86.86% 93.04% 91.67% 93.23% 86.51%
structbert.ch.large

CLUE benchmark

Model CMNLI OCNLI TNEWS AFQMC
structbert.ch.large 84.47% 81.28% 68.67% 76.11%
Example usage
Requirements and Installation
  • PyTorch version >= 1.0.1

  • Install other libraries via

pip install -r requirements.txt
  • For faster training install NVIDIA's apex library
Finetune MNLI
python run_classifier_multi_task.py \
  --task_name MNLI \
  --do_train \
  --do_eval \
  --do_test \
  --amp_type O1 \
  --lr_decay_factor 1 \
  --dropout 0.1 \
  --do_lower_case \
  --detach_index -1 \
  --core_encoder bert \
  --data_dir path_to_glue_data \
  --vocab_file config/vocab.txt \
  --bert_config_file config/large_bert_config.json \
  --init_checkpoint path_to_pretrained_model \
  --max_seq_length 128 \
  --train_batch_size 32 \
  --learning_rate 2e-5 \
  --num_train_epochs 3 \
  --fast_train \
  --gradient_accumulation_steps 1 \
  --output_dir path_to_output_dir 
Citation

If you use our work, please cite:

@article{wang2019structbert,
  title={Structbert: Incorporating language structures into pre-training for deep language understanding},
  author={Wang, Wei and Bi, Bin and Yan, Ming and Wu, Chen and Bao, Zuyi and Xia, Jiangnan and Peng, Liwei and Si, Luo},
  journal={arXiv preprint arXiv:1908.04577},
  year={2019}
}

Runs of junnyu structbert-large-zh on huggingface.co

24
Total runs
0
24-hour runs
0
3-day runs
-20
7-day runs
-31
30-day runs

More Information About structbert-large-zh huggingface.co Model

structbert-large-zh huggingface.co

structbert-large-zh huggingface.co is an AI model on huggingface.co that provides structbert-large-zh's model effect (), which can be used instantly with this junnyu structbert-large-zh model. huggingface.co supports a free trial of the structbert-large-zh model, and also provides paid use of the structbert-large-zh. Support call structbert-large-zh model through api, including Node.js, Python, http.

structbert-large-zh huggingface.co Url

https://huggingface.co/junnyu/structbert-large-zh

junnyu structbert-large-zh online free

structbert-large-zh huggingface.co is an online trial and call api platform, which integrates structbert-large-zh's modeling effects, including api services, and provides a free online trial of structbert-large-zh, you can try structbert-large-zh online for free by clicking the link below.

junnyu structbert-large-zh online free url in huggingface.co:

https://huggingface.co/junnyu/structbert-large-zh

structbert-large-zh install

structbert-large-zh is an open source model from GitHub that offers a free installation service, and any user can find structbert-large-zh on GitHub to install. At the same time, huggingface.co provides the effect of structbert-large-zh install, users can directly use structbert-large-zh installed effect in huggingface.co for debugging and trial. It also supports api for free installation.

structbert-large-zh install url in huggingface.co:

https://huggingface.co/junnyu/structbert-large-zh

Url of structbert-large-zh

structbert-large-zh huggingface.co Url

Provider of structbert-large-zh huggingface.co

junnyu
ORGANIZATIONS

Other API from junnyu

huggingface.co

Total runs: 8
Run Growth: 4
Growth Rate: 57.14%
Updated:July 21 2021
huggingface.co

Total runs: 3
Run Growth: 0
Growth Rate: 0.00%
Updated:February 22 2022
huggingface.co

Total runs: 0
Run Growth: 0
Growth Rate: 0.00%
Updated:February 20 2023
huggingface.co

Total runs: 0
Run Growth: 0
Growth Rate: 0.00%
Updated:February 17 2023
huggingface.co

Total runs: 0
Run Growth: 0
Growth Rate: 0.00%
Updated:February 17 2023