This model is a fine-tuned version of
microsoft/deberta-v3-small
on the GLUE COLA dataset.
It achieves the following results on the evaluation set:
Loss: 0.4051
Matthews Correlation: 0.6333
Model description
DeBERTa
improves the BERT and RoBERTa models using disentangled attention and enhanced mask decoder. With those two improvements, DeBERTa out perform RoBERTa on a majority of NLU tasks with 80GB training data.
In
DeBERTa V3
, we replaced the MLM objective with the RTD(Replaced Token Detection) objective introduced by ELECTRA for pre-training, as well as some innovations to be introduced in our upcoming paper. Compared to DeBERTa-V2, our V3 version significantly improves the model performance in downstream tasks. You can find a simple introduction about the model from the appendix A11 in our original
paper
, but we will provide more details in a separate write-up.
The DeBERTa V3 small model comes with 6 layers and a hidden size of 768. Its total parameter number is 143M since we use a vocabulary containing 128K tokens which introduce 98M parameters in the Embedding layer. This model was trained using the 160GB data as DeBERTa V2.
Intended uses & limitations
More information needed
Training and evaluation data
The Corpus of Linguistic Acceptability (CoLA) in its full form consists of 10657 sentences from 23 linguistics publications, expertly annotated for acceptability (grammaticality) by their original authors. The public version provided here contains 9594 sentences belonging to training and development sets, and excludes 1063 sentences belonging to a held out test set.
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
learning_rate: 3e-05
train_batch_size: 16
eval_batch_size: 8
seed: 42
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: linear
num_epochs: 5.0
Training results
Training Loss
Epoch
Step
Validation Loss
Matthews Correlation
No log
1.0
535
0.4051
0.6333
0.3371
2.0
1070
0.4455
0.6531
0.3371
3.0
1605
0.5755
0.6499
0.1305
4.0
2140
0.7188
0.6553
0.1305
5.0
2675
0.8047
0.6700
Framework versions
Transformers 4.13.0.dev0
Pytorch 1.10.0+cu111
Datasets 1.16.1
Tokenizers 0.10.3
Runs of mrm8488 deberta-v3-small-finetuned-cola on huggingface.co
160
Total runs
0
24-hour runs
-1
3-day runs
62
7-day runs
128
30-day runs
More Information About deberta-v3-small-finetuned-cola huggingface.co Model
More deberta-v3-small-finetuned-cola license Visit here:
deberta-v3-small-finetuned-cola huggingface.co is an AI model on huggingface.co that provides deberta-v3-small-finetuned-cola's model effect (), which can be used instantly with this mrm8488 deberta-v3-small-finetuned-cola model. huggingface.co supports a free trial of the deberta-v3-small-finetuned-cola model, and also provides paid use of the deberta-v3-small-finetuned-cola. Support call deberta-v3-small-finetuned-cola model through api, including Node.js, Python, http.
deberta-v3-small-finetuned-cola huggingface.co is an online trial and call api platform, which integrates deberta-v3-small-finetuned-cola's modeling effects, including api services, and provides a free online trial of deberta-v3-small-finetuned-cola, you can try deberta-v3-small-finetuned-cola online for free by clicking the link below.
mrm8488 deberta-v3-small-finetuned-cola online free url in huggingface.co:
deberta-v3-small-finetuned-cola is an open source model from GitHub that offers a free installation service, and any user can find deberta-v3-small-finetuned-cola on GitHub to install. At the same time, huggingface.co provides the effect of deberta-v3-small-finetuned-cola install, users can directly use deberta-v3-small-finetuned-cola installed effect in huggingface.co for debugging and trial. It also supports api for free installation.
deberta-v3-small-finetuned-cola install url in huggingface.co: