Introduction of mega-ar-small-4096-wikitext-103-raw-v1
Model Details of mega-ar-small-4096-wikitext-103-raw-v1
mega-ar-small-4096: MWE
mega-ar on wikitext-103-raw-v1 (document level)
This model is a fine-tuned version of
pszemraj/random-mega-ar-small-4096
on the
EleutherAI/wikitext_document_level
dataset (
wikitext-103-raw-v1
). This model has ~ 65M params.
It achieves the following results on the evaluation set:
Loss: 4.0338
Accuracy: 0.3243
Training procedure
This was tuned with
bf16
, while the authors recommend tuning with
fp32
. Will try
fp32
later.
Training hyperparameters
The following hyperparameters were used during training:
learning_rate: 0.0005
train_batch_size: 4
eval_batch_size: 1
seed: 80085
gradient_accumulation_steps: 32
total_train_batch_size: 128
optimizer: Adam with betas=(0.9,0.98) and epsilon=1e-07
lr_scheduler_type: linear
lr_scheduler_warmup_ratio: 0.05
num_epochs: 3.0
Training results
Training Loss
Epoch
Step
Validation Loss
Accuracy
7.3662
0.11
100
7.2782
0.0935
6.3064
0.22
200
6.2066
0.1634
5.8203
0.33
300
5.7299
0.1931
5.55
0.44
400
5.4173
0.2117
5.3194
0.55
500
5.1937
0.2278
5.1678
0.66
600
5.0206
0.2406
5.0375
0.77
700
4.8891
0.2508
4.9194
0.88
800
4.7592
0.2605
4.8272
0.99
900
4.6653
0.2681
4.7571
1.1
1000
4.5817
0.2754
4.6345
1.21
1100
4.5066
0.2820
4.6218
1.32
1200
4.4472
0.2867
4.5585
1.43
1300
4.3827
0.2923
4.5047
1.54
1400
4.3328
0.2963
4.4726
1.65
1500
4.2860
0.3002
4.4094
1.76
1600
4.2452
0.3038
4.3705
1.87
1700
4.2168
0.3062
4.3739
1.98
1800
4.1852
0.3095
4.2836
2.09
1900
4.1599
0.3112
4.302
2.2
2000
4.1307
0.3149
4.2847
2.31
2100
4.1113
0.3165
4.2348
2.42
2200
4.0925
0.3184
4.2837
2.53
2300
4.0743
0.3207
4.2058
2.64
2400
4.0612
0.3217
4.22
2.75
2500
4.0494
0.3224
4.1827
2.86
2600
4.0397
0.3237
4.1967
2.97
2700
4.0338
0.3243
Framework versions
Transformers 4.32.1
Pytorch 2.1.0.dev20230727+cu118
Datasets 2.13.1
Tokenizers 0.13.3
Runs of pszemraj mega-ar-small-4096-wikitext-103-raw-v1 on huggingface.co
14
Total runs
0
24-hour runs
0
3-day runs
3
7-day runs
6
30-day runs
More Information About mega-ar-small-4096-wikitext-103-raw-v1 huggingface.co Model
More mega-ar-small-4096-wikitext-103-raw-v1 license Visit here:
mega-ar-small-4096-wikitext-103-raw-v1 huggingface.co is an AI model on huggingface.co that provides mega-ar-small-4096-wikitext-103-raw-v1's model effect (), which can be used instantly with this pszemraj mega-ar-small-4096-wikitext-103-raw-v1 model. huggingface.co supports a free trial of the mega-ar-small-4096-wikitext-103-raw-v1 model, and also provides paid use of the mega-ar-small-4096-wikitext-103-raw-v1. Support call mega-ar-small-4096-wikitext-103-raw-v1 model through api, including Node.js, Python, http.
mega-ar-small-4096-wikitext-103-raw-v1 huggingface.co is an online trial and call api platform, which integrates mega-ar-small-4096-wikitext-103-raw-v1's modeling effects, including api services, and provides a free online trial of mega-ar-small-4096-wikitext-103-raw-v1, you can try mega-ar-small-4096-wikitext-103-raw-v1 online for free by clicking the link below.
pszemraj mega-ar-small-4096-wikitext-103-raw-v1 online free url in huggingface.co:
mega-ar-small-4096-wikitext-103-raw-v1 is an open source model from GitHub that offers a free installation service, and any user can find mega-ar-small-4096-wikitext-103-raw-v1 on GitHub to install. At the same time, huggingface.co provides the effect of mega-ar-small-4096-wikitext-103-raw-v1 install, users can directly use mega-ar-small-4096-wikitext-103-raw-v1 installed effect in huggingface.co for debugging and trial. It also supports api for free installation.
mega-ar-small-4096-wikitext-103-raw-v1 install url in huggingface.co: