Total runs:
627
Run Growth:
107
Growth Rate:
18.23%
Sponsored by
APIMart
BEE-spoke-data / bert-plus-L8-4096-v1.0
Model's Last Updated: December 29 2025
fill-mask
Introduction of bert-plus-L8-4096-v1.0
Model Details of bert-plus-L8-4096-v1.0
BEE-spoke-data/bert-plus-L8-4096-v1.0
still running some evals, etc. expect the model card to change a bit
* No additional code. This model uses
position_embedding_type="relative_key"
to help with long ctx.
this checkpoint
Further progression after multitask training etc. The most recent/last dataset it saw was the euirim/goodwiki dataset.
It achieves the following results on the evaluation set:
- Loss: 1.9835
- Accuracy: 0.6159
GLUE benchmark
WIP till this text is removed
Thus far, all completed in fp32 ( using nvidia tf32 dtype behind the scenes when supported )
| Model | Size | Avg | CoLA | SST2 | MRPC | STSB | QQP | MNLI | QNLI | RTE |
|---|---|---|---|---|---|---|---|---|---|---|
| bert-plus-L8-4096-v1.0 | 88.1M | 82.78 | 62.72 | 90.6 | 86.59 | 92.07 | 90.6 | 83.2 | 90.0 | 66.43 |
| bert_uncased_L-8_H-768_A-12 | 81.2M | 81.65 | 54.0 | 92.6 | 85.43 | 92.60 | 90.6 | 81.0 | 90.0 | 67.0 |
| bert-base-uncased | 110M | 79.05 | 52.1 | 93.5 | 88.9 | 85.8 | 71.2 | 84.0 | 90.5 | 66.4 |
and some comparisons to recent BERT models taken from nomic's blog post :
| Model | Size | Avg | CoLA | SST2 | MRPC | STSB | QQP | MNLI | QNLI | RTE |
|---|---|---|---|---|---|---|---|---|---|---|
| NomicBERT | 137M | 84.00 | 50.00 | 93.00 | 88.00 | 90.00 | 92.00 | 86.00 | 92.00 | 82.00 |
| RobertaBase | 125M | 86.00 | 64.00 | 95.00 | 90.00 | 91.00 | 92.00 | 88.00 | 93.00 | 79.00 |
| JinaBERTBase | 137M | 83.00 | 51.00 | 95.00 | 88.00 | 90.00 | 81.00 | 86.00 | 92.00 | 79.00 |
| MosaicBERT | 137M | 85.00 | 59.00 | 94.00 | 89.00 | 90.00 | 92.00 | 86.00 | 91.00 | 83.00 |
Observations:
-
Performance Variation Across Models and Tasks : The data highlights significant performance variability both across and within models for different GLUE tasks. This variability underscores the complexity of natural language understanding tasks and the need for models to be versatile in handling different types of linguistic challenges.
-
Model Size and Efficiency : Despite the differences in model size, there is not always a direct correlation between size and performance across tasks. For instance,
bert_uncased_L-8_H-768_A-12performs competitively with larger models in certain tasks, suggesting that efficiency in model architecture and training can compensate for smaller model sizes. -
Task-specific Challenges : Certain tasks, such as RTE, present considerable challenges to all models, indicating the difficulty of tasks that require deep understanding and reasoning over language. This suggests areas where further research and model innovation are needed to improve performance.
-
Overall Model Performance : Models like
roberta-baseshow strong performance across a broad spectrum of tasks, indicating the effectiveness of its architecture and pre-training methodology. Meanwhile, models such asBEE-spoke-data/bert-plus-L8-4096-v1.0showcase the potential for achieving competitive performance with relatively smaller sizes, emphasizing the importance of model design and optimization.
Training procedure
The below is auto-generated and just applies to the 'finishing touches' run on
goodwiki
.
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.0001
- train_batch_size: 4
- eval_batch_size: 4
- seed: 31010
- gradient_accumulation_steps: 16
- total_train_batch_size: 64
- optimizer: Adam with betas=(0.9,0.98) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 100
- num_epochs: 1.0
Training results
| Training Loss | Epoch | Step | Validation Loss | Accuracy |
|---|---|---|---|---|
| 2.1283 | 0.25 | 150 | 2.0892 | 0.6018 |
| 2.0999 | 0.5 | 300 | 2.0387 | 0.6084 |
| 2.0595 | 0.75 | 450 | 1.9971 | 0.6143 |
| 2.0481 | 1.0 | 600 | 1.9893 | 0.6152 |
Framework versions
- Transformers 4.37.2
- Pytorch 2.3.0.dev20240206+cu121
- Datasets 2.16.1
- Tokenizers 0.15.1
Runs of BEE-spoke-data bert-plus-L8-4096-v1.0 on huggingface.co
26
Total runs
-1
24-hour runs
0
3-day runs
-3
7-day runs
14
30-day runs
More Information About bert-plus-L8-4096-v1.0 huggingface.co Model
Url of bert-plus-L8-4096-v1.0
bert-plus-L8-4096-v1.0 huggingface.co Url
Provider of bert-plus-L8-4096-v1.0 huggingface.co
Other API from BEE-spoke-data
Updated:December 29 2025
Total runs:
604
Run Growth:
47
Growth Rate:
7.72%
Total runs:
344
Run Growth:
84
Growth Rate:
24.28%
Total runs:
207
Run Growth:
26
Growth Rate:
12.21%
Total runs:
191
Run Growth:
41
Growth Rate:
20.92%
Total runs:
115
Run Growth:
-114
Growth Rate:
-92.68%
Total runs:
102
Run Growth:
12
Growth Rate:
11.65%
Total runs:
97
Run Growth:
-59
Growth Rate:
-59.60%
Total runs:
95
Run Growth:
7
Growth Rate:
7.22%
Total runs:
87
Run Growth:
-43
Growth Rate:
-48.31%
Total runs:
81
Run Growth:
14
Growth Rate:
16.87%
Total runs:
71
Run Growth:
1
Growth Rate:
1.37%
Total runs:
63
Run Growth:
36
Growth Rate:
54.55%
Total runs:
54
Run Growth:
26
Growth Rate:
47.27%
Total runs:
54
Run Growth:
27
Growth Rate:
47.37%
Total runs:
45
Run Growth:
15
Growth Rate:
32.61%
Total runs:
42
Run Growth:
12
Growth Rate:
27.27%
Total runs:
42
Run Growth:
22
Growth Rate:
51.16%
Total runs:
37
Run Growth:
19
Growth Rate:
47.50%
Total runs:
36
Run Growth:
18
Growth Rate:
48.65%
Total runs:
34
Run Growth:
3
Growth Rate:
8.82%
Total runs:
33
Run Growth:
0
Growth Rate:
0.00%
Total runs:
30
Run Growth:
17
Growth Rate:
54.84%
Total runs:
30
Run Growth:
8
Growth Rate:
24.24%
Total runs:
29
Run Growth:
23
Growth Rate:
74.19%
Total runs:
29
Run Growth:
10
Growth Rate:
33.33%
Total runs:
29
Run Growth:
14
Growth Rate:
48.28%
Total runs:
25
Run Growth:
6
Growth Rate:
22.22%
Total runs:
25
Run Growth:
12
Growth Rate:
46.15%
Total runs:
25
Run Growth:
14
Growth Rate:
53.85%
Total runs:
24
Run Growth:
-5
Growth Rate:
-20.00%
Total runs:
24
Run Growth:
6
Growth Rate:
24.00%
Total runs:
24
Run Growth:
22
Growth Rate:
88.00%
Total runs:
24
Run Growth:
9
Growth Rate:
39.13%
Total runs:
24
Run Growth:
10
Growth Rate:
40.00%
Total runs:
24
Run Growth:
17
Growth Rate:
68.00%
Total runs:
24
Run Growth:
12
Growth Rate:
48.00%
Total runs:
22
Run Growth:
13
Growth Rate:
59.09%
Total runs:
20
Run Growth:
-4
Growth Rate:
-20.00%
Total runs:
17
Run Growth:
3
Growth Rate:
17.65%
Total runs:
16
Run Growth:
0
Growth Rate:
0.00%
Total runs:
15
Run Growth:
0
Growth Rate:
0.00%
Total runs:
15
Run Growth:
-1
Growth Rate:
-6.25%
Total runs:
11
Run Growth:
3
Growth Rate:
27.27%
Total runs:
9
Run Growth:
1
Growth Rate:
11.11%
Total runs:
9
Run Growth:
3
Growth Rate:
30.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%