Total runs:
109
Run Growth:
92
Growth Rate:
84.40%
Sponsored by
iMean AI Trip Planner
keras-io / deit
Model's Last Updated: July 05 2024
Introduction of deit
Model Details of deit
Model description
This model is implementation of the distillation recipe proposed in DeiT.
Visit Keras example on
Distilling Vision Transformers
.
Full credits to: Sayak Paul
In the original Vision Transformers (ViT) paper (Dosovitskiy et al.), the authors concluded that to perform on par with Convolutional Neural Networks (CNNs), ViTs need to be pre-trained on larger datasets. The larger the better. This is mainly due to the lack of inductive biases in the ViT architecture -- unlike CNNs, they don't have layers that exploit locality.
Many groups have proposed different ways to deal with the problem of data-intensiveness of ViT training. One such way was shown in the Data-efficient image Transformers, (DeiT) paper (Touvron et al.). The authors introduced a distillation technique that is specific to transformer-based vision models. DeiT is among the first works to show that it's possible to train ViTs well without using larger datasets.
Intended uses & limitations
The model is trained for demonstrative purposes and does not guarantee the best results in production.
For better results, follow & optimize the
Keras example
as per your need.
Training and evaluation data
The model is trained and evaluated on TF Flowers dataset
Training procedure
Training procedure is followed exactly as from the
keras example
.
The batch size is however decreased to 16 from the original 256 for accomodating the model in a single V100 GPU memory.
Training hyperparameters
The following hyperparameters were used during training:
| name | learning_rate | decay | beta_1 | beta_2 | epsilon | amsgrad | weight_decay | exclude_from_weight_decay | training_precision |
|---|---|---|---|---|---|---|---|---|---|
| AdamW | 6.25000029685907e-05 | 0.0 | 0.8999999761581421 | 0.9990000128746033 | 1e-07 | False | 9.999999747378752e-05 | None | float32 |
Model Plot
View Model Plot
Runs of keras-io deit on huggingface.co
10
Total runs
2
24-hour runs
3
3-day runs
5
7-day runs
-1
30-day runs
More Information About deit huggingface.co Model
Url of deit
deit huggingface.co Url
Provider of deit huggingface.co
Other API from keras-io
Updated:February 22 2022
Total runs:
105
Run Growth:
-49
Growth Rate:
-46.67%
Total runs:
72
Run Growth:
40
Growth Rate:
55.56%
Total runs:
53
Run Growth:
18
Growth Rate:
33.96%
Total runs:
44
Run Growth:
22
Growth Rate:
52.38%
Total runs:
44
Run Growth:
-40
Growth Rate:
-90.91%
Total runs:
42
Run Growth:
10
Growth Rate:
23.81%
Total runs:
41
Run Growth:
8
Growth Rate:
21.62%
Total runs:
37
Run Growth:
20
Growth Rate:
52.63%
Total runs:
35
Run Growth:
9
Growth Rate:
25.71%
Total runs:
35
Run Growth:
-6
Growth Rate:
-17.14%
Total runs:
35
Run Growth:
18
Growth Rate:
51.43%
Total runs:
33
Run Growth:
16
Growth Rate:
48.48%
Total runs:
32
Run Growth:
-27
Growth Rate:
-90.00%
Total runs:
31
Run Growth:
-12
Growth Rate:
-38.71%
Total runs:
31
Run Growth:
-22
Growth Rate:
-70.97%
Total runs:
31
Run Growth:
-36
Growth Rate:
-116.13%
Total runs:
30
Run Growth:
-19
Growth Rate:
-63.33%
Total runs:
29
Run Growth:
-36
Growth Rate:
-124.14%
Total runs:
29
Run Growth:
6
Growth Rate:
20.69%
Total runs:
27
Run Growth:
-22
Growth Rate:
-81.48%
Total runs:
25
Run Growth:
-12
Growth Rate:
-48.00%
Total runs:
24
Run Growth:
-10
Growth Rate:
-40.00%
Total runs:
21
Run Growth:
-7
Growth Rate:
-31.82%
Total runs:
21
Run Growth:
2
Growth Rate:
10.00%
Total runs:
21
Run Growth:
-4
Growth Rate:
-18.18%
Total runs:
20
Run Growth:
-2
Growth Rate:
-10.00%
Total runs:
20
Run Growth:
17
Growth Rate:
89.47%
Total runs:
19
Run Growth:
4
Growth Rate:
21.05%
Total runs:
19
Run Growth:
-7
Growth Rate:
-36.84%
Total runs:
19
Run Growth:
0
Growth Rate:
0.00%
Total runs:
18
Run Growth:
-6
Growth Rate:
-37.50%
Total runs:
18
Run Growth:
6
Growth Rate:
35.29%
Total runs:
16
Run Growth:
0
Growth Rate:
0.00%
Total runs:
15
Run Growth:
-2
Growth Rate:
-14.29%
Total runs:
15
Run Growth:
-7
Growth Rate:
-58.33%
Total runs:
15
Run Growth:
8
Growth Rate:
53.33%
Total runs:
15
Run Growth:
10
Growth Rate:
66.67%
Total runs:
13
Run Growth:
-12
Growth Rate:
-92.31%
Total runs:
12
Run Growth:
-14
Growth Rate:
-116.67%
Total runs:
12
Run Growth:
-13
Growth Rate:
-108.33%
Total runs:
12
Run Growth:
-17
Growth Rate:
-141.67%
Total runs:
12
Run Growth:
-2
Growth Rate:
-16.67%
Total runs:
11
Run Growth:
-16
Growth Rate:
-145.45%
Total runs:
11
Run Growth:
6
Growth Rate:
54.55%
Total runs:
11
Run Growth:
-3
Growth Rate:
-21.43%
Total runs:
11
Run Growth:
-2
Growth Rate:
-20.00%
Total runs:
10
Run Growth:
-5
Growth Rate:
-55.56%
Total runs:
10
Run Growth:
-17
Growth Rate:
-170.00%
Total runs:
7
Run Growth:
3
Growth Rate:
42.86%
Total runs:
7
Run Growth:
-6
Growth Rate:
-100.00%
Total runs:
5
Run Growth:
-5
Growth Rate:
-100.00%
Total runs:
4
Run Growth:
3
Growth Rate:
75.00%
Total runs:
4
Run Growth:
-1
Growth Rate:
-25.00%
Total runs:
3
Run Growth:
-16
Growth Rate:
-533.33%
Total runs:
3
Run Growth:
1
Growth Rate:
33.33%
Total runs:
0
Run Growth:
0
Growth Rate:
0.00%