Total runs:
2.0M
Run Growth:
-33.5K
Growth Rate:
-1.64%
Sponsored by
APIMart
stabilityai / sd-vae-ft-mse
Model's Last Updated: Junio 06 2023
Introduction of sd-vae-ft-mse
Model Details of sd-vae-ft-mse
Improved Autoencoders
Utilizing
These weights are intended to be used with the 🧨 diffusers library . If you are looking for the model to use with the original CompVis Stable Diffusion codebase , come here .
How to use with 🧨 diffusers
You can integrate this fine-tuned VAE decoder to your existing
diffusers
workflows, by including a
vae
argument to the
StableDiffusionPipeline
from diffusers.models import AutoencoderKL
from diffusers import StableDiffusionPipeline
model = "CompVis/stable-diffusion-v1-4"
vae = AutoencoderKL.from_pretrained("stabilityai/sd-vae-ft-mse")
pipe = StableDiffusionPipeline.from_pretrained(model, vae=vae)
Decoder Finetuning
We publish two kl-f8 autoencoder versions, finetuned from the original kl-f8 autoencoder on a 1:1 ratio of LAION-Aesthetics and LAION-Humans, an unreleased subset containing only SFW images of humans. The intent was to fine-tune on the Stable Diffusion training set (the autoencoder was originally trained on OpenImages) but also enrich the dataset with images of humans to improve the reconstruction of faces. The first, ft-EMA , was resumed from the original checkpoint, trained for 313198 steps and uses EMA weights. It uses the same loss configuration as the original checkpoint (L1 + LPIPS). The second, ft-MSE , was resumed from ft-EMA and uses EMA weights and was trained for another 280k steps using a different loss, with more emphasis on MSE reconstruction (MSE + 0.1 * LPIPS). It produces somewhat ``smoother'' outputs. The batch size for both versions was 192 (16 A100s, batch size 12 per GPU). To keep compatibility with existing models, only the decoder part was finetuned; the checkpoints can be used as a drop-in replacement for the existing autoencoder.
Original kl-f8 VAE vs f8-ft-EMA vs f8-ft-MSE
Evaluation
COCO 2017 (256x256, val, 5000 images)
| Model | train steps | rFID | PSNR | SSIM | PSIM | Link | Comments |
|---|---|---|---|---|---|---|---|
| original | 246803 | 4.99 | 23.4 +/- 3.8 | 0.69 +/- 0.14 | 1.01 +/- 0.28 | https://ommer-lab.com/files/latent-diffusion/kl-f8.zip | as used in SD |
| ft-EMA | 560001 | 4.42 | 23.8 +/- 3.9 | 0.69 +/- 0.13 | 0.96 +/- 0.27 | https://huggingface.co/stabilityai/sd-vae-ft-ema-original/resolve/main/vae-ft-ema-560000-ema-pruned.ckpt | slightly better overall, with EMA |
| ft-MSE | 840001 | 4.70 | 24.5 +/- 3.7 | 0.71 +/- 0.13 | 0.92 +/- 0.27 | https://huggingface.co/stabilityai/sd-vae-ft-mse-original/resolve/main/vae-ft-mse-840000-ema-pruned.ckpt | resumed with EMA from ft-EMA, emphasis on MSE (rec. loss = MSE + 0.1 * LPIPS), smoother outputs |
LAION-Aesthetics 5+ (256x256, subset, 10000 images)
| Model | train steps | rFID | PSNR | SSIM | PSIM | Link | Comments |
|---|---|---|---|---|---|---|---|
| original | 246803 | 2.61 | 26.0 +/- 4.4 | 0.81 +/- 0.12 | 0.75 +/- 0.36 | https://ommer-lab.com/files/latent-diffusion/kl-f8.zip | as used in SD |
| ft-EMA | 560001 | 1.77 | 26.7 +/- 4.8 | 0.82 +/- 0.12 | 0.67 +/- 0.34 | https://huggingface.co/stabilityai/sd-vae-ft-ema-original/resolve/main/vae-ft-ema-560000-ema-pruned.ckpt | slightly better overall, with EMA |
| ft-MSE | 840001 | 1.88 | 27.3 +/- 4.7 | 0.83 +/- 0.11 | 0.65 +/- 0.34 | https://huggingface.co/stabilityai/sd-vae-ft-mse-original/resolve/main/vae-ft-mse-840000-ema-pruned.ckpt | resumed with EMA from ft-EMA, emphasis on MSE (rec. loss = MSE + 0.1 * LPIPS), smoother outputs |
Visual
Visualization of reconstructions on 256x256 images from the COCO2017 validation dataset.
256x256: ft-EMA (left), ft-MSE (middle), original (right)
Runs of stabilityai sd-vae-ft-mse on huggingface.co
118.9K
Total runs
0
24-hour runs
0
3-day runs
587
7-day runs
-10.9K
30-day runs
More Information About sd-vae-ft-mse huggingface.co Model
Url of sd-vae-ft-mse
sd-vae-ft-mse huggingface.co Url
Provider of sd-vae-ft-mse huggingface.co
Other API from stabilityai
Updated:Octubre 31 2023
Total runs:
646.2K
Run Growth:
-143.8K
Growth Rate:
-22.25%
Total runs:
645.3K
Run Growth:
611.2K
Growth Rate:
94.72%
Total runs:
629.9K
Run Growth:
-60.3K
Growth Rate:
-9.57%
Total runs:
408.2K
Run Growth:
65.7K
Growth Rate:
16.08%
Total runs:
322.7K
Run Growth:
107.0K
Growth Rate:
34.04%
Total runs:
290.4K
Run Growth:
95.2K
Growth Rate:
32.78%
Total runs:
236.6K
Run Growth:
118.8K
Growth Rate:
50.23%
Total runs:
232.3K
Run Growth:
-44.9K
Growth Rate:
-19.31%
Total runs:
214.5K
Run Growth:
116.5K
Growth Rate:
54.32%
Total runs:
154.9K
Run Growth:
-13.7K
Growth Rate:
-8.87%
Total runs:
129.7K
Run Growth:
5.9K
Growth Rate:
4.53%
Total runs:
74.4K
Run Growth:
56.5K
Growth Rate:
75.94%
Total runs:
68.6K
Run Growth:
-15.4K
Growth Rate:
-22.44%
Total runs:
49.5K
Run Growth:
-35.5K
Growth Rate:
-70.01%
Total runs:
46.0K
Run Growth:
-99.7K
Growth Rate:
-217.35%
Total runs:
26.5K
Run Growth:
9.0K
Growth Rate:
34.14%
Total runs:
24.7K
Run Growth:
-20.9K
Growth Rate:
-84.57%
Total runs:
24.7K
Run Growth:
-20.2K
Growth Rate:
-81.90%
Total runs:
21.1K
Run Growth:
-7.9K
Growth Rate:
-37.97%
Total runs:
16.8K
Run Growth:
7.9K
Growth Rate:
46.87%
Total runs:
16.6K
Run Growth:
-3.0K
Growth Rate:
-17.87%
Total runs:
11.4K
Run Growth:
229
Growth Rate:
2.01%
Total runs:
10.6K
Run Growth:
376
Growth Rate:
3.54%
Total runs:
10.3K
Run Growth:
2.0K
Growth Rate:
19.16%
Total runs:
9.4K
Run Growth:
-414
Growth Rate:
-4.41%
Total runs:
8.7K
Run Growth:
-1.7K
Growth Rate:
-19.25%
Total runs:
8.1K
Run Growth:
-4.2K
Growth Rate:
-51.52%
Total runs:
8.0K
Run Growth:
2.7K
Growth Rate:
34.25%
Total runs:
5.9K
Run Growth:
1.7K
Growth Rate:
28.10%
Total runs:
4.7K
Run Growth:
-393
Growth Rate:
-8.41%
Total runs:
4.6K
Run Growth:
1.6K
Growth Rate:
31.27%
Total runs:
4.5K
Run Growth:
2.6K
Growth Rate:
54.97%
Total runs:
4.5K
Run Growth:
360
Growth Rate:
8.19%
Total runs:
4.3K
Run Growth:
2.6K
Growth Rate:
60.47%
Total runs:
4.2K
Run Growth:
1.9K
Growth Rate:
45.22%
Total runs:
4.2K
Run Growth:
-773
Growth Rate:
-18.55%
Total runs:
2.9K
Run Growth:
342
Growth Rate:
11.88%
Total runs:
2.3K
Run Growth:
313
Growth Rate:
13.69%
Total runs:
2.2K
Run Growth:
610
Growth Rate:
27.21%
Total runs:
2.1K
Run Growth:
91
Growth Rate:
4.43%
Total runs:
2.0K
Run Growth:
-209
Growth Rate:
-10.46%
Total runs:
1.9K
Run Growth:
514
Growth Rate:
27.28%
Total runs:
1.6K
Run Growth:
504
Growth Rate:
32.54%
Total runs:
1.5K
Run Growth:
66
Growth Rate:
4.31%
Total runs:
1.5K
Run Growth:
-2.6K
Growth Rate:
-184.66%
Total runs:
1.3K
Run Growth:
1.2K
Growth Rate:
92.35%
Total runs:
1.3K
Run Growth:
-134
Growth Rate:
-10.63%
Total runs:
1.1K
Run Growth:
-187
Growth Rate:
-16.62%
Total runs:
1.1K
Run Growth:
-878
Growth Rate:
-83.07%
Total runs:
973
Run Growth:
802
Growth Rate:
84.16%
Total runs:
690
Run Growth:
4
Growth Rate:
0.58%
Total runs:
648
Run Growth:
-1.5K
Growth Rate:
-231.79%
Total runs:
600
Run Growth:
0
Growth Rate:
0.00%
Total runs:
569
Run Growth:
0
Growth Rate:
0.00%
Total runs:
541
Run Growth:
352
Growth Rate:
65.06%
Total runs:
466
Run Growth:
302
Growth Rate:
64.81%
Total runs:
447
Run Growth:
-561
Growth Rate:
-120.65%
Total runs:
426
Run Growth:
-480
Growth Rate:
-107.14%
Total runs:
398
Run Growth:
-455
Growth Rate:
-114.32%
Total runs:
396
Run Growth:
192
Growth Rate:
48.85%
Total runs:
355
Run Growth:
-459
Growth Rate:
-122.40%
Total runs:
322
Run Growth:
197
Growth Rate:
61.18%
Total runs:
320
Run Growth:
-456
Growth Rate:
-133.72%
Total runs:
297
Run Growth:
-101
Growth Rate:
-34.01%
Total runs:
276
Run Growth:
-405
Growth Rate:
-146.74%
Total runs:
194
Run Growth:
-1
Growth Rate:
-0.52%
Total runs:
189
Run Growth:
98
Growth Rate:
51.85%
Total runs:
179
Run Growth:
3
Growth Rate:
1.69%
Total runs:
140
Run Growth:
114
Growth Rate:
82.61%
Total runs:
109
Run Growth:
20
Growth Rate:
18.87%
Total runs:
105
Run Growth:
34
Growth Rate:
32.38%
Total runs:
97
Run Growth:
66
Growth Rate:
70.21%
Total runs:
92
Run Growth:
58
Growth Rate:
65.17%
Total runs:
90
Run Growth:
55
Growth Rate:
62.50%
Total runs:
72
Run Growth:
-7
Growth Rate:
-9.72%
Total runs:
71
Run Growth:
20
Growth Rate:
28.17%
Total runs:
66
Run Growth:
15
Growth Rate:
22.73%
Total runs:
60
Run Growth:
0
Growth Rate:
0.00%
Total runs:
50
Run Growth:
7
Growth Rate:
14.00%
Total runs:
50
Run Growth:
50
Growth Rate:
100.00%
Total runs:
44
Run Growth:
-32
Growth Rate:
-72.73%
Total runs:
44
Run Growth:
-7
Growth Rate:
-15.91%
Total runs:
28
Run Growth:
3
Growth Rate:
10.34%
Total runs:
27
Run Growth:
-16
Growth Rate:
-59.26%
Total runs:
26
Run Growth:
-14
Growth Rate:
-37.84%
Total runs:
19
Run Growth:
-1
Growth Rate:
-5.26%
Total runs:
16
Run Growth:
16
Growth Rate:
100.00%
Total runs:
7
Run Growth:
-2
Growth Rate:
-28.57%
Total runs:
6
Run Growth:
6
Growth Rate:
100.00%
Total runs:
5
Run Growth:
4
Growth Rate:
80.00%
Total runs:
2
Run Growth:
-20
Growth Rate:
-1000.00%