Anzhc / MS-LC-EQ-D-VR_VAE

MS-LC-EQ-D-VR VAE: another reproduction of EQ-VAE on SDXL-VAE and then some

EQ-VAE paper: https://arxiv.org/abs/2502.09509
VIVAT paper: https://arxiv.org/pdf/2506.07863v1
Thanks to Kohaku and his reproduction that made me look into this: https://huggingface.co/KBlueLeaf/EQ-SDXL-VAE

Top: reconstructed by VAE image. Bottom: Latent to PCA
Upper one is original VAE, bottome one is EQ-VAE finetuned VAE.

Introduction

Refer to https://huggingface.co/KBlueLeaf/EQ-SDXL-VAE for introduction to EQ-VAE.

This implementation additionally utilizes some of fixes proposed in VIVAT paper, and custom in-house regularization techniques, as well as training implementation.

For additional examples and more information refer to: https://arcenciel.io/articles/20 and https://arcenciel.io/models/10994

Visual Examples

Usage

This is a finetuned SDXL VAE, adapted with new regularization, and other techniques. You can use this with your existing SDXL model, but image will be quite artefacting, particularly - oversharpening and ringing.

This VAE is supposed ot be used for finetune, after that images will become normal. But be aware, compatibility with old VAEs, that are not EQ, will be lost(They will become blurry).

Training Setup

• Base Model: SDXL-VAE
• Dataset: ~12.8k anime images
• Batch Size: 128 (bs 8, grad acc 16)
• Samples Seen: ~75k
• Loss Weights:

  • L1: 0.3

  • L2: 0.5

  • SSIM: 0.5

  • LPIPS: 0.5

  • KL: 0.000001

  • Consistency Loss: 0.75

  • Both Encoder and Decoder were trained.

Training Time : ~8-10 hours on 4060Ti

Evaluation Results

Im using small test set i have on me, separated into anime(434) and photo(500) images. Additionally, im measuring noise in latents. Sorgy for no larger test sets.

Results on small benchmark of 500 photos

Noise in latents

Results on a small benchmark of 434 anime arts

Noise in latents

KL loss suggests that this VAE implementation is much closer to SDXL, and likely will be a better candidate for further finetune, but that is just a theory.

References

[1] [2502.09509] EQ-VAE: Equivariance Regularized Latent Space for Improved Generative Image Modeling

[2] [2506.07863] VIVAT: VIRTUOUS IMPROVING VAE TRAINING THROUGH ARTIFACT MITIGATION

[3] sdxl-vae

Cite

@misc{anzhc_ms-lc-eq-d-vr_vae,
    author       = {Anzhc},
    title        = {MS-LC-EQ-D-VR VAE: another reproduction of EQ-VAE on SDXL-VAE and then some},
    year         = {2025},
    howpublished = {Hugging Face model card},
    url          = {https://huggingface.co/Anzhc/MS-LC-EQ-D-VR_VAE},
    note         = {Finetuned SDXL-VAE with EQ regularization and more, for improved latent representation.}
}

Acknowledgement

My friend Bluvoll, for no particular reason.