AbstractPhil / vae-lyra-sdxl-t5xl

VAE Lyra 🎵 - SDXL Edition

Multi-modal Variational Autoencoder for SDXL text embedding transformation using geometric fusion. Fuses CLIP-L, CLIP-G, and T5-XXL into a unified latent space.

Model Details

• Fusion Strategy : cantor
• Latent Dimension : 2048
• Training Steps : 7,814
• Best Loss : 0.4018

Architecture

• Modalities :
  • CLIP-L (768d) - SDXL text_encoder
  • CLIP-G (1280d) - SDXL text_encoder_2
  • T5-XXL (2048d) - Additional conditioning
• Encoder Layers : 3
• Decoder Layers : 3
• Hidden Dimension : 1024

SDXL Compatibility

This model outputs both CLIP embeddings needed for SDXL:

• clip_l : [batch, 77, 768] → text_encoder output
• clip_g : [batch, 77, 1280] → text_encoder_2 output

T5-XXL information is encoded into the latent space but not directly output.

Usage

from geovocab2.train.model.vae.vae_lyra import MultiModalVAE, MultiModalVAEConfig
from huggingface_hub import hf_hub_download
import torch

# Download model
model_path = hf_hub_download(
    repo_id="AbstractPhil/vae-lyra-sdxl-t5xl",
    filename="model.pt"
)

# Load checkpoint
checkpoint = torch.load(model_path)

# Create model
config = MultiModalVAEConfig(
    modality_dims={"clip_l": 768, "clip_g": 1280, "t5_xl": 2048},
    latent_dim=2048,
    fusion_strategy="cantor"
)

model = MultiModalVAE(config)
model.load_state_dict(checkpoint['model_state_dict'])
model.eval()

# Use model - train on all three
inputs = {
    "clip_l": clip_l_embeddings,   # [batch, 77, 768]
    "clip_g": clip_g_embeddings,   # [batch, 77, 1280]
    "t5_xl": t5_xl_embeddings      # [batch, 77, 2048]
}

# For SDXL inference - only decode CLIP outputs
recons, mu, logvar = model(inputs, target_modalities=["clip_l", "clip_g"])

# Use recons["clip_l"] and recons["clip_g"] with SDXL

Training Details

• Trained on 50,000 diverse prompts
• Mix of LAION flavors (95%) and synthetic prompts (5%)
• KL Annealing: True
• Learning Rate: 0.0001

Citation

@software{vae_lyra_sdxl_2025,
  author = {AbstractPhil},
  title = {VAE Lyra SDXL: Multi-Modal Variational Autoencoder},
  year = {2025},
  url = {https://huggingface.co/AbstractPhil/vae-lyra-sdxl-t5xl}
}