wangkanai / wan22-vae

---

license: other license_name: wan-license library_name: diffusers pipeline_tag: text-to-video tags: - video-generation - vae - wan - autoencoder - latent-space - video-compression base_model: wan-model/wan base_model_relation: component

WAN22 VAE - Video Autoencoder v1.0

High-performance Variational Autoencoder (VAE) component for the WAN (World Anything Now) video generation system. This VAE provides efficient latent space encoding and decoding for video content, enabling high-quality video generation with reduced computational requirements.

Model Description

The WAN22-VAE is a specialized variational autoencoder designed for video content processing in the WAN video generation pipeline. It compresses video frames into a compact latent representation and reconstructs them with high fidelity, enabling efficient text-to-video and image-to-video generation workflows.

Key Capabilities

• Video Compression : Efficient encoding of video frames into latent space representations
• High Fidelity Reconstruction : Accurate decoding back to pixel space with minimal quality loss
• Temporal Coherence : Maintains consistency across video frames during encoding/decoding
• Memory Efficient : Reduces VRAM requirements during video generation inference
• Compatible Pipeline Integration : Seamlessly integrates with WAN video generation models

Technical Highlights

• Optimized architecture for temporal video data processing
• Supports various frame rates and resolutions
• Low latency encoding/decoding for real-time applications
• Precision-optimized for stable inference on consumer hardware

Repository Contents

wan22-vae/
└── vae/
    └── wan/
        └── wan22-vae.safetensors    # 1.34 GB - Main VAE model weights

Total Repository Size : ~1.4 GB

File Details

Hardware Requirements

Minimum Requirements

• VRAM : 2 GB (VAE inference only)
• System RAM : 4 GB
• Disk Space : 1.5 GB free space
• GPU : CUDA-compatible GPU (NVIDIA) or compatible accelerator

Recommended Specifications

• VRAM : 4+ GB for comfortable operation with video generation pipeline
• System RAM : 16+ GB
• GPU : NVIDIA RTX 3060 or better
• Storage : SSD for faster model loading

Performance Notes

• VAE operations are typically memory-bound rather than compute-bound
• Larger batch sizes require proportionally more VRAM
• CPU inference is possible but significantly slower (30-50x)

Usage Examples

Basic Usage with Diffusers

import torch
from diffusers import AutoencoderKL

# Load the WAN22 VAE
vae_path = r"E:\huggingface\wan22-vae\vae\wan"
vae = AutoencoderKL.from_pretrained(
    vae_path,
    torch_dtype=torch.float16
)

# Move to GPU
device = "cuda" if torch.cuda.is_available() else "cpu"
vae = vae.to(device)

# Encode video frames to latent space
# video_frames: tensor of shape [batch, channels, height, width]
with torch.no_grad():
    latents = vae.encode(video_frames).latent_dist.sample()
    latents = latents * vae.config.scaling_factor

# Decode latents back to pixel space
with torch.no_grad():
    decoded_frames = vae.decode(latents / vae.config.scaling_factor).sample

Integration with WAN Video Generation Pipeline

import torch
from diffusers import DiffusionPipeline

# Load WAN video generation pipeline with custom VAE
pipeline = DiffusionPipeline.from_pretrained(
    "wan-model/wan-base",  # Replace with actual WAN model path
    vae=vae,  # Use the loaded WAN22-VAE
    torch_dtype=torch.float16
)
pipeline = pipeline.to("cuda")

# Generate video from text prompt
prompt = "A serene sunset over mountains with flowing clouds"
video_frames = pipeline(
    prompt=prompt,
    num_frames=24,
    height=512,
    width=512,
    num_inference_steps=50
).frames

Memory-Efficient Video Processing

import torch

# Enable memory-efficient attention for large videos
vae.enable_xformers_memory_efficient_attention()

# Process video in smaller chunks
def encode_video_chunks(video_tensor, chunk_size=8):
    """Encode video frames in chunks to reduce VRAM usage"""
    latents = []
    for i in range(0, video_tensor.shape[0], chunk_size):
        chunk = video_tensor[i:i+chunk_size].to(device)
        with torch.no_grad():
            chunk_latents = vae.encode(chunk).latent_dist.sample()
            latents.append(chunk_latents.cpu())
    return torch.cat(latents, dim=0)

Custom Latent Space Manipulation

import torch
import numpy as np

# Encode input video
latents = vae.encode(input_frames).latent_dist.sample()

# Apply transformations in latent space (e.g., interpolation)
latents_start = latents[0]
latents_end = latents[-1]

# Create smooth interpolation between frames
interpolated_latents = []
for alpha in np.linspace(0, 1, 16):
    interpolated = (1 - alpha) * latents_start + alpha * latents_end
    interpolated_latents.append(interpolated)

# Decode interpolated latents
smooth_video = vae.decode(torch.stack(interpolated_latents)).sample

Model Specifications

Architecture Details

• Model Type : Variational Autoencoder (VAE)
• Architecture : Convolutional encoder-decoder with KL divergence regularization
• Input Format : Video frames (RGB or grayscale)
• Latent Dimensions : Compressed spatial resolution with channel expansion
• Activation Functions : Mixed (SiLU, tanh for output)

Technical Specifications

• Format : SafeTensors (secure, efficient binary format)
• Precision : Mixed precision compatible (FP16/FP32)
• Framework : PyTorch-based, compatible with Diffusers library
• Parameters : ~335M parameters (1.34 GB in FP32)
• Compression Ratio : Approximately 8x spatial compression per dimension

Supported Input Resolutions

• Standard : 512x512, 768x768
• Extended : 256x256 to 1024x1024 (depending on VRAM)
• Aspect Ratios : Square and common video ratios (16:9, 4:3)

Performance Tips and Optimization

Memory Optimization

# Enable gradient checkpointing for training (if fine-tuning)
vae.enable_gradient_checkpointing()

# Use float16 for inference to reduce VRAM usage
vae = vae.half()

# Process frames in batches
batch_size = 4  # Adjust based on available VRAM

Speed Optimization

# Compile model with torch.compile (PyTorch 2.0+)
vae = torch.compile(vae, mode="reduce-overhead")

# Use channels_last memory format for better performance
vae = vae.to(memory_format=torch.channels_last)

# Enable TF32 on Ampere+ GPUs
torch.backends.cuda.matmul.allow_tf32 = True
torch.backends.cudnn.allow_tf32 = True

Quality vs Speed Trade-offs

• High Quality : Use FP32 precision, larger batch sizes, disable tiling
• Balanced : FP16 precision, moderate batch sizes (4-8 frames)
• Fast Inference : FP16 precision, smaller batches (1-2 frames), enable tiling

Best Practices

• Always use safetensors format for security and compatibility
• Monitor VRAM usage with torch.cuda.memory_allocated()
• Clear cache between large operations: torch.cuda.empty_cache()
• Use mixed precision training if fine-tuning the VAE
• Validate reconstruction quality with perceptual metrics (LPIPS, SSIM)

License

This model is released under a custom WAN license. Please review the license terms before use:

• Commercial Use : Subject to WAN license terms
• Research Use : Generally permitted with attribution
• Redistribution : Refer to original WAN model license
• Modifications : Check license for derivative work permissions

For complete license details, refer to the original WAN model repository or license documentation.

Citation

If you use this VAE in your research or projects, please cite:

@misc{wan22-vae,
  title={WAN22 VAE: Video Variational Autoencoder for WAN Video Generation},
  author={WAN Model Team},
  year={2024},
  publisher={Hugging Face},
  howpublished={\url{https://huggingface.co/wan-model/wan22-vae}}
}

Related Resources

Official Links

• WAN Base Model : WAN Model Repository
• Diffusers Documentation : https://huggingface.co/docs/diffusers
• Model Hub : https://huggingface.co/models

Community Resources

• WAN Community : Discussions and examples for WAN video generation
• Video Generation Papers : Research on video diffusion and VAE architectures
• Optimization Guides : Tips for efficient video processing with VAEs

Compatibility

• Required Libraries : torch>=2.0.0 , diffusers>=0.21.0 , transformers
• Compatible With : WAN video generation models, custom video pipelines
• Integration Examples : Check Diffusers documentation for VAE integration patterns

Technical Support

For technical issues, questions, or contributions:

1. Model Issues : Report to original WAN model repository
2. Integration Questions : Consult Diffusers documentation and community
3. Performance Optimization : Check PyTorch performance tuning guides
4. Local Setup : Verify CUDA installation and GPU compatibility

---

Version : v1.0 Last Updated : 2025-10-13 Model Format : SafeTensors Total Size : 1.4 GB

Changelog

v1.0 (Initial Release)

• Initial documentation for WAN22-VAE model
• Comprehensive usage examples for video encoding/decoding
• Hardware requirements and optimization guidelines
• Integration examples with Diffusers library
• Performance tuning recommendations