wangkanai / wan-2.1

WAN 2.1 - Video Generation with Camera Control

This repository contains WAN (Wan An) 2.1 models for advanced video generation with cinematic camera control capabilities. WAN 2.1 introduces LoRA adapters for precise camera movement control in generated videos.

Model Description

WAN 2.1 is a video generation model that extends traditional text-to-video capabilities with specialized camera control LoRAs. These LoRA adapters enable cinematic camera movements including rotation, arc shots, and drone perspectives.

Model Components

Total Size : ~1.3GB

VAE (Variational Autoencoder)

• File : vae/wan/wan21-vae.safetensors
• Size : 243 MB
• High-quality VAE for encoding and decoding video latents

Camera Control LoRAs

Three specialized LoRA models for precise camera movement control (all rank-16, 343MB each):

1. Camera Rotation LoRA

• File : loras/wan/wan21-camera-rotation-rank16-v1.safetensors
• Size : 343 MB
• Rank : 16
• Enables smooth rotational camera movements around subjects

2. Camera Arc Shot LoRA

• File : loras/wan/wan21-camera-arcshot-rank16-v1.safetensors
• Size : 343 MB
• Rank : 16
• Creates cinematic arc/circular camera movements

3. Camera Drone Shot LoRA

• File : loras/wan/wan21-camera-drone-rank16-v1.safetensors
• Size : 343 MB
• Rank : 16
• Simulates aerial drone-style camera perspectives and movements

Hardware Requirements

• VRAM : 12GB+ recommended for base model + LoRA
• Disk Space : 1.3GB
• LoRA Overhead : Minimal (343MB per LoRA)
• Memory : 16GB+ system RAM recommended

Usage

Loading the VAE

from diffusers import AutoencoderKL

vae = AutoencoderKL.from_single_file(
    "E:/huggingface/wan-2.1/vae/wan/wan21-vae.safetensors"
)

Basic Video Generation with Camera Control

from diffusers import DiffusionPipeline, AutoencoderKL
import torch

# Load your base video generation pipeline
pipe = DiffusionPipeline.from_pretrained(
    "your-base-video-model",
    torch_dtype=torch.float16
)

# Load WAN 2.1 VAE
pipe.vae = AutoencoderKL.from_single_file(
    "E:/huggingface/wan-2.1/vae/wan/wan21-vae.safetensors"
)

# Load desired camera control LoRA
pipe.load_lora_weights(
    "E:/huggingface/wan-2.1/loras/wan/wan21-camera-rotation-rank16-v1.safetensors"
)

pipe.to("cuda")

# Generate video with camera control
video = pipe(
    prompt="rotating camera, a beautiful garden with flowers",
    num_frames=24,
    num_inference_steps=50
).frames[0]

Camera Movement Prompting Tips

Rotation LoRA ( wan21-camera-rotation-rank16-v1.safetensors ):

• Prompts: "rotating camera", "camera circles around subject", "360-degree view"
• Effect: Orbital camera movement around the subject

Arc Shot LoRA ( wan21-camera-arcshot-rank16-v1.safetensors ):

• Prompts: "arc shot", "curved camera movement", "sweeping camera motion"
• Effect: Smooth curved dolly movements

Drone LoRA ( wan21-camera-drone-rank16-v1.safetensors ):

• Prompts: "aerial view", "drone shot", "bird's eye view", "flying camera"
• Effect: Aerial and elevated perspectives

Model Specifications

• VAE Architecture : Custom trained for video generation
• LoRA Rank : 16 (balanced between quality and efficiency)
• Format : SafeTensors (secure and efficient)
• Compatible With : Video diffusion models supporting LoRA and custom VAE
• Precision : Standard safetensors format

Combining Multiple LoRAs

# Load multiple camera control LoRAs
pipe.load_lora_weights(
    "E:/huggingface/wan-2.1/loras/wan/wan21-camera-rotation-rank16-v1.safetensors",
    adapter_name="rotation"
)
pipe.load_lora_weights(
    "E:/huggingface/wan-2.1/loras/wan/wan21-camera-drone-rank16-v1.safetensors",
    adapter_name="drone"
)

# Set different weights for each LoRA
pipe.set_adapters(["rotation", "drone"], adapter_weights=[0.7, 0.3])

Installation

# Install required dependencies
pip install diffusers transformers accelerate safetensors torch

Requirements

• Python 3.8+
• PyTorch 2.0+
• diffusers
• transformers
• accelerate
• safetensors

Performance Tips

1. Combining LoRAs : You can combine multiple camera LoRAs at lower weights for complex movements
2. Weight Adjustment : Adjust LoRA strength (typically 0.5-1.0) to control the intensity of camera effects
3. Hardware : Recommended 16GB+ VRAM for optimal performance

Version Comparison

WAN 2.1 vs WAN 2.2:

• WAN 2.1 : Original camera control LoRAs (v1)
• WAN 2.2 : Enhanced camera controls (v2) + additional enhancement LoRAs (lighting, faces, actions)

For improved camera control and additional enhancement features, see the WAN 2.2 collection which includes v2 variants with better quality.

License

Check the specific license terms for WAN models. This may differ from standard open-source licenses.

Citation

If you use this model in your research or projects, please cite:

@software{wan2.1,
  title={WAN 2.1: Video Generation with Camera Control},
  year={2024},
  note={Camera control LoRAs for cinematic video generation}
}

Model Card Contact

For questions about WAN 2.1 models, refer to the official WAN model documentation and community resources.

Related Resources

• WAN 2.2 Models - Enhanced camera controls and quality improvements
• WAN LightX2V LoRAs - CFG step distillation adapters

---

Note : This is a community model repository. Please use responsibly and in accordance with ethical AI guidelines.