wangkanai / wan22-fp16-encoders

WAN2.2 FP16 Text Encoders

High-precision FP16 text encoders for the WAN (Worldly Advanced Network) 2.2 text-to-video generation system. This repository contains the essential text encoding components required for WAN2.2 video generation workflows.

Model Description

This repository provides two specialized text encoder models optimized for video generation tasks:

• T5-XXL FP16 : Google's T5 (Text-to-Text Transfer Transformer) extra-extra-large encoder in 16-bit floating point precision
• UMT5-XXL FP16 : Universal Multilingual T5 extra-extra-large encoder in 16-bit floating point precision

These encoders are critical components of the WAN2.2 pipeline, responsible for transforming text prompts into high-dimensional semantic representations that guide the video generation process. The FP16 precision maintains excellent quality while reducing memory requirements compared to FP32 variants.

Key Features

• High Precision : FP16 format preserves text encoding quality with 50% memory reduction vs FP32
• Multilingual Support : UMT5-XXL provides robust multilingual text understanding
• Production Ready : Optimized for inference with safetensors format
• WAN2.2 Compatible : Designed specifically for WAN video generation workflows

Repository Contents

wan22-fp16-encoders/
└── text_encoders/
    ├── t5-xxl-fp16.safetensors       # 8.9 GB
    └── umt5-xxl-fp16.safetensors     # 11 GB

File Details

Total Repository Size : ~20 GB

Hardware Requirements

Minimum Requirements

• VRAM : 12 GB GPU memory (for text encoding alone)
• RAM : 16 GB system memory
• Disk Space : 25 GB free space (including working directory)
• GPU : CUDA-compatible GPU with compute capability 6.0+

Recommended Requirements

• VRAM : 16 GB+ GPU memory (for full WAN2.2 pipeline)
• RAM : 32 GB system memory
• Disk Space : 50 GB+ free space
• GPU : NVIDIA RTX 3090, RTX 4090, or A100

Performance Notes

• Both encoders can be loaded simultaneously with 24 GB+ VRAM
• Text encoding typically takes 1-5 seconds per prompt
• CPU offloading available but significantly slower (10-30x)

Usage Examples

Basic Text Encoding with Diffusers

from diffusers import DiffusionPipeline
import torch

# Load WAN2.2 pipeline with custom text encoders
pipe = DiffusionPipeline.from_pretrained(
    "your-wan22-model",
    text_encoder_path="E:/huggingface/wan22-fp16-encoders/text_encoders/t5-xxl-fp16.safetensors",
    torch_dtype=torch.float16,
    variant="fp16"
).to("cuda")

# Generate video from text
prompt = "A serene mountain landscape at sunset with flowing clouds"
video = pipe(prompt, num_frames=24, height=512, width=512).frames

# Save output
video[0].save("output_video.mp4")

Using UMT5 for Multilingual Prompts

from diffusers import DiffusionPipeline
import torch

# Load with multilingual encoder
pipe = DiffusionPipeline.from_pretrained(
    "your-wan22-model",
    text_encoder_path="E:/huggingface/wan22-fp16-encoders/text_encoders/umt5-xxl-fp16.safetensors",
    torch_dtype=torch.float16,
).to("cuda")

# Generate with multilingual prompt
prompt = "東京の夜景、ネオンライトと雨"  # Japanese: Tokyo nightscape with neon lights and rain
video = pipe(prompt, num_frames=48, height=768, width=768).frames

Memory-Optimized Loading

from diffusers import DiffusionPipeline
import torch

# Enable CPU offloading for lower VRAM systems
pipe = DiffusionPipeline.from_pretrained(
    "your-wan22-model",
    text_encoder_path="E:/huggingface/wan22-fp16-encoders/text_encoders/t5-xxl-fp16.safetensors",
    torch_dtype=torch.float16,
).to("cuda")

# Enable model CPU offload
pipe.enable_model_cpu_offload()

# Enable attention slicing for further memory reduction
pipe.enable_attention_slicing(1)

# Generate with reduced memory footprint
video = pipe(prompt, num_frames=16).frames

Direct Encoder Loading

from safetensors.torch import load_file
import torch

# Load encoder weights directly
encoder_weights = load_file(
    "E:/huggingface/wan22-fp16-encoders/text_encoders/t5-xxl-fp16.safetensors"
)

# Initialize your custom text encoder model
from transformers import T5EncoderModel

text_encoder = T5EncoderModel.from_pretrained("google/t5-v1_1-xxl", torch_dtype=torch.float16)
text_encoder.load_state_dict(encoder_weights)
text_encoder = text_encoder.to("cuda")

# Use encoder for custom workflows
# ... your custom inference code ...

Model Specifications

T5-XXL FP16

• Architecture : T5 (Text-to-Text Transfer Transformer)
• Model Size : Extra-Extra-Large (XXL)
• Parameters : ~11 billion
• Precision : FP16 (16-bit floating point)
• Format : SafeTensors
• Context Length : 512 tokens
• Embedding Dimension : 4096
• Language Support : English-focused, trained on C4 dataset

UMT5-XXL FP16

• Architecture : Universal Multilingual T5
• Model Size : Extra-Extra-Large (XXL)
• Parameters : ~13 billion
• Precision : FP16 (16-bit floating point)
• Format : SafeTensors
• Context Length : 512 tokens
• Embedding Dimension : 4096
• Language Support : 100+ languages (multilingual mC4 dataset)

Performance Tips and Optimization

Memory Optimization

1. Sequential Encoder Loading : Load encoders one at a time if VRAM is limited
2. CPU Offloading : Use enable_model_cpu_offload() for systems with <16 GB VRAM
3. Attention Slicing : Enable with enable_attention_slicing() to reduce peak memory
4. Batch Size : Process multiple prompts together for better GPU utilization

Speed Optimization

1. TensorRT Compilation : Convert encoders to TensorRT for 2-3x speedup
2. Flash Attention : Use xformers or flash-attention for faster inference
3. Model Quantization : Consider INT8 quantization for production deployment
4. Prompt Caching : Cache encoded prompts for repeated generations

Quality Optimization

1. Use UMT5 for Non-English : Better results with non-English prompts
2. Longer Prompts : These XXL models handle detailed descriptions well
3. Prompt Engineering : Structured, descriptive prompts yield best results
4. Negative Prompts : Combine with negative prompt encoding for better control

Best Practices

# Optimal configuration for quality and speed
pipe = DiffusionPipeline.from_pretrained(
    "wan22-model",
    text_encoder_path="E:/huggingface/wan22-fp16-encoders/text_encoders/t5-xxl-fp16.safetensors",
    torch_dtype=torch.float16,
    variant="fp16",
)

# Enable optimizations
pipe.enable_xformers_memory_efficient_attention()  # Flash attention
pipe.enable_attention_slicing(1)                   # Memory efficiency
pipe.to("cuda")

# Use compiled model for production (PyTorch 2.0+)
pipe.unet = torch.compile(pipe.unet, mode="reduce-overhead", fullgraph=True)

License

These text encoder models are provided under specific licensing terms. Please refer to the original model sources for detailed license information:

• T5-XXL : Apache 2.0 License (Google Research)
• UMT5-XXL : Apache 2.0 License (Google Research)
• WAN2.2 Pipeline : Please check WAN project license terms

Usage Restrictions : These models are intended for research and development purposes. Commercial usage should comply with respective license terms and any additional WAN project requirements.

Citation

If you use these text encoders in your research or projects, please cite the relevant papers:

T5 Citation

@article{raffel2020exploring,
  title={Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer},
  author={Raffel, Colin and Shazeer, Noam and Roberts, Adam and Lee, Katherine and Narang, Sharan and Matena, Michael and Zhou, Yanqi and Li, Wei and Liu, Peter J},
  journal={Journal of Machine Learning Research},
  volume={21},
  number={140},
  pages={1--67},
  year={2020}
}

mT5/UMT5 Citation

@inproceedings{xue2021mt5,
  title={mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer},
  author={Xue, Linting and Constant, Noah and Roberts, Adam and Kale, Mihir and Al-Rfou, Rami and Siddhant, Aditya and Barua, Aditya and Raffel, Colin},
  booktitle={Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  pages={483--498},
  year={2021}
}

WAN Project Citation

Please check the official WAN project repository for citation guidelines and additional references.

Related Resources

Official Documentation

• T5 Model Card
• mT5 Model Card
• Diffusers Documentation
• SafeTensors Format

WAN Project Resources

• WAN Official Repository: [Check official project page]
• WAN Documentation: [Check official documentation]
• Community Forum: [Check community channels]

Related Models

• WAN2.2 Base Models: E:/huggingface/wan22/
• WAN2.2 VAE: E:/huggingface/wan22-vae/
• Enhancement LoRAs: E:/huggingface/wan22-loras/

Technical Support

For issues specific to these text encoders:

• Check text encoder dimensions and compatibility with your WAN2.2 version
• Verify CUDA and PyTorch versions support FP16 operations
• Ensure sufficient VRAM for your chosen encoder(s)
• Review memory optimization strategies above

For WAN2.2 pipeline issues, please consult the main WAN project documentation and community resources.

---

Model Version : v1.2 Last Updated : 2025-10-28 Format : SafeTensors FP16 Compatibility : WAN2.2, Diffusers 0.21+, PyTorch 2.0+