sbapan41 / Text2Motion

Text2Motion

Text2Motion: Open and Advanced Large-Scale Video Generative Models

In this repository, we present Text2Motion , a comprehensive and open suite of video foundation models that pushes the boundaries of video generation. Text2Motion offers these key features:

• 👍 SOTA Performance : Text2Motion consistently outperforms existing open-source models and state-of-the-art commercial solutions across multiple benchmarks.
• 👍 Supports Consumer-grade GPUs : The T2V-1.3B model requires only 8.19 GB VRAM, making it compatible with almost all consumer-grade GPUs. It can generate a 5-second 480P video on an RTX 4090 in about 4 minutes (without optimization techniques like quantization). Its performance is even comparable to some closed-source models.
• 👍 Multiple Tasks : Text2Motion excels in Text-to-Video, Image-to-Video, Video Editing, Text-to-Image, and Video-to-Audio, advancing the field of video generation.
• 👍 Visual Text Generation : Text2Motion is the first video model capable of generating both Chinese and English text, featuring robust text generation that enhances its practical applications.
• 👍 Powerful Video VAE : Text2Motion-VAE delivers exceptional efficiency and performance, encoding and decoding 1080P videos of any length while preserving temporal information, making it an ideal foundation for video and image generation.

This repository features our T2V-14B model, which establishes a new SOTA performance benchmark among both open-source and closed-source models. It demonstrates exceptional capabilities in generating high-quality visuals with significant motion dynamics. It is also the only video model capable of producing both Chinese and English text and supports video generation at both 480P and 720P resolutions.

🔥 Latest News!!

• Feb 22, 2025: 👋 We've released the inference code and weights of Text2Motion.

📑 Todo List

• Text2Motion Text-to-Video
  • Multi-GPU Inference code of the 14B
  • Checkpoints of the 14B
  • Gradio demo
  • Diffusers integration
  • ComfyUI integration
• Text2Motion Image-to-Video
  • Multi-GPU Inference code of the 14B model
  • Checkpoints of the 14B model
  • Gradio demo
  • Diffusers integration
  • ComfyUI integration

Quickstart

Installation

Clone the repo:

git clone https://huggingface.co/sbapan41/Text2Motion
cd Text2Motion

Install dependencies:

# Ensure torch >= 2.4.0
pip install -r requirements.txt

Model Download

Download models using 🤗 huggingface-cli:

pip install "huggingface_hub[cli]"
huggingface-cli download sbapan41/Text2Motion --local-dir ./Text2Motion

Run Text-to-Video Generation

This repository supports two Text-to-Video models (14B) and two resolutions (480P and 720P). The parameters and configurations for these models are as follows:

(1) Without Prompt Extention

To facilitate implementation, we will start with a basic version of the inference process that skips the prompt extension step.

• Single-GPU inference

python generate.py  --task 14B --size 1280*720 --ckpt_dir ./Text2Motion --prompt "Two anthropomorphic cats in comfy boxing gear and bright gloves fight intensely on a spotlighted stage."

If you encounter OOM (Out-of-Memory) issues, you can use the --offload_model True and --t5_cpu options to reduce GPU memory usage. For example, on an RTX 4090 GPU:

- Multi-GPU inference using FSDP + xDiT USP

pip install "xfuser>=0.4.1" torchrun --nproc_per_node=8 generate.py --task 14B --size 1280*720 --ckpt_dir ./Text2Motion --dit_fsdp --t5_fsdp --ulysses_size 8 --prompt "Two anthropomorphic cats in comfy boxing gear and bright gloves fight intensely on a spotlighted stage."

| Model  | Dimension | Input Dimension | Output Dimension | Feedforward Dimension | Frequency Dimension | Number of Heads | Number of Layers |
|--------|-----------|-----------------|------------------|-----------------------|---------------------|-----------------|------------------|
| 14B    | 5120      | 16              | 16               | 13824                  | 256                 | 40              | 40              |