diffusers / matrix-game-2-modular

Matrix Game 2.0 Modular Pipeline

Set Up

uv venv -p 3.10
uv pip install -r requirements.txt
uv pip install git+https://github.com/huggingface/diffusers.git

How to Use

import torch
from diffusers import ModularPipelineBlocks
from diffusers.utils import export_to_video, load_image
from diffusers.modular_pipelines import WanModularPipeline

class MatrixGameWanModularPipeline(WanModularPipeline):
    """
    A ModularPipeline for MatrixGameWan.

    <Tip warning={true}>

        This is an experimental feature and is likely to change in the future.

    </Tip>
    """

    @property
    def default_sample_height(self):
        return 44

    @property
    def default_sample_width(self):
        return 80

# Download custom blocks for the pipeline
blocks = ModularPipelineBlocks.from_pretrained(
    "diffusers-internal-dev/matrix-game-2-modular",
    trust_remote_code=True,
)

# Initialize the pipeline runtime using the config in the repo
pipe = MatrixGameWanModularPipeline(blocks, "diffusers-internal-dev/matrix-game-2-modular")

# Load the model components of the pipeline
pipe.load_components(
    trust_remote_code=True,
    device_map="cuda",
    torch_dtype={"default": torch.bfloat16, "vae": torch.float32}
)

image = load_image("https://github.com/SkyworkAI/Matrix-Game/blob/main/Matrix-Game-2/demo_images/universal/0016.png?raw=true")
output = pipe(image=image, num_frames=141)
export_to_video(output.values['videos'][0], "matrix-game.mp4")

Providing Actions as Inputs

Each action is represented as a string. The available actions are:

Motion Actions: ["forward", "left", "right"] Camera Actions: ["camera_l", "camera_r", "camera_u", "camera_d"] Compound Actions: Combinations of motion and camera actions with an _ separating actions, e.g. "forward_left", "forward_left_camera_l"

image = load_image("https://github.com/SkyworkAI/Matrix-Game/blob/main/Matrix-Game-2/demo_images/universal/0016.png?raw=true")
output = pipe(image=image, actions=["forward", "camera_l"], num_frames=141)
export_to_video(output.values['videos'][0], "matrix-game.mp4")