nvidia / Diffusion_Renderer_Forward_Cosmos_7B

DiffusionRenderer (Cosmos): Neural Inverse and Forward Rendering with Video Diffusion Models

Ruofan Liang *, Zan Gojcic , Huan Ling , Jacob Munkberg , Jon Hasselgren , Zhi-Hao Lin , Jun Gao , Alexander Keller , Nandita Vijaykumar , Sanja Fidler , Zian Wang *

* indicates equal contribution

Paper | Project Page

Description:

Cosmos Diffusion Renderer is a video diffusion model that accurately estimates geometry and material buffers, and generates photorealistic images under specified lighting conditions, offering fundamental tools for image editing applications. This model is available for commercial use.

License/Terms of Use:

The model is distributed under NVIDIA Open Model License .

Deployment Geography:

Global

Use Case:

AI research, development and benchmarking for image/video delighting and relighting tasks.

Release Date:

Github 06/11/2025 via https://github.com/nv-tlabs/cosmos1-diffusion-renderer

Reference(s):

Project page: https://research.nvidia.com/labs/toronto-ai/DiffusionRenderer/

Model Architecture:

Architecture Type: Transformer.

Network Architecture: Transformer. This model has 7B model parameters.

Input:

Input Type(s): Image, Video

Input Format(s): Red, Green, Blue (RGB); Video input are frames of images.

Input Parameters: The input video data are five-dimensional, with the input dimension specified as [batch_size, num_frames, height, width, 3], where the three channels are Red, Green, Blue (RGB) channels.
The input image data are five-dimensional [batch_size, 1, height, width, 3], with the frame channel set to 1.

Other Properties Related to Input: The input resolution is 704 x 1280.

Output:

Output Type(s): Image, Video

Output Format(s): Red, Green, Blue (RGB); Output video is saved as image frames.

Output Parameters: The output video data are five-dimensional, with the dimension specified as [batch_size, num_frames, height, width, 3], where the three channels are Red, Green, Blue (RGB) channels.
The output image data are five-dimensional [batch_size, 1, height, width, 3], with the frame channel set to 1.

Other Properties Related to Output: The input resolution is 704 x 1280.

Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA’s hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.

Software Integration:

Runtime Engine(s): Not Applicable - Uses Python Scripts and Pytorch

Supported Hardware Microarchitecture Compatibility: NVIDIA Ampere (A100)

[Preferred/Supported] Operating System(s): ['Linux']

Model Version(s):

• Diffusion_Renderer_Inverse_Cosmos_7B : estimates geometry and material buffers from input image or video.
• Diffusion_Renderer_Forward_Cosmos_7B : generates photorealistic images or videos from G-buffers and environment map.

Training Dataset:

The training data contains samples with paired data of RGB video frames, G-buffers (base color, roughness, metallic, normals, depth), and lighting (HDR environment maps).

Data Collection Method : Synthetic

Labeling Method : Hybrid: Synthetic, Automated

Properties:

• 150k synthetic videos, 57 frames each at 704×1280 resolution.
• 150k auto-labeled real-videos, 57 frames each at 704×1280 resolution.

For additional details, please refer to the paper .

Inference:

Engine: Tensor(RT)

Test Hardware:

A100 GPUs

Ethical Considerations:

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.

Please report security vulnerabilities or NVIDIA AI Concerns here .