CompVis / schroedingers_cat

Schrödinger's Cat: Probabilistic Representation and Prediction of Potential Scene Kinematics

Timy Phan ^* · Jannik Wiese ^* · Björn Ommer

CompVis Group @ LMU Munich, Munich Center for Machine Learning (MCML)

ECCV 2026

^* Equal contribution

Official Code for the paper "Schrödinger's Cat: Probabilistic Representation and Prediction of Potential Scene Kinematics" accepted at ECCV 2026.

💡 TL;DR

GARFIELD learns a structured latent representation of possible scene kinematics from an image and optional sparse spatio-temporal constraints. The representation supports both joint trajectory sampling and direct point-wise density estimation.

📝 Overview

Goal Aware Representations of Future kInEmatic Latent Distributions (GARFIELD) represents future scene kinematics as localized motion distributions. Given an initial image and optional sparse constraints, a joint encoder produces spatio-temporal latents for scene elements over time. These latents encode uncertainty about where each element may move, and separate decoders expose that uncertainty as point samples, coherent trajectory rollouts, or probability heatmaps.

The full pipeline consists of four components:

• Encoder: The joint encoder combines image features and sparse kinematic constraints into structured latents, each tied to one scene element and timestep.
• Pointwise Decoder: The pointwise decoder trains the latents to represent localized distributions by sampling individual future track positions from each latent component.
• Full Decoder: The full decoder samples complete trajectories jointly, preserving dependencies across points, timesteps, and scene elements for coherent motion realizations.
• Density Decoder: The density decoder deterministically maps localized latents to probability heatmaps, enabling fast uncertainty inspection without Monte-Carlo sampling.

📊 Results

Motion Planning

GARFIELD infers accurate motion from very sparse information and, with only four conditioning points, outperforms Motion-I2V using sixteen.

Direct Density Decoding

The density decoder achieves superior discrete energy scores while estimating motion densities orders of magnitude faster than Monte-Carlo sampling.

Entropy-informed Conditioning

Selecting additional constraints by entropy collapses uncertainty without ground-truth errors and reaches comparable performance with fewer conditioning points.

🛠️ Usage

Weights and Inference Setup

For inference, you can clone the GitHub repository by running:

git clone https://github.com/CompVis/schroedingers_cat
cd schroedingers_cat

Then, download pretrained model weights from 🤗 huggingface running:

hf download CompVis/schroedingers_cat --include "*.pt" --local-dir ckpts

Finally, create an environment to run the repository:

Conda (recommended):

conda create -n garfield python=3.12 -y
conda activate garfield
python -m pip install --upgrade pip
pip install -r requirements.txt

Virtual environment:

python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
pip install -r requirements.txt

Citation

@inproceedings{phan2026schrodingerscat,
  title     = {Schrödinger's Cat: Probabilistic Representation and Prediction of Potential Scene Kinematics},
  author    = {Phan, Timy and Wiese, Jannik and Ommer, Björn},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year      = {2026}
}