# Warped Diffusion: Solving Temporally Correlated Inverse Problems with Image Diffusion Models

Anonymous Authors

Your browser does not support the video tag.
Input Video
Your browser does not support the video tag.
Noise Warping
Your browser does not support the video tag.
Video Output Latent Space
Your browser does not support the video tag.
Video Output Pixel Space

## Abstract

Using image models naively for solving inverse video problems often suffers from flickering, texture-sticking, and temporal inconsistency in generated videos. To tackle these problems, in this paper, we view frames as continuous functions in the 2D space and videos as a sequence of continuous transformations, i.e., warpings, between different frames. Given this perspective, we propose to train function space diffusion models for solving inverse problems only in \textit{images}. We then show that to apply these models to videos, the model should be equivariant with respect to the transformation observed in input videos. To ensure equivariance, we introduce two simple approaches including post-hoc test-time consistency guidance and training-time consistency augmentation. Our method allows us to use state-of-the-art latent diffusion models such as Stable Diffusion XL to solve video inverse problems. We demonstrate the effectiveness of our method for video inpainting and $8\times$ video super-resolution, outperforming state-of-the-art techniques based on noise transformation.

## Super-resolution for real videos

Your browser does not support the video tag.
Input Video
Your browser does not support the video tag.
Noise Warping
Your browser does not support the video tag.
Video Output Latent Space
Your browser does not support the video tag.
Video Output Pixel Space
Your browser does not support the video tag.
Input Video
Your browser does not support the video tag.
Noise Warping
Your browser does not support the video tag.
Video Output Latent Space
Your browser does not support the video tag.
Video Output Pixel Space
Your browser does not support the video tag.
Input Video
Your browser does not support the video tag.
Noise Warping
Your browser does not support the video tag.
Video Output Latent Space
Your browser does not support the video tag.
Video Output Pixel Space
Your browser does not support the video tag.
Input Video
Your browser does not support the video tag.
Noise Warping
Your browser does not support the video tag.
Video Output Latent Space
Your browser does not support the video tag.
Video Output Pixel Space
Your browser does not support the video tag.
Input Video
Your browser does not support the video tag.
Noise Warping
Your browser does not support the video tag.
Video Output Latent Space
Your browser does not support the video tag.
Video Output Pixel Space

## Comparisons

Your browser does not support the video tag.
Your browser does not support the video tag.
Your browser does not support the video tag.
Your browser does not support the video tag.
Your browser does not support the video tag.
Input Video
Your browser does not support the video tag.
Fixed Noise
Your browser does not support the video tag.
How I Warped Your Noise
Your browser does not support the video tag.
Warped Diffusion (Ours)
Your browser does not support the video tag.
Your browser does not support the video tag.
Your browser does not support the video tag.
Your browser does not support the video tag.