LADiBI: Blind Inverse Problem Solving Made Easy by Text-to-Image Latent Diffusion

Michail Dontas1, Yutong He1, Naoki Murata2, Yuki Mitsufuji2,3, J. Zico Kolter1,4, Ruslan Salakhutdinov1,
1Carnegie Mellon University, 2Sony AI, 3Sony Group Corporation, 4Bosch Center for AI
arXiv Paper Code

Our proposed LADiBI is a training-free blind inverse problem solving algorithm using large pre-trained text-to-image diffusion models. LADiBI is applicable to a wide variety of image distribution as well as operators with minimal modeling assumptions imposed.

Abstract

Blind inverse problems, where both the target data and forward operator are unknown, are crucial to many computer vision applications. Existing methods often depend on restrictive assumptions such as additional training, operator linearity, or narrow image distributions, thus limiting their generalizability. In this work, we present LADiBI, a training-free framework that uses large-scale text-to-image diffusion models to solve blind inverse problems with minimal assumptions. By leveraging natural language prompts, LADiBI jointly models priors for both the target image and operator, allowing for flexible adaptation across a variety of tasks. Additionally, we propose a novel posterior sampling approach that combines effective operator initialization with iterative refinement, enabling LADiBI to operate without predefined operator forms. Our experiments show that LADiBI is capable of solving a broad range of image restoration tasks, including both linear and nonlinear problems, on diverse target image distributions.

Proposed Method

A schematic overview of our algorithm LaDiBI (Algorithm 1). LADiBI does not requires model retraining or reselection for different target data distributions or operator functions; instead, all prior parameterization is encoded directly in the prompt, which users can adjust as needed. We also do not assume linearity of the operator, making LaDiBI, to the best of our knowledge, the most generalizable approach to blind inverse problem solving in image restoration.

An illustration of our general operator initialization algorithm (Algorithm 2). This new initialization scheme leverages the pseudo-supervision signals from multiple lower-quality data estimation generated by fast posterior diffusion sampling. This approach eliminates the assumption of a specific form of operators, allowing for nonlinear blind inverse problem solving and flexible operator parametrization.

Demo

Qualitative results on blind linear deblurring tasks. From top to bottom we showcase examples from motion deblur on FFHQ, Gaussian deblur on FFHQ, motion deblur on AFHQ, and Gaussian deblur on AFHQ respectively.

Qualitative results on the blind JPEG decompression task.

Qualitative results on blind deblurring paintings.

BibTeX


        @article{dontas2024blind,
          title={Blind Inverse Problem Solving Made Easy by Text-to-Image Latent Diffusion},
          author={Dontas, Michail and He, Yutong and Murata, Naoki and Mitsufuji, Yuki and Kolter, J Zico and Salakhutdinov, Ruslan},
          journal={arXiv preprint arXiv:2412.00557},
          year={2024}
        }