Low-Rank Residual Diffusion Models

Diffusion models have achieved remarkable progress in image generation and restoration. However, most frameworks assume a full-rank residual space, neglecting its inherent low-dimensional structure in near-domain transformations such as deraining and deblurring. We propose the Low-Rank Residual Diffusion Model (LRDM), which performs diffusion within a compact low-rank residual subspace for efficient and structure-preserving restoration. We establish the Low-Rank Residual Assumption, showing that the variational lower bound becomes tighter when residuals lie in a low-rank space. LRDM further introduces an Asymmetric Residual Diffusion Process, constraining the forward process in the low-rank domain while maintaining full-rank flexibility in the reverse direction. An Adaptive Rank Selection mechanism dynamically adjusts the rank across timesteps to capture varying residual complexity. Experiments on deraining, deblurring, and deshading benchmarks show that LRDM surpasses full-rank diffusion baselines and achieves state-of-the-art performance, validating the advantage of modeling diffusion in a low-rank residual space.