What Is It Like to Be a Noise? An Entropy-based Gaussian Noise Regularization for Diffusion Models

Optimizing noise latents in diffusion models is powerful for controllable generation, reward-guided sampling, and latent inversion, but the process is notoriously unstable. Without a principled regularizer, optimized latents drift away from the Gaussian prior, collapsing out of the typical set and producing severe artifacts. Existing constraints like norm-matching or simple KL divergence losses are often insufficient, as they fail to capture the full statistical properties of true Gaussian noise. We propose a principled, differentiable regularizer that correctly targets the high-mass typical set rather than the high-probability mode. Our energy function tractably approximates the KL divergence by matching low-order statistics. It combines a 1D marginal term to match the pixel-value histogram and a 2D spatial term to enforce decorrelation. By applying this in a multi-scale pyramid, our method penalizes correlations at all ranges, effectively projecting samples closer onto the true Gaussian typical set. We demonstrate its effectiveness for robust, artifact-free reward-guided generation and model-free latent inversion.