The Drift Kernel: Why Diffusion Models Change Even When Told Not To

Even when told to “do nothing,” modern diffusion models subtly alter their output relative to the input they are supposed to preserve. We call this effect **No-Op Drift**. We introduce the **Drift Kernel**$$K_M(\sigma)=\mathbb{E}[|I' - I_0|_2^2 \mid \sigma],$$the expected perceptual deviation induced when running a diffusion model at noise strength $\sigma$ under a null instruction. Using 120{,}000 baseline samples (30{,}000 per model across SD15, SD21, SDXL, and InstructPix2Pix) and 9{,}600 ablation samples (four strengths, null vs. strict copy prompts), we show that variance-driven diffusion models follow a quadratic form$$K_M(\sigma)\approx k_M\sigma^2 + c_M,$$with aggregate $R^2=0.97$. We derive this scaling from first principles via a Taylor expansion of the decoder, yielding $k_M=\mathrm{Tr}(J_D J_D^\top)$, which depends only on the decoder Jacobian—not on prompts. To validate mechanistic structure, we construct synthetic decoders that reproduce the two regimes seen in practice: quadratic variance-driven drift and flat, high-variance edit-driven drift. We show that prompt wording has negligible effect ($<17%$ coefficient difference), proving that drift is structural, not prompt-induced. We release **NoOp-Bench**, a benchmark with 10{,}000 inputs and full code for reproducible kernel estimation. Additional proofs, ablations, LPIPS/CLIP metrics, and extended visualizations appear in the Supplementary Material.