Abstract: How can we generate an image $B^\prime$ that satisfies $A:A^\prime::B:B^\prime$, given the input images $A$,$A^\prime$ and $B$?Recent works have tackled this challenge through approaches like visual in-context learning or visual instruction. However, these methods are typically limited to specific models (\eg InstructPix2Pix. Inpainting models) rather than general diffusion models (\eg Stable Diffusion, SDXL). This dependency may lead to inherited biases or lower editing capabilities. In this paper, we propose Difference Inversion, a method that isolates only the difference from $A$ and $A^\prime$ and applies it to $B$ to generate a plausible $B^\prime$. To address model dependency, it is crucial to structure prompts in the form of a "Full Prompt" suitable for input to stable diffusion models, rather than using an "Instruction Prompt". To this end, we accurately extract the Difference between $A$ and $A^\prime$ and combine it with the prompt of $B$, enabling a plug-and-play application of the difference. To extract a precise difference, we first identify it through 1) Delta Interpolation. Additionally, to ensure accurate training, we propose the 2) Token Consistency Loss and 3) Zero Initialization of Token Embeddings. Our extensive experiments demonstrate that Difference Inversion outperforms existing baselines both quantitatively and qualitatively, indicating its ability to generate more feasible $B^\prime$ in a model-agnostic manner.