Skip to yearly menu bar Skip to main content


Poster

Towards Transformer-Based Aligned Generation with Self-Coherence Guidance

Shulei Wang · w l · Hai Huang · Hanting Wang · Sihang Cai · WenKang Han · Tao Jin · Jingyuan Chen · Jiacheng Sun · Jieming Zhu · Zhou Zhao


Abstract:

We introduce a novel, training-free approach for enhancing alignment in Transformer-based Text-Guided Diffusion Models (TGDMs). Existing TGDMs often struggle to generate semantically aligned images, particularly when dealing with complex text prompts or multi-concept attribute binding challenges. Previous U-Net-based methods primarily optimized the latent space, but their direct application to Transformer-based architectures has shown limited effectiveness. Our method addresses these challenges by directly optimizing cross-attention maps during the generation process. Specifically, we introduce Self-Coherence Guidance, a method that dynamically refines attention maps using masks derived from previous denoising steps, ensuring precise alignment without additional training. To validate our approach, we constructed more challenging benchmarks for evaluating coarse-grained attribute binding, fine-grained attribute binding, and style binding. Experimental results demonstrate the superior performance of our method, significantly surpassing other state-of-the-art methods across all evaluated tasks. Our code is available at https://scg-diffusion.github.io/scg-diffusion.

Live content is unavailable. Log in and register to view live content