PersonaHOI: Effortlessly Improving Face Personalization in Human-Object Interaction Generation

We introduce PersonaHOI, a training- and tuning-free framework that fuses a general StableDiffusion model with a personalized face diffusion model to generate identity-consistent human-object interaction (HOI) images. While personalized face diffusion (PFD) models have advanced significantly, they often overfit facial features and fail to produce coherent full-body interactions with objects. To address this issue, PersonaHOI introduces an additional StableDiffusion (SD) branch to follow HOI-driven text descriptions in image generation. By incorporating proposed cross-attention constraints in the PFD branch, and spatial fusion strategies between SD and PFD branches at both the latent and residual level, PersonaHOI successfully blends personalized facial details with interactive non-facial regions, ensuring identity preservation and interaction coherence. Experiments, validated by a novel interaction alignment metric, demonstrate the superior realism and scalability of PersonaHOI, establishing a new standard for practical personalized face with HOI generation.