Skip to yearly menu bar Skip to main content


Template Free Reconstruction of Human-object Interaction with Procedural Interaction Generation

Xianghui Xie · Bharat Lal Bhatnagar · Jan Lenssen · Gerard Pons-Moll

Arch 4A-E Poster #34
award Highlight
[ ]
Thu 20 Jun 10:30 a.m. PDT — noon PDT


Reconstructing human-object interaction in 3D from a single RGB image is a challenging task and existing data driven methods do not generalize beyond the objects present in the carefully curated 3D interaction datasets.Capturing large-scale real data to learn strong interaction and 3D shape priors is very expensive due to the combinatorial nature of human-object interactions. In this paper, we propose ProciGen (Procedural interaction Generation), a method to procedurally generate datasets with both, plausible interaction and diverse object variation.We generate 1M+ human-object interaction pairs in 3D and leverage this large-scale data to train our HDM (Hierarchical Diffusion Model), a novel method to reconstruct interacting human and unseen objects, without any templates. Our HDM is an image-conditioned diffusion model that learns both realistic interaction and highly accurate human and object shapes.Experiments show that our HDM trained with ProciGen significantly outperforms prior methods that requires template meshes and that our dataset allows training methods with strong generalization ability to unseen object instances. Our code and data will be publicly released.

Live content is unavailable. Log in and register to view live content