TouchDream: 3D Object Completion through Imagined Touch
Abstract
Point cloud completion is crucial for robust 3D perception but remains challenging due to its ill-posed nature. Coarse-to-fine methods can lead to unconstrained local guesses in the absence of key structures, whereas diffusion-based approaches may introduce geometric inconsistencies. To overcome these limitations, we present TouchDream, a novel framework that leverages a diffusion model to 'dream' of tactile sensing on object surfaces, which reformulates the sensing process as a learnable generative modeling task. Unlike visual cues, tactile data provides rich local geometry that can be directly converted into 3D space for point fusion, offering a powerful guide for detail-aware completion. Specifically, our approach generate compact tactile latent representations conditioned on coarse points and sampled touch poses. A touch-guided refinement module then leverages touch features to optimize coarse points. Extensive experiments show that our TouchDream model achieves the state-of-the-art performance, significantly enhancing the recovery of local details.