Shape-of-You: Fused Gromov-Wasserstein Optimal Transport for Semantic Correspondence in-the-Wild

Establishing semantic correspondence without supervision is essential for handling diverse in-the-wild images where annotations are scarce.While recent 2D foundation models offer powerful features, adapting them for unsupervised learning via nearest-neighbor pseudo-labels has key limitations: it operates locally, ignoring structural relationships, and consequently its reliance on 2D appearance fails to resolve geometric ambiguities arising from symmetries or repetitive features.In this work, we address this by reformulating pseudo-label generation as a Fused Gromov-Wasserstein (FGW) problem, which jointly optimizes inter-feature similarity and intra-structural consistency. Our framework, Shape-of-You (SoY), leverages a 3D foundation model to define this intra-structure in the geometric space, resolving abovementioned ambiguity. However, since FGW is a computationally prohibitive quadratic problem, we approximate it through anchor-based linearization.The resulting probabilistic transport plan provides a structurally consistent, yet noisy, supervisory signal.We introduce a soft-target loss, which dynamically blends guidance from this plan with the network's current predictions, to build a learning framework robust to this noise.SoY achieves state-of-the-art performance on the SPair-71k and AP-10k datasets, establishing a new benchmark in unsupervised semantic correspondence.Code is in the supplement.