Two by Two: Learning Multi-Task Pairwise Objects Assembly for Generalizable Robot Manipulation

3D assembly tasks, such as furniture assembly and component fitting, play a crucial role in daily life and represent essential capabilities for future home robots. Existing benchmarks and datasets predominantly focus on assembling geometric fragments or factory parts, which fall short in addressing the complexities of everyday object interactions and assemblies. To bridge this gap, we present 2BY2, a large-scale annotated dataset for daily pairwise objects assembly, covering 18 fine-grained tasks that reflect real-life scenarios, such as plugging into sockets, arranging flowers in vases, and inserting bread into toasters. The 2BY2 dataset contains 1,034 different instances and 517 pairwise objects with pose and symmetry annotations, requiring approaches that align geometric shapes while accounting for functional and spatial relationships between objects. Leveraging the 2BY2 dataset, we introduce a multi-step paired SE(3) pose estimation method that utilizes equivariant geometric features to enforce assembly constraints. Compared to previous shape assembly methods, our approach achieves state-of-the-art performance across all 18 tasks in the 2BY2 dataset, reducing translation RMSE by an average of 0.046 and rotation RMSE by 11.44 across both inter-category and intra-category tasks. Additionally, robot experiments further validate the reliability and generalization ability of our method for complex 3D assembly tasks.