GENMANIP: LLM-driven Simulation for Generalizable Instruction-Following Manipulation

Robotic manipulation in real-world settings presents significant challenges, particularly in achieving reliable performance across diverse real-world conditions. However, existing simulation platforms often lack the necessary support for studying policy generalization across varied tasks and conditions, falling behind the growing interest in leveraging foundation models. To address these limitations, we introduce \textbf{GenManip}, a realistic tabletop simulation platform designed to study policy generalization. The platform features a \textbf{task-oriented scene graph}-based scenario generation driven by GPT capabilities, enabling large-scale everyday task synthesis using \textit{10K} 3D assets. To investigate the generalization of robotic manipulation, we introduce \textbf{GenManip-Bench}, a benchmark comprising 250 task scenarios derived from generated tasks and refined through human-in-the-loop correction. We focus on two key areas: a modular manipulation system that employs foundation models for component-specific analysis, and end-to-end policy exploration using the scalable data collection pipeline. Experimental results show that while data scaling benefits learning-based policies, their generalization remains limited compared to modular approaches using foundation models. We expect this platform to offer critical insights for advancing policy generalizability in realistic settings. All code will be made publicly available.