Zero-Shot Monocular Scene Flow Estimation in the Wild

Foundation models have shown generalization across datasets for many low-level vision tasks, like depth estimation, but no such model exists for scene flow.Even though scene flow has wide potential use, it is not used in practice because current predictive models do not generalize well.We solve three challenges to fix this problem.First, we create a method that jointly estimates geometry and motion for accurate prediction.Second, we alleviate scene flow data scarcity with a data recipe that affords us 1M annotated training samples across diverse synthetic scenes.Third, we evaluate different parameterizations for scene flow prediction and identify a natural and effective parameterization.Our resulting model outperforms existing methods as well baselines built on foundation models in term of 3D end-point error, and shows zero-shot generalization to the casually captured videos from DAVIS and the robotic manipulation scenes from RoboTAP.Overall, this makes scene flow prediction significantly more practical for in-the-wild use.