Recent methods have made significant progress in optical flow estimation. However, the evaluation of these methods mainly focus on improved accuracy in benchmarks and often overlook the analysis of the robustness or behavior of the networks, which could be important in safety-critical scenarios such as autonomous driving. In this paper, we propose a novel method for robustness evaluation by modifying data from original benchmarks. Unlike previous benchmarks that focus on complex scenes, we propose to modify key objects from the original images in order to analyze the sensitivity to these changes observed in the output. Our aim is to identify common failure cases of state-of-the-art (SOTA) methods to evaluate their robustness and understand their behaviors. We show that: Optical flow methods are more sensitive to shape changes than to texture changes; and optical flow methods tend to “remember” objects seen during training and may “ignore” the motion of unseen objects. Our experimental results and findings provide a more in-depth understanding of the behavior of recent optical flow methods, suggesting the need for more careful design, especially in safety-critical scenarios. The code and data will be made available.