Efficiently Mitigating Video Content Misalignment on Large Vision Model with Time-Series Data Alignment

Video understanding tasks tend to demand considerable computation and storage. Particularly as large vision models (LVMs) demonstrate increasing performance gains due to their large model size and a tremendous amount of training data. To achieve optimal task performance with minimal cost, LVMs can be trained with only the selected critical content during the training. However, during the inference, the LVMs will consume the entire lengthy videos without guidance. Such video content misalignment can jeopardize the LVM performance. Additionally, conventional efficient video understanding methods may be restricted due to a strong data assumption that may not be feasible when leveraging LVMs. Thus, we will first provide a preliminary study on the impact of such a misalignment challenge. Then, we will introduce a simple and efficient framework that leverages non-visual modalities to align the training and inference video content and use time-series data as an example to implement the framework. Experimental results on the Ego4D-derived dataset demonstrate the promising potential of the framework.