Skip to yearly menu bar Skip to main content


MoReVQA: Exploring Modular Reasoning Models for Video Question Answering

Juhong Min · Shyamal Buch · Arsha Nagrani · Minsu Cho · Cordelia Schmid

Arch 4A-E Poster #347
[ ]
Thu 20 Jun 10:30 a.m. PDT — noon PDT


This paper addresses the task of video question answering (videoQA) via a decomposed multi-stage, modular reasoning framework. Previous modular methods have shown promise with a single planning stage ungrounded in visual content. However, through a simple and effective baseline, we find that such systems can lead to brittle behavior in practice for challenging videoQA settings. Thus, unlike traditional single-stage planning methods, we propose a multi-stage system consisting of an event parser, a grounding stage, and a final reasoning stage in conjunction with an external memory. All stages are training-free, and performed using few-shot prompting of large models, creating interpretable intermediate outputs at each stage. By decomposing the underlying planning and task complexity, our method, MoReVQA, improves over prior work on standard videoQA benchmarks (NExT-QA, iVQA, EgoSchema, and ActivityNet-QA) with state-of-the-art results, and extensions to related tasks (grounded videoQA, paragraph captioning).

Live content is unavailable. Log in and register to view live content