Skip to yearly menu bar Skip to main content


RMem: Restricted Memory Banks Improve Video Object Segmentation

Junbao Zhou · Ziqi Pang · Yu-Xiong Wang

Arch 4A-E Poster #385
[ ]
Thu 20 Jun 5 p.m. PDT — 6:30 p.m. PDT


With recent video object segmentation (VOS) benchmarks evolving to challenging scenarios, we revisit a simple yet previously underestimated strategy: restricting the size of memory banks. This diverges from the prevalent practice of expanding memory banks to accommodate extensive historical information. Our specially designed "memory deciphering" study offers a pivotal insight underpinning such a strategy: expanding memory banks, while seemingly beneficial, actually increases the difficulty for VOS modules to decode relevant features due to the confusion from redundant information. By restricting memory banks to a limited number of essential frames, we achieve a notable improvement in VOS accuracy. This process balances the importance and freshness of frames to maintain an informative memory bank within a bounded capacity. Additionally, restricted memory banks reduce the training-inference discrepancy in memory lengths compared with continuous expansion. This fosters new opportunities in temporal reasoning and enables us to introduce the previously overlooked "temporal positional embedding." Finally, our insights are embodied in "RMem" ("R" for restricted), a simple yet effective VOS modification that excels at challenging VOS scenarios and establishes new state of the art for object state changes (VOST dataset) and long videos (the Long Videos dataset). Our codes are available at and our demo can be watched on

Live content is unavailable. Log in and register to view live content