Abstract: Rip currents are strong, localized and narrow currents of water that flow outwards into the sea causing numerous beach-related injuries and fatalities worldwide. Accurate identification of rip currents remains challenging due to their amorphous nature and a lack of annotated data, which often requires expert knowledge. To address these issues, we present RipVIS, a large-scale video instance segmentation benchmark explicitly designed for rip current segmentation. RipVIS is an order of magnitude larger than previous datasets, featuring 140 videos (204,056 frames), out of which 115 (161,600 frames) videos are with rip currents, collected from various sources, including drones, mobile phones, and fixed beach cameras. Our dataset encompasses diverse visual contexts, such as wave-breaking patterns, sediment flows, and water color variations, across multiple global locations, including the USA, Greece, Portugal, Romania, Costa Rica, Sri Lanka, New Zealand and Australia. Most videos are annotated at $5$ FPS to ensure accuracy in dynamic scenarios, supplemented by an additional 25 videos (42,456 frames) without rip currents. We conduct comprehensive ablation studies on Mask R-CNN, Cascade Mask R-CNN, YOLACT, and YOLO11, fine-tuning these models for the task of rip current segmentation. Baseline results are reported across standard metrics, with a particular focus on the F2 score to prioritize recall and reduce false negatives. To further enhance segmentation quality, we apply a post-processing step using Temporal Confidence Aggregation (TCA). RipVIS aims to set a new standard for rip current segmentation, contributing towards safer beach environments. We also offer a benchmark website to share data, models, and results with the research community, encouraging ongoing collaboration and future contributions at link.hidden.for.review.