Skip to yearly menu bar Skip to main content


Stronger Fewer & Superior: Harnessing Vision Foundation Models for Domain Generalized Semantic Segmentation

ZHIXIANG WEI · Lin Chen · Xiaoxiao Ma · Huaian Chen · Tianle Liu · Pengyang Ling · Jinjin Zheng · Ben Wang · Yi Jin

Arch 4A-E Poster #431
[ ] [ Project Page ]
Fri 21 Jun 5 p.m. PDT — 6:30 p.m. PDT

Abstract: In this paper, we first assess and harness various Vision Foundation Models (VFMs) in the context of Domain Generalized Semantic Segmentation (DGSS). Driven by the motivation that $\textbf{Leveraging Stronger pre-trained models and Fewer trainable parameters for Superior generalizability}$, we introduce a robust fine-tuning approach, namely "$\textbf{Rein}$", to parameter-efficiently harness VFMs for DGSS. Built upon a set of trainable tokens, each linked to distinct instances, Rein precisely refines and forwards the feature maps from each layer to the next layer within the backbone. This process produces diverse refinements for different categories within a single image. With fewer trainable parameters, Rein efficiently fine-tunes VFMs for DGSS tasks, surprisingly surpassing full parameter fine-tuning.Extensive experiments across various settings demonstrate that Rein significantly outperforms state-of-the-art methods. Remarkably, with just an extra $\textbf{1}$% of trainable parameters within the frozen backbone, Rein achieves a mIoU of $\textbf{68.1}$% on the Cityscapes, without accessing any real urban-scene datasets. Such an improvement boosts the state-of-the-art by a notable $\textbf{21.7}$% in mIoU with efficient training.

Live content is unavailable. Log in and register to view live content