Mixture of Prototypes for Test-time Adaptive Segmentation

Test-Time Adaptive Segmentation (TTA-Seg) aims to adapt a trained segmentation model to test data under distribution shift in an unsupervised manner. Existing approaches typically utilize class-wise prototypes to capture and transfer the source distribution, but inevitably neglecting the diversity within source samples. In this paper, we propose a new test-time adaptation paradigm based on the mixture-of-experts (MoE), where domain experts are designed to 1) better capture the source distribution, and 2) dynamically adjust their contribution in test case prediction. Specifically, during source training, prototypes are derived as the class-wise average for source pixel features. We then generate multiple experts through clustering these prototypes, providing each class with several experts with enhanced representativeness. At test time, each pixel's prediction is drawn from all experts' knowledge in an adaptive manner, \ie, a gating network assigns weight according to pixel-expert correlation. To optimize the system, we devise a min-max entropy optimization scheme for the gating network but keeping the rest frozen, minimizing the entropy of model prediction but maximizing the entropy in expert selection. Consequently, the model is urged to derived confident predictions with effective utilization of domain experts, hence promoting the adaptation. Experiments on two scenarios, Test-time Adaptation (TTA) and the more challenging continual TTA, demonstrate that our approach achieves the new state-of-the-art.