SURGEON: Memory-Adaptive Fully Test-Time Adaptation via Dynamic Activation Sparsity

Despite the growing integration of deep models into mobile and embedded terminals, the accuracy of these models often declines significantly during inference due to various deployment interferences. Test-time adaptation (TTA) has emerged as an effective strategy to improve the performance of deep models by adapting them to unlabeled target data online. Yet, the significant memory cost, particularly in memory-constrained IoT terminals, impedes the effective deployment of most backward-propagation-based TTA methods. To tackle memory constraints, we introduce SURGEON, a method that substantially reduces memory cost while preserving comparable accuracy improvements during fully test-time adaptation (FTTA) without relying on specific network architectures or modifications to the original training procedure. Specifically, we propose a novel dynamic activation sparsity strategy that directly prunes activations at layer-specific dynamic ratios, allowing for flexible control of learning ability and memory cost in a data-sensitive manner during adaptation. Among this, two metrics, Gradient Importance and Layer Activation Memory, are considered to determine the layer-wise activation pruning ratios, reflecting accuracy contribution and memory efficiency, respectively. Experimentally, our method surpasses previous TTA baselines by not only reducing memory usage but also achieving superior accuracy, delivering SOTA performance across diverse datasets, network architectures, and tasks.