Delving Aleatoric Uncertainty in Medical Image Segmentation via Vision Foundation Models

Medical image segmentation provides critical support for clinical workflows by precisely delineating anatomical structures and lesions. However, medical image datasets are often affected by acquisition noise and annotation ambiguity, leading to pervasive data uncertainty that substantially undermines model robustness. The existing study focuses primarily on model architectural improvements and predictive reliability estimation, while systematic exploration of the intrinsic data uncertainty remains insufficient. To address this gap, this work proposes leveraging the universal representation capabilities of visual foundation models to estimate inherent data uncertainty. Specifically, we analyze the feature diversity of the model's decoded representations and quantify their singular value energy to define the semantic perception scale for each class, thereby measuring sample difficulty and aleatoric uncertainty. Based on this foundation, we design two uncertainty-driven application strategies: (1) the aleatoric uncertainty-aware data filtering mechanism to eliminate potentially noisy samples and enhance model learning quality; (2) the dynamic uncertainty-aware optimization strategy that adaptively adjusts class-specific loss weights during training based on the semantic perception scale, combined with a label denoising mechanism to improve training stability. Experimental results on five public datasets encompassing CT and MRI modalities and involving multi-organ and tumor segmentation tasks demonstrate that our method achieves significant and robust performance improvements across various mainstream network architectures (CNN, Transformer, and Mamba), revealing the broad application potential of aleatoric uncertainty in medical image understanding and segmentation tasks. The code will be released.