Multi-task learning (MTL) is a paradigm that aims to improve the generalization of models by simultaneously learning multiple related tasks, leveraging shared representations and task-specific information to capture complex patterns and to enhance performance on individual tasks. However, existing work has discovered that MTL could possibly harm generalization, and one particular reason is the spurious correlations between tasks, where owing to the knowledge-sharing property, the task-specific predictors are more likely to develop reliance on spurious features. Most existing approaches address this issue through distributional robustness, aiming to maintain consistent performance across different distributions under unknown covariate shifts. Yet, this formulation lacks theoretical guarantee and can be sensitive to the construction of covariate shift. In this work, we propose a novel perspective, where we seek to directly identify the spurious correlations between tasks. Drawing inspirations from conventional formulations on spurious correlation, for each task, we propose to distinguish its spurious tasks using the difference in correlation coefficients between the empirical distribution and class-wise resampled distributions, thereby capturing the correlations between task labels w.r.t. each class. We prove theoretically the feasibility of such resampling strategy in characterizing the spurious correlation between tasks. Following the identification of task-specific spurious information, we propose a simple fine-tuning strategy regarding per-task predictors, debiased adversarial training, where the per-task predictors are adversarially trained to disregard information associated with their spurious tasks. Experimental results on four benchmark datasets show that our method effectively mitigates spurious correlations and outperforms state-of-the-art methods in improving generalization.