Analyzing gigapixel Whole Slide Images (WSIs) is challenging due to their complex pathological tissue environment and lack of target-driven domain knowledge.Previous approaches used additional priors to address this but required extra inference steps and specialized workflows, limiting scalability and the model's ability to uncover novel factors that might impact outcomes.To address these challenges, we propose a plug-and-play Pathology-Aware Mixture-of-Experts (PAMoE) module,which leverages a mixture of experts to learn pathology-related knowledge and filter out useful information.We train the experts to become 'specialists' in specific intratumoral tissues by learning to route each tissue patch to its mapped expert.In addition, to reduce the impact of irrelevant content on model performance, we introduce a new routing rule that discards patches in which none of the experts express interest,which enables the model to more easily capture the relationships between relevant patches.Through a comprehensive evaluation of PAMoE on survival task, we demonstrate that 1) Our module enhances the performance of baseline models in the majority of cases, and 2) The sparse expert processing across different tissues more effectively uncovers inter-tissue interactions. Examining the specific tissue preferences of individual experts offers novel insights into model interpretability.