Generative models have enabled the creation of highly realistic facial-synthetic images, raising significant concerns due to their potential for misuse. While research in Deepfake detection has advanced rapidly, many methods still struggle to generalize to unseen Deepfakes generated by novel synthesis techniques. To address this challenge, we propose a novel side-network-based decoder that extracts spatial and temporal cues based on the CLIP image encoder for generalized video-based Deepfake detection. Additionally, we introduce the Facial Component Guidance (FCG) to enhance the spatial learning generalizability by encouraging the model to focus on key facial regions. The cross-dataset evaluation demonstrates the superior performance of our approach, surpassing state-of-the-art methods on challenging datasets. Extensive experiments further validate the effectiveness of the proposed method in terms of data efficiency, parameter efficiency and model robustness.